Note: This is test shard 1 of 3.
[==========] Running 2 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 2 tests from CatalogManagerConfigurations/MasterStressTest
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20251029 02:51:46.150287 416 test_util.cc:276] Using random seed: 915644260
2025-10-29T02:51:46Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-29T02:51:46Z Disabled control of system clock
I20251029 02:51:46.184165 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskYVokEG/build/asan/bin/kudu
/tmp/dist-test-taskYVokEG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:42343
--webserver_interface=127.0.104.62
--webserver_port=0
--builtin_ntp_servers=127.0.104.20:44691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:42343,127.0.104.61:34239,127.0.104.60:41233
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20251029 02:51:46.507797 424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251029 02:51:46.508133 424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251029 02:51:46.508211 424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251029 02:51:46.516548 424 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251029 02:51:46.516639 424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251029 02:51:46.516688 424 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251029 02:51:46.516733 424 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251029 02:51:46.526525 424 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:44691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/master-0/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:42343,127.0.104.61:34239,127.0.104.60:41233
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:42343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 60f5e5267b92c39485a66121d3ce3cc7ef57b0e0
build type FASTDEBUG
built by None at 29 Oct 2025 02:43:25 UTC on 24a791456cd2
build id 8768
ASAN enabled
I20251029 02:51:46.528137 424 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251029 02:51:46.529819 424 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251029 02:51:46.538636 430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251029 02:51:46.539326 429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251029 02:51:46.539304 432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251029 02:51:46.540758 424 server_base.cc:1047] running on GCE node
I20251029 02:51:46.541519 424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251029 02:51:46.543421 424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251029 02:51:46.544777 424 hybrid_clock.cc:648] HybridClock initialized: now 1761706306544681 us; error 85 us; skew 500 ppm
I20251029 02:51:46.545140 424 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251029 02:51:46.547279 424 webserver.cc:492] Webserver started at http://127.0.104.62:46851/ using document root <none> and password file <none>
I20251029 02:51:46.547992 424 fs_manager.cc:362] Metadata directory not provided
I20251029 02:51:46.548117 424 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251029 02:51:46.548511 424 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251029 02:51:46.550907 424 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/master-0/data/instance:
uuid: "e9672114a6a742e2864c16ff60ba246e"
format_stamp: "Formatted at 2025-10-29 02:51:46 on dist-test-slave-qwrb"
I20251029 02:51:46.551617 424 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/master-0/wal/instance:
uuid: "e9672114a6a742e2864c16ff60ba246e"
format_stamp: "Formatted at 2025-10-29 02:51:46 on dist-test-slave-qwrb"
I20251029 02:51:46.556668 424 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20251029 02:51:46.560086 438 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251029 02:51:46.561405 424 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20251029 02:51:46.561559 424 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/master-0/wal
uuid: "e9672114a6a742e2864c16ff60ba246e"
format_stamp: "Formatted at 2025-10-29 02:51:46 on dist-test-slave-qwrb"
I20251029 02:51:46.561770 424 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251029 02:51:46.600800 424 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251029 02:51:46.601660 424 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251029 02:51:46.601938 424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251029 02:51:46.623116 424 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:42343
I20251029 02:51:46.623133 489 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:42343 every 8 connection(s)
I20251029 02:51:46.624625 424 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/master-0/data/info.pb
I20251029 02:51:46.628803 490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:46.630743 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskYVokEG/build/asan/bin/kudu as pid 424
I20251029 02:51:46.631064 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/master-0/wal/instance
I20251029 02:51:46.634447 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskYVokEG/build/asan/bin/kudu
/tmp/dist-test-taskYVokEG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:34239
--webserver_interface=127.0.104.61
--webserver_port=0
--builtin_ntp_servers=127.0.104.20:44691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:42343,127.0.104.61:34239,127.0.104.60:41233
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20251029 02:51:46.638293 490 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42343 } has no permanent_uuid. Determining permanent_uuid...
I20251029 02:51:46.659041 490 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34239 } has no permanent_uuid. Determining permanent_uuid...
W20251029 02:51:46.660143 440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.61:34239: connect: Connection refused (error 111)
W20251029 02:51:46.661590 490 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.61:34239: Network error: Client connection negotiation failed: client connection to 127.0.104.61:34239: connect: Connection refused (error 111)
I20251029 02:51:46.711048 490 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34239 } attempt: 1
W20251029 02:51:46.712847 490 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.61:34239: Network error: Client connection negotiation failed: client connection to 127.0.104.61:34239: connect: Connection refused (error 111)
I20251029 02:51:46.781177 490 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34239 } attempt: 2
W20251029 02:51:46.782946 490 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.61:34239: Network error: Client connection negotiation failed: client connection to 127.0.104.61:34239: connect: Connection refused (error 111)
I20251029 02:51:46.874253 490 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34239 } attempt: 3
W20251029 02:51:46.875982 490 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.61:34239: Network error: Client connection negotiation failed: client connection to 127.0.104.61:34239: connect: Connection refused (error 111)
W20251029 02:51:46.965437 492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251029 02:51:46.965728 492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251029 02:51:46.965844 492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251029 02:51:46.973767 492 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251029 02:51:46.973857 492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251029 02:51:46.973929 492 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251029 02:51:46.974004 492 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251029 02:51:46.983839 492 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:44691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/master-1/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:42343,127.0.104.61:34239,127.0.104.60:41233
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:34239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.104.61
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 60f5e5267b92c39485a66121d3ce3cc7ef57b0e0
build type FASTDEBUG
built by None at 29 Oct 2025 02:43:25 UTC on 24a791456cd2
build id 8768
ASAN enabled
I20251029 02:51:46.985406 492 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251029 02:51:46.987114 492 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251029 02:51:46.995117 500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251029 02:51:46.995380 501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251029 02:51:46.995751 503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251029 02:51:46.996382 492 server_base.cc:1047] running on GCE node
I20251029 02:51:46.997104 492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251029 02:51:46.998394 492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251029 02:51:46.999603 492 hybrid_clock.cc:648] HybridClock initialized: now 1761706306999567 us; error 50 us; skew 500 ppm
I20251029 02:51:46.999923 492 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251029 02:51:47.001955 492 webserver.cc:492] Webserver started at http://127.0.104.61:43475/ using document root <none> and password file <none>
I20251029 02:51:47.002697 492 fs_manager.cc:362] Metadata directory not provided
I20251029 02:51:47.002804 492 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251029 02:51:47.003160 492 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251029 02:51:47.005446 492 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/master-1/data/instance:
uuid: "e9261eb85ea443d5a3beb36bd0c549f0"
format_stamp: "Formatted at 2025-10-29 02:51:47 on dist-test-slave-qwrb"
I20251029 02:51:47.006219 492 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/master-1/wal/instance:
uuid: "e9261eb85ea443d5a3beb36bd0c549f0"
format_stamp: "Formatted at 2025-10-29 02:51:47 on dist-test-slave-qwrb"
I20251029 02:51:47.011427 492 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20251029 02:51:47.015264 509 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251029 02:51:47.016579 492 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20251029 02:51:47.016747 492 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/master-1/data,/tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/master-1/wal
uuid: "e9261eb85ea443d5a3beb36bd0c549f0"
format_stamp: "Formatted at 2025-10-29 02:51:47 on dist-test-slave-qwrb"
I20251029 02:51:47.017010 492 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251029 02:51:47.019260 490 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34239 } attempt: 4
W20251029 02:51:47.021131 490 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.61:34239: Network error: Client connection negotiation failed: client connection to 127.0.104.61:34239: connect: Connection refused (error 111)
I20251029 02:51:47.050906 492 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251029 02:51:47.051736 492 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251029 02:51:47.051964 492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251029 02:51:47.071908 492 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.61:34239
I20251029 02:51:47.071928 560 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.61:34239 every 8 connection(s)
I20251029 02:51:47.073419 492 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/master-1/data/info.pb
I20251029 02:51:47.077869 561 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:47.082239 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskYVokEG/build/asan/bin/kudu as pid 492
I20251029 02:51:47.082406 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/master-1/wal/instance
I20251029 02:51:47.083418 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskYVokEG/build/asan/bin/kudu
/tmp/dist-test-taskYVokEG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.60:41233
--webserver_interface=127.0.104.60
--webserver_port=0
--builtin_ntp_servers=127.0.104.20:44691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:42343,127.0.104.61:34239,127.0.104.60:41233
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20251029 02:51:47.083546 561 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42343 } has no permanent_uuid. Determining permanent_uuid...
I20251029 02:51:47.098737 561 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34239 } has no permanent_uuid. Determining permanent_uuid...
I20251029 02:51:47.110653 561 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41233 } has no permanent_uuid. Determining permanent_uuid...
W20251029 02:51:47.111773 512 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.60:41233: connect: Connection refused (error 111)
W20251029 02:51:47.113250 561 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.60:41233: Network error: Client connection negotiation failed: client connection to 127.0.104.60:41233: connect: Connection refused (error 111)
I20251029 02:51:47.162665 561 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41233 } attempt: 1
W20251029 02:51:47.164429 561 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.60:41233: Network error: Client connection negotiation failed: client connection to 127.0.104.60:41233: connect: Connection refused (error 111)
I20251029 02:51:47.232702 561 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41233 } attempt: 2
W20251029 02:51:47.234688 561 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.60:41233: Network error: Client connection negotiation failed: client connection to 127.0.104.60:41233: connect: Connection refused (error 111)
I20251029 02:51:47.320362 490 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34239 } attempt: 5
I20251029 02:51:47.325940 561 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41233 } attempt: 3
I20251029 02:51:47.326663 490 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41233 } has no permanent_uuid. Determining permanent_uuid...
W20251029 02:51:47.328289 561 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.60:41233: Network error: Client connection negotiation failed: client connection to 127.0.104.60:41233: connect: Connection refused (error 111)
W20251029 02:51:47.328540 490 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.60:41233: Network error: Client connection negotiation failed: client connection to 127.0.104.60:41233: connect: Connection refused (error 111)
I20251029 02:51:47.379732 490 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41233 } attempt: 1
W20251029 02:51:47.381584 490 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.60:41233: Network error: Client connection negotiation failed: client connection to 127.0.104.60:41233: connect: Connection refused (error 111)
W20251029 02:51:47.420004 563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251029 02:51:47.420342 563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251029 02:51:47.420464 563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251029 02:51:47.428468 563 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251029 02:51:47.428583 563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251029 02:51:47.428664 563 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251029 02:51:47.428730 563 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251029 02:51:47.438618 563 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:44691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/master-2/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:42343,127.0.104.61:34239,127.0.104.60:41233
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.60:41233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.104.60
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 60f5e5267b92c39485a66121d3ce3cc7ef57b0e0
build type FASTDEBUG
built by None at 29 Oct 2025 02:43:25 UTC on 24a791456cd2
build id 8768
ASAN enabled
I20251029 02:51:47.440344 563 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251029 02:51:47.442080 563 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251029 02:51:47.449793 490 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41233 } attempt: 2
W20251029 02:51:47.450755 571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251029 02:51:47.451296 572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251029 02:51:47.451876 490 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.60:41233: Network error: Client connection negotiation failed: client connection to 127.0.104.60:41233: connect: Connection refused (error 111)
W20251029 02:51:47.452034 574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251029 02:51:47.452244 563 server_base.cc:1047] running on GCE node
I20251029 02:51:47.452972 563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251029 02:51:47.454277 563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251029 02:51:47.455469 563 hybrid_clock.cc:648] HybridClock initialized: now 1761706307455410 us; error 54 us; skew 500 ppm
I20251029 02:51:47.455794 563 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251029 02:51:47.457891 563 webserver.cc:492] Webserver started at http://127.0.104.60:42893/ using document root <none> and password file <none>
I20251029 02:51:47.458663 563 fs_manager.cc:362] Metadata directory not provided
I20251029 02:51:47.458767 563 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251029 02:51:47.459112 563 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251029 02:51:47.461457 563 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/master-2/data/instance:
uuid: "e5e5222b26fe4145948620e51edf53fb"
format_stamp: "Formatted at 2025-10-29 02:51:47 on dist-test-slave-qwrb"
I20251029 02:51:47.462168 563 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/master-2/wal/instance:
uuid: "e5e5222b26fe4145948620e51edf53fb"
format_stamp: "Formatted at 2025-10-29 02:51:47 on dist-test-slave-qwrb"
I20251029 02:51:47.467826 563 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20251029 02:51:47.471066 580 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251029 02:51:47.471499 561 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41233 } attempt: 4
I20251029 02:51:47.472283 563 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20251029 02:51:47.472440 563 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/master-2/data,/tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/master-2/wal
uuid: "e5e5222b26fe4145948620e51edf53fb"
format_stamp: "Formatted at 2025-10-29 02:51:47 on dist-test-slave-qwrb"
I20251029 02:51:47.472626 563 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20251029 02:51:47.473182 561 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.60:41233: Network error: Client connection negotiation failed: client connection to 127.0.104.60:41233: connect: Connection refused (error 111)
I20251029 02:51:47.493072 563 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251029 02:51:47.493948 563 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251029 02:51:47.494203 563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251029 02:51:47.515242 563 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.60:41233
I20251029 02:51:47.515259 631 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.60:41233 every 8 connection(s)
I20251029 02:51:47.516605 563 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/master-2/data/info.pb
I20251029 02:51:47.521205 632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:47.521837 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskYVokEG/build/asan/bin/kudu as pid 563
I20251029 02:51:47.522037 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/master-2/wal/instance
I20251029 02:51:47.528568 632 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42343 } has no permanent_uuid. Determining permanent_uuid...
I20251029 02:51:47.541030 632 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34239 } has no permanent_uuid. Determining permanent_uuid...
I20251029 02:51:47.548369 632 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41233 } has no permanent_uuid. Determining permanent_uuid...
I20251029 02:51:47.558105 490 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41233 } attempt: 3
I20251029 02:51:47.561357 632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5e5222b26fe4145948620e51edf53fb: Bootstrap starting.
I20251029 02:51:47.566025 632 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5e5222b26fe4145948620e51edf53fb: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:47.567337 632 log.cc:826] T 00000000000000000000000000000000 P e5e5222b26fe4145948620e51edf53fb: Log is configured to *not* fsync() on all Append() calls
I20251029 02:51:47.569614 490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9672114a6a742e2864c16ff60ba246e: Bootstrap starting.
I20251029 02:51:47.570222 632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5e5222b26fe4145948620e51edf53fb: No bootstrap required, opened a new log
I20251029 02:51:47.573297 490 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9672114a6a742e2864c16ff60ba246e: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:47.575047 490 log.cc:826] T 00000000000000000000000000000000 P e9672114a6a742e2864c16ff60ba246e: Log is configured to *not* fsync() on all Append() calls
I20251029 02:51:47.577371 632 raft_consensus.cc:359] T 00000000000000000000000000000000 P e5e5222b26fe4145948620e51edf53fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9672114a6a742e2864c16ff60ba246e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42343 } } peers { permanent_uuid: "e9261eb85ea443d5a3beb36bd0c549f0" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34239 } } peers { permanent_uuid: "e5e5222b26fe4145948620e51edf53fb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41233 } }
I20251029 02:51:47.577732 632 raft_consensus.cc:385] T 00000000000000000000000000000000 P e5e5222b26fe4145948620e51edf53fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:47.577863 632 raft_consensus.cc:740] T 00000000000000000000000000000000 P e5e5222b26fe4145948620e51edf53fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5e5222b26fe4145948620e51edf53fb, State: Initialized, Role: FOLLOWER
I20251029 02:51:47.578094 490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9672114a6a742e2864c16ff60ba246e: No bootstrap required, opened a new log
I20251029 02:51:47.578557 632 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5e5222b26fe4145948620e51edf53fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9672114a6a742e2864c16ff60ba246e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42343 } } peers { permanent_uuid: "e9261eb85ea443d5a3beb36bd0c549f0" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34239 } } peers { permanent_uuid: "e5e5222b26fe4145948620e51edf53fb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41233 } }
I20251029 02:51:47.579396 642 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5e5222b26fe4145948620e51edf53fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9672114a6a742e2864c16ff60ba246e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42343 } } peers { permanent_uuid: "e9261eb85ea443d5a3beb36bd0c549f0" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34239 } } peers { permanent_uuid: "e5e5222b26fe4145948620e51edf53fb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41233 } } }
I20251029 02:51:47.579645 642 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5e5222b26fe4145948620e51edf53fb [sys.catalog]: This master's current role is: FOLLOWER
I20251029 02:51:47.580650 632 sys_catalog.cc:565] T 00000000000000000000000000000000 P e5e5222b26fe4145948620e51edf53fb [sys.catalog]: configured and running, proceeding with master startup.
I20251029 02:51:47.586086 490 raft_consensus.cc:359] T 00000000000000000000000000000000 P e9672114a6a742e2864c16ff60ba246e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9672114a6a742e2864c16ff60ba246e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42343 } } peers { permanent_uuid: "e9261eb85ea443d5a3beb36bd0c549f0" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34239 } } peers { permanent_uuid: "e5e5222b26fe4145948620e51edf53fb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41233 } }
I20251029 02:51:47.586462 490 raft_consensus.cc:385] T 00000000000000000000000000000000 P e9672114a6a742e2864c16ff60ba246e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:47.586614 490 raft_consensus.cc:740] T 00000000000000000000000000000000 P e9672114a6a742e2864c16ff60ba246e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9672114a6a742e2864c16ff60ba246e, State: Initialized, Role: FOLLOWER
I20251029 02:51:47.587412 490 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9672114a6a742e2864c16ff60ba246e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9672114a6a742e2864c16ff60ba246e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42343 } } peers { permanent_uuid: "e9261eb85ea443d5a3beb36bd0c549f0" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34239 } } peers { permanent_uuid: "e5e5222b26fe4145948620e51edf53fb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41233 } }
I20251029 02:51:47.588327 654 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9672114a6a742e2864c16ff60ba246e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9672114a6a742e2864c16ff60ba246e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42343 } } peers { permanent_uuid: "e9261eb85ea443d5a3beb36bd0c549f0" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34239 } } peers { permanent_uuid: "e5e5222b26fe4145948620e51edf53fb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41233 } } }
I20251029 02:51:47.588685 654 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9672114a6a742e2864c16ff60ba246e [sys.catalog]: This master's current role is: FOLLOWER
I20251029 02:51:47.589632 490 sys_catalog.cc:565] T 00000000000000000000000000000000 P e9672114a6a742e2864c16ff60ba246e [sys.catalog]: configured and running, proceeding with master startup.
W20251029 02:51:47.592533 653 catalog_manager.cc:1568] T 00000000000000000000000000000000 P e5e5222b26fe4145948620e51edf53fb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251029 02:51:47.592674 653 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251029 02:51:47.589133 563 master_runner.cc:186] Error getting master registration for 127.0.104.62:42343: OK, instance_id { permanent_uuid: "e9672114a6a742e2864c16ff60ba246e" instance_seqno: 1761706306617385 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20251029 02:51:47.598409 563 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251029 02:51:47.604717 444 tablet.cc:2378] T 00000000000000000000000000000000 P e9672114a6a742e2864c16ff60ba246e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251029 02:51:47.612577 666 catalog_manager.cc:1568] T 00000000000000000000000000000000 P e9672114a6a742e2864c16ff60ba246e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251029 02:51:47.612810 666 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251029 02:51:47.611701 424 master_runner.cc:186] Error getting master registration for 127.0.104.61:34239: OK, instance_id { permanent_uuid: "e9261eb85ea443d5a3beb36bd0c549f0" instance_seqno: 1761706307066410 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20251029 02:51:47.617988 424 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251029 02:51:47.746902 586 tablet.cc:2378] T 00000000000000000000000000000000 P e5e5222b26fe4145948620e51edf53fb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251029 02:51:47.772411 561 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41233 } attempt: 5
I20251029 02:51:47.785995 561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Bootstrap starting.
I20251029 02:51:47.790378 561 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:47.791783 561 log.cc:826] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Log is configured to *not* fsync() on all Append() calls
I20251029 02:51:47.794713 561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: No bootstrap required, opened a new log
I20251029 02:51:47.802613 561 raft_consensus.cc:359] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9672114a6a742e2864c16ff60ba246e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42343 } } peers { permanent_uuid: "e9261eb85ea443d5a3beb36bd0c549f0" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34239 } } peers { permanent_uuid: "e5e5222b26fe4145948620e51edf53fb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41233 } }
I20251029 02:51:47.802950 561 raft_consensus.cc:385] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:47.803069 561 raft_consensus.cc:740] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9261eb85ea443d5a3beb36bd0c549f0, State: Initialized, Role: FOLLOWER
I20251029 02:51:47.803766 561 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9672114a6a742e2864c16ff60ba246e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42343 } } peers { permanent_uuid: "e9261eb85ea443d5a3beb36bd0c549f0" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34239 } } peers { permanent_uuid: "e5e5222b26fe4145948620e51edf53fb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41233 } }
I20251029 02:51:47.805377 669 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9672114a6a742e2864c16ff60ba246e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42343 } } peers { permanent_uuid: "e9261eb85ea443d5a3beb36bd0c549f0" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34239 } } peers { permanent_uuid: "e5e5222b26fe4145948620e51edf53fb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41233 } } }
I20251029 02:51:47.805750 669 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0 [sys.catalog]: This master's current role is: FOLLOWER
I20251029 02:51:47.805907 561 sys_catalog.cc:565] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0 [sys.catalog]: configured and running, proceeding with master startup.
I20251029 02:51:47.826946 492 master_runner.cc:263] No leader master found from master e9261eb85ea443d5a3beb36bd0c549f0
I20251029 02:51:47.827083 492 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251029 02:51:47.829891 680 catalog_manager.cc:1568] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251029 02:51:47.830142 680 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251029 02:51:48.008999 642 raft_consensus.cc:493] T 00000000000000000000000000000000 P e5e5222b26fe4145948620e51edf53fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:48.009359 642 raft_consensus.cc:515] T 00000000000000000000000000000000 P e5e5222b26fe4145948620e51edf53fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9672114a6a742e2864c16ff60ba246e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42343 } } peers { permanent_uuid: "e9261eb85ea443d5a3beb36bd0c549f0" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34239 } } peers { permanent_uuid: "e5e5222b26fe4145948620e51edf53fb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41233 } }
I20251029 02:51:48.011334 642 leader_election.cc:290] T 00000000000000000000000000000000 P e5e5222b26fe4145948620e51edf53fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9672114a6a742e2864c16ff60ba246e (127.0.104.62:42343), e9261eb85ea443d5a3beb36bd0c549f0 (127.0.104.61:34239)
I20251029 02:51:48.012509 465 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e5e5222b26fe4145948620e51edf53fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9672114a6a742e2864c16ff60ba246e" is_pre_election: true
I20251029 02:51:48.013443 465 raft_consensus.cc:2468] T 00000000000000000000000000000000 P e9672114a6a742e2864c16ff60ba246e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5e5222b26fe4145948620e51edf53fb in term 0.
I20251029 02:51:48.014338 581 leader_election.cc:304] T 00000000000000000000000000000000 P e5e5222b26fe4145948620e51edf53fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e5e5222b26fe4145948620e51edf53fb, e9672114a6a742e2864c16ff60ba246e; no voters:
I20251029 02:51:48.015683 536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e5e5222b26fe4145948620e51edf53fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9261eb85ea443d5a3beb36bd0c549f0" is_pre_election: true
I20251029 02:51:48.016136 536 raft_consensus.cc:2468] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5e5222b26fe4145948620e51edf53fb in term 0.
I20251029 02:51:48.016587 642 raft_consensus.cc:2804] T 00000000000000000000000000000000 P e5e5222b26fe4145948620e51edf53fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:51:48.016734 642 raft_consensus.cc:493] T 00000000000000000000000000000000 P e5e5222b26fe4145948620e51edf53fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:51:48.016820 642 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e5e5222b26fe4145948620e51edf53fb [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:51:48.024993 642 raft_consensus.cc:515] T 00000000000000000000000000000000 P e5e5222b26fe4145948620e51edf53fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9672114a6a742e2864c16ff60ba246e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42343 } } peers { permanent_uuid: "e9261eb85ea443d5a3beb36bd0c549f0" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34239 } } peers { permanent_uuid: "e5e5222b26fe4145948620e51edf53fb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41233 } }
I20251029 02:51:48.026307 465 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e5e5222b26fe4145948620e51edf53fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9672114a6a742e2864c16ff60ba246e"
I20251029 02:51:48.026618 465 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e9672114a6a742e2864c16ff60ba246e [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:51:48.030025 465 raft_consensus.cc:2468] T 00000000000000000000000000000000 P e9672114a6a742e2864c16ff60ba246e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5e5222b26fe4145948620e51edf53fb in term 1.
I20251029 02:51:48.030530 581 leader_election.cc:304] T 00000000000000000000000000000000 P e5e5222b26fe4145948620e51edf53fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e5e5222b26fe4145948620e51edf53fb, e9672114a6a742e2864c16ff60ba246e; no voters:
I20251029 02:51:48.034848 536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e5e5222b26fe4145948620e51edf53fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9261eb85ea443d5a3beb36bd0c549f0"
I20251029 02:51:48.035089 536 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:51:48.038339 536 raft_consensus.cc:2468] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5e5222b26fe4145948620e51edf53fb in term 1.
I20251029 02:51:48.039476 642 leader_election.cc:290] T 00000000000000000000000000000000 P e5e5222b26fe4145948620e51edf53fb [CANDIDATE]: Term 1 election: Requested vote from peers e9672114a6a742e2864c16ff60ba246e (127.0.104.62:42343), e9261eb85ea443d5a3beb36bd0c549f0 (127.0.104.61:34239)
I20251029 02:51:48.039876 642 raft_consensus.cc:2804] T 00000000000000000000000000000000 P e5e5222b26fe4145948620e51edf53fb [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:51:48.041527 642 raft_consensus.cc:697] T 00000000000000000000000000000000 P e5e5222b26fe4145948620e51edf53fb [term 1 LEADER]: Becoming Leader. State: Replica: e5e5222b26fe4145948620e51edf53fb, State: Running, Role: LEADER
I20251029 02:51:48.042236 642 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5e5222b26fe4145948620e51edf53fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9672114a6a742e2864c16ff60ba246e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42343 } } peers { permanent_uuid: "e9261eb85ea443d5a3beb36bd0c549f0" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34239 } } peers { permanent_uuid: "e5e5222b26fe4145948620e51edf53fb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41233 } }
I20251029 02:51:48.051044 642 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5e5222b26fe4145948620e51edf53fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5e5222b26fe4145948620e51edf53fb. Latest consensus state: current_term: 1 leader_uuid: "e5e5222b26fe4145948620e51edf53fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9672114a6a742e2864c16ff60ba246e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42343 } } peers { permanent_uuid: "e9261eb85ea443d5a3beb36bd0c549f0" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34239 } } peers { permanent_uuid: "e5e5222b26fe4145948620e51edf53fb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41233 } } }
I20251029 02:51:48.051321 642 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5e5222b26fe4145948620e51edf53fb [sys.catalog]: This master's current role is: LEADER
W20251029 02:51:48.054389 515 tablet.cc:2378] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251029 02:51:48.056033 684 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251029 02:51:48.057461 684 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251029 02:51:48.064534 465 raft_consensus.cc:1275] T 00000000000000000000000000000000 P e9672114a6a742e2864c16ff60ba246e [term 1 FOLLOWER]: Refusing update from remote peer e5e5222b26fe4145948620e51edf53fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251029 02:51:48.064535 536 raft_consensus.cc:1275] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0 [term 1 FOLLOWER]: Refusing update from remote peer e5e5222b26fe4145948620e51edf53fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251029 02:51:48.065451 642 consensus_queue.cc:1048] T 00000000000000000000000000000000 P e5e5222b26fe4145948620e51edf53fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9261eb85ea443d5a3beb36bd0c549f0" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:51:48.066161 642 consensus_queue.cc:1048] T 00000000000000000000000000000000 P e5e5222b26fe4145948620e51edf53fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9672114a6a742e2864c16ff60ba246e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:51:48.111873 669 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5e5222b26fe4145948620e51edf53fb. Latest consensus state: current_term: 1 leader_uuid: "e5e5222b26fe4145948620e51edf53fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9672114a6a742e2864c16ff60ba246e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42343 } } peers { permanent_uuid: "e9261eb85ea443d5a3beb36bd0c549f0" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34239 } } peers { permanent_uuid: "e5e5222b26fe4145948620e51edf53fb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41233 } } }
I20251029 02:51:48.112151 669 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0 [sys.catalog]: This master's current role is: FOLLOWER
I20251029 02:51:48.113446 654 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9672114a6a742e2864c16ff60ba246e [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5e5222b26fe4145948620e51edf53fb. Latest consensus state: current_term: 1 leader_uuid: "e5e5222b26fe4145948620e51edf53fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9672114a6a742e2864c16ff60ba246e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42343 } } peers { permanent_uuid: "e9261eb85ea443d5a3beb36bd0c549f0" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34239 } } peers { permanent_uuid: "e5e5222b26fe4145948620e51edf53fb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41233 } } }
I20251029 02:51:48.113662 654 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9672114a6a742e2864c16ff60ba246e [sys.catalog]: This master's current role is: FOLLOWER
I20251029 02:51:48.115048 682 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5e5222b26fe4145948620e51edf53fb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e5e5222b26fe4145948620e51edf53fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9672114a6a742e2864c16ff60ba246e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42343 } } peers { permanent_uuid: "e9261eb85ea443d5a3beb36bd0c549f0" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34239 } } peers { permanent_uuid: "e5e5222b26fe4145948620e51edf53fb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41233 } } }
I20251029 02:51:48.115330 682 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5e5222b26fe4145948620e51edf53fb [sys.catalog]: This master's current role is: LEADER
I20251029 02:51:48.115561 682 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5e5222b26fe4145948620e51edf53fb [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e5e5222b26fe4145948620e51edf53fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9672114a6a742e2864c16ff60ba246e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42343 } } peers { permanent_uuid: "e9261eb85ea443d5a3beb36bd0c549f0" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34239 } } peers { permanent_uuid: "e5e5222b26fe4145948620e51edf53fb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41233 } } }
I20251029 02:51:48.115715 682 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5e5222b26fe4145948620e51edf53fb [sys.catalog]: This master's current role is: LEADER
I20251029 02:51:48.117887 684 catalog_manager.cc:1357] Generated new cluster ID: 96c87f4c6dea451cb609ec9ee56d85bf
I20251029 02:51:48.118003 684 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251029 02:51:48.118494 654 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9672114a6a742e2864c16ff60ba246e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e5e5222b26fe4145948620e51edf53fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9672114a6a742e2864c16ff60ba246e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42343 } } peers { permanent_uuid: "e9261eb85ea443d5a3beb36bd0c549f0" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34239 } } peers { permanent_uuid: "e5e5222b26fe4145948620e51edf53fb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41233 } } }
I20251029 02:51:48.118696 654 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9672114a6a742e2864c16ff60ba246e [sys.catalog]: This master's current role is: FOLLOWER
I20251029 02:51:48.121520 669 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e5e5222b26fe4145948620e51edf53fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9672114a6a742e2864c16ff60ba246e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42343 } } peers { permanent_uuid: "e9261eb85ea443d5a3beb36bd0c549f0" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34239 } } peers { permanent_uuid: "e5e5222b26fe4145948620e51edf53fb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41233 } } }
I20251029 02:51:48.121742 669 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0 [sys.catalog]: This master's current role is: FOLLOWER
I20251029 02:51:48.132320 684 catalog_manager.cc:1380] Generated new certificate authority record
I20251029 02:51:48.133991 684 catalog_manager.cc:1514] Loading token signing keys...
I20251029 02:51:48.148347 684 catalog_manager.cc:6022] T 00000000000000000000000000000000 P e5e5222b26fe4145948620e51edf53fb: Generated new TSK 0
I20251029 02:51:48.149503 684 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251029 02:51:48.183889 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskYVokEG/build/asan/bin/kudu
/tmp/dist-test-taskYVokEG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.1:0
--local_ip_for_outbound_sockets=127.0.104.1
--webserver_interface=127.0.104.1
--webserver_port=0
--tserver_master_addrs=127.0.104.62:42343,127.0.104.61:34239,127.0.104.60:41233
--builtin_ntp_servers=127.0.104.20:44691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
W20251029 02:51:48.532145 694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251029 02:51:48.532912 694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251029 02:51:48.533066 694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251029 02:51:48.541838 694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251029 02:51:48.542043 694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.1
I20251029 02:51:48.552451 694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:44691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.104.1
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.0.104.62:42343,127.0.104.61:34239,127.0.104.60:41233
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.1
--log_dir=/tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 60f5e5267b92c39485a66121d3ce3cc7ef57b0e0
build type FASTDEBUG
built by None at 29 Oct 2025 02:43:25 UTC on 24a791456cd2
build id 8768
ASAN enabled
I20251029 02:51:48.554075 694 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251029 02:51:48.555979 694 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251029 02:51:48.565507 699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251029 02:51:48.565541 700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251029 02:51:48.566563 702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251029 02:51:48.567606 694 server_base.cc:1047] running on GCE node
I20251029 02:51:48.568367 694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251029 02:51:48.569347 694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251029 02:51:48.570619 694 hybrid_clock.cc:648] HybridClock initialized: now 1761706308570570 us; error 46 us; skew 500 ppm
I20251029 02:51:48.570942 694 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251029 02:51:48.573215 694 webserver.cc:492] Webserver started at http://127.0.104.1:43541/ using document root <none> and password file <none>
I20251029 02:51:48.573968 694 fs_manager.cc:362] Metadata directory not provided
I20251029 02:51:48.574075 694 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251029 02:51:48.574443 694 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251029 02:51:48.576861 694 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/data/instance:
uuid: "b495a49349324f7d81908f2da700f0d9"
format_stamp: "Formatted at 2025-10-29 02:51:48 on dist-test-slave-qwrb"
I20251029 02:51:48.577612 694 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/instance:
uuid: "b495a49349324f7d81908f2da700f0d9"
format_stamp: "Formatted at 2025-10-29 02:51:48 on dist-test-slave-qwrb"
I20251029 02:51:48.582988 694 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20251029 02:51:48.586391 708 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251029 02:51:48.587778 694 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20251029 02:51:48.587947 694 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal
uuid: "b495a49349324f7d81908f2da700f0d9"
format_stamp: "Formatted at 2025-10-29 02:51:48 on dist-test-slave-qwrb"
I20251029 02:51:48.588163 694 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251029 02:51:48.614864 666 catalog_manager.cc:1269] Loaded cluster ID: 96c87f4c6dea451cb609ec9ee56d85bf
I20251029 02:51:48.614979 666 catalog_manager.cc:1562] T 00000000000000000000000000000000 P e9672114a6a742e2864c16ff60ba246e: loading cluster ID for follower catalog manager: success
I20251029 02:51:48.617795 666 catalog_manager.cc:1584] T 00000000000000000000000000000000 P e9672114a6a742e2864c16ff60ba246e: acquiring CA information for follower catalog manager: success
I20251029 02:51:48.620287 666 catalog_manager.cc:1612] T 00000000000000000000000000000000 P e9672114a6a742e2864c16ff60ba246e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251029 02:51:48.625910 694 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251029 02:51:48.626828 694 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251029 02:51:48.627089 694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251029 02:51:48.628096 694 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251029 02:51:48.630111 694 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251029 02:51:48.630227 694 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251029 02:51:48.630348 694 ts_tablet_manager.cc:616] Registered 0 tablets
I20251029 02:51:48.630426 694 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251029 02:51:48.668403 694 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:38481
I20251029 02:51:48.668471 820 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:38481 every 8 connection(s)
I20251029 02:51:48.669790 694 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/data/info.pb
I20251029 02:51:48.673336 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskYVokEG/build/asan/bin/kudu as pid 694
I20251029 02:51:48.673575 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/instance
I20251029 02:51:48.680020 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskYVokEG/build/asan/bin/kudu
/tmp/dist-test-taskYVokEG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.2:0
--local_ip_for_outbound_sockets=127.0.104.2
--webserver_interface=127.0.104.2
--webserver_port=0
--tserver_master_addrs=127.0.104.62:42343,127.0.104.61:34239,127.0.104.60:41233
--builtin_ntp_servers=127.0.104.20:44691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20251029 02:51:48.691449 821 heartbeater.cc:344] Connected to a master server at 127.0.104.62:42343
I20251029 02:51:48.691901 821 heartbeater.cc:461] Registering TS with master...
I20251029 02:51:48.693017 823 heartbeater.cc:344] Connected to a master server at 127.0.104.61:34239
I20251029 02:51:48.693073 821 heartbeater.cc:507] Master 127.0.104.62:42343 requested a full tablet report, sending...
I20251029 02:51:48.693188 823 heartbeater.cc:461] Registering TS with master...
I20251029 02:51:48.693652 823 heartbeater.cc:507] Master 127.0.104.61:34239 requested a full tablet report, sending...
I20251029 02:51:48.695709 526 ts_manager.cc:194] Registered new tserver with Master: b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:51:48.695730 455 ts_manager.cc:194] Registered new tserver with Master: b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:51:48.702240 822 heartbeater.cc:344] Connected to a master server at 127.0.104.60:41233
I20251029 02:51:48.702380 822 heartbeater.cc:461] Registering TS with master...
I20251029 02:51:48.702849 822 heartbeater.cc:507] Master 127.0.104.60:41233 requested a full tablet report, sending...
I20251029 02:51:48.704355 597 ts_manager.cc:194] Registered new tserver with Master: b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:51:48.705912 597 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.104.1:56119
I20251029 02:51:48.832053 680 catalog_manager.cc:1269] Loaded cluster ID: 96c87f4c6dea451cb609ec9ee56d85bf
I20251029 02:51:48.832151 680 catalog_manager.cc:1562] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: loading cluster ID for follower catalog manager: success
I20251029 02:51:48.834646 680 catalog_manager.cc:1584] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: acquiring CA information for follower catalog manager: success
I20251029 02:51:48.836364 680 catalog_manager.cc:1612] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20251029 02:51:49.036342 831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251029 02:51:49.036698 831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251029 02:51:49.036875 831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251029 02:51:49.045576 831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251029 02:51:49.045816 831 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.2
I20251029 02:51:49.055805 831 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:44691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.104.2
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.0.104.62:42343,127.0.104.61:34239,127.0.104.60:41233
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.2
--log_dir=/tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 60f5e5267b92c39485a66121d3ce3cc7ef57b0e0
build type FASTDEBUG
built by None at 29 Oct 2025 02:43:25 UTC on 24a791456cd2
build id 8768
ASAN enabled
I20251029 02:51:49.057578 831 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251029 02:51:49.059450 831 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251029 02:51:49.069521 837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251029 02:51:49.069458 836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251029 02:51:49.070739 839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251029 02:51:49.071060 831 server_base.cc:1047] running on GCE node
I20251029 02:51:49.071720 831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251029 02:51:49.072708 831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251029 02:51:49.074040 831 hybrid_clock.cc:648] HybridClock initialized: now 1761706309073982 us; error 45 us; skew 500 ppm
I20251029 02:51:49.074368 831 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251029 02:51:49.077140 831 webserver.cc:492] Webserver started at http://127.0.104.2:37935/ using document root <none> and password file <none>
I20251029 02:51:49.078003 831 fs_manager.cc:362] Metadata directory not provided
I20251029 02:51:49.078116 831 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251029 02:51:49.078473 831 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251029 02:51:49.080955 831 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/data/instance:
uuid: "c756528642a543eeacd93cef3e335a7d"
format_stamp: "Formatted at 2025-10-29 02:51:49 on dist-test-slave-qwrb"
I20251029 02:51:49.081739 831 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/instance:
uuid: "c756528642a543eeacd93cef3e335a7d"
format_stamp: "Formatted at 2025-10-29 02:51:49 on dist-test-slave-qwrb"
I20251029 02:51:49.087472 831 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20251029 02:51:49.091612 845 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251029 02:51:49.093118 831 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20251029 02:51:49.093286 831 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal
uuid: "c756528642a543eeacd93cef3e335a7d"
format_stamp: "Formatted at 2025-10-29 02:51:49 on dist-test-slave-qwrb"
I20251029 02:51:49.093513 831 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251029 02:51:49.128278 831 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251029 02:51:49.129185 831 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251029 02:51:49.129456 831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251029 02:51:49.130620 850 raft_consensus.cc:493] T 00000000000000000000000000000000 P e9672114a6a742e2864c16ff60ba246e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e5e5222b26fe4145948620e51edf53fb)
I20251029 02:51:49.130831 831 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251029 02:51:49.130846 850 raft_consensus.cc:515] T 00000000000000000000000000000000 P e9672114a6a742e2864c16ff60ba246e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9672114a6a742e2864c16ff60ba246e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42343 } } peers { permanent_uuid: "e9261eb85ea443d5a3beb36bd0c549f0" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34239 } } peers { permanent_uuid: "e5e5222b26fe4145948620e51edf53fb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41233 } }
I20251029 02:51:49.132084 850 leader_election.cc:290] T 00000000000000000000000000000000 P e9672114a6a742e2864c16ff60ba246e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e9261eb85ea443d5a3beb36bd0c549f0 (127.0.104.61:34239), e5e5222b26fe4145948620e51edf53fb (127.0.104.60:41233)
I20251029 02:51:49.132784 536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e9672114a6a742e2864c16ff60ba246e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e9261eb85ea443d5a3beb36bd0c549f0" is_pre_election: true
I20251029 02:51:49.132865 607 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e9672114a6a742e2864c16ff60ba246e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e5e5222b26fe4145948620e51edf53fb" is_pre_election: true
I20251029 02:51:49.133163 831 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251029 02:51:49.133327 831 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20251029 02:51:49.133486 831 ts_tablet_manager.cc:616] Registered 0 tablets
I20251029 02:51:49.133620 831 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251029 02:51:49.133633 441 leader_election.cc:304] T 00000000000000000000000000000000 P e9672114a6a742e2864c16ff60ba246e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e9672114a6a742e2864c16ff60ba246e; no voters: e5e5222b26fe4145948620e51edf53fb, e9261eb85ea443d5a3beb36bd0c549f0
I20251029 02:51:49.134008 850 raft_consensus.cc:2749] T 00000000000000000000000000000000 P e9672114a6a742e2864c16ff60ba246e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251029 02:51:49.138803 855 raft_consensus.cc:493] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e5e5222b26fe4145948620e51edf53fb)
I20251029 02:51:49.139119 855 raft_consensus.cc:515] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9672114a6a742e2864c16ff60ba246e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42343 } } peers { permanent_uuid: "e9261eb85ea443d5a3beb36bd0c549f0" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34239 } } peers { permanent_uuid: "e5e5222b26fe4145948620e51edf53fb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41233 } }
I20251029 02:51:49.141126 607 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e9261eb85ea443d5a3beb36bd0c549f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e5e5222b26fe4145948620e51edf53fb" is_pre_election: true
I20251029 02:51:49.141151 465 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e9261eb85ea443d5a3beb36bd0c549f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e9672114a6a742e2864c16ff60ba246e" is_pre_election: true
I20251029 02:51:49.141357 465 raft_consensus.cc:2468] T 00000000000000000000000000000000 P e9672114a6a742e2864c16ff60ba246e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9261eb85ea443d5a3beb36bd0c549f0 in term 1.
I20251029 02:51:49.141912 510 leader_election.cc:304] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e9261eb85ea443d5a3beb36bd0c549f0, e9672114a6a742e2864c16ff60ba246e; no voters:
I20251029 02:51:49.142264 855 leader_election.cc:290] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e9672114a6a742e2864c16ff60ba246e (127.0.104.62:42343), e5e5222b26fe4145948620e51edf53fb (127.0.104.60:41233)
I20251029 02:51:49.142690 855 raft_consensus.cc:2804] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251029 02:51:49.142876 855 raft_consensus.cc:493] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader e5e5222b26fe4145948620e51edf53fb)
I20251029 02:51:49.143061 855 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0 [term 1 FOLLOWER]: Advancing to term 2
I20251029 02:51:49.147431 855 raft_consensus.cc:515] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9672114a6a742e2864c16ff60ba246e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42343 } } peers { permanent_uuid: "e9261eb85ea443d5a3beb36bd0c549f0" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34239 } } peers { permanent_uuid: "e5e5222b26fe4145948620e51edf53fb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41233 } }
I20251029 02:51:49.149231 607 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e9261eb85ea443d5a3beb36bd0c549f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e5e5222b26fe4145948620e51edf53fb"
I20251029 02:51:49.150669 465 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e9261eb85ea443d5a3beb36bd0c549f0" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e9672114a6a742e2864c16ff60ba246e"
I20251029 02:51:49.150758 855 leader_election.cc:290] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0 [CANDIDATE]: Term 2 election: Requested vote from peers e9672114a6a742e2864c16ff60ba246e (127.0.104.62:42343), e5e5222b26fe4145948620e51edf53fb (127.0.104.60:41233)
I20251029 02:51:49.150892 465 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e9672114a6a742e2864c16ff60ba246e [term 1 FOLLOWER]: Advancing to term 2
I20251029 02:51:49.152715 536 raft_consensus.cc:1240] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0 [term 2 FOLLOWER]: Rejecting Update request from peer e5e5222b26fe4145948620e51edf53fb for earlier term 1. Current term is 2. Ops: []
I20251029 02:51:49.153440 872 consensus_queue.cc:1059] T 00000000000000000000000000000000 P e5e5222b26fe4145948620e51edf53fb [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e9261eb85ea443d5a3beb36bd0c549f0" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34239 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251029 02:51:49.154068 465 raft_consensus.cc:2468] T 00000000000000000000000000000000 P e9672114a6a742e2864c16ff60ba246e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9261eb85ea443d5a3beb36bd0c549f0 in term 2.
I20251029 02:51:49.154731 510 leader_election.cc:304] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e9261eb85ea443d5a3beb36bd0c549f0, e9672114a6a742e2864c16ff60ba246e; no voters: e5e5222b26fe4145948620e51edf53fb
I20251029 02:51:49.155076 855 raft_consensus.cc:2804] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0 [term 2 FOLLOWER]: Leader election won for term 2
I20251029 02:51:49.155218 855 raft_consensus.cc:697] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0 [term 2 LEADER]: Becoming Leader. State: Replica: e9261eb85ea443d5a3beb36bd0c549f0, State: Running, Role: LEADER
I20251029 02:51:49.155857 880 raft_consensus.cc:3055] T 00000000000000000000000000000000 P e5e5222b26fe4145948620e51edf53fb [term 1 LEADER]: Stepping down as leader of term 1
I20251029 02:51:49.155745 855 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9672114a6a742e2864c16ff60ba246e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42343 } } peers { permanent_uuid: "e9261eb85ea443d5a3beb36bd0c549f0" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34239 } } peers { permanent_uuid: "e5e5222b26fe4145948620e51edf53fb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41233 } }
I20251029 02:51:49.156077 880 raft_consensus.cc:740] T 00000000000000000000000000000000 P e5e5222b26fe4145948620e51edf53fb [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e5e5222b26fe4145948620e51edf53fb, State: Running, Role: LEADER
I20251029 02:51:49.156488 880 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5e5222b26fe4145948620e51edf53fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9672114a6a742e2864c16ff60ba246e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42343 } } peers { permanent_uuid: "e9261eb85ea443d5a3beb36bd0c549f0" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34239 } } peers { permanent_uuid: "e5e5222b26fe4145948620e51edf53fb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41233 } }
I20251029 02:51:49.157092 880 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e5e5222b26fe4145948620e51edf53fb [term 1 FOLLOWER]: Advancing to term 2
I20251029 02:51:49.159579 868 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9261eb85ea443d5a3beb36bd0c549f0. Latest consensus state: current_term: 2 leader_uuid: "e9261eb85ea443d5a3beb36bd0c549f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9672114a6a742e2864c16ff60ba246e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42343 } } peers { permanent_uuid: "e9261eb85ea443d5a3beb36bd0c549f0" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34239 } } peers { permanent_uuid: "e5e5222b26fe4145948620e51edf53fb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41233 } } }
I20251029 02:51:49.159840 868 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0 [sys.catalog]: This master's current role is: LEADER
I20251029 02:51:49.161798 897 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251029 02:51:49.163929 897 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251029 02:51:49.165611 897 catalog_manager.cc:1269] Loaded cluster ID: 96c87f4c6dea451cb609ec9ee56d85bf
I20251029 02:51:49.165709 897 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251029 02:51:49.167402 897 catalog_manager.cc:1514] Loading token signing keys...
I20251029 02:51:49.168434 897 catalog_manager.cc:6033] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Loaded TSK: 0
I20251029 02:51:49.169351 897 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251029 02:51:49.185878 831 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.2:41701
I20251029 02:51:49.185947 964 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.2:41701 every 8 connection(s)
I20251029 02:51:49.187381 831 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/data/info.pb
I20251029 02:51:49.194559 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskYVokEG/build/asan/bin/kudu as pid 831
I20251029 02:51:49.194777 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/instance
I20251029 02:51:49.201416 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskYVokEG/build/asan/bin/kudu
/tmp/dist-test-taskYVokEG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.3:0
--local_ip_for_outbound_sockets=127.0.104.3
--webserver_interface=127.0.104.3
--webserver_port=0
--tserver_master_addrs=127.0.104.62:42343,127.0.104.61:34239,127.0.104.60:41233
--builtin_ntp_servers=127.0.104.20:44691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20251029 02:51:49.207003 966 heartbeater.cc:344] Connected to a master server at 127.0.104.60:41233
I20251029 02:51:49.207477 966 heartbeater.cc:461] Registering TS with master...
I20251029 02:51:49.207791 965 heartbeater.cc:344] Connected to a master server at 127.0.104.62:42343
I20251029 02:51:49.207908 965 heartbeater.cc:461] Registering TS with master...
I20251029 02:51:49.208808 966 heartbeater.cc:507] Master 127.0.104.60:41233 requested a full tablet report, sending...
I20251029 02:51:49.208837 965 heartbeater.cc:507] Master 127.0.104.62:42343 requested a full tablet report, sending...
I20251029 02:51:49.208863 822 heartbeater.cc:499] Master 127.0.104.60:41233 was elected leader, sending a full tablet report...
I20251029 02:51:49.209110 967 heartbeater.cc:344] Connected to a master server at 127.0.104.61:34239
I20251029 02:51:49.209255 967 heartbeater.cc:461] Registering TS with master...
I20251029 02:51:49.209825 967 heartbeater.cc:507] Master 127.0.104.61:34239 requested a full tablet report, sending...
I20251029 02:51:49.210701 525 ts_manager.cc:194] Registered new tserver with Master: c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:51:49.210712 455 ts_manager.cc:194] Registered new tserver with Master: c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:51:49.210891 597 ts_manager.cc:194] Registered new tserver with Master: c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:51:49.212347 525 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.104.2:37633
W20251029 02:51:49.552732 973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251029 02:51:49.553045 973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251029 02:51:49.553139 973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251029 02:51:49.561769 973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251029 02:51:49.561949 973 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.3
I20251029 02:51:49.572005 973 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:44691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.104.3
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.0.104.62:42343,127.0.104.61:34239,127.0.104.60:41233
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.3
--log_dir=/tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 60f5e5267b92c39485a66121d3ce3cc7ef57b0e0
build type FASTDEBUG
built by None at 29 Oct 2025 02:43:25 UTC on 24a791456cd2
build id 8768
ASAN enabled
I20251029 02:51:49.574148 973 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251029 02:51:49.576020 973 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251029 02:51:49.585973 979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251029 02:51:49.586004 978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251029 02:51:49.586170 981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251029 02:51:49.587111 973 server_base.cc:1047] running on GCE node
I20251029 02:51:49.587728 973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251029 02:51:49.588704 973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251029 02:51:49.589910 973 hybrid_clock.cc:648] HybridClock initialized: now 1761706309589874 us; error 48 us; skew 500 ppm
I20251029 02:51:49.590217 973 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251029 02:51:49.592552 973 webserver.cc:492] Webserver started at http://127.0.104.3:45387/ using document root <none> and password file <none>
I20251029 02:51:49.593354 973 fs_manager.cc:362] Metadata directory not provided
I20251029 02:51:49.593542 973 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251029 02:51:49.593976 973 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251029 02:51:49.594918 653 catalog_manager.cc:1612] T 00000000000000000000000000000000 P e5e5222b26fe4145948620e51edf53fb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251029 02:51:49.596431 973 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/data/instance:
uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
format_stamp: "Formatted at 2025-10-29 02:51:49 on dist-test-slave-qwrb"
I20251029 02:51:49.597198 973 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/instance:
uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
format_stamp: "Formatted at 2025-10-29 02:51:49 on dist-test-slave-qwrb"
I20251029 02:51:49.602916 973 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20251029 02:51:49.606456 987 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251029 02:51:49.607875 973 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20251029 02:51:49.608031 973 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal
uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
format_stamp: "Formatted at 2025-10-29 02:51:49 on dist-test-slave-qwrb"
I20251029 02:51:49.608244 973 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251029 02:51:49.652323 973 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251029 02:51:49.653226 973 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251029 02:51:49.653494 973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251029 02:51:49.654645 973 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251029 02:51:49.655177 607 raft_consensus.cc:1275] T 00000000000000000000000000000000 P e5e5222b26fe4145948620e51edf53fb [term 2 FOLLOWER]: Refusing update from remote peer e9261eb85ea443d5a3beb36bd0c549f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251029 02:51:49.655951 868 consensus_queue.cc:1048] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5e5222b26fe4145948620e51edf53fb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251029 02:51:49.656687 973 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251029 02:51:49.656787 973 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251029 02:51:49.656895 973 ts_tablet_manager.cc:616] Registered 0 tablets
I20251029 02:51:49.656982 973 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251029 02:51:49.657964 880 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5e5222b26fe4145948620e51edf53fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9261eb85ea443d5a3beb36bd0c549f0. Latest consensus state: current_term: 2 leader_uuid: "e9261eb85ea443d5a3beb36bd0c549f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9672114a6a742e2864c16ff60ba246e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42343 } } peers { permanent_uuid: "e9261eb85ea443d5a3beb36bd0c549f0" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34239 } } peers { permanent_uuid: "e5e5222b26fe4145948620e51edf53fb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41233 } } }
I20251029 02:51:49.658129 880 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5e5222b26fe4145948620e51edf53fb [sys.catalog]: This master's current role is: FOLLOWER
I20251029 02:51:49.659811 855 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "e9261eb85ea443d5a3beb36bd0c549f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9672114a6a742e2864c16ff60ba246e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42343 } } peers { permanent_uuid: "e9261eb85ea443d5a3beb36bd0c549f0" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34239 } } peers { permanent_uuid: "e5e5222b26fe4145948620e51edf53fb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41233 } } }
I20251029 02:51:49.660116 855 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0 [sys.catalog]: This master's current role is: LEADER
I20251029 02:51:49.661011 465 raft_consensus.cc:1275] T 00000000000000000000000000000000 P e9672114a6a742e2864c16ff60ba246e [term 2 FOLLOWER]: Refusing update from remote peer e9261eb85ea443d5a3beb36bd0c549f0: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20251029 02:51:49.661423 880 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5e5222b26fe4145948620e51edf53fb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "e9261eb85ea443d5a3beb36bd0c549f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9672114a6a742e2864c16ff60ba246e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42343 } } peers { permanent_uuid: "e9261eb85ea443d5a3beb36bd0c549f0" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34239 } } peers { permanent_uuid: "e5e5222b26fe4145948620e51edf53fb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41233 } } }
I20251029 02:51:49.661626 880 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5e5222b26fe4145948620e51edf53fb [sys.catalog]: This master's current role is: FOLLOWER
I20251029 02:51:49.661607 855 consensus_queue.cc:1048] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9672114a6a742e2864c16ff60ba246e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20251029 02:51:49.664415 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9672114a6a742e2864c16ff60ba246e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "e9261eb85ea443d5a3beb36bd0c549f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9672114a6a742e2864c16ff60ba246e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42343 } } peers { permanent_uuid: "e9261eb85ea443d5a3beb36bd0c549f0" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34239 } } peers { permanent_uuid: "e5e5222b26fe4145948620e51edf53fb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41233 } } }
I20251029 02:51:49.664615 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9672114a6a742e2864c16ff60ba246e [sys.catalog]: This master's current role is: FOLLOWER
I20251029 02:51:49.664618 999 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9672114a6a742e2864c16ff60ba246e [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9261eb85ea443d5a3beb36bd0c549f0. Latest consensus state: current_term: 2 leader_uuid: "e9261eb85ea443d5a3beb36bd0c549f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9672114a6a742e2864c16ff60ba246e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42343 } } peers { permanent_uuid: "e9261eb85ea443d5a3beb36bd0c549f0" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34239 } } peers { permanent_uuid: "e5e5222b26fe4145948620e51edf53fb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41233 } } }
I20251029 02:51:49.665103 999 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9672114a6a742e2864c16ff60ba246e [sys.catalog]: This master's current role is: FOLLOWER
I20251029 02:51:49.699015 973 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.3:45003
I20251029 02:51:49.699075 1106 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.3:45003 every 8 connection(s)
I20251029 02:51:49.700332 823 heartbeater.cc:499] Master 127.0.104.61:34239 was elected leader, sending a full tablet report...
I20251029 02:51:49.700575 973 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/data/info.pb
I20251029 02:51:49.706341 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskYVokEG/build/asan/bin/kudu as pid 973
I20251029 02:51:49.706584 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/instance
I20251029 02:51:49.715143 967 heartbeater.cc:499] Master 127.0.104.61:34239 was elected leader, sending a full tablet report...
I20251029 02:51:49.725064 1108 heartbeater.cc:344] Connected to a master server at 127.0.104.60:41233
I20251029 02:51:49.725468 1108 heartbeater.cc:461] Registering TS with master...
I20251029 02:51:49.725957 1109 heartbeater.cc:344] Connected to a master server at 127.0.104.61:34239
I20251029 02:51:49.726094 1109 heartbeater.cc:461] Registering TS with master...
I20251029 02:51:49.726462 1108 heartbeater.cc:507] Master 127.0.104.60:41233 requested a full tablet report, sending...
I20251029 02:51:49.726596 1109 heartbeater.cc:507] Master 127.0.104.61:34239 requested a full tablet report, sending...
I20251029 02:51:49.727252 1107 heartbeater.cc:344] Connected to a master server at 127.0.104.62:42343
I20251029 02:51:49.727375 1107 heartbeater.cc:461] Registering TS with master...
I20251029 02:51:49.727834 1107 heartbeater.cc:507] Master 127.0.104.62:42343 requested a full tablet report, sending...
I20251029 02:51:49.728080 597 ts_manager.cc:194] Registered new tserver with Master: 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:51:49.728130 525 ts_manager.cc:194] Registered new tserver with Master: 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:51:49.728837 455 ts_manager.cc:194] Registered new tserver with Master: 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:51:49.729151 525 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.104.3:48627
I20251029 02:51:49.735117 416 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251029 02:51:49.785063 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskYVokEG/build/asan/bin/kudu pid 563
I20251029 02:51:49.778254 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_c63b7cadeb794da8898043bf1d17d825"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:49.778578 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_046964a136d243398aa29780b335a81f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:49.778333 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_876ae95cfcec456baa0e6aa194a41cbf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:49.786988 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c63b7cadeb794da8898043bf1d17d825 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:51:49.786993 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_046964a136d243398aa29780b335a81f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:49.778789 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_ba822c78e7944fdd938333aab3bf3ec6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:49.787556 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ba822c78e7944fdd938333aab3bf3ec6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:49.779008 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_4b2eefdc44dc47e8ae3d6c99796e70cb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:49.788400 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4b2eefdc44dc47e8ae3d6c99796e70cb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:49.779198 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_6680c24512764984801e90e4cc1ed97e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:49.789166 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6680c24512764984801e90e4cc1ed97e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:49.779419 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_32be3ff89a7b43a8887395cdc00ab91d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:49.790670 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_32be3ff89a7b43a8887395cdc00ab91d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:49.778082 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_1be9773476d14e20bbf0dd51bdec4514"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:49.792140 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_876ae95cfcec456baa0e6aa194a41cbf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:51:49.792452 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1be9773476d14e20bbf0dd51bdec4514 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:49.778038 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_34496044f52d481fa74884859fbf8da0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:49.794613 627 generic_service.cc:196] Checking for leaks (request via RPC)
I20251029 02:51:49.779626 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_bb9a5bf1703d426989326891b2df5b6b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:49.797295 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bb9a5bf1703d426989326891b2df5b6b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:51:49.799536 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_34496044f52d481fa74884859fbf8da0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:49.808386 855 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "e9261eb85ea443d5a3beb36bd0c549f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9672114a6a742e2864c16ff60ba246e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42343 } } peers { permanent_uuid: "e9261eb85ea443d5a3beb36bd0c549f0" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34239 } } peers { permanent_uuid: "e5e5222b26fe4145948620e51edf53fb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41233 } } }
I20251029 02:51:49.808665 855 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0 [sys.catalog]: This master's current role is: LEADER
I20251029 02:51:49.830139 521 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_c63b7cadeb794da8898043bf1d17d825" } new_table_name: "default.Table_6d2603a353964b8b821b14a31b259c95" modify_external_catalogs: true
I20251029 02:51:49.825026 525 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_6680c24512764984801e90e4cc1ed97e" } new_table_name: "default.Table_7a7b22a6e750401c9751e3ad5fbb42ed" modify_external_catalogs: true
I20251029 02:51:49.836531 522 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_4b2eefdc44dc47e8ae3d6c99796e70cb" } new_table_name: "default.Table_1dc0fd70708a4fa6828722209bb71dcb" modify_external_catalogs: true
I20251029 02:51:49.836614 519 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_046964a136d243398aa29780b335a81f" } new_table_name: "default.Table_83374ee2c0324d7c8bc8ecefed677919" modify_external_catalogs: true
I20251029 02:51:49.837059 524 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_bb9a5bf1703d426989326891b2df5b6b" } modify_external_catalogs: true
I20251029 02:51:49.837262 524 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_bb9a5bf1703d426989326891b2df5b6b" } modify_external_catalogs: true
I20251029 02:51:49.838001 520 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_876ae95cfcec456baa0e6aa194a41cbf" } modify_external_catalogs: true
I20251029 02:51:49.838192 520 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_876ae95cfcec456baa0e6aa194a41cbf" } modify_external_catalogs: true
I20251029 02:51:49.838346 526 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_ba822c78e7944fdd938333aab3bf3ec6" } modify_external_catalogs: true
I20251029 02:51:49.838456 517 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_1be9773476d14e20bbf0dd51bdec4514" } modify_external_catalogs: true
I20251029 02:51:49.838596 526 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_ba822c78e7944fdd938333aab3bf3ec6" } modify_external_catalogs: true
I20251029 02:51:49.838672 517 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_1be9773476d14e20bbf0dd51bdec4514" } modify_external_catalogs: true
I20251029 02:51:49.839040 523 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_34496044f52d481fa74884859fbf8da0" } modify_external_catalogs: true
I20251029 02:51:49.839226 523 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_34496044f52d481fa74884859fbf8da0" } modify_external_catalogs: true
I20251029 02:51:49.839704 518 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_32be3ff89a7b43a8887395cdc00ab91d" } new_table_name: "default.Table_f971a5ce4a2641c3ae82667fd94a2e62" modify_external_catalogs: true
W20251029 02:51:49.848248 521 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8cb92fa59f114b01aa27a28358ecb4e7 has no consensus state
W20251029 02:51:49.849495 525 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 91aa5291a1d442598f7dcbcf0f1cbd2f has no consensus state
I20251029 02:51:49.850610 520 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 822ff4457c31438e8e9fda06645cb1ca
W20251029 02:51:49.849815 519 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 24114a60aeb24653aa7384df675aac76 has no consensus state
W20251029 02:51:49.850106 522 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a6d728bd089841a1a5ba997f8a71862a has no consensus state
I20251029 02:51:49.863817 517 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 40e6e6a59d854c72a5d5fcfa6269648d
I20251029 02:51:49.863854 523 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet b75816e2405144c2a4683f8e095a05f8
I20251029 02:51:49.864523 524 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet ea15704bb4804e6ca09c0b9bf6920701
I20251029 02:51:49.864684 526 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 8b217f889f6c4937a2928cf7f931d387
W20251029 02:51:49.864759 518 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 09d2c3ee989d477ba0100d41d6f214b5 has no consensus state
W20251029 02:51:49.878787 513 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 91aa5291a1d442598f7dcbcf0f1cbd2f (table default.Table_7a7b22a6e750401c9751e3ad5fbb42ed [id=18cf9b24efed4f1095ff99a7fe2022fb], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 91aa5291a1d442598f7dcbcf0f1cbd2f consensus state has no leader
I20251029 02:51:49.890918 1038 tablet_service.cc:1505] Processing CreateTablet for tablet a6d728bd089841a1a5ba997f8a71862a (DEFAULT_TABLE table=default.Table_1dc0fd70708a4fa6828722209bb71dcb [id=1a1aeb237dad4464b717bbae5d4001b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:49.890805 1042 tablet_service.cc:1505] Processing CreateTablet for tablet 91aa5291a1d442598f7dcbcf0f1cbd2f (DEFAULT_TABLE table=default.Table_7a7b22a6e750401c9751e3ad5fbb42ed [id=18cf9b24efed4f1095ff99a7fe2022fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:49.892500 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6d728bd089841a1a5ba997f8a71862a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:49.892921 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91aa5291a1d442598f7dcbcf0f1cbd2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:49.894824 1039 tablet_service.cc:1505] Processing CreateTablet for tablet 09d2c3ee989d477ba0100d41d6f214b5 (DEFAULT_TABLE table=default.Table_f971a5ce4a2641c3ae82667fd94a2e62 [id=d0cd3d1567c54463947443927e18169f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:49.895257 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09d2c3ee989d477ba0100d41d6f214b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:49.896759 898 tablet_service.cc:1505] Processing CreateTablet for tablet 24114a60aeb24653aa7384df675aac76 (DEFAULT_TABLE table=default.Table_83374ee2c0324d7c8bc8ecefed677919 [id=782a7aaf782749d384c78095cda17dc4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:49.897855 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24114a60aeb24653aa7384df675aac76. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:49.898648 1041 tablet_service.cc:1505] Processing CreateTablet for tablet 8cb92fa59f114b01aa27a28358ecb4e7 (DEFAULT_TABLE table=default.Table_6d2603a353964b8b821b14a31b259c95 [id=78615d8807a74b4496a3223ef6fe709d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:49.898872 896 tablet_service.cc:1505] Processing CreateTablet for tablet 09d2c3ee989d477ba0100d41d6f214b5 (DEFAULT_TABLE table=default.Table_f971a5ce4a2641c3ae82667fd94a2e62 [id=d0cd3d1567c54463947443927e18169f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:49.899092 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8cb92fa59f114b01aa27a28358ecb4e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:49.899266 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09d2c3ee989d477ba0100d41d6f214b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:49.902873 900 tablet_service.cc:1505] Processing CreateTablet for tablet 91aa5291a1d442598f7dcbcf0f1cbd2f (DEFAULT_TABLE table=default.Table_7a7b22a6e750401c9751e3ad5fbb42ed [id=18cf9b24efed4f1095ff99a7fe2022fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:49.902956 895 tablet_service.cc:1505] Processing CreateTablet for tablet a6d728bd089841a1a5ba997f8a71862a (DEFAULT_TABLE table=default.Table_1dc0fd70708a4fa6828722209bb71dcb [id=1a1aeb237dad4464b717bbae5d4001b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:49.903470 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91aa5291a1d442598f7dcbcf0f1cbd2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:49.903538 899 tablet_service.cc:1505] Processing CreateTablet for tablet 8cb92fa59f114b01aa27a28358ecb4e7 (DEFAULT_TABLE table=default.Table_6d2603a353964b8b821b14a31b259c95 [id=78615d8807a74b4496a3223ef6fe709d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:49.904034 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8cb92fa59f114b01aa27a28358ecb4e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:49.906706 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6d728bd089841a1a5ba997f8a71862a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:49.910610 1040 tablet_service.cc:1505] Processing CreateTablet for tablet 24114a60aeb24653aa7384df675aac76 (DEFAULT_TABLE table=default.Table_83374ee2c0324d7c8bc8ecefed677919 [id=782a7aaf782749d384c78095cda17dc4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:49.911089 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24114a60aeb24653aa7384df675aac76. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:49.931546 1167 tablet_bootstrap.cc:492] T a6d728bd089841a1a5ba997f8a71862a P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:51:49.931676 1166 tablet_bootstrap.cc:492] T 91aa5291a1d442598f7dcbcf0f1cbd2f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:51:49.936483 1167 tablet_bootstrap.cc:654] T a6d728bd089841a1a5ba997f8a71862a P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:49.937873 1166 tablet_bootstrap.cc:654] T 91aa5291a1d442598f7dcbcf0f1cbd2f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:49.938058 1167 log.cc:826] T a6d728bd089841a1a5ba997f8a71862a P c756528642a543eeacd93cef3e335a7d: Log is configured to *not* fsync() on all Append() calls
I20251029 02:51:49.939620 1166 log.cc:826] T 91aa5291a1d442598f7dcbcf0f1cbd2f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Log is configured to *not* fsync() on all Append() calls
I20251029 02:51:49.944398 756 tablet_service.cc:1505] Processing CreateTablet for tablet 91aa5291a1d442598f7dcbcf0f1cbd2f (DEFAULT_TABLE table=default.Table_7a7b22a6e750401c9751e3ad5fbb42ed [id=18cf9b24efed4f1095ff99a7fe2022fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:49.946221 1166 tablet_bootstrap.cc:492] T 91aa5291a1d442598f7dcbcf0f1cbd2f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:51:49.946661 1166 ts_tablet_manager.cc:1403] T 91aa5291a1d442598f7dcbcf0f1cbd2f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.007s
I20251029 02:51:49.945410 755 tablet_service.cc:1505] Processing CreateTablet for tablet 8cb92fa59f114b01aa27a28358ecb4e7 (DEFAULT_TABLE table=default.Table_6d2603a353964b8b821b14a31b259c95 [id=78615d8807a74b4496a3223ef6fe709d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:49.948102 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8cb92fa59f114b01aa27a28358ecb4e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:49.945842 754 tablet_service.cc:1505] Processing CreateTablet for tablet 24114a60aeb24653aa7384df675aac76 (DEFAULT_TABLE table=default.Table_83374ee2c0324d7c8bc8ecefed677919 [id=782a7aaf782749d384c78095cda17dc4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:49.950946 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24114a60aeb24653aa7384df675aac76. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:49.946238 753 tablet_service.cc:1505] Processing CreateTablet for tablet 09d2c3ee989d477ba0100d41d6f214b5 (DEFAULT_TABLE table=default.Table_f971a5ce4a2641c3ae82667fd94a2e62 [id=d0cd3d1567c54463947443927e18169f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:49.951725 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09d2c3ee989d477ba0100d41d6f214b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:49.946630 752 tablet_service.cc:1505] Processing CreateTablet for tablet a6d728bd089841a1a5ba997f8a71862a (DEFAULT_TABLE table=default.Table_1dc0fd70708a4fa6828722209bb71dcb [id=1a1aeb237dad4464b717bbae5d4001b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:49.952458 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a6d728bd089841a1a5ba997f8a71862a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:49.957371 1167 tablet_bootstrap.cc:492] T a6d728bd089841a1a5ba997f8a71862a P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:51:49.957744 1167 ts_tablet_manager.cc:1403] T a6d728bd089841a1a5ba997f8a71862a P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.026s user 0.017s sys 0.008s
I20251029 02:51:49.957530 1166 raft_consensus.cc:359] T 91aa5291a1d442598f7dcbcf0f1cbd2f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:49.958027 1166 raft_consensus.cc:385] T 91aa5291a1d442598f7dcbcf0f1cbd2f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:49.958205 1166 raft_consensus.cc:740] T 91aa5291a1d442598f7dcbcf0f1cbd2f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:51:49.959200 1166 consensus_queue.cc:260] T 91aa5291a1d442598f7dcbcf0f1cbd2f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:49.966873 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91aa5291a1d442598f7dcbcf0f1cbd2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:49.967573 1167 raft_consensus.cc:359] T a6d728bd089841a1a5ba997f8a71862a P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:49.968031 1167 raft_consensus.cc:385] T a6d728bd089841a1a5ba997f8a71862a P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:49.968192 1167 raft_consensus.cc:740] T a6d728bd089841a1a5ba997f8a71862a P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:51:49.968924 1167 consensus_queue.cc:260] T a6d728bd089841a1a5ba997f8a71862a P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:49.977216 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskYVokEG/build/asan/bin/kudu with pid 563
I20251029 02:51:49.978900 518 master_service.cc:954] ReplaceTablet: received request to replace tablet 09d2c3ee989d477ba0100d41d6f214b5 from {username='slave'} at 127.0.0.1:35314
I20251029 02:51:49.991899 526 master_service.cc:954] ReplaceTablet: received request to replace tablet 91aa5291a1d442598f7dcbcf0f1cbd2f from {username='slave'} at 127.0.0.1:35314
I20251029 02:51:49.993463 524 master_service.cc:954] ReplaceTablet: received request to replace tablet a6d728bd089841a1a5ba997f8a71862a from {username='slave'} at 127.0.0.1:35314
I20251029 02:51:49.994839 523 master_service.cc:954] ReplaceTablet: received request to replace tablet 09d2c3ee989d477ba0100d41d6f214b5 from {username='slave'} at 127.0.0.1:35314
I20251029 02:51:49.995136 517 master_service.cc:954] ReplaceTablet: received request to replace tablet 91aa5291a1d442598f7dcbcf0f1cbd2f from {username='slave'} at 127.0.0.1:35314
I20251029 02:51:49.996246 1172 tablet_bootstrap.cc:492] T 09d2c3ee989d477ba0100d41d6f214b5 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:51:49.997942 1109 heartbeater.cc:499] Master 127.0.104.61:34239 was elected leader, sending a full tablet report...
I20251029 02:51:50.000161 1172 tablet_bootstrap.cc:654] T 09d2c3ee989d477ba0100d41d6f214b5 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:50.001523 1172 log.cc:826] T 09d2c3ee989d477ba0100d41d6f214b5 P b495a49349324f7d81908f2da700f0d9: Log is configured to *not* fsync() on all Append() calls
I20251029 02:51:50.005421 524 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet a6d728bd089841a1a5ba997f8a71862a
I20251029 02:51:50.006279 524 catalog_manager.cc:6719] ReplaceTablet: tablet a6d728bd089841a1a5ba997f8a71862a deleted and replaced by tablet 02d3583e77714d08ae476cbfdb2583c9
I20251029 02:51:50.007884 900 tablet_service.cc:1552] Processing DeleteTablet for tablet a6d728bd089841a1a5ba997f8a71862a with delete_type TABLET_DATA_DELETED (replaced by tablet 02d3583e77714d08ae476cbfdb2583c9) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:50.009251 753 tablet_service.cc:1552] Processing DeleteTablet for tablet a6d728bd089841a1a5ba997f8a71862a with delete_type TABLET_DATA_DELETED (replaced by tablet 02d3583e77714d08ae476cbfdb2583c9) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:50.009647 1166 ts_tablet_manager.cc:1434] T 91aa5291a1d442598f7dcbcf0f1cbd2f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.063s user 0.017s sys 0.001s
I20251029 02:51:50.010627 1040 tablet_service.cc:1552] Processing DeleteTablet for tablet a6d728bd089841a1a5ba997f8a71862a with delete_type TABLET_DATA_DELETED (replaced by tablet 02d3583e77714d08ae476cbfdb2583c9) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:50.011061 526 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 91aa5291a1d442598f7dcbcf0f1cbd2f
W20251029 02:51:50.011180 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet a6d728bd089841a1a5ba997f8a71862a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a6d728bd089841a1a5ba997f8a71862a already in progress: creating tablet
I20251029 02:51:50.011602 526 catalog_manager.cc:6719] ReplaceTablet: tablet 91aa5291a1d442598f7dcbcf0f1cbd2f deleted and replaced by tablet 2d2ed6d2389c4f1e91e6f6423b89b3bc
I20251029 02:51:50.012614 895 tablet_service.cc:1552] Processing DeleteTablet for tablet 91aa5291a1d442598f7dcbcf0f1cbd2f with delete_type TABLET_DATA_DELETED (replaced by tablet 2d2ed6d2389c4f1e91e6f6423b89b3bc) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:50.013146 753 tablet_service.cc:1552] Processing DeleteTablet for tablet 91aa5291a1d442598f7dcbcf0f1cbd2f with delete_type TABLET_DATA_DELETED (replaced by tablet 2d2ed6d2389c4f1e91e6f6423b89b3bc) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:50.013185 1038 tablet_service.cc:1552] Processing DeleteTablet for tablet 91aa5291a1d442598f7dcbcf0f1cbd2f with delete_type TABLET_DATA_DELETED (replaced by tablet 2d2ed6d2389c4f1e91e6f6423b89b3bc) from {username='slave'} at 127.0.0.1:37626
W20251029 02:51:50.014104 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 91aa5291a1d442598f7dcbcf0f1cbd2f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 91aa5291a1d442598f7dcbcf0f1cbd2f already in progress: creating tablet
I20251029 02:51:50.017715 518 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 09d2c3ee989d477ba0100d41d6f214b5
I20251029 02:51:50.018301 518 catalog_manager.cc:6719] ReplaceTablet: tablet 09d2c3ee989d477ba0100d41d6f214b5 deleted and replaced by tablet 648ad50f1dca4c0f96a5e5da089d12d1
I20251029 02:51:50.019697 521 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet 8cb92fa59f114b01aa27a28358ecb4e7 (table default.Table_6d2603a353964b8b821b14a31b259c95 [id=78615d8807a74b4496a3223ef6fe709d]). Expected version 1 got 0
I20251029 02:51:50.019994 521 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet 24114a60aeb24653aa7384df675aac76 (table default.Table_83374ee2c0324d7c8bc8ecefed677919 [id=782a7aaf782749d384c78095cda17dc4]). Expected version 1 got 0
I20251029 02:51:50.022466 1166 tablet_bootstrap.cc:492] T 09d2c3ee989d477ba0100d41d6f214b5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:51:50.024616 1166 tablet_bootstrap.cc:654] T 09d2c3ee989d477ba0100d41d6f214b5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
W20251029 02:51:50.025615 990 connection.cc:537] client connection to 127.0.104.60:41233 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251029 02:51:50.026351 1172 tablet_bootstrap.cc:492] T 09d2c3ee989d477ba0100d41d6f214b5 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
W20251029 02:51:50.026480 1108 heartbeater.cc:646] Failed to heartbeat to 127.0.104.60:41233 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251029 02:51:50.026698 1172 ts_tablet_manager.cc:1403] T 09d2c3ee989d477ba0100d41d6f214b5 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.031s user 0.007s sys 0.007s
I20251029 02:51:50.031967 1176 tablet_replica.cc:333] T 91aa5291a1d442598f7dcbcf0f1cbd2f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:51:50.032550 1176 raft_consensus.cc:2243] T 91aa5291a1d442598f7dcbcf0f1cbd2f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:50.033047 1176 raft_consensus.cc:2272] T 91aa5291a1d442598f7dcbcf0f1cbd2f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:50.033008 895 tablet_service.cc:1552] Processing DeleteTablet for tablet 09d2c3ee989d477ba0100d41d6f214b5 with delete_type TABLET_DATA_DELETED (replaced by tablet 648ad50f1dca4c0f96a5e5da089d12d1) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:50.034574 753 tablet_service.cc:1552] Processing DeleteTablet for tablet 09d2c3ee989d477ba0100d41d6f214b5 with delete_type TABLET_DATA_DELETED (replaced by tablet 648ad50f1dca4c0f96a5e5da089d12d1) from {username='slave'} at 127.0.0.1:44580
W20251029 02:51:50.035449 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet a6d728bd089841a1a5ba997f8a71862a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a6d728bd089841a1a5ba997f8a71862a already in progress: creating tablet
I20251029 02:51:50.037663 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_c4aefe30acb547f5b9bf3b1f645f3e4e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:50.037834 1113 negotiation.cc:337] Failed RPC negotiation. Trace:
1029 02:51:50.027982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:41233 (local address 127.0.104.3:52887)
1029 02:51:50.028637 (+ 655us) negotiation.cc:107] Waiting for socket to connect
1029 02:51:50.028785 (+ 148us) client_negotiation.cc:174] Beginning negotiation
1029 02:51:50.029149 (+ 364us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1029 02:51:50.036663 (+ 7514us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:41233: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":149}
W20251029 02:51:50.038487 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c4aefe30acb547f5b9bf3b1f645f3e4e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:50.040035 1166 tablet_bootstrap.cc:492] T 09d2c3ee989d477ba0100d41d6f214b5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:51:50.040272 1166 ts_tablet_manager.cc:1403] T 09d2c3ee989d477ba0100d41d6f214b5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.004s
I20251029 02:51:50.040752 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_19b1787d124341939d34a6e721a87ae9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:50.041421 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_19b1787d124341939d34a6e721a87ae9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:50.042855 1176 ts_tablet_manager.cc:1916] T 91aa5291a1d442598f7dcbcf0f1cbd2f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:50.043334 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_8312118d2b5e40748caaa1d849c5af51"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:50.043984 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8312118d2b5e40748caaa1d849c5af51 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:51:50.045107 512 consensus_peers.cc:597] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0 -> Peer e5e5222b26fe4145948620e51edf53fb (127.0.104.60:41233): Couldn't send request to peer e5e5222b26fe4145948620e51edf53fb. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.60:41233: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251029 02:51:50.043435 1166 raft_consensus.cc:359] T 09d2c3ee989d477ba0100d41d6f214b5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:50.045326 1166 raft_consensus.cc:385] T 09d2c3ee989d477ba0100d41d6f214b5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:50.045455 1166 raft_consensus.cc:740] T 09d2c3ee989d477ba0100d41d6f214b5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:51:50.045758 1166 consensus_queue.cc:260] T 09d2c3ee989d477ba0100d41d6f214b5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:50.046146 1040 tablet_service.cc:1552] Processing DeleteTablet for tablet 09d2c3ee989d477ba0100d41d6f214b5 with delete_type TABLET_DATA_DELETED (replaced by tablet 648ad50f1dca4c0f96a5e5da089d12d1) from {username='slave'} at 127.0.0.1:37626
W20251029 02:51:50.046339 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 09d2c3ee989d477ba0100d41d6f214b5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 09d2c3ee989d477ba0100d41d6f214b5 already in progress: creating tablet
I20251029 02:51:50.047448 1166 ts_tablet_manager.cc:1434] T 09d2c3ee989d477ba0100d41d6f214b5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20251029 02:51:50.050081 1172 raft_consensus.cc:359] T 09d2c3ee989d477ba0100d41d6f214b5 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:50.050490 1172 raft_consensus.cc:385] T 09d2c3ee989d477ba0100d41d6f214b5 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:50.050633 1172 raft_consensus.cc:740] T 09d2c3ee989d477ba0100d41d6f214b5 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:51:50.051331 1172 consensus_queue.cc:260] T 09d2c3ee989d477ba0100d41d6f214b5 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:50.053021 1166 tablet_bootstrap.cc:492] T 8cb92fa59f114b01aa27a28358ecb4e7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:51:50.055179 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_c6dccb4bd9f4464bbd23d27234ae267e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:50.055706 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c6dccb4bd9f4464bbd23d27234ae267e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:50.056139 1167 ts_tablet_manager.cc:1434] T a6d728bd089841a1a5ba997f8a71862a P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.098s user 0.045s sys 0.046s
I20251029 02:51:50.056167 1176 ts_tablet_manager.cc:1929] T 91aa5291a1d442598f7dcbcf0f1cbd2f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:50.056636 1167 tablet_bootstrap.cc:492] T 09d2c3ee989d477ba0100d41d6f214b5 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:51:50.057034 1166 tablet_bootstrap.cc:654] T 8cb92fa59f114b01aa27a28358ecb4e7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:50.058531 1167 tablet_bootstrap.cc:654] T 09d2c3ee989d477ba0100d41d6f214b5 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:50.060835 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_b13dc3449d0c412fad8f65f6e9ff6a99"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:50.061501 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b13dc3449d0c412fad8f65f6e9ff6a99 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:50.056308 1176 log.cc:1199] T 91aa5291a1d442598f7dcbcf0f1cbd2f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/91aa5291a1d442598f7dcbcf0f1cbd2f
I20251029 02:51:50.063043 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_a85023af366f423b8a637b058d6c65b4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:50.063186 1176 ts_tablet_manager.cc:1950] T 91aa5291a1d442598f7dcbcf0f1cbd2f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
W20251029 02:51:50.063905 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a85023af366f423b8a637b058d6c65b4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:50.067296 1172 ts_tablet_manager.cc:1434] T 09d2c3ee989d477ba0100d41d6f214b5 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.040s user 0.010s sys 0.005s
I20251029 02:51:50.064311 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_9c3c143c7d464c32acb4176464f0fb64"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:50.068256 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9c3c143c7d464c32acb4176464f0fb64 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:50.074383 1176 tablet_replica.cc:333] T 09d2c3ee989d477ba0100d41d6f214b5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:51:50.075240 1176 raft_consensus.cc:2243] T 09d2c3ee989d477ba0100d41d6f214b5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:50.075707 1176 raft_consensus.cc:2272] T 09d2c3ee989d477ba0100d41d6f214b5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:50.076715 1176 ts_tablet_manager.cc:1916] T 09d2c3ee989d477ba0100d41d6f214b5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:50.081081 1172 tablet_bootstrap.cc:492] T 24114a60aeb24653aa7384df675aac76 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:51:50.083182 1172 tablet_bootstrap.cc:654] T 24114a60aeb24653aa7384df675aac76 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
W20251029 02:51:50.084743 966 heartbeater.cc:646] Failed to heartbeat to 127.0.104.60:41233 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.104.60:41233: connect: Connection refused (error 111)
I20251029 02:51:50.085763 895 tablet_service.cc:1505] Processing CreateTablet for tablet 02d3583e77714d08ae476cbfdb2583c9 (DEFAULT_TABLE table=default.Table_1dc0fd70708a4fa6828722209bb71dcb [id=1a1aeb237dad4464b717bbae5d4001b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:50.086144 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02d3583e77714d08ae476cbfdb2583c9. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:51:50.095100 822 heartbeater.cc:646] Failed to heartbeat to 127.0.104.60:41233 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.104.60:41233: connect: Connection refused (error 111)
I20251029 02:51:50.095820 1166 tablet_bootstrap.cc:492] T 8cb92fa59f114b01aa27a28358ecb4e7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:51:50.096060 1166 ts_tablet_manager.cc:1403] T 8cb92fa59f114b01aa27a28358ecb4e7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.043s user 0.008s sys 0.004s
I20251029 02:51:50.097003 1166 raft_consensus.cc:359] T 8cb92fa59f114b01aa27a28358ecb4e7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:50.097189 1176 ts_tablet_manager.cc:1929] T 09d2c3ee989d477ba0100d41d6f214b5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:50.097288 1166 raft_consensus.cc:385] T 8cb92fa59f114b01aa27a28358ecb4e7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:50.097379 1176 log.cc:1199] T 09d2c3ee989d477ba0100d41d6f214b5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/09d2c3ee989d477ba0100d41d6f214b5
I20251029 02:51:50.097743 1167 tablet_bootstrap.cc:492] T 09d2c3ee989d477ba0100d41d6f214b5 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:51:50.097797 1176 ts_tablet_manager.cc:1950] T 09d2c3ee989d477ba0100d41d6f214b5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:51:50.097988 1167 ts_tablet_manager.cc:1403] T 09d2c3ee989d477ba0100d41d6f214b5 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.041s user 0.017s sys 0.021s
I20251029 02:51:50.097426 1166 raft_consensus.cc:740] T 8cb92fa59f114b01aa27a28358ecb4e7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:51:50.098847 1166 consensus_queue.cc:260] T 8cb92fa59f114b01aa27a28358ecb4e7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:50.099349 1166 ts_tablet_manager.cc:1434] T 8cb92fa59f114b01aa27a28358ecb4e7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20251029 02:51:50.099345 1167 raft_consensus.cc:359] T 09d2c3ee989d477ba0100d41d6f214b5 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:50.099632 1167 raft_consensus.cc:385] T 09d2c3ee989d477ba0100d41d6f214b5 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:50.099771 1167 raft_consensus.cc:740] T 09d2c3ee989d477ba0100d41d6f214b5 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:51:50.099943 1166 tablet_bootstrap.cc:492] T a6d728bd089841a1a5ba997f8a71862a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:51:50.100132 1167 consensus_queue.cc:260] T 09d2c3ee989d477ba0100d41d6f214b5 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:50.100704 1167 ts_tablet_manager.cc:1434] T 09d2c3ee989d477ba0100d41d6f214b5 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251029 02:51:50.101649 1166 tablet_bootstrap.cc:654] T a6d728bd089841a1a5ba997f8a71862a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:50.102749 1167 tablet_bootstrap.cc:492] T 8cb92fa59f114b01aa27a28358ecb4e7 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:51:50.104547 1167 tablet_bootstrap.cc:654] T 8cb92fa59f114b01aa27a28358ecb4e7 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:50.107144 1172 tablet_bootstrap.cc:492] T 24114a60aeb24653aa7384df675aac76 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:51:50.107488 1172 ts_tablet_manager.cc:1403] T 24114a60aeb24653aa7384df675aac76 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.002s
I20251029 02:51:50.066303 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_78f7811a55f243faa1be1c3f90754dcf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:50.108604 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_78f7811a55f243faa1be1c3f90754dcf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:50.108846 1172 raft_consensus.cc:359] T 24114a60aeb24653aa7384df675aac76 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:50.109086 1172 raft_consensus.cc:385] T 24114a60aeb24653aa7384df675aac76 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:50.109186 1172 raft_consensus.cc:740] T 24114a60aeb24653aa7384df675aac76 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:51:50.065339 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_fd9d401f3f92450cab95de4f0895a1e1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:50.109777 1172 consensus_queue.cc:260] T 24114a60aeb24653aa7384df675aac76 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
W20251029 02:51:50.110343 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fd9d401f3f92450cab95de4f0895a1e1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:50.075567 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 91aa5291a1d442598f7dcbcf0f1cbd2f (table default.Table_7a7b22a6e750401c9751e3ad5fbb42ed [id=18cf9b24efed4f1095ff99a7fe2022fb]) successfully deleted
I20251029 02:51:50.112830 1172 ts_tablet_manager.cc:1434] T 24114a60aeb24653aa7384df675aac76 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251029 02:51:50.112895 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 09d2c3ee989d477ba0100d41d6f214b5 (table default.Table_f971a5ce4a2641c3ae82667fd94a2e62 [id=d0cd3d1567c54463947443927e18169f]) successfully deleted
I20251029 02:51:50.115109 1172 tablet_bootstrap.cc:492] T 8cb92fa59f114b01aa27a28358ecb4e7 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:51:50.116916 1040 tablet_service.cc:1505] Processing CreateTablet for tablet 02d3583e77714d08ae476cbfdb2583c9 (DEFAULT_TABLE table=default.Table_1dc0fd70708a4fa6828722209bb71dcb [id=1a1aeb237dad4464b717bbae5d4001b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:50.117461 1172 tablet_bootstrap.cc:654] T 8cb92fa59f114b01aa27a28358ecb4e7 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:50.117493 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02d3583e77714d08ae476cbfdb2583c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:50.094136 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_14462d3ac1c24574963a67d66a1cb274"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:50.124282 1172 tablet_bootstrap.cc:492] T 8cb92fa59f114b01aa27a28358ecb4e7 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
W20251029 02:51:50.124275 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_14462d3ac1c24574963a67d66a1cb274 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:50.124619 1172 ts_tablet_manager.cc:1403] T 8cb92fa59f114b01aa27a28358ecb4e7 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.003s
I20251029 02:51:50.125675 1172 raft_consensus.cc:359] T 8cb92fa59f114b01aa27a28358ecb4e7 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:50.126037 1172 raft_consensus.cc:385] T 8cb92fa59f114b01aa27a28358ecb4e7 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:50.126201 1172 raft_consensus.cc:740] T 8cb92fa59f114b01aa27a28358ecb4e7 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:51:50.126538 753 tablet_service.cc:1505] Processing CreateTablet for tablet 02d3583e77714d08ae476cbfdb2583c9 (DEFAULT_TABLE table=default.Table_1dc0fd70708a4fa6828722209bb71dcb [id=1a1aeb237dad4464b717bbae5d4001b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:50.126653 1172 consensus_queue.cc:260] T 8cb92fa59f114b01aa27a28358ecb4e7 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:50.127085 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02d3583e77714d08ae476cbfdb2583c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:50.130183 1172 ts_tablet_manager.cc:1434] T 8cb92fa59f114b01aa27a28358ecb4e7 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251029 02:51:50.131047 754 tablet_service.cc:1552] Processing DeleteTablet for tablet 09d2c3ee989d477ba0100d41d6f214b5 with delete_type TABLET_DATA_DELETED (replaced by tablet 648ad50f1dca4c0f96a5e5da089d12d1) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:50.131429 1175 tablet_replica.cc:333] T 09d2c3ee989d477ba0100d41d6f214b5 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:51:50.131938 1175 raft_consensus.cc:2243] T 09d2c3ee989d477ba0100d41d6f214b5 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:50.132390 1175 raft_consensus.cc:2272] T 09d2c3ee989d477ba0100d41d6f214b5 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:50.132985 1172 tablet_bootstrap.cc:492] T 91aa5291a1d442598f7dcbcf0f1cbd2f P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:51:50.134245 1175 ts_tablet_manager.cc:1916] T 09d2c3ee989d477ba0100d41d6f214b5 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:50.135318 1172 tablet_bootstrap.cc:654] T 91aa5291a1d442598f7dcbcf0f1cbd2f P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:50.139397 1166 tablet_bootstrap.cc:492] T a6d728bd089841a1a5ba997f8a71862a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:51:50.139823 1166 ts_tablet_manager.cc:1403] T a6d728bd089841a1a5ba997f8a71862a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.040s user 0.010s sys 0.023s
I20251029 02:51:50.140955 1166 raft_consensus.cc:359] T a6d728bd089841a1a5ba997f8a71862a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:50.141336 1166 raft_consensus.cc:385] T a6d728bd089841a1a5ba997f8a71862a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:50.141508 1166 raft_consensus.cc:740] T a6d728bd089841a1a5ba997f8a71862a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:51:50.141593 1175 ts_tablet_manager.cc:1929] T 09d2c3ee989d477ba0100d41d6f214b5 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:50.141827 1175 log.cc:1199] T 09d2c3ee989d477ba0100d41d6f214b5 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/09d2c3ee989d477ba0100d41d6f214b5
I20251029 02:51:50.141922 1166 consensus_queue.cc:260] T a6d728bd089841a1a5ba997f8a71862a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:50.142328 1175 ts_tablet_manager.cc:1950] T 09d2c3ee989d477ba0100d41d6f214b5 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:51:50.146414 1166 ts_tablet_manager.cc:1434] T a6d728bd089841a1a5ba997f8a71862a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251029 02:51:50.150033 1166 tablet_bootstrap.cc:492] T 24114a60aeb24653aa7384df675aac76 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:51:50.152210 1166 tablet_bootstrap.cc:654] T 24114a60aeb24653aa7384df675aac76 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:50.159466 1166 tablet_bootstrap.cc:492] T 24114a60aeb24653aa7384df675aac76 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:51:50.159724 1166 ts_tablet_manager.cc:1403] T 24114a60aeb24653aa7384df675aac76 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.002s
I20251029 02:51:50.160612 1166 raft_consensus.cc:359] T 24114a60aeb24653aa7384df675aac76 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:50.097637 517 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_7a7b22a6e750401c9751e3ad5fbb42ed" } modify_external_catalogs: true
I20251029 02:51:50.160878 1172 tablet_bootstrap.cc:492] T 91aa5291a1d442598f7dcbcf0f1cbd2f P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:51:50.160892 1166 raft_consensus.cc:385] T 24114a60aeb24653aa7384df675aac76 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:50.160964 1166 raft_consensus.cc:740] T 24114a60aeb24653aa7384df675aac76 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:51:50.099831 519 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_6d2603a353964b8b821b14a31b259c95" } new_table_name: "default.Table_4bdbcbb87e5b40aeb9f98eb1aa0e6fe3" modify_external_catalogs: true
I20251029 02:51:50.161136 517 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_7a7b22a6e750401c9751e3ad5fbb42ed" } modify_external_catalogs: true
I20251029 02:51:50.161199 1166 consensus_queue.cc:260] T 24114a60aeb24653aa7384df675aac76 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:50.161710 1166 ts_tablet_manager.cc:1434] T 24114a60aeb24653aa7384df675aac76 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251029 02:51:50.161027 1172 ts_tablet_manager.cc:1403] T 91aa5291a1d442598f7dcbcf0f1cbd2f P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.028s user 0.007s sys 0.013s
I20251029 02:51:50.163564 1172 raft_consensus.cc:359] T 91aa5291a1d442598f7dcbcf0f1cbd2f P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:50.163916 1172 raft_consensus.cc:385] T 91aa5291a1d442598f7dcbcf0f1cbd2f P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:50.164078 1172 raft_consensus.cc:740] T 91aa5291a1d442598f7dcbcf0f1cbd2f P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:51:50.164495 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 09d2c3ee989d477ba0100d41d6f214b5 (table default.Table_f971a5ce4a2641c3ae82667fd94a2e62 [id=d0cd3d1567c54463947443927e18169f]) successfully deleted
I20251029 02:51:50.169000 1172 consensus_queue.cc:260] T 91aa5291a1d442598f7dcbcf0f1cbd2f P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:50.173521 1172 ts_tablet_manager.cc:1434] T 91aa5291a1d442598f7dcbcf0f1cbd2f P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
W20251029 02:51:50.174407 824 tablet.cc:2378] T 24114a60aeb24653aa7384df675aac76 P b495a49349324f7d81908f2da700f0d9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251029 02:51:50.177348 1172 tablet_bootstrap.cc:492] T a6d728bd089841a1a5ba997f8a71862a P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:51:50.177390 526 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_83374ee2c0324d7c8bc8ecefed677919" } new_table_name: "default.Table_f4a825a9cffc4afea35104d88069df1e" modify_external_catalogs: true
I20251029 02:51:50.178026 1166 tablet_bootstrap.cc:492] T 02d3583e77714d08ae476cbfdb2583c9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:51:50.179828 1166 tablet_bootstrap.cc:654] T 02d3583e77714d08ae476cbfdb2583c9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:50.180493 1172 tablet_bootstrap.cc:654] T a6d728bd089841a1a5ba997f8a71862a P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:50.181571 1174 tablet_replica.cc:333] T a6d728bd089841a1a5ba997f8a71862a P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:51:50.182092 1174 raft_consensus.cc:2243] T a6d728bd089841a1a5ba997f8a71862a P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:50.182572 1174 raft_consensus.cc:2272] T a6d728bd089841a1a5ba997f8a71862a P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:50.187959 1174 ts_tablet_manager.cc:1916] T a6d728bd089841a1a5ba997f8a71862a P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:50.191587 524 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet 24114a60aeb24653aa7384df675aac76 (table default.Table_83374ee2c0324d7c8bc8ecefed677919 [id=782a7aaf782749d384c78095cda17dc4]). Expected version 1 got 0
W20251029 02:51:50.191635 968 tablet.cc:2378] T 09d2c3ee989d477ba0100d41d6f214b5 P c756528642a543eeacd93cef3e335a7d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251029 02:51:50.193034 1040 tablet_service.cc:1505] Processing CreateTablet for tablet 81c770c1c9a84637b71989257cb4453a (DEFAULT_TABLE table=default.Table_19b1787d124341939d34a6e721a87ae9 [id=33950ad96f8140fbb26ab0cd8b2b2b97]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:50.193461 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81c770c1c9a84637b71989257cb4453a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:50.194903 522 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet 8cb92fa59f114b01aa27a28358ecb4e7 (table default.Table_6d2603a353964b8b821b14a31b259c95 [id=78615d8807a74b4496a3223ef6fe709d]). Expected version 1 got 0
I20251029 02:51:50.197331 753 tablet_service.cc:1505] Processing CreateTablet for tablet 81c770c1c9a84637b71989257cb4453a (DEFAULT_TABLE table=default.Table_19b1787d124341939d34a6e721a87ae9 [id=33950ad96f8140fbb26ab0cd8b2b2b97]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:50.197865 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81c770c1c9a84637b71989257cb4453a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:50.201001 1167 tablet_bootstrap.cc:492] T 8cb92fa59f114b01aa27a28358ecb4e7 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:51:50.201213 1167 ts_tablet_manager.cc:1403] T 8cb92fa59f114b01aa27a28358ecb4e7 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.099s user 0.024s sys 0.046s
I20251029 02:51:50.202050 1167 raft_consensus.cc:359] T 8cb92fa59f114b01aa27a28358ecb4e7 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:50.202333 1167 raft_consensus.cc:385] T 8cb92fa59f114b01aa27a28358ecb4e7 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:50.202488 1167 raft_consensus.cc:740] T 8cb92fa59f114b01aa27a28358ecb4e7 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:51:50.202855 1167 consensus_queue.cc:260] T 8cb92fa59f114b01aa27a28358ecb4e7 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:50.203162 752 tablet_service.cc:1505] Processing CreateTablet for tablet 2d2ed6d2389c4f1e91e6f6423b89b3bc (DEFAULT_TABLE table=default.Table_7a7b22a6e750401c9751e3ad5fbb42ed [id=18cf9b24efed4f1095ff99a7fe2022fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:50.203573 1167 ts_tablet_manager.cc:1434] T 8cb92fa59f114b01aa27a28358ecb4e7 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 02:51:50.203670 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d2ed6d2389c4f1e91e6f6423b89b3bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:50.204097 1167 tablet_bootstrap.cc:492] T 24114a60aeb24653aa7384df675aac76 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:51:50.205750 1167 tablet_bootstrap.cc:654] T 24114a60aeb24653aa7384df675aac76 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:50.207079 895 tablet_service.cc:1505] Processing CreateTablet for tablet 81c770c1c9a84637b71989257cb4453a (DEFAULT_TABLE table=default.Table_19b1787d124341939d34a6e721a87ae9 [id=33950ad96f8140fbb26ab0cd8b2b2b97]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:50.207585 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81c770c1c9a84637b71989257cb4453a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:50.210950 898 tablet_service.cc:1505] Processing CreateTablet for tablet 8aaa0a9dd62443d68f96d7bd44a0c02b (DEFAULT_TABLE table=default.Table_c4aefe30acb547f5b9bf3b1f645f3e4e [id=2ce1f471df9d4210af37a1b4758daaf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:50.211412 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8aaa0a9dd62443d68f96d7bd44a0c02b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:50.212034 1172 tablet_bootstrap.cc:492] T a6d728bd089841a1a5ba997f8a71862a P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:51:50.212268 1172 ts_tablet_manager.cc:1403] T a6d728bd089841a1a5ba997f8a71862a P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.007s
I20251029 02:51:50.213294 899 tablet_service.cc:1505] Processing CreateTablet for tablet 2d2ed6d2389c4f1e91e6f6423b89b3bc (DEFAULT_TABLE table=default.Table_7a7b22a6e750401c9751e3ad5fbb42ed [id=18cf9b24efed4f1095ff99a7fe2022fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:50.213303 1172 raft_consensus.cc:359] T a6d728bd089841a1a5ba997f8a71862a P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:50.213533 1172 raft_consensus.cc:385] T a6d728bd089841a1a5ba997f8a71862a P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:50.213622 1172 raft_consensus.cc:740] T a6d728bd089841a1a5ba997f8a71862a P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:51:50.213706 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d2ed6d2389c4f1e91e6f6423b89b3bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:50.213917 1172 consensus_queue.cc:260] T a6d728bd089841a1a5ba997f8a71862a P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:50.218106 1172 ts_tablet_manager.cc:1434] T a6d728bd089841a1a5ba997f8a71862a P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
W20251029 02:51:50.219168 1110 tablet.cc:2378] T 24114a60aeb24653aa7384df675aac76 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251029 02:51:50.220871 1172 tablet_bootstrap.cc:492] T 02d3583e77714d08ae476cbfdb2583c9 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:51:50.222803 754 tablet_service.cc:1505] Processing CreateTablet for tablet 8aaa0a9dd62443d68f96d7bd44a0c02b (DEFAULT_TABLE table=default.Table_c4aefe30acb547f5b9bf3b1f645f3e4e [id=2ce1f471df9d4210af37a1b4758daaf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:50.226248 1040 tablet_service.cc:1505] Processing CreateTablet for tablet 8aaa0a9dd62443d68f96d7bd44a0c02b (DEFAULT_TABLE table=default.Table_c4aefe30acb547f5b9bf3b1f645f3e4e [id=2ce1f471df9d4210af37a1b4758daaf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:50.226390 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8aaa0a9dd62443d68f96d7bd44a0c02b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:50.226687 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8aaa0a9dd62443d68f96d7bd44a0c02b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:50.226284 1038 tablet_service.cc:1505] Processing CreateTablet for tablet 2d2ed6d2389c4f1e91e6f6423b89b3bc (DEFAULT_TABLE table=default.Table_7a7b22a6e750401c9751e3ad5fbb42ed [id=18cf9b24efed4f1095ff99a7fe2022fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:50.230870 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d2ed6d2389c4f1e91e6f6423b89b3bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:50.238332 1172 tablet_bootstrap.cc:654] T 02d3583e77714d08ae476cbfdb2583c9 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:50.242273 756 tablet_service.cc:1505] Processing CreateTablet for tablet 648ad50f1dca4c0f96a5e5da089d12d1 (DEFAULT_TABLE table=default.Table_f971a5ce4a2641c3ae82667fd94a2e62 [id=d0cd3d1567c54463947443927e18169f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:50.242602 1174 ts_tablet_manager.cc:1929] T a6d728bd089841a1a5ba997f8a71862a P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:50.242820 1174 log.cc:1199] T a6d728bd089841a1a5ba997f8a71862a P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/a6d728bd089841a1a5ba997f8a71862a
I20251029 02:51:50.242846 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 648ad50f1dca4c0f96a5e5da089d12d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:50.243413 1174 ts_tablet_manager.cc:1950] T a6d728bd089841a1a5ba997f8a71862a P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:51:50.245702 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet a6d728bd089841a1a5ba997f8a71862a (table default.Table_1dc0fd70708a4fa6828722209bb71dcb [id=1a1aeb237dad4464b717bbae5d4001b0]) successfully deleted
I20251029 02:51:50.246196 1174 tablet_replica.cc:333] T 09d2c3ee989d477ba0100d41d6f214b5 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
W20251029 02:51:50.246284 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 91aa5291a1d442598f7dcbcf0f1cbd2f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 91aa5291a1d442598f7dcbcf0f1cbd2f already in progress: creating tablet
I20251029 02:51:50.246387 1174 raft_consensus.cc:2243] T 09d2c3ee989d477ba0100d41d6f214b5 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:50.248011 896 tablet_service.cc:1505] Processing CreateTablet for tablet 648ad50f1dca4c0f96a5e5da089d12d1 (DEFAULT_TABLE table=default.Table_f971a5ce4a2641c3ae82667fd94a2e62 [id=d0cd3d1567c54463947443927e18169f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:50.248415 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 648ad50f1dca4c0f96a5e5da089d12d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:50.248531 1172 tablet_bootstrap.cc:492] T 02d3583e77714d08ae476cbfdb2583c9 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:51:50.248660 1172 ts_tablet_manager.cc:1403] T 02d3583e77714d08ae476cbfdb2583c9 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.028s user 0.003s sys 0.001s
I20251029 02:51:50.249509 1172 raft_consensus.cc:359] T 02d3583e77714d08ae476cbfdb2583c9 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:50.249719 1172 raft_consensus.cc:385] T 02d3583e77714d08ae476cbfdb2583c9 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:50.249804 1172 raft_consensus.cc:740] T 02d3583e77714d08ae476cbfdb2583c9 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:51:50.250025 1172 consensus_queue.cc:260] T 02d3583e77714d08ae476cbfdb2583c9 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:50.253010 1040 tablet_service.cc:1505] Processing CreateTablet for tablet 648ad50f1dca4c0f96a5e5da089d12d1 (DEFAULT_TABLE table=default.Table_f971a5ce4a2641c3ae82667fd94a2e62 [id=d0cd3d1567c54463947443927e18169f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:50.253391 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 648ad50f1dca4c0f96a5e5da089d12d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:50.258769 1172 ts_tablet_manager.cc:1434] T 02d3583e77714d08ae476cbfdb2583c9 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20251029 02:51:50.262758 1167 tablet_bootstrap.cc:492] T 24114a60aeb24653aa7384df675aac76 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:51:50.262969 1167 ts_tablet_manager.cc:1403] T 24114a60aeb24653aa7384df675aac76 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.059s user 0.021s sys 0.006s
I20251029 02:51:50.263844 1167 raft_consensus.cc:359] T 24114a60aeb24653aa7384df675aac76 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:50.264089 1167 raft_consensus.cc:385] T 24114a60aeb24653aa7384df675aac76 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:50.264226 1167 raft_consensus.cc:740] T 24114a60aeb24653aa7384df675aac76 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:51:50.264550 1167 consensus_queue.cc:260] T 24114a60aeb24653aa7384df675aac76 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:50.265378 1174 raft_consensus.cc:2272] T 09d2c3ee989d477ba0100d41d6f214b5 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:50.266103 1174 ts_tablet_manager.cc:1916] T 09d2c3ee989d477ba0100d41d6f214b5 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:50.266790 1167 ts_tablet_manager.cc:1434] T 24114a60aeb24653aa7384df675aac76 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251029 02:51:50.267921 1167 tablet_bootstrap.cc:492] T 91aa5291a1d442598f7dcbcf0f1cbd2f P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:51:50.270659 1167 tablet_bootstrap.cc:654] T 91aa5291a1d442598f7dcbcf0f1cbd2f P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:50.278267 1174 ts_tablet_manager.cc:1929] T 09d2c3ee989d477ba0100d41d6f214b5 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:50.278564 1174 log.cc:1199] T 09d2c3ee989d477ba0100d41d6f214b5 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/09d2c3ee989d477ba0100d41d6f214b5
I20251029 02:51:50.279112 1174 ts_tablet_manager.cc:1950] T 09d2c3ee989d477ba0100d41d6f214b5 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:51:50.279734 1172 tablet_bootstrap.cc:492] T 2d2ed6d2389c4f1e91e6f6423b89b3bc P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:51:50.280576 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 09d2c3ee989d477ba0100d41d6f214b5 (table default.Table_f971a5ce4a2641c3ae82667fd94a2e62 [id=d0cd3d1567c54463947443927e18169f]) successfully deleted
I20251029 02:51:50.281843 1167 tablet_bootstrap.cc:492] T 91aa5291a1d442598f7dcbcf0f1cbd2f P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:51:50.281999 1167 ts_tablet_manager.cc:1403] T 91aa5291a1d442598f7dcbcf0f1cbd2f P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.010s
I20251029 02:51:50.282238 1172 tablet_bootstrap.cc:654] T 2d2ed6d2389c4f1e91e6f6423b89b3bc P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:50.282691 1167 raft_consensus.cc:359] T 91aa5291a1d442598f7dcbcf0f1cbd2f P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:50.283178 1167 raft_consensus.cc:385] T 91aa5291a1d442598f7dcbcf0f1cbd2f P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:50.293893 1172 tablet_bootstrap.cc:492] T 2d2ed6d2389c4f1e91e6f6423b89b3bc P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:51:50.294126 1172 ts_tablet_manager.cc:1403] T 2d2ed6d2389c4f1e91e6f6423b89b3bc P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.004s
I20251029 02:51:50.294447 1167 raft_consensus.cc:740] T 91aa5291a1d442598f7dcbcf0f1cbd2f P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:51:50.295011 1172 raft_consensus.cc:359] T 2d2ed6d2389c4f1e91e6f6423b89b3bc P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:50.295223 1172 raft_consensus.cc:385] T 2d2ed6d2389c4f1e91e6f6423b89b3bc P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:50.295310 1172 raft_consensus.cc:740] T 2d2ed6d2389c4f1e91e6f6423b89b3bc P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:51:50.295475 1167 consensus_queue.cc:260] T 91aa5291a1d442598f7dcbcf0f1cbd2f P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:50.296077 1166 tablet_bootstrap.cc:492] T 02d3583e77714d08ae476cbfdb2583c9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:51:50.296083 1167 ts_tablet_manager.cc:1434] T 91aa5291a1d442598f7dcbcf0f1cbd2f P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20251029 02:51:50.296277 1166 ts_tablet_manager.cc:1403] T 02d3583e77714d08ae476cbfdb2583c9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.118s user 0.032s sys 0.080s
I20251029 02:51:50.296968 1167 tablet_bootstrap.cc:492] T 02d3583e77714d08ae476cbfdb2583c9 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:51:50.297108 1166 raft_consensus.cc:359] T 02d3583e77714d08ae476cbfdb2583c9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:50.297338 1166 raft_consensus.cc:385] T 02d3583e77714d08ae476cbfdb2583c9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:50.297464 1166 raft_consensus.cc:740] T 02d3583e77714d08ae476cbfdb2583c9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:51:50.297740 1166 consensus_queue.cc:260] T 02d3583e77714d08ae476cbfdb2583c9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:50.298030 1172 consensus_queue.cc:260] T 2d2ed6d2389c4f1e91e6f6423b89b3bc P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:50.298801 1167 tablet_bootstrap.cc:654] T 02d3583e77714d08ae476cbfdb2583c9 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:50.299139 520 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet 8cb92fa59f114b01aa27a28358ecb4e7 (table default.Table_6d2603a353964b8b821b14a31b259c95 [id=78615d8807a74b4496a3223ef6fe709d]). Expected version 1 got 0
I20251029 02:51:50.300192 1166 ts_tablet_manager.cc:1434] T 02d3583e77714d08ae476cbfdb2583c9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251029 02:51:50.300674 1172 ts_tablet_manager.cc:1434] T 2d2ed6d2389c4f1e91e6f6423b89b3bc P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20251029 02:51:50.301106 1172 tablet_bootstrap.cc:492] T 8aaa0a9dd62443d68f96d7bd44a0c02b P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:51:50.302938 1172 tablet_bootstrap.cc:654] T 8aaa0a9dd62443d68f96d7bd44a0c02b P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:50.303560 525 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_f971a5ce4a2641c3ae82667fd94a2e62" } new_table_name: "default.Table_9f11b26c8c144d7a8f6e2fbb3c6fe36e" modify_external_catalogs: true
I20251029 02:51:50.305526 521 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_1dc0fd70708a4fa6828722209bb71dcb" } new_table_name: "default.Table_5b3a7640a9e54e7a9868985322ade0cb" modify_external_catalogs: true
I20251029 02:51:50.307117 523 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_c4aefe30acb547f5b9bf3b1f645f3e4e" } modify_external_catalogs: true
I20251029 02:51:50.307312 523 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_c4aefe30acb547f5b9bf3b1f645f3e4e" } modify_external_catalogs: true
I20251029 02:51:50.308363 1166 tablet_bootstrap.cc:492] T 81c770c1c9a84637b71989257cb4453a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:51:50.309172 518 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_19b1787d124341939d34a6e721a87ae9" } modify_external_catalogs: true
I20251029 02:51:50.309369 518 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_19b1787d124341939d34a6e721a87ae9" } modify_external_catalogs: true
I20251029 02:51:50.310653 1166 tablet_bootstrap.cc:654] T 81c770c1c9a84637b71989257cb4453a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:50.345065 855 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "e9261eb85ea443d5a3beb36bd0c549f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9672114a6a742e2864c16ff60ba246e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42343 } } peers { permanent_uuid: "e9261eb85ea443d5a3beb36bd0c549f0" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34239 } } peers { permanent_uuid: "e5e5222b26fe4145948620e51edf53fb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41233 } } }
I20251029 02:51:50.345391 855 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0 [sys.catalog]: This master's current role is: LEADER
I20251029 02:51:50.350958 1171 raft_consensus.cc:493] T 24114a60aeb24653aa7384df675aac76 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:50.351254 1171 raft_consensus.cc:515] T 24114a60aeb24653aa7384df675aac76 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:50.353443 1171 leader_election.cc:290] T 24114a60aeb24653aa7384df675aac76 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:51:50.353644 517 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 2d2ed6d2389c4f1e91e6f6423b89b3bc
W20251029 02:51:50.353941 526 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 24114a60aeb24653aa7384df675aac76 consensus state has no leader
I20251029 02:51:50.354399 526 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_8312118d2b5e40748caaa1d849c5af51" } modify_external_catalogs: true
I20251029 02:51:50.354642 517 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_b13dc3449d0c412fad8f65f6e9ff6a99" } modify_external_catalogs: true
I20251029 02:51:50.354892 517 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_b13dc3449d0c412fad8f65f6e9ff6a99" } modify_external_catalogs: true
I20251029 02:51:50.355222 524 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_c6dccb4bd9f4464bbd23d27234ae267e" } new_table_name: "default.Table_83ad3367766c49269223bd91b060a972" modify_external_catalogs: true
I20251029 02:51:50.356021 522 master_service.cc:954] ReplaceTablet: received request to replace tablet 8cb92fa59f114b01aa27a28358ecb4e7 from {username='slave'} at 127.0.0.1:35314
I20251029 02:51:50.357704 756 tablet_service.cc:1552] Processing DeleteTablet for tablet 2d2ed6d2389c4f1e91e6f6423b89b3bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:50 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:50.357859 520 master_service.cc:954] ReplaceTablet: received request to replace tablet a6d728bd089841a1a5ba997f8a71862a from {username='slave'} at 127.0.0.1:35314
I20251029 02:51:50.358381 520 master_service.cc:954] ReplaceTablet: received request to replace tablet 02d3583e77714d08ae476cbfdb2583c9 from {username='slave'} at 127.0.0.1:35314
I20251029 02:51:50.358580 1175 tablet_replica.cc:333] T 2d2ed6d2389c4f1e91e6f6423b89b3bc P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:51:50.358891 1175 raft_consensus.cc:2243] T 2d2ed6d2389c4f1e91e6f6423b89b3bc P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:50.359133 1175 raft_consensus.cc:2272] T 2d2ed6d2389c4f1e91e6f6423b89b3bc P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:50.359622 1040 tablet_service.cc:1552] Processing DeleteTablet for tablet 2d2ed6d2389c4f1e91e6f6423b89b3bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:50 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:50.360002 1175 ts_tablet_manager.cc:1916] T 2d2ed6d2389c4f1e91e6f6423b89b3bc P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:50.360620 898 tablet_service.cc:1552] Processing DeleteTablet for tablet 2d2ed6d2389c4f1e91e6f6423b89b3bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:50 UTC) from {username='slave'} at 127.0.0.1:35702
W20251029 02:51:50.353681 519 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8cb92fa59f114b01aa27a28358ecb4e7 consensus state has no leader
I20251029 02:51:50.361353 519 master_service.cc:954] ReplaceTablet: received request to replace tablet 8aaa0a9dd62443d68f96d7bd44a0c02b from {username='slave'} at 127.0.0.1:35314
W20251029 02:51:50.362416 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 2d2ed6d2389c4f1e91e6f6423b89b3bc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2d2ed6d2389c4f1e91e6f6423b89b3bc already in progress: creating tablet
I20251029 02:51:50.354640 526 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_8312118d2b5e40748caaa1d849c5af51" } modify_external_catalogs: true
W20251029 02:51:50.366909 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 2d2ed6d2389c4f1e91e6f6423b89b3bc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2d2ed6d2389c4f1e91e6f6423b89b3bc already in progress: creating tablet
I20251029 02:51:50.367651 1175 ts_tablet_manager.cc:1929] T 2d2ed6d2389c4f1e91e6f6423b89b3bc P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:50.367781 1175 log.cc:1199] T 2d2ed6d2389c4f1e91e6f6423b89b3bc P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/2d2ed6d2389c4f1e91e6f6423b89b3bc
I20251029 02:51:50.368129 1175 ts_tablet_manager.cc:1950] T 2d2ed6d2389c4f1e91e6f6423b89b3bc P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:51:50.369275 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 2d2ed6d2389c4f1e91e6f6423b89b3bc (table default.Table_7a7b22a6e750401c9751e3ad5fbb42ed [id=18cf9b24efed4f1095ff99a7fe2022fb]) successfully deleted
I20251029 02:51:50.376380 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24114a60aeb24653aa7384df675aac76" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:51:50.376942 776 raft_consensus.cc:2468] T 24114a60aeb24653aa7384df675aac76 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:51:50.377987 847 leader_election.cc:304] T 24114a60aeb24653aa7384df675aac76 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:51:50.378554 1171 raft_consensus.cc:2804] T 24114a60aeb24653aa7384df675aac76 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:51:50.378736 1171 raft_consensus.cc:493] T 24114a60aeb24653aa7384df675aac76 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:51:50.378849 1171 raft_consensus.cc:3060] T 24114a60aeb24653aa7384df675aac76 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:51:50.383459 1171 raft_consensus.cc:515] T 24114a60aeb24653aa7384df675aac76 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
W20251029 02:51:50.386098 525 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 648ad50f1dca4c0f96a5e5da089d12d1 consensus state has no leader
I20251029 02:51:50.387456 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24114a60aeb24653aa7384df675aac76" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:51:50.387754 523 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 8aaa0a9dd62443d68f96d7bd44a0c02b
I20251029 02:51:50.387705 776 raft_consensus.cc:3060] T 24114a60aeb24653aa7384df675aac76 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:51:50.389405 518 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 81c770c1c9a84637b71989257cb4453a
I20251029 02:51:50.392109 898 tablet_service.cc:1552] Processing DeleteTablet for tablet 8aaa0a9dd62443d68f96d7bd44a0c02b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:50 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:50.392237 776 raft_consensus.cc:2468] T 24114a60aeb24653aa7384df675aac76 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:51:50.392810 847 leader_election.cc:304] T 24114a60aeb24653aa7384df675aac76 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:51:50.393400 756 tablet_service.cc:1552] Processing DeleteTablet for tablet 8aaa0a9dd62443d68f96d7bd44a0c02b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:50 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:50.392112 895 tablet_service.cc:1552] Processing DeleteTablet for tablet 81c770c1c9a84637b71989257cb4453a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:50 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:50.393581 753 tablet_service.cc:1552] Processing DeleteTablet for tablet 81c770c1c9a84637b71989257cb4453a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:50 UTC) from {username='slave'} at 127.0.0.1:44580
W20251029 02:51:50.394254 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 8aaa0a9dd62443d68f96d7bd44a0c02b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8aaa0a9dd62443d68f96d7bd44a0c02b already in progress: creating tablet
W20251029 02:51:50.394579 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 81c770c1c9a84637b71989257cb4453a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 81c770c1c9a84637b71989257cb4453a already in progress: creating tablet
I20251029 02:51:50.395228 1040 tablet_service.cc:1552] Processing DeleteTablet for tablet 8aaa0a9dd62443d68f96d7bd44a0c02b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:50 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:50.395553 1038 tablet_service.cc:1552] Processing DeleteTablet for tablet 81c770c1c9a84637b71989257cb4453a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:50 UTC) from {username='slave'} at 127.0.0.1:37626
W20251029 02:51:50.396124 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 8aaa0a9dd62443d68f96d7bd44a0c02b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8aaa0a9dd62443d68f96d7bd44a0c02b already in progress: creating tablet
W20251029 02:51:50.396416 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 81c770c1c9a84637b71989257cb4453a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 81c770c1c9a84637b71989257cb4453a already in progress: creating tablet
W20251029 02:51:50.396121 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 8aaa0a9dd62443d68f96d7bd44a0c02b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8aaa0a9dd62443d68f96d7bd44a0c02b already in progress: creating tablet
W20251029 02:51:50.397658 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 81c770c1c9a84637b71989257cb4453a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 81c770c1c9a84637b71989257cb4453a already in progress: creating tablet
I20251029 02:51:50.398903 1171 leader_election.cc:290] T 24114a60aeb24653aa7384df675aac76 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:51:50.400893 1171 raft_consensus.cc:2804] T 24114a60aeb24653aa7384df675aac76 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:51:50.401389 1171 raft_consensus.cc:697] T 24114a60aeb24653aa7384df675aac76 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:51:50.401968 1166 tablet_bootstrap.cc:492] T 81c770c1c9a84637b71989257cb4453a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:51:50.402200 1166 ts_tablet_manager.cc:1403] T 81c770c1c9a84637b71989257cb4453a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.094s user 0.034s sys 0.047s
I20251029 02:51:50.403326 1166 raft_consensus.cc:359] T 81c770c1c9a84637b71989257cb4453a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } }
I20251029 02:51:50.403615 1172 tablet_bootstrap.cc:492] T 8aaa0a9dd62443d68f96d7bd44a0c02b P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:51:50.403633 1166 raft_consensus.cc:385] T 81c770c1c9a84637b71989257cb4453a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:50.403725 1166 raft_consensus.cc:740] T 81c770c1c9a84637b71989257cb4453a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:51:50.403760 1172 ts_tablet_manager.cc:1403] T 8aaa0a9dd62443d68f96d7bd44a0c02b P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.103s user 0.024s sys 0.026s
I20251029 02:51:50.404023 1166 consensus_queue.cc:260] T 81c770c1c9a84637b71989257cb4453a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } }
I20251029 02:51:50.404722 1166 ts_tablet_manager.cc:1434] T 81c770c1c9a84637b71989257cb4453a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251029 02:51:50.402029 1171 consensus_queue.cc:237] T 24114a60aeb24653aa7384df675aac76 P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:50.407496 1172 raft_consensus.cc:359] T 8aaa0a9dd62443d68f96d7bd44a0c02b P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } }
I20251029 02:51:50.407711 1167 tablet_bootstrap.cc:492] T 02d3583e77714d08ae476cbfdb2583c9 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:51:50.407711 1172 raft_consensus.cc:385] T 8aaa0a9dd62443d68f96d7bd44a0c02b P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:50.407819 1172 raft_consensus.cc:740] T 8aaa0a9dd62443d68f96d7bd44a0c02b P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:51:50.407902 1167 ts_tablet_manager.cc:1403] T 02d3583e77714d08ae476cbfdb2583c9 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.111s user 0.030s sys 0.049s
I20251029 02:51:50.408092 1172 consensus_queue.cc:260] T 8aaa0a9dd62443d68f96d7bd44a0c02b P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } }
I20251029 02:51:50.408725 1172 ts_tablet_manager.cc:1434] T 8aaa0a9dd62443d68f96d7bd44a0c02b P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251029 02:51:50.409065 1172 tablet_bootstrap.cc:492] T 81c770c1c9a84637b71989257cb4453a P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:51:50.411116 1172 tablet_bootstrap.cc:654] T 81c770c1c9a84637b71989257cb4453a P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:50.413543 1062 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24114a60aeb24653aa7384df675aac76" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:51:50.413931 1167 raft_consensus.cc:359] T 02d3583e77714d08ae476cbfdb2583c9 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:50.414083 1062 raft_consensus.cc:2468] T 24114a60aeb24653aa7384df675aac76 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:51:50.414158 1167 raft_consensus.cc:385] T 02d3583e77714d08ae476cbfdb2583c9 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:50.414264 1167 raft_consensus.cc:740] T 02d3583e77714d08ae476cbfdb2583c9 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:51:50.414497 1166 tablet_bootstrap.cc:492] T 2d2ed6d2389c4f1e91e6f6423b89b3bc P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:51:50.414536 1167 consensus_queue.cc:260] T 02d3583e77714d08ae476cbfdb2583c9 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:50.414465 1061 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24114a60aeb24653aa7384df675aac76" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:51:50.415385 1061 raft_consensus.cc:3060] T 24114a60aeb24653aa7384df675aac76 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:51:50.416414 1167 ts_tablet_manager.cc:1434] T 02d3583e77714d08ae476cbfdb2583c9 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.008s user 0.000s sys 0.003s
I20251029 02:51:50.416693 1166 tablet_bootstrap.cc:654] T 2d2ed6d2389c4f1e91e6f6423b89b3bc P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:50.416728 1167 tablet_bootstrap.cc:492] T 648ad50f1dca4c0f96a5e5da089d12d1 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:51:50.419154 1167 tablet_bootstrap.cc:654] T 648ad50f1dca4c0f96a5e5da089d12d1 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:50.421188 1061 raft_consensus.cc:2468] T 24114a60aeb24653aa7384df675aac76 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:51:50.424074 1172 tablet_bootstrap.cc:492] T 81c770c1c9a84637b71989257cb4453a P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:51:50.424396 1172 ts_tablet_manager.cc:1403] T 81c770c1c9a84637b71989257cb4453a P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.001s
I20251029 02:51:50.426040 1172 raft_consensus.cc:359] T 81c770c1c9a84637b71989257cb4453a P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } }
I20251029 02:51:50.426472 1172 raft_consensus.cc:385] T 81c770c1c9a84637b71989257cb4453a P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:50.426661 1172 raft_consensus.cc:740] T 81c770c1c9a84637b71989257cb4453a P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:51:50.427075 1172 consensus_queue.cc:260] T 81c770c1c9a84637b71989257cb4453a P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } }
I20251029 02:51:50.427862 1172 ts_tablet_manager.cc:1434] T 81c770c1c9a84637b71989257cb4453a P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251029 02:51:50.428352 1172 tablet_bootstrap.cc:492] T 648ad50f1dca4c0f96a5e5da089d12d1 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:51:50.430776 1172 tablet_bootstrap.cc:654] T 648ad50f1dca4c0f96a5e5da089d12d1 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
W20251029 02:51:50.434984 524 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2dfa48aa89074a53bc13ed5ea199490e has no consensus state
I20251029 02:51:50.435274 524 master_service.cc:954] ReplaceTablet: received request to replace tablet 02d3583e77714d08ae476cbfdb2583c9 from {username='slave'} at 127.0.0.1:35314
I20251029 02:51:50.435348 1166 tablet_bootstrap.cc:492] T 2d2ed6d2389c4f1e91e6f6423b89b3bc P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:51:50.435398 1172 tablet_bootstrap.cc:492] T 648ad50f1dca4c0f96a5e5da089d12d1 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:51:50.435513 1166 ts_tablet_manager.cc:1403] T 2d2ed6d2389c4f1e91e6f6423b89b3bc P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.013s
I20251029 02:51:50.435534 1172 ts_tablet_manager.cc:1403] T 648ad50f1dca4c0f96a5e5da089d12d1 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.003s
I20251029 02:51:50.435704 522 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 8cb92fa59f114b01aa27a28358ecb4e7
I20251029 02:51:50.436159 522 catalog_manager.cc:6719] ReplaceTablet: tablet 8cb92fa59f114b01aa27a28358ecb4e7 deleted and replaced by tablet 3caf6aa732e6460e96145c0f4105a826
I20251029 02:51:50.436305 1166 raft_consensus.cc:359] T 2d2ed6d2389c4f1e91e6f6423b89b3bc P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:50.436307 1172 raft_consensus.cc:359] T 648ad50f1dca4c0f96a5e5da089d12d1 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:50.436478 1166 raft_consensus.cc:385] T 2d2ed6d2389c4f1e91e6f6423b89b3bc P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:50.436494 1172 raft_consensus.cc:385] T 648ad50f1dca4c0f96a5e5da089d12d1 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:50.436578 1172 raft_consensus.cc:740] T 648ad50f1dca4c0f96a5e5da089d12d1 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:51:50.436569 1166 raft_consensus.cc:740] T 2d2ed6d2389c4f1e91e6f6423b89b3bc P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:51:50.436517 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_5e80187ca8bf484dae849e39465263c8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:50.436811 1172 consensus_queue.cc:260] T 648ad50f1dca4c0f96a5e5da089d12d1 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:50.436931 526 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet db280d36b1ff41c08ab62627be1c3251
W20251029 02:51:50.437110 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5e80187ca8bf484dae849e39465263c8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:50.437427 1166 consensus_queue.cc:260] T 2d2ed6d2389c4f1e91e6f6423b89b3bc P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:50.437958 1166 ts_tablet_manager.cc:1434] T 2d2ed6d2389c4f1e91e6f6423b89b3bc P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 02:51:50.438486 1166 tablet_bootstrap.cc:492] T 8aaa0a9dd62443d68f96d7bd44a0c02b P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:51:50.439291 1038 tablet_service.cc:1552] Processing DeleteTablet for tablet 8cb92fa59f114b01aa27a28358ecb4e7 with delete_type TABLET_DATA_DELETED (replaced by tablet 3caf6aa732e6460e96145c0f4105a826) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:50.439594 1176 tablet_replica.cc:333] T 8cb92fa59f114b01aa27a28358ecb4e7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:51:50.439780 1176 raft_consensus.cc:2243] T 8cb92fa59f114b01aa27a28358ecb4e7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:50.439977 1176 raft_consensus.cc:2272] T 8cb92fa59f114b01aa27a28358ecb4e7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:50.440903 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_631cf79ec867461baecbc486773b88c4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:50.441525 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_631cf79ec867461baecbc486773b88c4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:50.441776 1166 tablet_bootstrap.cc:654] T 8aaa0a9dd62443d68f96d7bd44a0c02b P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:50.442386 517 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 951c8863725f412ba3a4c7829583d185
I20251029 02:51:50.442903 1176 ts_tablet_manager.cc:1916] T 8cb92fa59f114b01aa27a28358ecb4e7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:50.444875 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_d10be51e146c4d849c7cd7308ebeae63"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:50.445360 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d10be51e146c4d849c7cd7308ebeae63 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:50.447162 895 tablet_service.cc:1552] Processing DeleteTablet for tablet 8cb92fa59f114b01aa27a28358ecb4e7 with delete_type TABLET_DATA_DELETED (replaced by tablet 3caf6aa732e6460e96145c0f4105a826) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:50.447495 1174 tablet_replica.cc:333] T 8cb92fa59f114b01aa27a28358ecb4e7 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:51:50.447685 1174 raft_consensus.cc:2243] T 8cb92fa59f114b01aa27a28358ecb4e7 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:50.447885 1174 raft_consensus.cc:2272] T 8cb92fa59f114b01aa27a28358ecb4e7 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:50.450598 1174 ts_tablet_manager.cc:1916] T 8cb92fa59f114b01aa27a28358ecb4e7 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:50.453191 1176 ts_tablet_manager.cc:1929] T 8cb92fa59f114b01aa27a28358ecb4e7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:50.453338 1176 log.cc:1199] T 8cb92fa59f114b01aa27a28358ecb4e7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/8cb92fa59f114b01aa27a28358ecb4e7
I20251029 02:51:50.453738 1176 ts_tablet_manager.cc:1950] T 8cb92fa59f114b01aa27a28358ecb4e7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:51:50.455044 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 8cb92fa59f114b01aa27a28358ecb4e7 (table default.Table_4bdbcbb87e5b40aeb9f98eb1aa0e6fe3 [id=78615d8807a74b4496a3223ef6fe709d]) successfully deleted
I20251029 02:51:50.439091 753 tablet_service.cc:1552] Processing DeleteTablet for tablet 8cb92fa59f114b01aa27a28358ecb4e7 with delete_type TABLET_DATA_DELETED (replaced by tablet 3caf6aa732e6460e96145c0f4105a826) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:50.455753 1175 tablet_replica.cc:333] T 8cb92fa59f114b01aa27a28358ecb4e7 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:51:50.456000 1175 raft_consensus.cc:2243] T 8cb92fa59f114b01aa27a28358ecb4e7 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:50.456218 1175 raft_consensus.cc:2272] T 8cb92fa59f114b01aa27a28358ecb4e7 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:50.457326 1175 ts_tablet_manager.cc:1916] T 8cb92fa59f114b01aa27a28358ecb4e7 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:50.437604 1172 ts_tablet_manager.cc:1434] T 648ad50f1dca4c0f96a5e5da089d12d1 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 02:51:50.467363 1175 ts_tablet_manager.cc:1929] T 8cb92fa59f114b01aa27a28358ecb4e7 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:50.467553 1175 log.cc:1199] T 8cb92fa59f114b01aa27a28358ecb4e7 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/8cb92fa59f114b01aa27a28358ecb4e7
I20251029 02:51:50.468127 1175 ts_tablet_manager.cc:1950] T 8cb92fa59f114b01aa27a28358ecb4e7 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:51:50.469442 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 8cb92fa59f114b01aa27a28358ecb4e7 (table default.Table_4bdbcbb87e5b40aeb9f98eb1aa0e6fe3 [id=78615d8807a74b4496a3223ef6fe709d]) successfully deleted
I20251029 02:51:50.471097 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_cd5277bcf2e24e189f8b1ded2f593c83"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:50.471690 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cd5277bcf2e24e189f8b1ded2f593c83 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:50.473476 753 tablet_service.cc:1505] Processing CreateTablet for tablet 2dfa48aa89074a53bc13ed5ea199490e (DEFAULT_TABLE table=default.Table_83ad3367766c49269223bd91b060a972 [id=3bd791c439904f8c9e66b9e4dd107a78]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:50.473680 1038 tablet_service.cc:1505] Processing CreateTablet for tablet 2dfa48aa89074a53bc13ed5ea199490e (DEFAULT_TABLE table=default.Table_83ad3367766c49269223bd91b060a972 [id=3bd791c439904f8c9e66b9e4dd107a78]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:50.473912 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2dfa48aa89074a53bc13ed5ea199490e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:50.473819 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_134014ec659240928f77016dc4e6c4ff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:50.474085 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2dfa48aa89074a53bc13ed5ea199490e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:50.479350 895 tablet_service.cc:1505] Processing CreateTablet for tablet 2dfa48aa89074a53bc13ed5ea199490e (DEFAULT_TABLE table=default.Table_83ad3367766c49269223bd91b060a972 [id=3bd791c439904f8c9e66b9e4dd107a78]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:50.479851 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2dfa48aa89074a53bc13ed5ea199490e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:50.481909 1179 raft_consensus.cc:493] T 8aaa0a9dd62443d68f96d7bd44a0c02b P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:50.482090 1179 raft_consensus.cc:515] T 8aaa0a9dd62443d68f96d7bd44a0c02b P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } }
W20251029 02:51:50.483356 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_134014ec659240928f77016dc4e6c4ff in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:50.484244 1179 leader_election.cc:290] T 8aaa0a9dd62443d68f96d7bd44a0c02b P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:51:50.502688 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_114cb1da4678460ab5267cff0eac3c9c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:50.503384 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_114cb1da4678460ab5267cff0eac3c9c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:50.514993 1172 tablet_bootstrap.cc:492] T 2dfa48aa89074a53bc13ed5ea199490e P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:51:50.516367 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_0be0c9e3b0fb4574b38412eed122847c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:50.517012 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0be0c9e3b0fb4574b38412eed122847c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:50.517050 1172 tablet_bootstrap.cc:654] T 2dfa48aa89074a53bc13ed5ea199490e P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:50.518368 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_011e3bd22b8a47b1b33695daa8d322dd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:50.518980 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_011e3bd22b8a47b1b33695daa8d322dd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:50.520305 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_7e818850675e4c5c973db094e7223348"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:50.520828 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7e818850675e4c5c973db094e7223348 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:50.522500 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_25688635c7a948f1b010aa30b88a3b6c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:50.523097 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_25688635c7a948f1b010aa30b88a3b6c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:50.529212 523 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet 24114a60aeb24653aa7384df675aac76 (table default.Table_f4a825a9cffc4afea35104d88069df1e [id=782a7aaf782749d384c78095cda17dc4]). Expected version 2 got 0
I20251029 02:51:50.530720 525 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet 02d3583e77714d08ae476cbfdb2583c9 (table default.Table_1dc0fd70708a4fa6828722209bb71dcb [id=1a1aeb237dad4464b717bbae5d4001b0]). Expected version 1 got 0
I20251029 02:51:50.530897 525 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet 24114a60aeb24653aa7384df675aac76 (table default.Table_f4a825a9cffc4afea35104d88069df1e [id=782a7aaf782749d384c78095cda17dc4]). Expected version 2 got 0
I20251029 02:51:50.531805 518 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet 02d3583e77714d08ae476cbfdb2583c9 (table default.Table_1dc0fd70708a4fa6828722209bb71dcb [id=1a1aeb237dad4464b717bbae5d4001b0]). Expected version 1 got 0
W20251029 02:51:50.532529 521 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 02d3583e77714d08ae476cbfdb2583c9 consensus state has no leader
I20251029 02:51:50.538061 754 tablet_service.cc:1552] Processing DeleteTablet for tablet a6d728bd089841a1a5ba997f8a71862a with delete_type TABLET_DATA_DELETED (replaced by tablet 02d3583e77714d08ae476cbfdb2583c9) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:50.538156 756 tablet_service.cc:1552] Processing DeleteTablet for tablet 91aa5291a1d442598f7dcbcf0f1cbd2f with delete_type TABLET_DATA_DELETED (replaced by tablet 2d2ed6d2389c4f1e91e6f6423b89b3bc) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:50.538380 752 tablet_service.cc:1552] Processing DeleteTablet for tablet 2d2ed6d2389c4f1e91e6f6423b89b3bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:50 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:50.538676 1175 tablet_replica.cc:333] T a6d728bd089841a1a5ba997f8a71862a P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:51:50.538936 1175 raft_consensus.cc:2243] T a6d728bd089841a1a5ba997f8a71862a P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:50.539141 1175 raft_consensus.cc:2272] T a6d728bd089841a1a5ba997f8a71862a P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:50.539588 898 tablet_service.cc:1505] Processing CreateTablet for tablet 94edf3f2e0854b07bbd348ecab82d3ca (DEFAULT_TABLE table=default.Table_d10be51e146c4d849c7cd7308ebeae63 [id=13b28fb78eb54543b973bfda2fb1aad8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:50.540032 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94edf3f2e0854b07bbd348ecab82d3ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:50.543287 1175 ts_tablet_manager.cc:1916] T a6d728bd089841a1a5ba997f8a71862a P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:50.545770 1174 ts_tablet_manager.cc:1929] T 8cb92fa59f114b01aa27a28358ecb4e7 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:50.545917 1174 log.cc:1199] T 8cb92fa59f114b01aa27a28358ecb4e7 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/8cb92fa59f114b01aa27a28358ecb4e7
I20251029 02:51:50.546344 1174 ts_tablet_manager.cc:1950] T 8cb92fa59f114b01aa27a28358ecb4e7 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:51:50.547701 894 tablet_service.cc:1505] Processing CreateTablet for tablet 3caf6aa732e6460e96145c0f4105a826 (DEFAULT_TABLE table=default.Table_4bdbcbb87e5b40aeb9f98eb1aa0e6fe3 [id=78615d8807a74b4496a3223ef6fe709d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:50.547791 899 tablet_service.cc:1505] Processing CreateTablet for tablet 494b058c819548dab52917d7690a03ca (DEFAULT_TABLE table=default.Table_9c3c143c7d464c32acb4176464f0fb64 [id=ded0c39ca7954a40933ccc61c5d32909]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:50.548254 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3caf6aa732e6460e96145c0f4105a826. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:50.548224 900 tablet_service.cc:1505] Processing CreateTablet for tablet 0c0aeaabe14740718f75f723023aa087 (DEFAULT_TABLE table=default.Table_fd9d401f3f92450cab95de4f0895a1e1 [id=feb1f56ae44249c6adce65637017bb43]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:50.548604 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c0aeaabe14740718f75f723023aa087. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:50.548949 893 tablet_service.cc:1505] Processing CreateTablet for tablet 355f26adb2b44b98bedcc97072d951ad (DEFAULT_TABLE table=default.Table_a85023af366f423b8a637b058d6c65b4 [id=144a24e58314435d85f9c759ef79fa60]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:50.549539 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 355f26adb2b44b98bedcc97072d951ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:50.550818 896 tablet_service.cc:1505] Processing CreateTablet for tablet 6a378024f1fb45d086297968ab504156 (DEFAULT_TABLE table=default.Table_14462d3ac1c24574963a67d66a1cb274 [id=2c98232622ff4370a069b00d8d787f18]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:50.551183 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a378024f1fb45d086297968ab504156. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:50.554718 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 494b058c819548dab52917d7690a03ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:50.554890 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 8cb92fa59f114b01aa27a28358ecb4e7 (table default.Table_4bdbcbb87e5b40aeb9f98eb1aa0e6fe3 [id=78615d8807a74b4496a3223ef6fe709d]) successfully deleted
I20251029 02:51:50.556107 752 tablet_service.cc:1505] Processing CreateTablet for tablet 94edf3f2e0854b07bbd348ecab82d3ca (DEFAULT_TABLE table=default.Table_d10be51e146c4d849c7cd7308ebeae63 [id=13b28fb78eb54543b973bfda2fb1aad8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:50.556185 755 tablet_service.cc:1505] Processing CreateTablet for tablet 6a378024f1fb45d086297968ab504156 (DEFAULT_TABLE table=default.Table_14462d3ac1c24574963a67d66a1cb274 [id=2c98232622ff4370a069b00d8d787f18]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:50.556550 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94edf3f2e0854b07bbd348ecab82d3ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:50.556814 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a378024f1fb45d086297968ab504156. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:50.557047 754 tablet_service.cc:1505] Processing CreateTablet for tablet 0c0aeaabe14740718f75f723023aa087 (DEFAULT_TABLE table=default.Table_fd9d401f3f92450cab95de4f0895a1e1 [id=feb1f56ae44249c6adce65637017bb43]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:50.557354 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c0aeaabe14740718f75f723023aa087. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:50.559321 751 tablet_service.cc:1505] Processing CreateTablet for tablet 3caf6aa732e6460e96145c0f4105a826 (DEFAULT_TABLE table=default.Table_4bdbcbb87e5b40aeb9f98eb1aa0e6fe3 [id=78615d8807a74b4496a3223ef6fe709d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:50.559916 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3caf6aa732e6460e96145c0f4105a826. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:50.563125 756 tablet_service.cc:1505] Processing CreateTablet for tablet 494b058c819548dab52917d7690a03ca (DEFAULT_TABLE table=default.Table_9c3c143c7d464c32acb4176464f0fb64 [id=ded0c39ca7954a40933ccc61c5d32909]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:50.563521 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 494b058c819548dab52917d7690a03ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:50.569296 1170 raft_consensus.cc:493] T 81c770c1c9a84637b71989257cb4453a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:50.569540 1170 raft_consensus.cc:515] T 81c770c1c9a84637b71989257cb4453a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } }
I20251029 02:51:50.571942 1170 leader_election.cc:290] T 81c770c1c9a84637b71989257cb4453a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:51:50.578140 1170 raft_consensus.cc:493] T a6d728bd089841a1a5ba997f8a71862a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:50.578356 1170 raft_consensus.cc:515] T a6d728bd089841a1a5ba997f8a71862a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:50.579133 1170 leader_election.cc:290] T a6d728bd089841a1a5ba997f8a71862a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:51:50.585134 525 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_9c3c143c7d464c32acb4176464f0fb64" } modify_external_catalogs: true
I20251029 02:51:50.585372 525 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_9c3c143c7d464c32acb4176464f0fb64" } modify_external_catalogs: true
I20251029 02:51:50.585665 1041 tablet_service.cc:1505] Processing CreateTablet for tablet 494b058c819548dab52917d7690a03ca (DEFAULT_TABLE table=default.Table_9c3c143c7d464c32acb4176464f0fb64 [id=ded0c39ca7954a40933ccc61c5d32909]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:50.586159 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 494b058c819548dab52917d7690a03ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:50.586278 1040 tablet_service.cc:1505] Processing CreateTablet for tablet 94edf3f2e0854b07bbd348ecab82d3ca (DEFAULT_TABLE table=default.Table_d10be51e146c4d849c7cd7308ebeae63 [id=13b28fb78eb54543b973bfda2fb1aad8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:50.586726 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94edf3f2e0854b07bbd348ecab82d3ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:50.586938 1037 tablet_service.cc:1505] Processing CreateTablet for tablet 3caf6aa732e6460e96145c0f4105a826 (DEFAULT_TABLE table=default.Table_4bdbcbb87e5b40aeb9f98eb1aa0e6fe3 [id=78615d8807a74b4496a3223ef6fe709d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:50.587452 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3caf6aa732e6460e96145c0f4105a826. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:50.588335 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet 2d2ed6d2389c4f1e91e6f6423b89b3bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:50 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:50.588686 1176 tablet_replica.cc:333] T 2d2ed6d2389c4f1e91e6f6423b89b3bc P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:51:50.588917 1176 raft_consensus.cc:2243] T 2d2ed6d2389c4f1e91e6f6423b89b3bc P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:50.589102 1176 raft_consensus.cc:2272] T 2d2ed6d2389c4f1e91e6f6423b89b3bc P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:50.589802 1039 tablet_service.cc:1505] Processing CreateTablet for tablet 6a378024f1fb45d086297968ab504156 (DEFAULT_TABLE table=default.Table_14462d3ac1c24574963a67d66a1cb274 [id=2c98232622ff4370a069b00d8d787f18]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:50.590209 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a378024f1fb45d086297968ab504156. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:50.590854 1042 tablet_service.cc:1505] Processing CreateTablet for tablet 0c0aeaabe14740718f75f723023aa087 (DEFAULT_TABLE table=default.Table_fd9d401f3f92450cab95de4f0895a1e1 [id=feb1f56ae44249c6adce65637017bb43]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:50.591323 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c0aeaabe14740718f75f723023aa087. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:50.592685 521 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_114cb1da4678460ab5267cff0eac3c9c" } new_table_name: "default.Table_fb13d20c094a41d88582ee4fcc83020f" modify_external_catalogs: true
I20251029 02:51:50.593616 751 tablet_service.cc:1505] Processing CreateTablet for tablet 355f26adb2b44b98bedcc97072d951ad (DEFAULT_TABLE table=default.Table_a85023af366f423b8a637b058d6c65b4 [id=144a24e58314435d85f9c759ef79fa60]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:50.593829 1036 tablet_service.cc:1505] Processing CreateTablet for tablet 355f26adb2b44b98bedcc97072d951ad (DEFAULT_TABLE table=default.Table_a85023af366f423b8a637b058d6c65b4 [id=144a24e58314435d85f9c759ef79fa60]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:50.594099 523 master_service.cc:954] ReplaceTablet: received request to replace tablet 24114a60aeb24653aa7384df675aac76 from {username='slave'} at 127.0.0.1:35314
I20251029 02:51:50.594363 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 355f26adb2b44b98bedcc97072d951ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:50.594744 1034 tablet_service.cc:1552] Processing DeleteTablet for tablet 81c770c1c9a84637b71989257cb4453a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:50 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:50.595189 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 355f26adb2b44b98bedcc97072d951ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:50.595594 1176 ts_tablet_manager.cc:1916] T 2d2ed6d2389c4f1e91e6f6423b89b3bc P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:50.605894 526 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet 648ad50f1dca4c0f96a5e5da089d12d1 (table default.Table_9f11b26c8c144d7a8f6e2fbb3c6fe36e [id=d0cd3d1567c54463947443927e18169f]). Expected version 2 got 0
I20251029 02:51:50.607374 755 tablet_service.cc:1552] Processing DeleteTablet for tablet 8aaa0a9dd62443d68f96d7bd44a0c02b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:50 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:50.608217 755 tablet_service.cc:1552] Processing DeleteTablet for tablet 81c770c1c9a84637b71989257cb4453a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:50 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:50.610138 518 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_d10be51e146c4d849c7cd7308ebeae63" } new_table_name: "default.Table_1b27eebc21ef4d1aa39f04fd2e3c3388" modify_external_catalogs: true
I20251029 02:51:50.611949 519 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_14462d3ac1c24574963a67d66a1cb274" } modify_external_catalogs: true
I20251029 02:51:50.612147 519 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_14462d3ac1c24574963a67d66a1cb274" } modify_external_catalogs: true
I20251029 02:51:50.613607 517 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_5e80187ca8bf484dae849e39465263c8" } modify_external_catalogs: true
I20251029 02:51:50.613780 517 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_5e80187ca8bf484dae849e39465263c8" } modify_external_catalogs: true
I20251029 02:51:50.616052 1175 ts_tablet_manager.cc:1929] T a6d728bd089841a1a5ba997f8a71862a P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:50.616245 1175 log.cc:1199] T a6d728bd089841a1a5ba997f8a71862a P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/a6d728bd089841a1a5ba997f8a71862a
I20251029 02:51:50.616653 1175 ts_tablet_manager.cc:1950] T a6d728bd089841a1a5ba997f8a71862a P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:51:50.617651 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet a6d728bd089841a1a5ba997f8a71862a (table default.Table_5b3a7640a9e54e7a9868985322ade0cb [id=1a1aeb237dad4464b717bbae5d4001b0]) successfully deleted
I20251029 02:51:50.627264 525 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 494b058c819548dab52917d7690a03ca
I20251029 02:51:50.627336 520 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_cd5277bcf2e24e189f8b1ded2f593c83" } new_table_name: "default.Table_418d4e45a6364eaba65e55ffb9aeccbb" modify_external_catalogs: true
I20251029 02:51:50.628825 524 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 02d3583e77714d08ae476cbfdb2583c9
I20251029 02:51:50.629293 524 catalog_manager.cc:6719] ReplaceTablet: tablet 02d3583e77714d08ae476cbfdb2583c9 deleted and replaced by tablet e6bcea41d5d4480eaf5d6986a93f5e45
I20251029 02:51:50.630079 752 tablet_service.cc:1552] Processing DeleteTablet for tablet 02d3583e77714d08ae476cbfdb2583c9 with delete_type TABLET_DATA_DELETED (replaced by tablet e6bcea41d5d4480eaf5d6986a93f5e45) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:50.636704 1176 ts_tablet_manager.cc:1929] T 2d2ed6d2389c4f1e91e6f6423b89b3bc P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:50.636881 1176 log.cc:1199] T 2d2ed6d2389c4f1e91e6f6423b89b3bc P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/2d2ed6d2389c4f1e91e6f6423b89b3bc
I20251029 02:51:50.637306 1176 ts_tablet_manager.cc:1950] T 2d2ed6d2389c4f1e91e6f6423b89b3bc P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:51:50.637774 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet 02d3583e77714d08ae476cbfdb2583c9 with delete_type TABLET_DATA_DELETED (replaced by tablet e6bcea41d5d4480eaf5d6986a93f5e45) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:50.638080 1040 tablet_service.cc:1552] Processing DeleteTablet for tablet 494b058c819548dab52917d7690a03ca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:50 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:50.640261 522 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_0be0c9e3b0fb4574b38412eed122847c" } new_table_name: "default.Table_9becb486767e4662a93114d2242f031d" modify_external_catalogs: true
I20251029 02:51:50.644968 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 2d2ed6d2389c4f1e91e6f6423b89b3bc (table default.Table_7a7b22a6e750401c9751e3ad5fbb42ed [id=18cf9b24efed4f1095ff99a7fe2022fb]) successfully deleted
I20251029 02:51:50.645062 1176 tablet_replica.cc:333] T 81c770c1c9a84637b71989257cb4453a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:51:50.645277 1176 raft_consensus.cc:2243] T 81c770c1c9a84637b71989257cb4453a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:50.645465 1176 raft_consensus.cc:2272] T 81c770c1c9a84637b71989257cb4453a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:50.645480 524 master_service.cc:954] ReplaceTablet: received request to replace tablet 24114a60aeb24653aa7384df675aac76 from {username='slave'} at 127.0.0.1:35314
I20251029 02:51:50.646148 525 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_f4a825a9cffc4afea35104d88069df1e" } modify_external_catalogs: true
I20251029 02:51:50.646224 1176 ts_tablet_manager.cc:1916] T 81c770c1c9a84637b71989257cb4453a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:50.646363 525 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_f4a825a9cffc4afea35104d88069df1e" } modify_external_catalogs: true
I20251029 02:51:50.648032 756 tablet_service.cc:1552] Processing DeleteTablet for tablet 494b058c819548dab52917d7690a03ca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:50 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:50.648962 1038 tablet_service.cc:1552] Processing DeleteTablet for tablet a6d728bd089841a1a5ba997f8a71862a with delete_type TABLET_DATA_DELETED (replaced by tablet 02d3583e77714d08ae476cbfdb2583c9) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:50.649909 753 tablet_service.cc:1552] Processing DeleteTablet for tablet 8cb92fa59f114b01aa27a28358ecb4e7 with delete_type TABLET_DATA_DELETED (replaced by tablet 3caf6aa732e6460e96145c0f4105a826) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:50.534195 895 tablet_service.cc:1552] Processing DeleteTablet for tablet 91aa5291a1d442598f7dcbcf0f1cbd2f with delete_type TABLET_DATA_DELETED (replaced by tablet 2d2ed6d2389c4f1e91e6f6423b89b3bc) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:50.651851 1175 tablet_replica.cc:333] T 91aa5291a1d442598f7dcbcf0f1cbd2f P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:51:50.652055 1175 raft_consensus.cc:2243] T 91aa5291a1d442598f7dcbcf0f1cbd2f P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:50.652253 1175 raft_consensus.cc:2272] T 91aa5291a1d442598f7dcbcf0f1cbd2f P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:50.652669 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskYVokEG/build/asan/bin/kudu
/tmp/dist-test-taskYVokEG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.60:41233
--webserver_interface=127.0.104.60
--webserver_port=42893
--builtin_ntp_servers=127.0.104.20:44691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:42343,127.0.104.61:34239,127.0.104.60:41233
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20251029 02:51:50.653386 1175 ts_tablet_manager.cc:1916] T 91aa5291a1d442598f7dcbcf0f1cbd2f P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:50.654189 1176 ts_tablet_manager.cc:1929] T 81c770c1c9a84637b71989257cb4453a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:50.654335 1176 log.cc:1199] T 81c770c1c9a84637b71989257cb4453a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/81c770c1c9a84637b71989257cb4453a
I20251029 02:51:50.656469 1037 tablet_service.cc:1505] Processing CreateTablet for tablet 22be6bc81efb434ab7fc3242193789c3 (DEFAULT_TABLE table=default.Table_78f7811a55f243faa1be1c3f90754dcf [id=cf081cb5322d40c3bdd89cb9e0f58a99]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:50.656953 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22be6bc81efb434ab7fc3242193789c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:50.658847 753 tablet_service.cc:1505] Processing CreateTablet for tablet 22be6bc81efb434ab7fc3242193789c3 (DEFAULT_TABLE table=default.Table_78f7811a55f243faa1be1c3f90754dcf [id=cf081cb5322d40c3bdd89cb9e0f58a99]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:50.659207 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22be6bc81efb434ab7fc3242193789c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:50.665830 1175 ts_tablet_manager.cc:1929] T 91aa5291a1d442598f7dcbcf0f1cbd2f P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:50.667677 1176 ts_tablet_manager.cc:1950] T 81c770c1c9a84637b71989257cb4453a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:51:50.667940 1166 tablet_bootstrap.cc:492] T 8aaa0a9dd62443d68f96d7bd44a0c02b P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:51:50.668210 1166 ts_tablet_manager.cc:1403] T 8aaa0a9dd62443d68f96d7bd44a0c02b P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.230s user 0.059s sys 0.098s
I20251029 02:51:50.668404 1176 tablet_replica.cc:333] T 02d3583e77714d08ae476cbfdb2583c9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:51:50.668617 1176 raft_consensus.cc:2243] T 02d3583e77714d08ae476cbfdb2583c9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:50.668599 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 81c770c1c9a84637b71989257cb4453a (table default.Table_19b1787d124341939d34a6e721a87ae9 [id=33950ad96f8140fbb26ab0cd8b2b2b97]) successfully deleted
I20251029 02:51:50.668787 1176 raft_consensus.cc:2272] T 02d3583e77714d08ae476cbfdb2583c9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:50.669097 1166 raft_consensus.cc:359] T 8aaa0a9dd62443d68f96d7bd44a0c02b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } }
I20251029 02:51:50.669297 1166 raft_consensus.cc:385] T 8aaa0a9dd62443d68f96d7bd44a0c02b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:50.669355 1166 raft_consensus.cc:740] T 8aaa0a9dd62443d68f96d7bd44a0c02b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:51:50.669626 1166 consensus_queue.cc:260] T 8aaa0a9dd62443d68f96d7bd44a0c02b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } }
I20251029 02:51:50.670485 1174 tablet_replica.cc:333] T 91aa5291a1d442598f7dcbcf0f1cbd2f P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:51:50.670622 1171 raft_consensus.cc:493] T 91aa5291a1d442598f7dcbcf0f1cbd2f P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:50.670809 1171 raft_consensus.cc:515] T 91aa5291a1d442598f7dcbcf0f1cbd2f P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:50.671492 1171 leader_election.cc:290] T 91aa5291a1d442598f7dcbcf0f1cbd2f P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
W20251029 02:51:50.674482 849 leader_election.cc:343] T 91aa5291a1d442598f7dcbcf0f1cbd2f P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Not found: Tablet not found: 91aa5291a1d442598f7dcbcf0f1cbd2f
I20251029 02:51:50.676538 1166 ts_tablet_manager.cc:1434] T 8aaa0a9dd62443d68f96d7bd44a0c02b P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20251029 02:51:50.678846 1166 tablet_bootstrap.cc:492] T 648ad50f1dca4c0f96a5e5da089d12d1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
W20251029 02:51:50.679736 521 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8f7c746f8b6948b58676298c2c2ea221 has no consensus state
I20251029 02:51:50.680567 1166 tablet_bootstrap.cc:654] T 648ad50f1dca4c0f96a5e5da089d12d1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:50.681272 521 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_011e3bd22b8a47b1b33695daa8d322dd" } modify_external_catalogs: true
I20251029 02:51:50.681458 521 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_011e3bd22b8a47b1b33695daa8d322dd" } modify_external_catalogs: true
I20251029 02:51:50.681496 1174 raft_consensus.cc:2243] T 91aa5291a1d442598f7dcbcf0f1cbd2f P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:50.682219 1174 raft_consensus.cc:2272] T 91aa5291a1d442598f7dcbcf0f1cbd2f P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:50.683107 1174 ts_tablet_manager.cc:1916] T 91aa5291a1d442598f7dcbcf0f1cbd2f P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:50.690301 1174 ts_tablet_manager.cc:1929] T 91aa5291a1d442598f7dcbcf0f1cbd2f P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:50.690678 896 tablet_service.cc:1552] Processing DeleteTablet for tablet 02d3583e77714d08ae476cbfdb2583c9 with delete_type TABLET_DATA_DELETED (replaced by tablet e6bcea41d5d4480eaf5d6986a93f5e45) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:50.691354 896 tablet_service.cc:1505] Processing CreateTablet for tablet 22be6bc81efb434ab7fc3242193789c3 (DEFAULT_TABLE table=default.Table_78f7811a55f243faa1be1c3f90754dcf [id=cf081cb5322d40c3bdd89cb9e0f58a99]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:50.691780 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 22be6bc81efb434ab7fc3242193789c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:50.674005 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "91aa5291a1d442598f7dcbcf0f1cbd2f" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:51:50.677378 1179 raft_consensus.cc:493] T 648ad50f1dca4c0f96a5e5da089d12d1 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:50.694278 1179 raft_consensus.cc:515] T 648ad50f1dca4c0f96a5e5da089d12d1 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:50.694932 895 tablet_service.cc:1552] Processing DeleteTablet for tablet 494b058c819548dab52917d7690a03ca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:50 UTC) from {username='slave'} at 127.0.0.1:35702
W20251029 02:51:50.695322 847 leader_election.cc:343] T 91aa5291a1d442598f7dcbcf0f1cbd2f P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Illegal state: Tablet not RUNNING: STOPPED
I20251029 02:51:50.695456 847 leader_election.cc:304] T 91aa5291a1d442598f7dcbcf0f1cbd2f P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9
W20251029 02:51:50.695675 847 raft_consensus.cc:2686] T 91aa5291a1d442598f7dcbcf0f1cbd2f P c756528642a543eeacd93cef3e335a7d: unable to run election callback: Service unavailable: Thread pool token was shut down
I20251029 02:51:50.695905 1179 leader_election.cc:290] T 648ad50f1dca4c0f96a5e5da089d12d1 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:51:50.696266 1175 log.cc:1199] T 91aa5291a1d442598f7dcbcf0f1cbd2f P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/91aa5291a1d442598f7dcbcf0f1cbd2f
I20251029 02:51:50.696798 1166 tablet_bootstrap.cc:492] T 648ad50f1dca4c0f96a5e5da089d12d1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
W20251029 02:51:50.696930 520 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ba014ecfec7f4602ae1c09d931bac824 has no consensus state
I20251029 02:51:50.697113 519 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 6a378024f1fb45d086297968ab504156
I20251029 02:51:50.697934 517 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 978fdeffaf9e48acb5d9ddf09c3de928
I20251029 02:51:50.698175 523 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 24114a60aeb24653aa7384df675aac76
I20251029 02:51:50.698673 523 catalog_manager.cc:6719] ReplaceTablet: tablet 24114a60aeb24653aa7384df675aac76 deleted and replaced by tablet b3cf1f5cbff24f48a4c80971b6a979dd
I20251029 02:51:50.699345 1166 ts_tablet_manager.cc:1403] T 648ad50f1dca4c0f96a5e5da089d12d1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.002s
W20251029 02:51:50.699918 522 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a0cee0a6eb5a435b9bb3295cbe242b70 has no consensus state
I20251029 02:51:50.700225 1166 raft_consensus.cc:359] T 648ad50f1dca4c0f96a5e5da089d12d1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:50.700428 1166 raft_consensus.cc:385] T 648ad50f1dca4c0f96a5e5da089d12d1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:50.700539 1166 raft_consensus.cc:740] T 648ad50f1dca4c0f96a5e5da089d12d1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:51:50.700799 1166 consensus_queue.cc:260] T 648ad50f1dca4c0f96a5e5da089d12d1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:50.702347 895 tablet_service.cc:1552] Processing DeleteTablet for tablet 24114a60aeb24653aa7384df675aac76 with delete_type TABLET_DATA_DELETED (replaced by tablet b3cf1f5cbff24f48a4c80971b6a979dd) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:50.703480 1166 ts_tablet_manager.cc:1434] T 648ad50f1dca4c0f96a5e5da089d12d1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251029 02:51:50.703644 1172 tablet_bootstrap.cc:492] T 2dfa48aa89074a53bc13ed5ea199490e P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:51:50.703794 520 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet 648ad50f1dca4c0f96a5e5da089d12d1 (table default.Table_9f11b26c8c144d7a8f6e2fbb3c6fe36e [id=d0cd3d1567c54463947443927e18169f]). Expected version 2 got 0
I20251029 02:51:50.703860 1172 ts_tablet_manager.cc:1403] T 2dfa48aa89074a53bc13ed5ea199490e P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.189s user 0.006s sys 0.006s
I20251029 02:51:50.704805 1172 raft_consensus.cc:359] T 2dfa48aa89074a53bc13ed5ea199490e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:50.705068 1172 raft_consensus.cc:385] T 2dfa48aa89074a53bc13ed5ea199490e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:50.705193 1172 raft_consensus.cc:740] T 2dfa48aa89074a53bc13ed5ea199490e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:51:50.705551 1172 consensus_queue.cc:260] T 2dfa48aa89074a53bc13ed5ea199490e P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:50.706238 756 tablet_service.cc:1552] Processing DeleteTablet for tablet 24114a60aeb24653aa7384df675aac76 with delete_type TABLET_DATA_DELETED (replaced by tablet b3cf1f5cbff24f48a4c80971b6a979dd) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:50.707773 520 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet 648ad50f1dca4c0f96a5e5da089d12d1 (table default.Table_9f11b26c8c144d7a8f6e2fbb3c6fe36e [id=d0cd3d1567c54463947443927e18169f]). Expected version 2 got 0
I20251029 02:51:50.708973 1166 tablet_bootstrap.cc:492] T 2dfa48aa89074a53bc13ed5ea199490e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:51:50.710762 1166 tablet_bootstrap.cc:654] T 2dfa48aa89074a53bc13ed5ea199490e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:50.715011 1175 ts_tablet_manager.cc:1950] T 91aa5291a1d442598f7dcbcf0f1cbd2f P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:51:50.715397 1041 tablet_service.cc:1552] Processing DeleteTablet for tablet 24114a60aeb24653aa7384df675aac76 with delete_type TABLET_DATA_DELETED (replaced by tablet b3cf1f5cbff24f48a4c80971b6a979dd) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:50.716167 1174 log.cc:1199] T 91aa5291a1d442598f7dcbcf0f1cbd2f P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/91aa5291a1d442598f7dcbcf0f1cbd2f
I20251029 02:51:50.716630 1174 ts_tablet_manager.cc:1950] T 91aa5291a1d442598f7dcbcf0f1cbd2f P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:51:50.717465 1175 tablet_replica.cc:333] T 8aaa0a9dd62443d68f96d7bd44a0c02b P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:51:50.717605 1174 tablet_replica.cc:333] T 02d3583e77714d08ae476cbfdb2583c9 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:51:50.717643 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 91aa5291a1d442598f7dcbcf0f1cbd2f (table default.Table_7a7b22a6e750401c9751e3ad5fbb42ed [id=18cf9b24efed4f1095ff99a7fe2022fb]) successfully deleted
I20251029 02:51:50.717809 1174 raft_consensus.cc:2243] T 02d3583e77714d08ae476cbfdb2583c9 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:50.718027 1174 raft_consensus.cc:2272] T 02d3583e77714d08ae476cbfdb2583c9 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:50.718391 520 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet 2dfa48aa89074a53bc13ed5ea199490e (table default.Table_83ad3367766c49269223bd91b060a972 [id=3bd791c439904f8c9e66b9e4dd107a78]). Expected version 1 got 0
I20251029 02:51:50.718827 1175 raft_consensus.cc:2243] T 8aaa0a9dd62443d68f96d7bd44a0c02b P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:50.719183 1175 raft_consensus.cc:2272] T 8aaa0a9dd62443d68f96d7bd44a0c02b P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:50.719561 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_fb62281fb765427dae4f824d094a9155"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:50.719951 1176 ts_tablet_manager.cc:1916] T 02d3583e77714d08ae476cbfdb2583c9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 02:51:50.720165 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fb62281fb765427dae4f824d094a9155 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:50.720789 1175 ts_tablet_manager.cc:1916] T 8aaa0a9dd62443d68f96d7bd44a0c02b P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:50.721586 523 master_service.cc:954] ReplaceTablet: received request to replace tablet 94edf3f2e0854b07bbd348ecab82d3ca from {username='slave'} at 127.0.0.1:35314
I20251029 02:51:50.722265 1166 tablet_bootstrap.cc:492] T 2dfa48aa89074a53bc13ed5ea199490e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:51:50.722461 1166 ts_tablet_manager.cc:1403] T 2dfa48aa89074a53bc13ed5ea199490e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.003s
I20251029 02:51:50.723023 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_35aaa3609a3c43e4b75780f3480723f2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:50.723271 1172 ts_tablet_manager.cc:1434] T 2dfa48aa89074a53bc13ed5ea199490e P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.019s user 0.001s sys 0.001s
I20251029 02:51:50.723481 1166 raft_consensus.cc:359] T 2dfa48aa89074a53bc13ed5ea199490e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
W20251029 02:51:50.723613 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_35aaa3609a3c43e4b75780f3480723f2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:50.723659 1172 tablet_bootstrap.cc:492] T 6a378024f1fb45d086297968ab504156 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:51:50.723676 1166 raft_consensus.cc:385] T 2dfa48aa89074a53bc13ed5ea199490e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:50.723753 1166 raft_consensus.cc:740] T 2dfa48aa89074a53bc13ed5ea199490e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:51:50.724018 1166 consensus_queue.cc:260] T 2dfa48aa89074a53bc13ed5ea199490e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:50.725359 1172 tablet_bootstrap.cc:654] T 6a378024f1fb45d086297968ab504156 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:50.728274 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_6add80f3d25b4701b9a0fa5639dec66c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:50.728854 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6add80f3d25b4701b9a0fa5639dec66c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:50.729471 524 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet 2dfa48aa89074a53bc13ed5ea199490e (table default.Table_83ad3367766c49269223bd91b060a972 [id=3bd791c439904f8c9e66b9e4dd107a78]). Expected version 1 got 0
I20251029 02:51:50.730409 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_bf3710e423ae4ab6838435bdf16dc7d0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:50.730981 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_8169731c33644ab78428f0527930e4bc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:50.731475 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8169731c33644ab78428f0527930e4bc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:51:50.731086 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bf3710e423ae4ab6838435bdf16dc7d0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:51:50.732736 518 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 94edf3f2e0854b07bbd348ecab82d3ca consensus state has no leader
I20251029 02:51:50.733079 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_1a524bb5f82a4041b8008ff91a0b9d27"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:50.733582 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1a524bb5f82a4041b8008ff91a0b9d27 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:50.733914 1166 ts_tablet_manager.cc:1434] T 2dfa48aa89074a53bc13ed5ea199490e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20251029 02:51:50.735030 1174 ts_tablet_manager.cc:1916] T 02d3583e77714d08ae476cbfdb2583c9 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:50.735180 1042 tablet_service.cc:1552] Processing DeleteTablet for tablet 6a378024f1fb45d086297968ab504156 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:50 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:50.735433 1038 tablet_service.cc:1552] Processing DeleteTablet for tablet 8aaa0a9dd62443d68f96d7bd44a0c02b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:50 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:50.735700 751 tablet_service.cc:1552] Processing DeleteTablet for tablet 6a378024f1fb45d086297968ab504156 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:50 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:50.737250 1167 tablet_bootstrap.cc:492] T 648ad50f1dca4c0f96a5e5da089d12d1 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:51:50.737443 1167 ts_tablet_manager.cc:1403] T 648ad50f1dca4c0f96a5e5da089d12d1 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.321s user 0.057s sys 0.078s
I20251029 02:51:50.738013 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 91aa5291a1d442598f7dcbcf0f1cbd2f (table default.Table_7a7b22a6e750401c9751e3ad5fbb42ed [id=18cf9b24efed4f1095ff99a7fe2022fb]) successfully deleted
I20251029 02:51:50.738309 1167 raft_consensus.cc:359] T 648ad50f1dca4c0f96a5e5da089d12d1 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
W20251029 02:51:50.738497 511 catalog_manager.cc:4952] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 2d2ed6d2389c4f1e91e6f6423b89b3bc because the tablet was not found. No further retry: Not found: Tablet not found: 2d2ed6d2389c4f1e91e6f6423b89b3bc
I20251029 02:51:50.738498 1167 raft_consensus.cc:385] T 648ad50f1dca4c0f96a5e5da089d12d1 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:50.738610 1167 raft_consensus.cc:740] T 648ad50f1dca4c0f96a5e5da089d12d1 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:51:50.738874 1167 consensus_queue.cc:260] T 648ad50f1dca4c0f96a5e5da089d12d1 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:50.740070 896 tablet_service.cc:1552] Processing DeleteTablet for tablet 6a378024f1fb45d086297968ab504156 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:50 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:50.739905 919 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "648ad50f1dca4c0f96a5e5da089d12d1" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:51:50.740458 919 raft_consensus.cc:2468] T 648ad50f1dca4c0f96a5e5da089d12d1 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b495a49349324f7d81908f2da700f0d9 in term 0.
I20251029 02:51:50.740476 1167 ts_tablet_manager.cc:1434] T 648ad50f1dca4c0f96a5e5da089d12d1 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20251029 02:51:50.741271 1167 tablet_bootstrap.cc:492] T 2d2ed6d2389c4f1e91e6f6423b89b3bc P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:51:50.742652 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8aaa0a9dd62443d68f96d7bd44a0c02b" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:51:50.743098 1167 tablet_bootstrap.cc:654] T 2d2ed6d2389c4f1e91e6f6423b89b3bc P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:50.744709 711 leader_election.cc:304] T 648ad50f1dca4c0f96a5e5da089d12d1 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:51:50.744843 1061 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8aaa0a9dd62443d68f96d7bd44a0c02b" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:51:50.745072 1061 raft_consensus.cc:2468] T 8aaa0a9dd62443d68f96d7bd44a0c02b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b495a49349324f7d81908f2da700f0d9 in term 0.
I20251029 02:51:50.745150 521 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet c7d00bd332f241c1b1fde8ce7fa054f5
I20251029 02:51:50.745151 1179 raft_consensus.cc:2804] T 648ad50f1dca4c0f96a5e5da089d12d1 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20251029 02:51:50.745262 711 leader_election.cc:343] T 8aaa0a9dd62443d68f96d7bd44a0c02b P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:50.745267 1179 raft_consensus.cc:493] T 648ad50f1dca4c0f96a5e5da089d12d1 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:51:50.745406 1062 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "648ad50f1dca4c0f96a5e5da089d12d1" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:51:50.745546 1179 raft_consensus.cc:3060] T 648ad50f1dca4c0f96a5e5da089d12d1 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:51:50.745575 1062 raft_consensus.cc:2468] T 648ad50f1dca4c0f96a5e5da089d12d1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b495a49349324f7d81908f2da700f0d9 in term 0.
I20251029 02:51:50.748560 1179 raft_consensus.cc:515] T 648ad50f1dca4c0f96a5e5da089d12d1 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:50.749581 1179 leader_election.cc:290] T 648ad50f1dca4c0f96a5e5da089d12d1 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 election: Requested vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:51:50.750022 521 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet 2dfa48aa89074a53bc13ed5ea199490e (table default.Table_83ad3367766c49269223bd91b060a972 [id=3bd791c439904f8c9e66b9e4dd107a78]). Expected version 1 got 0
I20251029 02:51:50.750093 712 leader_election.cc:304] T 8aaa0a9dd62443d68f96d7bd44a0c02b P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters: c756528642a543eeacd93cef3e335a7d
W20251029 02:51:50.750432 712 raft_consensus.cc:2686] T 8aaa0a9dd62443d68f96d7bd44a0c02b P b495a49349324f7d81908f2da700f0d9: unable to run election callback: Service unavailable: Thread pool token was shut down
I20251029 02:51:50.750415 1062 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "648ad50f1dca4c0f96a5e5da089d12d1" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:51:50.750725 1062 raft_consensus.cc:3060] T 648ad50f1dca4c0f96a5e5da089d12d1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:51:50.751576 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "81c770c1c9a84637b71989257cb4453a" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:51:50.751770 776 raft_consensus.cc:2468] T 81c770c1c9a84637b71989257cb4453a P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 0.
I20251029 02:51:50.752218 1167 tablet_bootstrap.cc:492] T 2d2ed6d2389c4f1e91e6f6423b89b3bc P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:51:50.752310 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "648ad50f1dca4c0f96a5e5da089d12d1" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
I20251029 02:51:50.752564 920 raft_consensus.cc:3060] T 648ad50f1dca4c0f96a5e5da089d12d1 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:51:50.752599 1167 ts_tablet_manager.cc:1403] T 2d2ed6d2389c4f1e91e6f6423b89b3bc P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.003s
I20251029 02:51:50.753484 1167 raft_consensus.cc:359] T 2d2ed6d2389c4f1e91e6f6423b89b3bc P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:50.753926 1167 raft_consensus.cc:385] T 2d2ed6d2389c4f1e91e6f6423b89b3bc P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:50.754091 1167 raft_consensus.cc:740] T 2d2ed6d2389c4f1e91e6f6423b89b3bc P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:51:50.754137 1062 raft_consensus.cc:2468] T 648ad50f1dca4c0f96a5e5da089d12d1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b495a49349324f7d81908f2da700f0d9 in term 1.
I20251029 02:51:50.754469 1167 consensus_queue.cc:260] T 2d2ed6d2389c4f1e91e6f6423b89b3bc P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
W20251029 02:51:50.754742 989 leader_election.cc:343] T a6d728bd089841a1a5ba997f8a71862a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: a6d728bd089841a1a5ba997f8a71862a
I20251029 02:51:50.755069 989 leader_election.cc:304] T 81c770c1c9a84637b71989257cb4453a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters:
W20251029 02:51:50.755393 989 raft_consensus.cc:2686] T 81c770c1c9a84637b71989257cb4453a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: unable to run election callback: Service unavailable: Thread pool token was shut down
I20251029 02:51:50.755445 1172 tablet_bootstrap.cc:492] T 6a378024f1fb45d086297968ab504156 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:51:50.755591 1172 ts_tablet_manager.cc:1403] T 6a378024f1fb45d086297968ab504156 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.032s user 0.002s sys 0.001s
I20251029 02:51:50.755722 920 raft_consensus.cc:2468] T 648ad50f1dca4c0f96a5e5da089d12d1 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b495a49349324f7d81908f2da700f0d9 in term 1.
I20251029 02:51:50.756023 712 leader_election.cc:304] T 648ad50f1dca4c0f96a5e5da089d12d1 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters:
I20251029 02:51:50.756307 1179 raft_consensus.cc:2804] T 648ad50f1dca4c0f96a5e5da089d12d1 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:51:50.756394 1172 raft_consensus.cc:359] T 6a378024f1fb45d086297968ab504156 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:50.756548 1167 ts_tablet_manager.cc:1434] T 2d2ed6d2389c4f1e91e6f6423b89b3bc P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251029 02:51:50.756564 1172 raft_consensus.cc:385] T 6a378024f1fb45d086297968ab504156 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:50.756649 1172 raft_consensus.cc:740] T 6a378024f1fb45d086297968ab504156 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:51:50.756918 1172 consensus_queue.cc:260] T 6a378024f1fb45d086297968ab504156 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:50.758009 521 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet 2dfa48aa89074a53bc13ed5ea199490e (table default.Table_83ad3367766c49269223bd91b060a972 [id=3bd791c439904f8c9e66b9e4dd107a78]). Expected version 1 got 0
I20251029 02:51:50.759972 1166 tablet_bootstrap.cc:492] T 355f26adb2b44b98bedcc97072d951ad P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:51:50.762212 521 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_1b27eebc21ef4d1aa39f04fd2e3c3388" } new_table_name: "default.Table_bd27b9a173a243b0aa50adcf36f2a06d" modify_external_catalogs: true
I20251029 02:51:50.763413 1175 ts_tablet_manager.cc:1929] T 8aaa0a9dd62443d68f96d7bd44a0c02b P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:50.763567 1175 log.cc:1199] T 8aaa0a9dd62443d68f96d7bd44a0c02b P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/8aaa0a9dd62443d68f96d7bd44a0c02b
I20251029 02:51:50.763938 1175 ts_tablet_manager.cc:1950] T 8aaa0a9dd62443d68f96d7bd44a0c02b P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:51:50.764366 1176 ts_tablet_manager.cc:1929] T 02d3583e77714d08ae476cbfdb2583c9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:50.764508 1176 log.cc:1199] T 02d3583e77714d08ae476cbfdb2583c9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/02d3583e77714d08ae476cbfdb2583c9
I20251029 02:51:50.764909 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 8aaa0a9dd62443d68f96d7bd44a0c02b (table default.Table_c4aefe30acb547f5b9bf3b1f645f3e4e [id=2ce1f471df9d4210af37a1b4758daaf4]) successfully deleted
I20251029 02:51:50.765184 1176 ts_tablet_manager.cc:1950] T 02d3583e77714d08ae476cbfdb2583c9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:51:50.766261 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 02d3583e77714d08ae476cbfdb2583c9 (table default.Table_5b3a7640a9e54e7a9868985322ade0cb [id=1a1aeb237dad4464b717bbae5d4001b0]) successfully deleted
I20251029 02:51:50.767154 525 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet b3cf1f5cbff24f48a4c80971b6a979dd
I20251029 02:51:50.768332 1167 tablet_bootstrap.cc:492] T 81c770c1c9a84637b71989257cb4453a P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:51:50.768551 1166 tablet_bootstrap.cc:654] T 355f26adb2b44b98bedcc97072d951ad P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:50.770130 1167 tablet_bootstrap.cc:654] T 81c770c1c9a84637b71989257cb4453a P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:50.770287 1172 ts_tablet_manager.cc:1434] T 6a378024f1fb45d086297968ab504156 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.014s user 0.001s sys 0.001s
W20251029 02:51:50.771080 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 494b058c819548dab52917d7690a03ca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 494b058c819548dab52917d7690a03ca already in progress: creating tablet
I20251029 02:51:50.771687 1172 tablet_bootstrap.cc:492] T 3caf6aa732e6460e96145c0f4105a826 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:51:50.774307 1172 tablet_bootstrap.cc:654] T 3caf6aa732e6460e96145c0f4105a826 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:50.775820 1176 tablet_replica.cc:333] T a6d728bd089841a1a5ba997f8a71862a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:51:50.767141 1174 ts_tablet_manager.cc:1929] T 02d3583e77714d08ae476cbfdb2583c9 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:50.776360 1176 raft_consensus.cc:2243] T a6d728bd089841a1a5ba997f8a71862a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:50.776569 1174 log.cc:1199] T 02d3583e77714d08ae476cbfdb2583c9 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/02d3583e77714d08ae476cbfdb2583c9
I20251029 02:51:50.777040 1176 raft_consensus.cc:2272] T a6d728bd089841a1a5ba997f8a71862a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:50.777122 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "81c770c1c9a84637b71989257cb4453a" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:51:50.777540 1174 ts_tablet_manager.cc:1950] T 02d3583e77714d08ae476cbfdb2583c9 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:51:50.778543 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 02d3583e77714d08ae476cbfdb2583c9 (table default.Table_5b3a7640a9e54e7a9868985322ade0cb [id=1a1aeb237dad4464b717bbae5d4001b0]) successfully deleted
I20251029 02:51:50.779274 1176 ts_tablet_manager.cc:1916] T a6d728bd089841a1a5ba997f8a71862a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 02:51:50.778043 990 leader_election.cc:343] T 81c770c1c9a84637b71989257cb4453a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:50.782763 1175 tablet_replica.cc:333] T 81c770c1c9a84637b71989257cb4453a P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
W20251029 02:51:50.782820 990 leader_election.cc:343] T a6d728bd089841a1a5ba997f8a71862a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Not found: Tablet not found: a6d728bd089841a1a5ba997f8a71862a
I20251029 02:51:50.782950 990 leader_election.cc:304] T a6d728bd089841a1a5ba997f8a71862a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5; no voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d
I20251029 02:51:50.783015 1175 raft_consensus.cc:2243] T 81c770c1c9a84637b71989257cb4453a P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
W20251029 02:51:50.783200 990 raft_consensus.cc:2686] T a6d728bd089841a1a5ba997f8a71862a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: unable to run election callback: Service unavailable: Thread pool token was shut down
I20251029 02:51:50.783262 1175 raft_consensus.cc:2272] T 81c770c1c9a84637b71989257cb4453a P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251029 02:51:50.783895 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 494b058c819548dab52917d7690a03ca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 494b058c819548dab52917d7690a03ca already in progress: creating tablet
I20251029 02:51:50.784094 1175 ts_tablet_manager.cc:1916] T 81c770c1c9a84637b71989257cb4453a P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:50.784292 1174 tablet_replica.cc:333] T 24114a60aeb24653aa7384df675aac76 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:51:50.784549 1174 raft_consensus.cc:2243] T 24114a60aeb24653aa7384df675aac76 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:51:50.785022 1174 pending_rounds.cc:70] T 24114a60aeb24653aa7384df675aac76 P c756528642a543eeacd93cef3e335a7d: Trying to abort 1 pending ops.
I20251029 02:51:50.785246 1174 pending_rounds.cc:77] T 24114a60aeb24653aa7384df675aac76 P c756528642a543eeacd93cef3e335a7d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215949047433568256 op_type: NO_OP noop_request { }
I20251029 02:51:50.785430 1174 raft_consensus.cc:2889] T 24114a60aeb24653aa7384df675aac76 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 02:51:50.785565 1174 raft_consensus.cc:2272] T 24114a60aeb24653aa7384df675aac76 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:50.787145 1174 ts_tablet_manager.cc:1916] T 24114a60aeb24653aa7384df675aac76 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:50.787627 1166 tablet_bootstrap.cc:492] T 355f26adb2b44b98bedcc97072d951ad P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:51:50.787817 1166 ts_tablet_manager.cc:1403] T 355f26adb2b44b98bedcc97072d951ad P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.028s user 0.002s sys 0.003s
I20251029 02:51:50.788612 1166 raft_consensus.cc:359] T 355f26adb2b44b98bedcc97072d951ad P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:50.788864 1166 raft_consensus.cc:385] T 355f26adb2b44b98bedcc97072d951ad P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:50.788983 1166 raft_consensus.cc:740] T 355f26adb2b44b98bedcc97072d951ad P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:51:50.789381 1166 consensus_queue.cc:260] T 355f26adb2b44b98bedcc97072d951ad P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:50.795885 523 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 94edf3f2e0854b07bbd348ecab82d3ca
I20251029 02:51:50.796381 523 catalog_manager.cc:6719] ReplaceTablet: tablet 94edf3f2e0854b07bbd348ecab82d3ca deleted and replaced by tablet 86f8776b5c4c4082bf276199feaaefea
I20251029 02:51:50.797091 896 tablet_service.cc:1552] Processing DeleteTablet for tablet 94edf3f2e0854b07bbd348ecab82d3ca with delete_type TABLET_DATA_DELETED (replaced by tablet 86f8776b5c4c4082bf276199feaaefea) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:50.797408 753 tablet_service.cc:1552] Processing DeleteTablet for tablet 94edf3f2e0854b07bbd348ecab82d3ca with delete_type TABLET_DATA_DELETED (replaced by tablet 86f8776b5c4c4082bf276199feaaefea) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:50.799816 1175 ts_tablet_manager.cc:1929] T 81c770c1c9a84637b71989257cb4453a P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:50.799981 1175 log.cc:1199] T 81c770c1c9a84637b71989257cb4453a P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/81c770c1c9a84637b71989257cb4453a
I20251029 02:51:50.800391 1175 ts_tablet_manager.cc:1950] T 81c770c1c9a84637b71989257cb4453a P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:51:50.801479 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 81c770c1c9a84637b71989257cb4453a (table default.Table_19b1787d124341939d34a6e721a87ae9 [id=33950ad96f8140fbb26ab0cd8b2b2b97]) successfully deleted
I20251029 02:51:50.802286 1038 tablet_service.cc:1552] Processing DeleteTablet for tablet 94edf3f2e0854b07bbd348ecab82d3ca with delete_type TABLET_DATA_DELETED (replaced by tablet 86f8776b5c4c4082bf276199feaaefea) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:50.802623 1171 raft_consensus.cc:493] T 2d2ed6d2389c4f1e91e6f6423b89b3bc P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:50.802845 1166 ts_tablet_manager.cc:1434] T 355f26adb2b44b98bedcc97072d951ad P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20251029 02:51:50.802825 1171 raft_consensus.cc:515] T 2d2ed6d2389c4f1e91e6f6423b89b3bc P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:50.803514 1171 leader_election.cc:290] T 2d2ed6d2389c4f1e91e6f6423b89b3bc P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
W20251029 02:51:50.805400 849 leader_election.cc:343] T 2d2ed6d2389c4f1e91e6f6423b89b3bc P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Not found: Tablet not found: 2d2ed6d2389c4f1e91e6f6423b89b3bc
W20251029 02:51:50.805830 847 leader_election.cc:343] T 2d2ed6d2389c4f1e91e6f6423b89b3bc P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 2d2ed6d2389c4f1e91e6f6423b89b3bc
I20251029 02:51:50.805976 847 leader_election.cc:304] T 2d2ed6d2389c4f1e91e6f6423b89b3bc P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9
I20251029 02:51:50.806439 1171 raft_consensus.cc:2749] T 2d2ed6d2389c4f1e91e6f6423b89b3bc P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:51:50.807271 1176 ts_tablet_manager.cc:1929] T a6d728bd089841a1a5ba997f8a71862a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:50.807695 1167 tablet_bootstrap.cc:492] T 81c770c1c9a84637b71989257cb4453a P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:51:50.807859 1167 ts_tablet_manager.cc:1403] T 81c770c1c9a84637b71989257cb4453a P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.040s user 0.000s sys 0.002s
I20251029 02:51:50.808218 1176 log.cc:1199] T a6d728bd089841a1a5ba997f8a71862a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/a6d728bd089841a1a5ba997f8a71862a
I20251029 02:51:50.808745 1167 raft_consensus.cc:359] T 81c770c1c9a84637b71989257cb4453a P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } }
I20251029 02:51:50.808835 1179 raft_consensus.cc:697] T 648ad50f1dca4c0f96a5e5da089d12d1 P b495a49349324f7d81908f2da700f0d9 [term 1 LEADER]: Becoming Leader. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Running, Role: LEADER
I20251029 02:51:50.808947 1167 raft_consensus.cc:385] T 81c770c1c9a84637b71989257cb4453a P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:50.809023 1167 raft_consensus.cc:740] T 81c770c1c9a84637b71989257cb4453a P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:51:50.809266 1167 consensus_queue.cc:260] T 81c770c1c9a84637b71989257cb4453a P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } }
I20251029 02:51:50.809377 1179 consensus_queue.cc:237] T 648ad50f1dca4c0f96a5e5da089d12d1 P b495a49349324f7d81908f2da700f0d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:50.809590 1176 ts_tablet_manager.cc:1950] T a6d728bd089841a1a5ba997f8a71862a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:51:50.812081 1167 ts_tablet_manager.cc:1434] T 81c770c1c9a84637b71989257cb4453a P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251029 02:51:50.812086 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet a6d728bd089841a1a5ba997f8a71862a (table default.Table_5b3a7640a9e54e7a9868985322ade0cb [id=1a1aeb237dad4464b717bbae5d4001b0]) successfully deleted
I20251029 02:51:50.815054 1172 tablet_bootstrap.cc:492] T 3caf6aa732e6460e96145c0f4105a826 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:51:50.815290 1172 ts_tablet_manager.cc:1403] T 3caf6aa732e6460e96145c0f4105a826 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.044s user 0.007s sys 0.003s
I20251029 02:51:50.816128 1172 raft_consensus.cc:359] T 3caf6aa732e6460e96145c0f4105a826 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:50.816397 1172 raft_consensus.cc:385] T 3caf6aa732e6460e96145c0f4105a826 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:50.816529 1172 raft_consensus.cc:740] T 3caf6aa732e6460e96145c0f4105a826 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:51:50.816819 1172 consensus_queue.cc:260] T 3caf6aa732e6460e96145c0f4105a826 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:50.817369 1172 ts_tablet_manager.cc:1434] T 3caf6aa732e6460e96145c0f4105a826 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251029 02:51:50.817759 1172 tablet_bootstrap.cc:492] T 494b058c819548dab52917d7690a03ca P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:51:50.813298 1176 tablet_replica.cc:333] T 24114a60aeb24653aa7384df675aac76 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:51:50.818223 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_512c767544874054a9fddf1c846db270"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:50.818717 1176 raft_consensus.cc:2243] T 24114a60aeb24653aa7384df675aac76 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus shutting down.
W20251029 02:51:50.818820 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_512c767544874054a9fddf1c846db270 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:50.819285 1167 tablet_bootstrap.cc:492] T 8aaa0a9dd62443d68f96d7bd44a0c02b P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:51:50.818975 1176 raft_consensus.cc:2272] T 24114a60aeb24653aa7384df675aac76 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:50.820433 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_d1a375b674704cddb1c3bb13c566729e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:50.820926 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d1a375b674704cddb1c3bb13c566729e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:50.821085 1167 tablet_bootstrap.cc:654] T 8aaa0a9dd62443d68f96d7bd44a0c02b P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:50.821183 1174 ts_tablet_manager.cc:1929] T 24114a60aeb24653aa7384df675aac76 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:51:50.821333 1174 log.cc:1199] T 24114a60aeb24653aa7384df675aac76 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/24114a60aeb24653aa7384df675aac76
I20251029 02:51:50.821807 1174 ts_tablet_manager.cc:1950] T 24114a60aeb24653aa7384df675aac76 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:51:50.822024 1166 tablet_bootstrap.cc:492] T 0c0aeaabe14740718f75f723023aa087 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:51:50.823802 1166 tablet_bootstrap.cc:654] T 0c0aeaabe14740718f75f723023aa087 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:50.823848 1175 tablet_replica.cc:333] T 02d3583e77714d08ae476cbfdb2583c9 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:51:50.824057 1175 raft_consensus.cc:2243] T 02d3583e77714d08ae476cbfdb2583c9 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:50.824167 1172 tablet_bootstrap.cc:654] T 494b058c819548dab52917d7690a03ca P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:50.824254 1175 raft_consensus.cc:2272] T 02d3583e77714d08ae476cbfdb2583c9 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:50.824846 1175 ts_tablet_manager.cc:1916] T 02d3583e77714d08ae476cbfdb2583c9 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:50.826138 1176 ts_tablet_manager.cc:1916] T 24114a60aeb24653aa7384df675aac76 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 02:51:50.830327 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 6a378024f1fb45d086297968ab504156 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6a378024f1fb45d086297968ab504156 already in progress: creating tablet
W20251029 02:51:50.831633 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 94edf3f2e0854b07bbd348ecab82d3ca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 94edf3f2e0854b07bbd348ecab82d3ca already in progress: creating tablet
I20251029 02:51:50.832600 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_9588744bba32427a9fcdbdcd19582c7f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:50.833204 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9588744bba32427a9fcdbdcd19582c7f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:50.839403 1166 tablet_bootstrap.cc:492] T 0c0aeaabe14740718f75f723023aa087 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:51:50.839607 1166 ts_tablet_manager.cc:1403] T 0c0aeaabe14740718f75f723023aa087 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.002s
I20251029 02:51:50.840463 1166 raft_consensus.cc:359] T 0c0aeaabe14740718f75f723023aa087 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:50.840706 1166 raft_consensus.cc:385] T 0c0aeaabe14740718f75f723023aa087 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:50.840797 1166 raft_consensus.cc:740] T 0c0aeaabe14740718f75f723023aa087 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:51:50.840709 520 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_35aaa3609a3c43e4b75780f3480723f2" } modify_external_catalogs: true
I20251029 02:51:50.840903 1175 ts_tablet_manager.cc:1929] T 02d3583e77714d08ae476cbfdb2583c9 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:50.840932 520 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_35aaa3609a3c43e4b75780f3480723f2" } modify_external_catalogs: true
I20251029 02:51:50.841043 1175 log.cc:1199] T 02d3583e77714d08ae476cbfdb2583c9 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/02d3583e77714d08ae476cbfdb2583c9
I20251029 02:51:50.841467 1175 ts_tablet_manager.cc:1950] T 02d3583e77714d08ae476cbfdb2583c9 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:51:50.842092 1166 consensus_queue.cc:260] T 0c0aeaabe14740718f75f723023aa087 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:50.842288 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_c60a78e1cea24e8e864803a5c6124df1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:50.842650 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 02d3583e77714d08ae476cbfdb2583c9 (table default.Table_5b3a7640a9e54e7a9868985322ade0cb [id=1a1aeb237dad4464b717bbae5d4001b0]) successfully deleted
W20251029 02:51:50.842905 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c60a78e1cea24e8e864803a5c6124df1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:50.844655 1172 tablet_bootstrap.cc:492] T 494b058c819548dab52917d7690a03ca P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:51:50.844823 1172 ts_tablet_manager.cc:1403] T 494b058c819548dab52917d7690a03ca P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.027s user 0.001s sys 0.002s
I20251029 02:51:50.845079 1166 ts_tablet_manager.cc:1434] T 0c0aeaabe14740718f75f723023aa087 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20251029 02:51:50.845777 1172 raft_consensus.cc:359] T 494b058c819548dab52917d7690a03ca P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:50.845958 1172 raft_consensus.cc:385] T 494b058c819548dab52917d7690a03ca P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:50.846055 1172 raft_consensus.cc:740] T 494b058c819548dab52917d7690a03ca P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:51:50.846283 1172 consensus_queue.cc:260] T 494b058c819548dab52917d7690a03ca P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:50.846784 1172 ts_tablet_manager.cc:1434] T 494b058c819548dab52917d7690a03ca P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 02:51:50.846998 1166 tablet_bootstrap.cc:492] T 94edf3f2e0854b07bbd348ecab82d3ca P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:51:50.847077 1172 tablet_bootstrap.cc:492] T 94edf3f2e0854b07bbd348ecab82d3ca P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:51:50.849624 524 master_service.cc:954] ReplaceTablet: received request to replace tablet 648ad50f1dca4c0f96a5e5da089d12d1 from {username='slave'} at 127.0.0.1:35314
I20251029 02:51:50.851141 1172 tablet_bootstrap.cc:654] T 94edf3f2e0854b07bbd348ecab82d3ca P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:50.851727 1166 tablet_bootstrap.cc:654] T 94edf3f2e0854b07bbd348ecab82d3ca P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:50.853045 1176 ts_tablet_manager.cc:1929] T 24114a60aeb24653aa7384df675aac76 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:50.853202 1176 log.cc:1199] T 24114a60aeb24653aa7384df675aac76 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/24114a60aeb24653aa7384df675aac76
I20251029 02:51:50.853612 1176 ts_tablet_manager.cc:1950] T 24114a60aeb24653aa7384df675aac76 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:51:50.854971 1176 tablet_replica.cc:333] T 8aaa0a9dd62443d68f96d7bd44a0c02b P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:51:50.855157 1176 raft_consensus.cc:2243] T 8aaa0a9dd62443d68f96d7bd44a0c02b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:50.855356 1176 raft_consensus.cc:2272] T 8aaa0a9dd62443d68f96d7bd44a0c02b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251029 02:51:50.856496 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 6a378024f1fb45d086297968ab504156 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6a378024f1fb45d086297968ab504156 already in progress: creating tablet
I20251029 02:51:50.858165 1172 tablet_bootstrap.cc:492] T 94edf3f2e0854b07bbd348ecab82d3ca P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:51:50.858358 1172 ts_tablet_manager.cc:1403] T 94edf3f2e0854b07bbd348ecab82d3ca P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20251029 02:51:50.858820 518 master_service.cc:954] ReplaceTablet: received request to replace tablet 355f26adb2b44b98bedcc97072d951ad from {username='slave'} at 127.0.0.1:35314
I20251029 02:51:50.859280 1172 raft_consensus.cc:359] T 94edf3f2e0854b07bbd348ecab82d3ca P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:50.859517 1172 raft_consensus.cc:385] T 94edf3f2e0854b07bbd348ecab82d3ca P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:50.859606 1172 raft_consensus.cc:740] T 94edf3f2e0854b07bbd348ecab82d3ca P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:51:50.860425 1176 ts_tablet_manager.cc:1916] T 8aaa0a9dd62443d68f96d7bd44a0c02b P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:50.861390 1172 consensus_queue.cc:260] T 94edf3f2e0854b07bbd348ecab82d3ca P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:50.862807 1175 tablet_replica.cc:333] T 494b058c819548dab52917d7690a03ca P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:51:50.863222 1175 raft_consensus.cc:2243] T 494b058c819548dab52917d7690a03ca P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:50.863459 1175 raft_consensus.cc:2272] T 494b058c819548dab52917d7690a03ca P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:50.861920 1172 ts_tablet_manager.cc:1434] T 94edf3f2e0854b07bbd348ecab82d3ca P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20251029 02:51:50.865694 1172 tablet_bootstrap.cc:492] T 0c0aeaabe14740718f75f723023aa087 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:51:50.866873 1175 ts_tablet_manager.cc:1916] T 494b058c819548dab52917d7690a03ca P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 02:51:50.868691 521 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 86f8776b5c4c4082bf276199feaaefea has no consensus state
I20251029 02:51:50.868975 1172 tablet_bootstrap.cc:654] T 0c0aeaabe14740718f75f723023aa087 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:50.874248 1172 tablet_bootstrap.cc:492] T 0c0aeaabe14740718f75f723023aa087 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:51:50.874430 1172 ts_tablet_manager.cc:1403] T 0c0aeaabe14740718f75f723023aa087 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20251029 02:51:50.875355 1172 raft_consensus.cc:359] T 0c0aeaabe14740718f75f723023aa087 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:50.875537 1172 raft_consensus.cc:385] T 0c0aeaabe14740718f75f723023aa087 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:50.875625 1167 tablet_bootstrap.cc:492] T 8aaa0a9dd62443d68f96d7bd44a0c02b P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:51:50.875633 1172 raft_consensus.cc:740] T 0c0aeaabe14740718f75f723023aa087 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:51:50.875787 1167 ts_tablet_manager.cc:1403] T 8aaa0a9dd62443d68f96d7bd44a0c02b P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.057s user 0.006s sys 0.011s
I20251029 02:51:50.875892 1172 consensus_queue.cc:260] T 0c0aeaabe14740718f75f723023aa087 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:50.876557 1167 raft_consensus.cc:359] T 8aaa0a9dd62443d68f96d7bd44a0c02b P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } }
I20251029 02:51:50.876749 1167 raft_consensus.cc:385] T 8aaa0a9dd62443d68f96d7bd44a0c02b P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:50.876834 1167 raft_consensus.cc:740] T 8aaa0a9dd62443d68f96d7bd44a0c02b P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:51:50.877071 1167 consensus_queue.cc:260] T 8aaa0a9dd62443d68f96d7bd44a0c02b P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } }
I20251029 02:51:50.879626 1167 ts_tablet_manager.cc:1434] T 8aaa0a9dd62443d68f96d7bd44a0c02b P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251029 02:51:50.879974 1172 ts_tablet_manager.cc:1434] T 0c0aeaabe14740718f75f723023aa087 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251029 02:51:50.883167 1167 tablet_bootstrap.cc:492] T 2dfa48aa89074a53bc13ed5ea199490e P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:51:50.883217 1175 ts_tablet_manager.cc:1929] T 494b058c819548dab52917d7690a03ca P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:50.883327 1175 log.cc:1199] T 494b058c819548dab52917d7690a03ca P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/494b058c819548dab52917d7690a03ca
I20251029 02:51:50.883598 1172 tablet_bootstrap.cc:492] T 355f26adb2b44b98bedcc97072d951ad P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:51:50.884456 753 tablet_service.cc:1505] Processing CreateTablet for tablet ba014ecfec7f4602ae1c09d931bac824 (DEFAULT_TABLE table=default.Table_418d4e45a6364eaba65e55ffb9aeccbb [id=7bb8b246fbf1431bb71bbb7c5520e301]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:50.884847 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba014ecfec7f4602ae1c09d931bac824. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:50.886644 1176 ts_tablet_manager.cc:1929] T 8aaa0a9dd62443d68f96d7bd44a0c02b P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:50.886775 1176 log.cc:1199] T 8aaa0a9dd62443d68f96d7bd44a0c02b P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/8aaa0a9dd62443d68f96d7bd44a0c02b
I20251029 02:51:50.887156 1176 ts_tablet_manager.cc:1950] T 8aaa0a9dd62443d68f96d7bd44a0c02b P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:51:50.887176 756 tablet_service.cc:1505] Processing CreateTablet for tablet 94854145ac0d4be38780d4cdb721725e (DEFAULT_TABLE table=default.Table_631cf79ec867461baecbc486773b88c4 [id=7a493d6672624c90a81a647d7785181c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:50.887401 754 tablet_service.cc:1505] Processing CreateTablet for tablet a0cee0a6eb5a435b9bb3295cbe242b70 (DEFAULT_TABLE table=default.Table_9becb486767e4662a93114d2242f031d [id=c2475b97464f45da8990e004660c42b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:50.887516 526 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet 648ad50f1dca4c0f96a5e5da089d12d1 (table default.Table_9f11b26c8c144d7a8f6e2fbb3c6fe36e [id=d0cd3d1567c54463947443927e18169f]). Expected version 2 got 0
I20251029 02:51:50.887943 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94854145ac0d4be38780d4cdb721725e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:50.889300 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 8aaa0a9dd62443d68f96d7bd44a0c02b (table default.Table_c4aefe30acb547f5b9bf3b1f645f3e4e [id=2ce1f471df9d4210af37a1b4758daaf4]) successfully deleted
I20251029 02:51:50.889470 1038 tablet_service.cc:1505] Processing CreateTablet for tablet ba014ecfec7f4602ae1c09d931bac824 (DEFAULT_TABLE table=default.Table_418d4e45a6364eaba65e55ffb9aeccbb [id=7bb8b246fbf1431bb71bbb7c5520e301]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:50.889897 1037 tablet_service.cc:1505] Processing CreateTablet for tablet 94854145ac0d4be38780d4cdb721725e (DEFAULT_TABLE table=default.Table_631cf79ec867461baecbc486773b88c4 [id=7a493d6672624c90a81a647d7785181c]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 02:51:50.890209 512 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 86f8776b5c4c4082bf276199feaaefea (table default.Table_bd27b9a173a243b0aa50adcf36f2a06d [id=13b28fb78eb54543b973bfda2fb1aad8], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 86f8776b5c4c4082bf276199feaaefea has no consensus state
I20251029 02:51:50.890223 1042 tablet_service.cc:1505] Processing CreateTablet for tablet 55c1b243d8a94938a5caa58754092cec (DEFAULT_TABLE table=default.Table_134014ec659240928f77016dc4e6c4ff [id=bd2335c28dd44b5584ab71694cebda2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:50.890972 1041 tablet_service.cc:1505] Processing CreateTablet for tablet a0cee0a6eb5a435b9bb3295cbe242b70 (DEFAULT_TABLE table=default.Table_9becb486767e4662a93114d2242f031d [id=c2475b97464f45da8990e004660c42b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:50.889915 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba014ecfec7f4602ae1c09d931bac824. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:50.892045 896 tablet_service.cc:1505] Processing CreateTablet for tablet ba014ecfec7f4602ae1c09d931bac824 (DEFAULT_TABLE table=default.Table_418d4e45a6364eaba65e55ffb9aeccbb [id=7bb8b246fbf1431bb71bbb7c5520e301]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:50.892422 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba014ecfec7f4602ae1c09d931bac824. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:50.894820 895 tablet_service.cc:1505] Processing CreateTablet for tablet 55c1b243d8a94938a5caa58754092cec (DEFAULT_TABLE table=default.Table_134014ec659240928f77016dc4e6c4ff [id=bd2335c28dd44b5584ab71694cebda2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:50.894968 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 2d2ed6d2389c4f1e91e6f6423b89b3bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:50 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:50.895190 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55c1b243d8a94938a5caa58754092cec. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:50.896209 1174 tablet_replica.cc:333] T 2d2ed6d2389c4f1e91e6f6423b89b3bc P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:51:50.896440 1174 raft_consensus.cc:2243] T 2d2ed6d2389c4f1e91e6f6423b89b3bc P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:50.896651 1174 raft_consensus.cc:2272] T 2d2ed6d2389c4f1e91e6f6423b89b3bc P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:50.894832 899 tablet_service.cc:1505] Processing CreateTablet for tablet e6bcea41d5d4480eaf5d6986a93f5e45 (DEFAULT_TABLE table=default.Table_5b3a7640a9e54e7a9868985322ade0cb [id=1a1aeb237dad4464b717bbae5d4001b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:50.897680 1175 ts_tablet_manager.cc:1950] T 494b058c819548dab52917d7690a03ca P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:51:50.897805 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6bcea41d5d4480eaf5d6986a93f5e45. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:50.898826 900 tablet_service.cc:1505] Processing CreateTablet for tablet 94854145ac0d4be38780d4cdb721725e (DEFAULT_TABLE table=default.Table_631cf79ec867461baecbc486773b88c4 [id=7a493d6672624c90a81a647d7785181c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:50.899190 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94854145ac0d4be38780d4cdb721725e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:50.899263 1175 tablet_replica.cc:333] T 24114a60aeb24653aa7384df675aac76 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:51:50.899901 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 494b058c819548dab52917d7690a03ca (table default.Table_9c3c143c7d464c32acb4176464f0fb64 [id=ded0c39ca7954a40933ccc61c5d32909]) successfully deleted
I20251029 02:51:50.901247 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0cee0a6eb5a435b9bb3295cbe242b70. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:51:50.901738 511 catalog_manager.cc:4952] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 8cb92fa59f114b01aa27a28358ecb4e7 because the tablet was not found. No further retry: Not found: Tablet not found: 8cb92fa59f114b01aa27a28358ecb4e7
I20251029 02:51:50.902825 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0cee0a6eb5a435b9bb3295cbe242b70. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:50.902801 898 tablet_service.cc:1505] Processing CreateTablet for tablet a0cee0a6eb5a435b9bb3295cbe242b70 (DEFAULT_TABLE table=default.Table_9becb486767e4662a93114d2242f031d [id=c2475b97464f45da8990e004660c42b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:50.903116 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0cee0a6eb5a435b9bb3295cbe242b70. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:50.903945 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55c1b243d8a94938a5caa58754092cec. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:50.903882 751 tablet_service.cc:1505] Processing CreateTablet for tablet 55c1b243d8a94938a5caa58754092cec (DEFAULT_TABLE table=default.Table_134014ec659240928f77016dc4e6c4ff [id=bd2335c28dd44b5584ab71694cebda2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:50.907011 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94854145ac0d4be38780d4cdb721725e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:50.909090 1174 ts_tablet_manager.cc:1916] T 2d2ed6d2389c4f1e91e6f6423b89b3bc P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:50.909948 752 tablet_service.cc:1505] Processing CreateTablet for tablet e6bcea41d5d4480eaf5d6986a93f5e45 (DEFAULT_TABLE table=default.Table_5b3a7640a9e54e7a9868985322ade0cb [id=1a1aeb237dad4464b717bbae5d4001b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:50.910429 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55c1b243d8a94938a5caa58754092cec. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:50.910440 755 tablet_service.cc:1505] Processing CreateTablet for tablet 8f7c746f8b6948b58676298c2c2ea221 (DEFAULT_TABLE table=default.Table_fb13d20c094a41d88582ee4fcc83020f [id=669ae9dd0a5749b1aa3557ab8183a057]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:50.910939 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f7c746f8b6948b58676298c2c2ea221. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:50.910579 750 tablet_service.cc:1505] Processing CreateTablet for tablet f04473c3a5f74bdba196b91c2538537e (DEFAULT_TABLE table=default.Table_7e818850675e4c5c973db094e7223348 [id=ec8cdbf19343401da05dec0c8c060ae6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:50.911466 1039 tablet_service.cc:1505] Processing CreateTablet for tablet e6bcea41d5d4480eaf5d6986a93f5e45 (DEFAULT_TABLE table=default.Table_5b3a7640a9e54e7a9868985322ade0cb [id=1a1aeb237dad4464b717bbae5d4001b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:50.911849 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6bcea41d5d4480eaf5d6986a93f5e45. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:50.914067 894 tablet_service.cc:1505] Processing CreateTablet for tablet 8f7c746f8b6948b58676298c2c2ea221 (DEFAULT_TABLE table=default.Table_fb13d20c094a41d88582ee4fcc83020f [id=669ae9dd0a5749b1aa3557ab8183a057]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:50.914467 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f7c746f8b6948b58676298c2c2ea221. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:50.915846 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6bcea41d5d4480eaf5d6986a93f5e45. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:50.915789 1175 raft_consensus.cc:2243] T 24114a60aeb24653aa7384df675aac76 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:50.916031 1175 raft_consensus.cc:2272] T 24114a60aeb24653aa7384df675aac76 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:50.917081 1175 ts_tablet_manager.cc:1916] T 24114a60aeb24653aa7384df675aac76 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:50.917138 893 tablet_service.cc:1505] Processing CreateTablet for tablet f04473c3a5f74bdba196b91c2538537e (DEFAULT_TABLE table=default.Table_7e818850675e4c5c973db094e7223348 [id=ec8cdbf19343401da05dec0c8c060ae6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:50.917580 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f04473c3a5f74bdba196b91c2538537e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:50.918062 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f04473c3a5f74bdba196b91c2538537e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:50.921592 892 tablet_service.cc:1505] Processing CreateTablet for tablet 8b89ee9017b244c5b866b1fc7154ffe3 (DEFAULT_TABLE table=default.Table_25688635c7a948f1b010aa30b88a3b6c [id=052abf3dcc904bdcb869d5b885723dab]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:50.922114 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b89ee9017b244c5b866b1fc7154ffe3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:50.924029 1174 ts_tablet_manager.cc:1929] T 2d2ed6d2389c4f1e91e6f6423b89b3bc P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:50.924188 1174 log.cc:1199] T 2d2ed6d2389c4f1e91e6f6423b89b3bc P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/2d2ed6d2389c4f1e91e6f6423b89b3bc
I20251029 02:51:50.924580 1174 ts_tablet_manager.cc:1950] T 2d2ed6d2389c4f1e91e6f6423b89b3bc P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:51:50.924979 890 tablet_service.cc:1552] Processing DeleteTablet for tablet 8aaa0a9dd62443d68f96d7bd44a0c02b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:50 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:50.925282 891 tablet_service.cc:1552] Processing DeleteTablet for tablet 81c770c1c9a84637b71989257cb4453a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:50 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:50.926101 1167 tablet_bootstrap.cc:654] T 2dfa48aa89074a53bc13ed5ea199490e P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:50.928149 1166 tablet_bootstrap.cc:492] T 94edf3f2e0854b07bbd348ecab82d3ca P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:51:50.928347 1166 ts_tablet_manager.cc:1403] T 94edf3f2e0854b07bbd348ecab82d3ca P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.081s user 0.006s sys 0.001s
I20251029 02:51:50.929054 1167 tablet_bootstrap.cc:492] T 2dfa48aa89074a53bc13ed5ea199490e P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:51:50.929200 1166 raft_consensus.cc:359] T 94edf3f2e0854b07bbd348ecab82d3ca P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:50.929311 1167 ts_tablet_manager.cc:1403] T 2dfa48aa89074a53bc13ed5ea199490e P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.046s user 0.001s sys 0.003s
I20251029 02:51:50.929415 1166 raft_consensus.cc:385] T 94edf3f2e0854b07bbd348ecab82d3ca P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:50.929527 1166 raft_consensus.cc:740] T 94edf3f2e0854b07bbd348ecab82d3ca P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:51:50.929790 1166 consensus_queue.cc:260] T 94edf3f2e0854b07bbd348ecab82d3ca P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:50.930524 1167 raft_consensus.cc:359] T 2dfa48aa89074a53bc13ed5ea199490e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:50.930783 1167 raft_consensus.cc:385] T 2dfa48aa89074a53bc13ed5ea199490e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:50.930896 1167 raft_consensus.cc:740] T 2dfa48aa89074a53bc13ed5ea199490e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:51:50.931231 1167 consensus_queue.cc:260] T 2dfa48aa89074a53bc13ed5ea199490e P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:50.931470 1166 ts_tablet_manager.cc:1434] T 94edf3f2e0854b07bbd348ecab82d3ca P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251029 02:51:50.931797 1166 tablet_bootstrap.cc:492] T 6a378024f1fb45d086297968ab504156 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:51:50.936465 526 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_8169731c33644ab78428f0527930e4bc" } modify_external_catalogs: true
I20251029 02:51:50.936748 526 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_8169731c33644ab78428f0527930e4bc" } modify_external_catalogs: true
I20251029 02:51:50.937220 749 tablet_service.cc:1505] Processing CreateTablet for tablet 8b89ee9017b244c5b866b1fc7154ffe3 (DEFAULT_TABLE table=default.Table_25688635c7a948f1b010aa30b88a3b6c [id=052abf3dcc904bdcb869d5b885723dab]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:50.938215 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b89ee9017b244c5b866b1fc7154ffe3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:50.940903 1174 tablet_replica.cc:333] T 8aaa0a9dd62443d68f96d7bd44a0c02b P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:51:50.941082 1174 raft_consensus.cc:2243] T 8aaa0a9dd62443d68f96d7bd44a0c02b P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:50.941258 1174 raft_consensus.cc:2272] T 8aaa0a9dd62443d68f96d7bd44a0c02b P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:50.941701 1167 ts_tablet_manager.cc:1434] T 2dfa48aa89074a53bc13ed5ea199490e P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20251029 02:51:50.942093 522 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet 2dfa48aa89074a53bc13ed5ea199490e (table default.Table_83ad3367766c49269223bd91b060a972 [id=3bd791c439904f8c9e66b9e4dd107a78]). Expected version 1 got 0
I20251029 02:51:50.942260 1167 tablet_bootstrap.cc:492] T 494b058c819548dab52917d7690a03ca P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:51:50.942291 524 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 648ad50f1dca4c0f96a5e5da089d12d1
I20251029 02:51:50.942834 524 catalog_manager.cc:6719] ReplaceTablet: tablet 648ad50f1dca4c0f96a5e5da089d12d1 deleted and replaced by tablet 11f4252b5e694312aaa142468f8e68b4
I20251029 02:51:50.942940 522 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_134014ec659240928f77016dc4e6c4ff" } new_table_name: "default.Table_823c69ddf86f442baf2cfc774320e6e4" modify_external_catalogs: true
I20251029 02:51:50.943164 524 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_9f11b26c8c144d7a8f6e2fbb3c6fe36e" } new_table_name: "default.Table_a80bdae756454abe89c1b71249975f9e" modify_external_catalogs: true
I20251029 02:51:50.943907 756 tablet_service.cc:1552] Processing DeleteTablet for tablet 648ad50f1dca4c0f96a5e5da089d12d1 with delete_type TABLET_DATA_DELETED (replaced by tablet 11f4252b5e694312aaa142468f8e68b4) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:50.944595 520 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 1448a96dfe9e49278928d60fe247b78a
I20251029 02:51:50.944787 1167 tablet_bootstrap.cc:654] T 494b058c819548dab52917d7690a03ca P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:50.944893 520 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_bf3710e423ae4ab6838435bdf16dc7d0" } modify_external_catalogs: true
I20251029 02:51:50.945044 520 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_bf3710e423ae4ab6838435bdf16dc7d0" } modify_external_catalogs: true
I20251029 02:51:50.946288 518 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 355f26adb2b44b98bedcc97072d951ad
I20251029 02:51:50.946810 518 catalog_manager.cc:6719] ReplaceTablet: tablet 355f26adb2b44b98bedcc97072d951ad deleted and replaced by tablet deca0c4a114441079fce22e74b403050
I20251029 02:51:50.947213 518 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_418d4e45a6364eaba65e55ffb9aeccbb" } new_table_name: "default.Table_64e512f8ce034224aea9b887ed0d09dc" modify_external_catalogs: true
I20251029 02:51:50.948952 756 tablet_service.cc:1552] Processing DeleteTablet for tablet 355f26adb2b44b98bedcc97072d951ad with delete_type TABLET_DATA_DELETED (replaced by tablet deca0c4a114441079fce22e74b403050) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:50.949818 517 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_78f7811a55f243faa1be1c3f90754dcf" } new_table_name: "default.Table_61af959119eb4551940deeb613b069d9" modify_external_catalogs: true
W20251029 02:51:50.953673 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 94edf3f2e0854b07bbd348ecab82d3ca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 94edf3f2e0854b07bbd348ecab82d3ca already in progress: creating tablet
I20251029 02:51:50.954723 1167 tablet_bootstrap.cc:492] T 494b058c819548dab52917d7690a03ca P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:51:50.954869 1167 ts_tablet_manager.cc:1403] T 494b058c819548dab52917d7690a03ca P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.006s
I20251029 02:51:50.955360 525 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_83ad3367766c49269223bd91b060a972" } modify_external_catalogs: true
I20251029 02:51:50.955540 525 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_83ad3367766c49269223bd91b060a972" } modify_external_catalogs: true
I20251029 02:51:50.955650 1167 raft_consensus.cc:359] T 494b058c819548dab52917d7690a03ca P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:50.955880 1167 raft_consensus.cc:385] T 494b058c819548dab52917d7690a03ca P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:50.955987 1167 raft_consensus.cc:740] T 494b058c819548dab52917d7690a03ca P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:51:50.956202 896 tablet_service.cc:1552] Processing DeleteTablet for tablet 648ad50f1dca4c0f96a5e5da089d12d1 with delete_type TABLET_DATA_DELETED (replaced by tablet 11f4252b5e694312aaa142468f8e68b4) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:50.956456 895 tablet_service.cc:1552] Processing DeleteTablet for tablet 355f26adb2b44b98bedcc97072d951ad with delete_type TABLET_DATA_DELETED (replaced by tablet deca0c4a114441079fce22e74b403050) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:50.956729 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 2d2ed6d2389c4f1e91e6f6423b89b3bc (table default.Table_7a7b22a6e750401c9751e3ad5fbb42ed [id=18cf9b24efed4f1095ff99a7fe2022fb]) successfully deleted
I20251029 02:51:50.956993 1167 consensus_queue.cc:260] T 494b058c819548dab52917d7690a03ca P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:50.958559 1179 raft_consensus.cc:493] T 94edf3f2e0854b07bbd348ecab82d3ca P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:50.958742 1179 raft_consensus.cc:515] T 94edf3f2e0854b07bbd348ecab82d3ca P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:50.958853 1167 ts_tablet_manager.cc:1434] T 494b058c819548dab52917d7690a03ca P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251029 02:51:50.959278 1167 tablet_bootstrap.cc:492] T 355f26adb2b44b98bedcc97072d951ad P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:51:50.959678 1179 leader_election.cc:290] T 94edf3f2e0854b07bbd348ecab82d3ca P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:51:50.959805 1172 tablet_bootstrap.cc:654] T 355f26adb2b44b98bedcc97072d951ad P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:50.960071 919 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94edf3f2e0854b07bbd348ecab82d3ca" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:51:50.961784 711 leader_election.cc:343] T 94edf3f2e0854b07bbd348ecab82d3ca P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:50.963102 1174 ts_tablet_manager.cc:1916] T 8aaa0a9dd62443d68f96d7bd44a0c02b P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:50.966298 1170 raft_consensus.cc:493] T 355f26adb2b44b98bedcc97072d951ad P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:50.966528 1170 raft_consensus.cc:515] T 355f26adb2b44b98bedcc97072d951ad P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:50.967389 1040 tablet_service.cc:1505] Processing CreateTablet for tablet 8f7c746f8b6948b58676298c2c2ea221 (DEFAULT_TABLE table=default.Table_fb13d20c094a41d88582ee4fcc83020f [id=669ae9dd0a5749b1aa3557ab8183a057]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:50.969422 1175 ts_tablet_manager.cc:1929] T 24114a60aeb24653aa7384df675aac76 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:50.969982 1175 log.cc:1199] T 24114a60aeb24653aa7384df675aac76 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/24114a60aeb24653aa7384df675aac76
I20251029 02:51:50.970501 1174 ts_tablet_manager.cc:1929] T 8aaa0a9dd62443d68f96d7bd44a0c02b P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:50.971505 919 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "355f26adb2b44b98bedcc97072d951ad" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:51:50.972314 520 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet b79449f853984d71a466f3d75a5a1289
W20251029 02:51:50.972427 518 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ba014ecfec7f4602ae1c09d931bac824 consensus state has no leader
I20251029 02:51:50.972425 1174 log.cc:1199] T 8aaa0a9dd62443d68f96d7bd44a0c02b P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/8aaa0a9dd62443d68f96d7bd44a0c02b
I20251029 02:51:50.972734 520 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_512c767544874054a9fddf1c846db270" } modify_external_catalogs: true
I20251029 02:51:50.972877 1174 ts_tablet_manager.cc:1950] T 8aaa0a9dd62443d68f96d7bd44a0c02b P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:51:50.972926 520 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_512c767544874054a9fddf1c846db270" } modify_external_catalogs: true
I20251029 02:51:50.973837 1174 tablet_replica.cc:333] T 81c770c1c9a84637b71989257cb4453a P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:51:50.973943 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 8aaa0a9dd62443d68f96d7bd44a0c02b (table default.Table_c4aefe30acb547f5b9bf3b1f645f3e4e [id=2ce1f471df9d4210af37a1b4758daaf4]) successfully deleted
I20251029 02:51:50.974020 1174 raft_consensus.cc:2243] T 81c770c1c9a84637b71989257cb4453a P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:50.974192 1174 raft_consensus.cc:2272] T 81c770c1c9a84637b71989257cb4453a P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus is shut down!
W20251029 02:51:50.974732 522 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 55c1b243d8a94938a5caa58754092cec consensus state has no leader
W20251029 02:51:50.976168 990 leader_election.cc:343] T 355f26adb2b44b98bedcc97072d951ad P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:50.976168 1170 leader_election.cc:290] T 355f26adb2b44b98bedcc97072d951ad P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:51:50.976420 1036 tablet_service.cc:1505] Processing CreateTablet for tablet f04473c3a5f74bdba196b91c2538537e (DEFAULT_TABLE table=default.Table_7e818850675e4c5c973db094e7223348 [id=ec8cdbf19343401da05dec0c8c060ae6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:50.976661 1167 tablet_bootstrap.cc:654] T 355f26adb2b44b98bedcc97072d951ad P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
W20251029 02:51:50.977439 524 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 11f4252b5e694312aaa142468f8e68b4 has no consensus state
I20251029 02:51:50.978605 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "355f26adb2b44b98bedcc97072d951ad" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:51:50.974753 526 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 539631c341b34f2f861a833f9f17db48
I20251029 02:51:50.981367 1175 ts_tablet_manager.cc:1950] T 24114a60aeb24653aa7384df675aac76 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:51:50.983819 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_cf38449ac74341ddb95386e510cc9683"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:50.984483 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cf38449ac74341ddb95386e510cc9683 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:51:50.987560 517 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 22be6bc81efb434ab7fc3242193789c3 consensus state has no leader
I20251029 02:51:50.988206 1167 tablet_bootstrap.cc:492] T 355f26adb2b44b98bedcc97072d951ad P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:51:50.988315 1175 tablet_replica.cc:333] T 6a378024f1fb45d086297968ab504156 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:51:50.988445 1167 ts_tablet_manager.cc:1403] T 355f26adb2b44b98bedcc97072d951ad P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.029s user 0.009s sys 0.000s
I20251029 02:51:50.988570 1175 raft_consensus.cc:2243] T 6a378024f1fb45d086297968ab504156 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:50.988790 1175 raft_consensus.cc:2272] T 6a378024f1fb45d086297968ab504156 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:50.989528 1167 raft_consensus.cc:359] T 355f26adb2b44b98bedcc97072d951ad P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:50.989795 1167 raft_consensus.cc:385] T 355f26adb2b44b98bedcc97072d951ad P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:50.989871 1167 raft_consensus.cc:740] T 355f26adb2b44b98bedcc97072d951ad P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:51:50.990108 1167 consensus_queue.cc:260] T 355f26adb2b44b98bedcc97072d951ad P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:50.990646 1175 ts_tablet_manager.cc:1916] T 6a378024f1fb45d086297968ab504156 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:50.990851 1172 tablet_bootstrap.cc:492] T 355f26adb2b44b98bedcc97072d951ad P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:51:50.991058 1172 ts_tablet_manager.cc:1403] T 355f26adb2b44b98bedcc97072d951ad P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.109s user 0.006s sys 0.004s
I20251029 02:51:50.976401 1034 tablet_service.cc:1505] Processing CreateTablet for tablet 8b89ee9017b244c5b866b1fc7154ffe3 (DEFAULT_TABLE table=default.Table_25688635c7a948f1b010aa30b88a3b6c [id=052abf3dcc904bdcb869d5b885723dab]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:50.991915 1172 raft_consensus.cc:359] T 355f26adb2b44b98bedcc97072d951ad P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:50.992121 1172 raft_consensus.cc:385] T 355f26adb2b44b98bedcc97072d951ad P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:50.992210 1172 raft_consensus.cc:740] T 355f26adb2b44b98bedcc97072d951ad P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:51:50.992506 1172 consensus_queue.cc:260] T 355f26adb2b44b98bedcc97072d951ad P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:50.995630 1172 ts_tablet_manager.cc:1434] T 355f26adb2b44b98bedcc97072d951ad P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251029 02:51:50.996522 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f04473c3a5f74bdba196b91c2538537e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:50.998454 1172 tablet_bootstrap.cc:492] T 22be6bc81efb434ab7fc3242193789c3 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:51:50.998565 520 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet ca4f64dc98d7420893359b12b2bd57b4
I20251029 02:51:50.999140 1174 ts_tablet_manager.cc:1916] T 81c770c1c9a84637b71989257cb4453a P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:51.000140 1172 tablet_bootstrap.cc:654] T 22be6bc81efb434ab7fc3242193789c3 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:51.005889 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f7c746f8b6948b58676298c2c2ea221. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.009773 526 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet 3caf6aa732e6460e96145c0f4105a826 (table default.Table_4bdbcbb87e5b40aeb9f98eb1aa0e6fe3 [id=78615d8807a74b4496a3223ef6fe709d]). Expected version 2 got 0
W20251029 02:51:51.010710 526 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet 3caf6aa732e6460e96145c0f4105a826 (table default.Table_4bdbcbb87e5b40aeb9f98eb1aa0e6fe3 [id=78615d8807a74b4496a3223ef6fe709d], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet 3caf6aa732e6460e96145c0f4105a826 consensus state has no leader
I20251029 02:51:51.013794 749 tablet_service.cc:1505] Processing CreateTablet for tablet bc490b1327e44059a57e6cd7b63350e3 (DEFAULT_TABLE table=default.Table_6add80f3d25b4701b9a0fa5639dec66c [id=14bb2451b1c6422bb5fe048cd378d1b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.014240 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8b89ee9017b244c5b866b1fc7154ffe3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.014401 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc490b1327e44059a57e6cd7b63350e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.014322 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet 648ad50f1dca4c0f96a5e5da089d12d1 with delete_type TABLET_DATA_DELETED (replaced by tablet 11f4252b5e694312aaa142468f8e68b4) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:51.014889 1033 tablet_service.cc:1552] Processing DeleteTablet for tablet 355f26adb2b44b98bedcc97072d951ad with delete_type TABLET_DATA_DELETED (replaced by tablet deca0c4a114441079fce22e74b403050) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:51.014986 1176 tablet_replica.cc:333] T 648ad50f1dca4c0f96a5e5da089d12d1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:51:51.015192 1176 raft_consensus.cc:2243] T 648ad50f1dca4c0f96a5e5da089d12d1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:51.015367 1176 raft_consensus.cc:2272] T 648ad50f1dca4c0f96a5e5da089d12d1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:51.015693 893 tablet_service.cc:1505] Processing CreateTablet for tablet bc490b1327e44059a57e6cd7b63350e3 (DEFAULT_TABLE table=default.Table_6add80f3d25b4701b9a0fa5639dec66c [id=14bb2451b1c6422bb5fe048cd378d1b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.015762 894 tablet_service.cc:1505] Processing CreateTablet for tablet 42f463dc3cc1434b8ee5ba268cb9fc9a (DEFAULT_TABLE table=default.Table_fb62281fb765427dae4f824d094a9155 [id=9898cffc2b964059ba3163ee4d1395c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.015936 1172 tablet_bootstrap.cc:492] T 22be6bc81efb434ab7fc3242193789c3 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:51:51.016079 1172 ts_tablet_manager.cc:1403] T 22be6bc81efb434ab7fc3242193789c3 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.004s
I20251029 02:51:51.016129 1176 ts_tablet_manager.cc:1916] T 648ad50f1dca4c0f96a5e5da089d12d1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:51.016155 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42f463dc3cc1434b8ee5ba268cb9fc9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.016341 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc490b1327e44059a57e6cd7b63350e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.016932 1172 raft_consensus.cc:359] T 22be6bc81efb434ab7fc3242193789c3 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:51.017107 1172 raft_consensus.cc:385] T 22be6bc81efb434ab7fc3242193789c3 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:51.017190 1172 raft_consensus.cc:740] T 22be6bc81efb434ab7fc3242193789c3 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:51:51.017468 1172 consensus_queue.cc:260] T 22be6bc81efb434ab7fc3242193789c3 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:51.018781 1062 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94edf3f2e0854b07bbd348ecab82d3ca" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:51:51.019376 1062 raft_consensus.cc:2468] T 94edf3f2e0854b07bbd348ecab82d3ca P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b495a49349324f7d81908f2da700f0d9 in term 0.
I20251029 02:51:51.020556 1172 ts_tablet_manager.cc:1434] T 22be6bc81efb434ab7fc3242193789c3 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251029 02:51:51.023098 1172 tablet_bootstrap.cc:492] T 94854145ac0d4be38780d4cdb721725e P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:51:51.024901 1172 tablet_bootstrap.cc:654] T 94854145ac0d4be38780d4cdb721725e P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
W20251029 02:51:51.025630 989 leader_election.cc:343] T 355f26adb2b44b98bedcc97072d951ad P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:51.026050 712 leader_election.cc:304] T 94edf3f2e0854b07bbd348ecab82d3ca P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters: c756528642a543eeacd93cef3e335a7d
I20251029 02:51:51.026482 1179 raft_consensus.cc:2804] T 94edf3f2e0854b07bbd348ecab82d3ca P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:51:51.026655 1179 raft_consensus.cc:493] T 94edf3f2e0854b07bbd348ecab82d3ca P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:51:51.026755 1179 raft_consensus.cc:3060] T 94edf3f2e0854b07bbd348ecab82d3ca P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:51:51.027032 989 leader_election.cc:304] T 355f26adb2b44b98bedcc97072d951ad P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5; no voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d
I20251029 02:51:51.027305 1170 raft_consensus.cc:2749] T 355f26adb2b44b98bedcc97072d951ad P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:51:51.027441 1172 tablet_bootstrap.cc:492] T 94854145ac0d4be38780d4cdb721725e P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:51:51.027621 1172 ts_tablet_manager.cc:1403] T 94854145ac0d4be38780d4cdb721725e P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20251029 02:51:51.028509 1172 raft_consensus.cc:359] T 94854145ac0d4be38780d4cdb721725e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:51.028708 1172 raft_consensus.cc:385] T 94854145ac0d4be38780d4cdb721725e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:51.028780 1172 raft_consensus.cc:740] T 94854145ac0d4be38780d4cdb721725e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:51:51.029008 1172 consensus_queue.cc:260] T 94854145ac0d4be38780d4cdb721725e P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:51.029660 1179 raft_consensus.cc:515] T 94edf3f2e0854b07bbd348ecab82d3ca P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:51.030478 893 tablet_service.cc:1505] Processing CreateTablet for tablet 86f8776b5c4c4082bf276199feaaefea (DEFAULT_TABLE table=default.Table_bd27b9a173a243b0aa50adcf36f2a06d [id=13b28fb78eb54543b973bfda2fb1aad8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.031008 1033 tablet_service.cc:1505] Processing CreateTablet for tablet bc490b1327e44059a57e6cd7b63350e3 (DEFAULT_TABLE table=default.Table_6add80f3d25b4701b9a0fa5639dec66c [id=14bb2451b1c6422bb5fe048cd378d1b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.031674 1179 leader_election.cc:290] T 94edf3f2e0854b07bbd348ecab82d3ca P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 election: Requested vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:51:51.032079 1172 ts_tablet_manager.cc:1434] T 94854145ac0d4be38780d4cdb721725e P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251029 02:51:51.032850 523 master_service.cc:954] ReplaceTablet: received request to replace tablet 3caf6aa732e6460e96145c0f4105a826 from {username='slave'} at 127.0.0.1:35314
I20251029 02:51:51.032940 526 master_service.cc:954] ReplaceTablet: received request to replace tablet 648ad50f1dca4c0f96a5e5da089d12d1 from {username='slave'} at 127.0.0.1:35314
I20251029 02:51:51.034618 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_3276fcafb63848adaa3184302586391c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:51.034822 750 tablet_service.cc:1505] Processing CreateTablet for tablet 86f8776b5c4c4082bf276199feaaefea (DEFAULT_TABLE table=default.Table_bd27b9a173a243b0aa50adcf36f2a06d [id=13b28fb78eb54543b973bfda2fb1aad8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.034981 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc490b1327e44059a57e6cd7b63350e3. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:51:51.035184 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3276fcafb63848adaa3184302586391c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:51.035202 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86f8776b5c4c4082bf276199feaaefea. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.036238 919 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94edf3f2e0854b07bbd348ecab82d3ca" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
W20251029 02:51:51.037628 711 leader_election.cc:343] T 94edf3f2e0854b07bbd348ecab82d3ca P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:51.037796 526 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet 22be6bc81efb434ab7fc3242193789c3 (table default.Table_61af959119eb4551940deeb613b069d9 [id=cf081cb5322d40c3bdd89cb9e0f58a99]). Expected version 1 got 0
I20251029 02:51:51.038703 755 tablet_service.cc:1505] Processing CreateTablet for tablet 42f463dc3cc1434b8ee5ba268cb9fc9a (DEFAULT_TABLE table=default.Table_fb62281fb765427dae4f824d094a9155 [id=9898cffc2b964059ba3163ee4d1395c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.039098 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42f463dc3cc1434b8ee5ba268cb9fc9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.039763 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_77c77066213c4d05bdf61bc81facdebd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:51.039907 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_4c5fd91d917e4f1a99baef291b2d77e0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:51.040195 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86f8776b5c4c4082bf276199feaaefea. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:51:51.040328 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_77c77066213c4d05bdf61bc81facdebd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:51:51.040732 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4c5fd91d917e4f1a99baef291b2d77e0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:51.041580 1035 tablet_service.cc:1505] Processing CreateTablet for tablet 42f463dc3cc1434b8ee5ba268cb9fc9a (DEFAULT_TABLE table=default.Table_fb62281fb765427dae4f824d094a9155 [id=9898cffc2b964059ba3163ee4d1395c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.041843 894 tablet_service.cc:1505] Processing CreateTablet for tablet 2ebde39f813d46e898ad1754325ad1a0 (DEFAULT_TABLE table=default.Table_1a524bb5f82a4041b8008ff91a0b9d27 [id=5db5ea0ab3e249f4acea117bb07187f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.042550 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ebde39f813d46e898ad1754325ad1a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.044291 1062 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94edf3f2e0854b07bbd348ecab82d3ca" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:51:51.045032 1062 raft_consensus.cc:3060] T 94edf3f2e0854b07bbd348ecab82d3ca P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:51:51.047211 525 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 2dfa48aa89074a53bc13ed5ea199490e
I20251029 02:51:51.048144 752 tablet_service.cc:1552] Processing DeleteTablet for tablet 2dfa48aa89074a53bc13ed5ea199490e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:50 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:51.048877 755 tablet_service.cc:1505] Processing CreateTablet for tablet 2ebde39f813d46e898ad1754325ad1a0 (DEFAULT_TABLE table=default.Table_1a524bb5f82a4041b8008ff91a0b9d27 [id=5db5ea0ab3e249f4acea117bb07187f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.049294 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ebde39f813d46e898ad1754325ad1a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.049455 1062 raft_consensus.cc:2468] T 94edf3f2e0854b07bbd348ecab82d3ca P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b495a49349324f7d81908f2da700f0d9 in term 1.
I20251029 02:51:51.049739 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42f463dc3cc1434b8ee5ba268cb9fc9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.053740 1175 ts_tablet_manager.cc:1929] T 6a378024f1fb45d086297968ab504156 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:51.053897 1175 log.cc:1199] T 6a378024f1fb45d086297968ab504156 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/6a378024f1fb45d086297968ab504156
I20251029 02:51:51.054368 1175 ts_tablet_manager.cc:1950] T 6a378024f1fb45d086297968ab504156 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:51:51.054333 892 tablet_service.cc:1552] Processing DeleteTablet for tablet 2dfa48aa89074a53bc13ed5ea199490e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:50 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:51.055035 1172 tablet_bootstrap.cc:492] T ba014ecfec7f4602ae1c09d931bac824 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:51:51.057394 1032 tablet_service.cc:1505] Processing CreateTablet for tablet 86f8776b5c4c4082bf276199feaaefea (DEFAULT_TABLE table=default.Table_bd27b9a173a243b0aa50adcf36f2a06d [id=13b28fb78eb54543b973bfda2fb1aad8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.058769 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_47a1a59db0444fa49f7274509385ff0c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:51.059360 712 leader_election.cc:304] T 94edf3f2e0854b07bbd348ecab82d3ca P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters: c756528642a543eeacd93cef3e335a7d
I20251029 02:51:51.059608 1167 ts_tablet_manager.cc:1434] T 355f26adb2b44b98bedcc97072d951ad P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.071s user 0.003s sys 0.001s
I20251029 02:51:51.059820 1172 tablet_bootstrap.cc:654] T ba014ecfec7f4602ae1c09d931bac824 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:51.060848 1167 tablet_bootstrap.cc:492] T 3caf6aa732e6460e96145c0f4105a826 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
W20251029 02:51:51.060976 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_47a1a59db0444fa49f7274509385ff0c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:51.062287 1179 raft_consensus.cc:2804] T 94edf3f2e0854b07bbd348ecab82d3ca P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:51:51.062603 1167 tablet_bootstrap.cc:654] T 3caf6aa732e6460e96145c0f4105a826 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:51.063009 1179 raft_consensus.cc:697] T 94edf3f2e0854b07bbd348ecab82d3ca P b495a49349324f7d81908f2da700f0d9 [term 1 LEADER]: Becoming Leader. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Running, Role: LEADER
I20251029 02:51:51.063333 1179 consensus_queue.cc:237] T 94edf3f2e0854b07bbd348ecab82d3ca P b495a49349324f7d81908f2da700f0d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:51.066834 519 master_service.cc:954] ReplaceTablet: received request to replace tablet 3caf6aa732e6460e96145c0f4105a826 from {username='slave'} at 127.0.0.1:35314
I20251029 02:51:51.067256 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_e6ca0a151fc14c5085c6ea454f460242"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:51.067860 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e6ca0a151fc14c5085c6ea454f460242 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:51.068353 1172 tablet_bootstrap.cc:492] T ba014ecfec7f4602ae1c09d931bac824 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:51:51.068589 1172 ts_tablet_manager.cc:1403] T ba014ecfec7f4602ae1c09d931bac824 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.004s
I20251029 02:51:51.069693 1172 raft_consensus.cc:359] T ba014ecfec7f4602ae1c09d931bac824 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:51.070267 1172 raft_consensus.cc:385] T ba014ecfec7f4602ae1c09d931bac824 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:51.070427 1172 raft_consensus.cc:740] T ba014ecfec7f4602ae1c09d931bac824 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:51:51.070581 1030 tablet_service.cc:1552] Processing DeleteTablet for tablet 2dfa48aa89074a53bc13ed5ea199490e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:50 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:51.070595 1174 ts_tablet_manager.cc:1929] T 81c770c1c9a84637b71989257cb4453a P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:51.070740 1174 log.cc:1199] T 81c770c1c9a84637b71989257cb4453a P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/81c770c1c9a84637b71989257cb4453a
I20251029 02:51:51.070849 1172 consensus_queue.cc:260] T ba014ecfec7f4602ae1c09d931bac824 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:51.071146 1174 ts_tablet_manager.cc:1950] T 81c770c1c9a84637b71989257cb4453a P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:51:51.071375 1167 tablet_bootstrap.cc:492] T 3caf6aa732e6460e96145c0f4105a826 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:51:51.071524 1167 ts_tablet_manager.cc:1403] T 3caf6aa732e6460e96145c0f4105a826 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.003s
I20251029 02:51:51.072152 1174 tablet_replica.cc:333] T 648ad50f1dca4c0f96a5e5da089d12d1 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:51:51.072337 1174 raft_consensus.cc:2243] T 648ad50f1dca4c0f96a5e5da089d12d1 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:51.072517 1174 raft_consensus.cc:2272] T 648ad50f1dca4c0f96a5e5da089d12d1 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:51.073129 1038 tablet_service.cc:1552] Processing DeleteTablet for tablet 94edf3f2e0854b07bbd348ecab82d3ca with delete_type TABLET_DATA_DELETED (replaced by tablet 86f8776b5c4c4082bf276199feaaefea) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:51.073150 524 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_fd9d401f3f92450cab95de4f0895a1e1" } new_table_name: "default.Table_e2703b6254854a0980e73e76f49db366" modify_external_catalogs: true
I20251029 02:51:51.073493 1175 tablet_replica.cc:333] T 94edf3f2e0854b07bbd348ecab82d3ca P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:51:51.074321 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86f8776b5c4c4082bf276199feaaefea. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.075435 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 6a378024f1fb45d086297968ab504156 (table default.Table_14462d3ac1c24574963a67d66a1cb274 [id=2c98232622ff4370a069b00d8d787f18]) successfully deleted
I20251029 02:51:51.075933 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 81c770c1c9a84637b71989257cb4453a (table default.Table_19b1787d124341939d34a6e721a87ae9 [id=33950ad96f8140fbb26ab0cd8b2b2b97]) successfully deleted
I20251029 02:51:51.072713 1031 tablet_service.cc:1505] Processing CreateTablet for tablet 2ebde39f813d46e898ad1754325ad1a0 (DEFAULT_TABLE table=default.Table_1a524bb5f82a4041b8008ff91a0b9d27 [id=5db5ea0ab3e249f4acea117bb07187f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.079699 1166 tablet_bootstrap.cc:654] T 6a378024f1fb45d086297968ab504156 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:51.081656 1174 ts_tablet_manager.cc:1916] T 648ad50f1dca4c0f96a5e5da089d12d1 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:51.081840 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ebde39f813d46e898ad1754325ad1a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.082078 1175 raft_consensus.cc:2243] T 94edf3f2e0854b07bbd348ecab82d3ca P b495a49349324f7d81908f2da700f0d9 [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:51:51.082612 1175 pending_rounds.cc:70] T 94edf3f2e0854b07bbd348ecab82d3ca P b495a49349324f7d81908f2da700f0d9: Trying to abort 1 pending ops.
I20251029 02:51:51.082903 1175 pending_rounds.cc:77] T 94edf3f2e0854b07bbd348ecab82d3ca P b495a49349324f7d81908f2da700f0d9: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215949050118664192 op_type: NO_OP noop_request { }
I20251029 02:51:51.083144 1175 raft_consensus.cc:2889] T 94edf3f2e0854b07bbd348ecab82d3ca P b495a49349324f7d81908f2da700f0d9 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 02:51:51.083354 1175 raft_consensus.cc:2272] T 94edf3f2e0854b07bbd348ecab82d3ca P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:51.084564 1175 ts_tablet_manager.cc:1916] T 94edf3f2e0854b07bbd348ecab82d3ca P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:51.086748 1172 ts_tablet_manager.cc:1434] T ba014ecfec7f4602ae1c09d931bac824 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.018s user 0.001s sys 0.001s
I20251029 02:51:51.087507 1176 ts_tablet_manager.cc:1929] T 648ad50f1dca4c0f96a5e5da089d12d1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:51.087692 523 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 3caf6aa732e6460e96145c0f4105a826
I20251029 02:51:51.087770 1176 log.cc:1199] T 648ad50f1dca4c0f96a5e5da089d12d1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/648ad50f1dca4c0f96a5e5da089d12d1
I20251029 02:51:51.088069 1172 tablet_bootstrap.cc:492] T a0cee0a6eb5a435b9bb3295cbe242b70 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:51:51.088132 1176 ts_tablet_manager.cc:1950] T 648ad50f1dca4c0f96a5e5da089d12d1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:51:51.088295 523 catalog_manager.cc:6719] ReplaceTablet: tablet 3caf6aa732e6460e96145c0f4105a826 deleted and replaced by tablet bf210435010d4dde8903fd116eb468e3
I20251029 02:51:51.089342 1176 tablet_replica.cc:333] T 355f26adb2b44b98bedcc97072d951ad P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:51:51.089519 1176 raft_consensus.cc:2243] T 355f26adb2b44b98bedcc97072d951ad P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:51.093356 755 tablet_service.cc:1552] Processing DeleteTablet for tablet 3caf6aa732e6460e96145c0f4105a826 with delete_type TABLET_DATA_DELETED (replaced by tablet bf210435010d4dde8903fd116eb468e3) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:51.093640 1176 raft_consensus.cc:2272] T 355f26adb2b44b98bedcc97072d951ad P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:51.095502 1172 tablet_bootstrap.cc:654] T a0cee0a6eb5a435b9bb3295cbe242b70 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:51.095531 1174 ts_tablet_manager.cc:1929] T 648ad50f1dca4c0f96a5e5da089d12d1 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:51.095691 1174 log.cc:1199] T 648ad50f1dca4c0f96a5e5da089d12d1 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/648ad50f1dca4c0f96a5e5da089d12d1
I20251029 02:51:51.096086 1174 ts_tablet_manager.cc:1950] T 648ad50f1dca4c0f96a5e5da089d12d1 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:51:51.097059 1174 tablet_replica.cc:333] T 355f26adb2b44b98bedcc97072d951ad P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:51:51.097266 1174 raft_consensus.cc:2243] T 355f26adb2b44b98bedcc97072d951ad P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:51.097453 1174 raft_consensus.cc:2272] T 355f26adb2b44b98bedcc97072d951ad P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:51.097904 1176 ts_tablet_manager.cc:1916] T 355f26adb2b44b98bedcc97072d951ad P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:51.098946 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 648ad50f1dca4c0f96a5e5da089d12d1 (table default.Table_a80bdae756454abe89c1b71249975f9e [id=d0cd3d1567c54463947443927e18169f]) successfully deleted
I20251029 02:51:51.100845 1040 tablet_service.cc:1552] Processing DeleteTablet for tablet 3caf6aa732e6460e96145c0f4105a826 with delete_type TABLET_DATA_DELETED (replaced by tablet bf210435010d4dde8903fd116eb468e3) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:51.100831 1167 raft_consensus.cc:359] T 3caf6aa732e6460e96145c0f4105a826 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:51.101008 1167 raft_consensus.cc:385] T 3caf6aa732e6460e96145c0f4105a826 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:51.101091 1167 raft_consensus.cc:740] T 3caf6aa732e6460e96145c0f4105a826 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
W20251029 02:51:51.101974 524 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0c0aeaabe14740718f75f723023aa087 consensus state has no leader
I20251029 02:51:51.103454 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 648ad50f1dca4c0f96a5e5da089d12d1 (table default.Table_a80bdae756454abe89c1b71249975f9e [id=d0cd3d1567c54463947443927e18169f]) successfully deleted
I20251029 02:51:51.103986 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 3caf6aa732e6460e96145c0f4105a826 with delete_type TABLET_DATA_DELETED (replaced by tablet bf210435010d4dde8903fd116eb468e3) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:51.104460 893 tablet_service.cc:1552] Processing DeleteTablet for tablet 494b058c819548dab52917d7690a03ca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:50 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:51.105222 1175 ts_tablet_manager.cc:1929] T 94edf3f2e0854b07bbd348ecab82d3ca P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:51:51.105394 1175 log.cc:1199] T 94edf3f2e0854b07bbd348ecab82d3ca P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/94edf3f2e0854b07bbd348ecab82d3ca
I20251029 02:51:51.105618 1167 consensus_queue.cc:260] T 3caf6aa732e6460e96145c0f4105a826 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:51.105806 1175 ts_tablet_manager.cc:1950] T 94edf3f2e0854b07bbd348ecab82d3ca P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:51:51.107388 1167 ts_tablet_manager.cc:1434] T 3caf6aa732e6460e96145c0f4105a826 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.036s user 0.001s sys 0.001s
I20251029 02:51:51.107697 1167 tablet_bootstrap.cc:492] T 6a378024f1fb45d086297968ab504156 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:51:51.107982 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 94edf3f2e0854b07bbd348ecab82d3ca (table default.Table_bd27b9a173a243b0aa50adcf36f2a06d [id=13b28fb78eb54543b973bfda2fb1aad8]) successfully deleted
I20251029 02:51:51.109391 1167 tablet_bootstrap.cc:654] T 6a378024f1fb45d086297968ab504156 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:51.113925 1174 ts_tablet_manager.cc:1916] T 355f26adb2b44b98bedcc97072d951ad P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:51.119079 525 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet ba014ecfec7f4602ae1c09d931bac824 (table default.Table_64e512f8ce034224aea9b887ed0d09dc [id=7bb8b246fbf1431bb71bbb7c5520e301]). Expected version 2 got 0
I20251029 02:51:51.120668 1174 ts_tablet_manager.cc:1929] T 355f26adb2b44b98bedcc97072d951ad P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:51.120836 1174 log.cc:1199] T 355f26adb2b44b98bedcc97072d951ad P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/355f26adb2b44b98bedcc97072d951ad
I20251029 02:51:51.121038 895 tablet_service.cc:1505] Processing CreateTablet for tablet deca0c4a114441079fce22e74b403050 (DEFAULT_TABLE table=default.Table_a85023af366f423b8a637b058d6c65b4 [id=144a24e58314435d85f9c759ef79fa60]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.121213 1174 ts_tablet_manager.cc:1950] T 355f26adb2b44b98bedcc97072d951ad P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:51:51.121423 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet deca0c4a114441079fce22e74b403050. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.121505 892 tablet_service.cc:1505] Processing CreateTablet for tablet d32532973b3f46dd8158a2ce42d26365 (DEFAULT_TABLE table=default.Table_d1a375b674704cddb1c3bb13c566729e [id=909ecde47c7245b2b84dcd665e779b1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.121848 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d32532973b3f46dd8158a2ce42d26365. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.122210 1174 tablet_replica.cc:333] T 2dfa48aa89074a53bc13ed5ea199490e P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:51:51.122407 1174 raft_consensus.cc:2243] T 2dfa48aa89074a53bc13ed5ea199490e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:51.122603 894 tablet_service.cc:1505] Processing CreateTablet for tablet 5886b8d39f384426bcbed7289ae72e51 (DEFAULT_TABLE table=default.Table_9588744bba32427a9fcdbdcd19582c7f [id=ace1554749d24504aad642b368b1781f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.122982 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5886b8d39f384426bcbed7289ae72e51. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.123412 1172 tablet_bootstrap.cc:492] T a0cee0a6eb5a435b9bb3295cbe242b70 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:51:51.123463 1167 tablet_bootstrap.cc:492] T 6a378024f1fb45d086297968ab504156 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:51:51.123577 1172 ts_tablet_manager.cc:1403] T a0cee0a6eb5a435b9bb3295cbe242b70 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.036s user 0.001s sys 0.002s
I20251029 02:51:51.123631 1167 ts_tablet_manager.cc:1403] T 6a378024f1fb45d086297968ab504156 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.001s
I20251029 02:51:51.124351 1172 raft_consensus.cc:359] T a0cee0a6eb5a435b9bb3295cbe242b70 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:51.124564 1172 raft_consensus.cc:385] T a0cee0a6eb5a435b9bb3295cbe242b70 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:51.124475 1167 raft_consensus.cc:359] T 6a378024f1fb45d086297968ab504156 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:51.124632 1172 raft_consensus.cc:740] T a0cee0a6eb5a435b9bb3295cbe242b70 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:51:51.124670 1167 raft_consensus.cc:385] T 6a378024f1fb45d086297968ab504156 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:51.124740 1167 raft_consensus.cc:740] T 6a378024f1fb45d086297968ab504156 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:51:51.124905 1172 consensus_queue.cc:260] T a0cee0a6eb5a435b9bb3295cbe242b70 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:51.124969 1167 consensus_queue.cc:260] T 6a378024f1fb45d086297968ab504156 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:51.125494 1167 ts_tablet_manager.cc:1434] T 6a378024f1fb45d086297968ab504156 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251029 02:51:51.126161 1167 tablet_bootstrap.cc:492] T 94edf3f2e0854b07bbd348ecab82d3ca P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:51:51.121227 893 tablet_service.cc:1505] Processing CreateTablet for tablet 0dd31dc5873d407d958a0f5199f78782 (DEFAULT_TABLE table=default.Table_c60a78e1cea24e8e864803a5c6124df1 [id=99445ef970c644f8a39388b2015da927]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.126900 1172 ts_tablet_manager.cc:1434] T a0cee0a6eb5a435b9bb3295cbe242b70 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251029 02:51:51.127009 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0dd31dc5873d407d958a0f5199f78782. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.127947 1172 tablet_bootstrap.cc:492] T 8f7c746f8b6948b58676298c2c2ea221 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:51:51.128366 1167 tablet_bootstrap.cc:654] T 94edf3f2e0854b07bbd348ecab82d3ca P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:51.129586 1172 tablet_bootstrap.cc:654] T 8f7c746f8b6948b58676298c2c2ea221 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:51.130445 1175 tablet_replica.cc:333] T 648ad50f1dca4c0f96a5e5da089d12d1 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:51:51.130676 1175 raft_consensus.cc:2243] T 648ad50f1dca4c0f96a5e5da089d12d1 P b495a49349324f7d81908f2da700f0d9 [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:51:51.130918 1175 pending_rounds.cc:70] T 648ad50f1dca4c0f96a5e5da089d12d1 P b495a49349324f7d81908f2da700f0d9: Trying to abort 1 pending ops.
I20251029 02:51:51.131009 1175 pending_rounds.cc:77] T 648ad50f1dca4c0f96a5e5da089d12d1 P b495a49349324f7d81908f2da700f0d9: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215949049081135104 op_type: NO_OP noop_request { }
I20251029 02:51:51.131122 1175 raft_consensus.cc:2889] T 648ad50f1dca4c0f96a5e5da089d12d1 P b495a49349324f7d81908f2da700f0d9 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 02:51:51.131230 1175 raft_consensus.cc:2272] T 648ad50f1dca4c0f96a5e5da089d12d1 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:51.131793 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 355f26adb2b44b98bedcc97072d951ad (table default.Table_a85023af366f423b8a637b058d6c65b4 [id=144a24e58314435d85f9c759ef79fa60]) successfully deleted
I20251029 02:51:51.131983 1175 ts_tablet_manager.cc:1916] T 648ad50f1dca4c0f96a5e5da089d12d1 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:51.136142 755 tablet_service.cc:1505] Processing CreateTablet for tablet 0dd31dc5873d407d958a0f5199f78782 (DEFAULT_TABLE table=default.Table_c60a78e1cea24e8e864803a5c6124df1 [id=99445ef970c644f8a39388b2015da927]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.136422 750 tablet_service.cc:1505] Processing CreateTablet for tablet d32532973b3f46dd8158a2ce42d26365 (DEFAULT_TABLE table=default.Table_d1a375b674704cddb1c3bb13c566729e [id=909ecde47c7245b2b84dcd665e779b1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.136634 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0dd31dc5873d407d958a0f5199f78782. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.136970 752 tablet_service.cc:1505] Processing CreateTablet for tablet deca0c4a114441079fce22e74b403050 (DEFAULT_TABLE table=default.Table_a85023af366f423b8a637b058d6c65b4 [id=144a24e58314435d85f9c759ef79fa60]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.137483 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet deca0c4a114441079fce22e74b403050. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.137676 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d32532973b3f46dd8158a2ce42d26365. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.138343 749 tablet_service.cc:1505] Processing CreateTablet for tablet 5886b8d39f384426bcbed7289ae72e51 (DEFAULT_TABLE table=default.Table_9588744bba32427a9fcdbdcd19582c7f [id=ace1554749d24504aad642b368b1781f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.139434 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5886b8d39f384426bcbed7289ae72e51. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.139516 1166 tablet_bootstrap.cc:492] T 6a378024f1fb45d086297968ab504156 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:51:51.139685 1166 ts_tablet_manager.cc:1403] T 6a378024f1fb45d086297968ab504156 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.208s user 0.001s sys 0.008s
I20251029 02:51:51.140381 1166 raft_consensus.cc:359] T 6a378024f1fb45d086297968ab504156 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:51.140599 1166 raft_consensus.cc:385] T 6a378024f1fb45d086297968ab504156 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:51.140662 1166 raft_consensus.cc:740] T 6a378024f1fb45d086297968ab504156 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:51:51.143939 1176 ts_tablet_manager.cc:1929] T 355f26adb2b44b98bedcc97072d951ad P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:51.144061 1176 log.cc:1199] T 355f26adb2b44b98bedcc97072d951ad P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/355f26adb2b44b98bedcc97072d951ad
I20251029 02:51:51.144405 1176 ts_tablet_manager.cc:1950] T 355f26adb2b44b98bedcc97072d951ad P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:51:51.144698 1166 consensus_queue.cc:260] T 6a378024f1fb45d086297968ab504156 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:51.145876 1166 ts_tablet_manager.cc:1434] T 6a378024f1fb45d086297968ab504156 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20251029 02:51:51.146162 1166 tablet_bootstrap.cc:492] T 494b058c819548dab52917d7690a03ca P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:51:51.148037 1166 tablet_bootstrap.cc:654] T 494b058c819548dab52917d7690a03ca P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:51.149807 1176 tablet_replica.cc:333] T 2dfa48aa89074a53bc13ed5ea199490e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:51:51.149999 1176 raft_consensus.cc:2243] T 2dfa48aa89074a53bc13ed5ea199490e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:51.150161 1176 raft_consensus.cc:2272] T 2dfa48aa89074a53bc13ed5ea199490e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:51.122627 1174 raft_consensus.cc:2272] T 2dfa48aa89074a53bc13ed5ea199490e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:51.150921 1176 ts_tablet_manager.cc:1916] T 2dfa48aa89074a53bc13ed5ea199490e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:51.151091 1174 ts_tablet_manager.cc:1916] T 2dfa48aa89074a53bc13ed5ea199490e P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:51.152230 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 355f26adb2b44b98bedcc97072d951ad (table default.Table_a85023af366f423b8a637b058d6c65b4 [id=144a24e58314435d85f9c759ef79fa60]) successfully deleted
I20251029 02:51:51.152381 1031 tablet_service.cc:1505] Processing CreateTablet for tablet 5886b8d39f384426bcbed7289ae72e51 (DEFAULT_TABLE table=default.Table_9588744bba32427a9fcdbdcd19582c7f [id=ace1554749d24504aad642b368b1781f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.152784 1039 tablet_service.cc:1505] Processing CreateTablet for tablet 9be9add0274d41b688edd6292a253a06 (DEFAULT_TABLE table=default.Table_cf38449ac74341ddb95386e510cc9683 [id=f45ce142f37f4febbceb4e568e35e0f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.152997 751 tablet_service.cc:1505] Processing CreateTablet for tablet 11f4252b5e694312aaa142468f8e68b4 (DEFAULT_TABLE table=default.Table_a80bdae756454abe89c1b71249975f9e [id=d0cd3d1567c54463947443927e18169f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.153332 1037 tablet_service.cc:1505] Processing CreateTablet for tablet deca0c4a114441079fce22e74b403050 (DEFAULT_TABLE table=default.Table_a85023af366f423b8a637b058d6c65b4 [id=144a24e58314435d85f9c759ef79fa60]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.154310 1034 tablet_service.cc:1505] Processing CreateTablet for tablet 11f4252b5e694312aaa142468f8e68b4 (DEFAULT_TABLE table=default.Table_a80bdae756454abe89c1b71249975f9e [id=d0cd3d1567c54463947443927e18169f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.154862 1032 tablet_service.cc:1505] Processing CreateTablet for tablet d32532973b3f46dd8158a2ce42d26365 (DEFAULT_TABLE table=default.Table_d1a375b674704cddb1c3bb13c566729e [id=909ecde47c7245b2b84dcd665e779b1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.155309 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5886b8d39f384426bcbed7289ae72e51. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.156941 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_77a87f09abd140cb8ef5d7a47097912e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:51.156980 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_06e55ce591df4a9ba1d6abea1bf01c3a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:51.157577 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_06e55ce591df4a9ba1d6abea1bf01c3a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:51:51.157598 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_77a87f09abd140cb8ef5d7a47097912e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:51.159492 525 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_61af959119eb4551940deeb613b069d9" } modify_external_catalogs: true
I20251029 02:51:51.159719 525 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_61af959119eb4551940deeb613b069d9" } modify_external_catalogs: true
I20251029 02:51:51.161149 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11f4252b5e694312aaa142468f8e68b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.162549 754 tablet_service.cc:1505] Processing CreateTablet for tablet 9be9add0274d41b688edd6292a253a06 (DEFAULT_TABLE table=default.Table_cf38449ac74341ddb95386e510cc9683 [id=f45ce142f37f4febbceb4e568e35e0f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.163012 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9be9add0274d41b688edd6292a253a06. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.166132 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_42a71464d71d487baa7245b5c3bdbbcd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:51.167452 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_e5a6d34625a6441fa8a50e020642ff79"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:51.168236 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e5a6d34625a6441fa8a50e020642ff79 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:51.175455 521 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_fb62281fb765427dae4f824d094a9155" } new_table_name: "default.Table_03f7d6f02b064939ba7464c5173e02e3" modify_external_catalogs: true
I20251029 02:51:51.175482 523 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_bd27b9a173a243b0aa50adcf36f2a06d" } modify_external_catalogs: true
I20251029 02:51:51.178156 523 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_bd27b9a173a243b0aa50adcf36f2a06d" } modify_external_catalogs: true
W20251029 02:51:51.178712 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_42a71464d71d487baa7245b5c3bdbbcd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:51.181496 522 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_1a524bb5f82a4041b8008ff91a0b9d27" } modify_external_catalogs: true
I20251029 02:51:51.181710 522 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_1a524bb5f82a4041b8008ff91a0b9d27" } modify_external_catalogs: true
I20251029 02:51:51.182304 518 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_9588744bba32427a9fcdbdcd19582c7f" } new_table_name: "default.Table_839fd4e3c7874233afc8cea6d4c17026" modify_external_catalogs: true
I20251029 02:51:51.187650 520 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_cf38449ac74341ddb95386e510cc9683" } new_table_name: "default.Table_c7daf9eda79e470186f5774de2b824d3" modify_external_catalogs: true
I20251029 02:51:51.187922 1175 ts_tablet_manager.cc:1929] T 648ad50f1dca4c0f96a5e5da089d12d1 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:51:51.188694 1175 log.cc:1199] T 648ad50f1dca4c0f96a5e5da089d12d1 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/648ad50f1dca4c0f96a5e5da089d12d1
I20251029 02:51:51.191972 1175 ts_tablet_manager.cc:1950] T 648ad50f1dca4c0f96a5e5da089d12d1 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:51:51.194597 519 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_fb13d20c094a41d88582ee4fcc83020f" } new_table_name: "default.Table_c1952ce7217c4687bee6f2c2c4381b33" modify_external_catalogs: true
I20251029 02:51:51.196588 1172 tablet_bootstrap.cc:492] T 8f7c746f8b6948b58676298c2c2ea221 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:51:51.196756 1172 ts_tablet_manager.cc:1403] T 8f7c746f8b6948b58676298c2c2ea221 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.069s user 0.003s sys 0.002s
I20251029 02:51:51.197235 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 648ad50f1dca4c0f96a5e5da089d12d1 (table default.Table_a80bdae756454abe89c1b71249975f9e [id=d0cd3d1567c54463947443927e18169f]) successfully deleted
I20251029 02:51:51.197835 1172 raft_consensus.cc:359] T 8f7c746f8b6948b58676298c2c2ea221 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:51.198062 1172 raft_consensus.cc:385] T 8f7c746f8b6948b58676298c2c2ea221 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:51.198161 1172 raft_consensus.cc:740] T 8f7c746f8b6948b58676298c2c2ea221 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
W20251029 02:51:51.199687 521 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 42f463dc3cc1434b8ee5ba268cb9fc9a consensus state has no leader
I20251029 02:51:51.200721 521 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_77a87f09abd140cb8ef5d7a47097912e" } modify_external_catalogs: true
I20251029 02:51:51.200955 521 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_77a87f09abd140cb8ef5d7a47097912e" } modify_external_catalogs: true
I20251029 02:51:51.198369 1175 tablet_replica.cc:333] T 355f26adb2b44b98bedcc97072d951ad P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:51:51.198454 1172 consensus_queue.cc:260] T 8f7c746f8b6948b58676298c2c2ea221 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:51.205164 1175 raft_consensus.cc:2243] T 355f26adb2b44b98bedcc97072d951ad P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:51.206827 1175 raft_consensus.cc:2272] T 355f26adb2b44b98bedcc97072d951ad P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:51.207346 1170 raft_consensus.cc:493] T 0c0aeaabe14740718f75f723023aa087 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:51.207525 1170 raft_consensus.cc:515] T 0c0aeaabe14740718f75f723023aa087 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:51.208443 1170 leader_election.cc:290] T 0c0aeaabe14740718f75f723023aa087 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:51:51.208762 1033 tablet_service.cc:1505] Processing CreateTablet for tablet 0dd31dc5873d407d958a0f5199f78782 (DEFAULT_TABLE table=default.Table_c60a78e1cea24e8e864803a5c6124df1 [id=99445ef970c644f8a39388b2015da927]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.210687 1176 ts_tablet_manager.cc:1929] T 2dfa48aa89074a53bc13ed5ea199490e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:51.210987 1176 log.cc:1199] T 2dfa48aa89074a53bc13ed5ea199490e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/2dfa48aa89074a53bc13ed5ea199490e
I20251029 02:51:51.212759 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9be9add0274d41b688edd6292a253a06. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.212983 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c0aeaabe14740718f75f723023aa087" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:51:51.213313 776 raft_consensus.cc:2468] T 0c0aeaabe14740718f75f723023aa087 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 0.
I20251029 02:51:51.214530 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11f4252b5e694312aaa142468f8e68b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.215241 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0dd31dc5873d407d958a0f5199f78782. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.215895 989 leader_election.cc:304] T 0c0aeaabe14740718f75f723023aa087 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters:
I20251029 02:51:51.217821 1170 raft_consensus.cc:2804] T 0c0aeaabe14740718f75f723023aa087 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:51:51.217959 1170 raft_consensus.cc:493] T 0c0aeaabe14740718f75f723023aa087 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:51:51.218067 1170 raft_consensus.cc:3060] T 0c0aeaabe14740718f75f723023aa087 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:51:51.220414 1031 tablet_service.cc:1552] Processing DeleteTablet for tablet 6a378024f1fb45d086297968ab504156 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:50 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:51.207186 1172 ts_tablet_manager.cc:1434] T 8f7c746f8b6948b58676298c2c2ea221 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20251029 02:51:51.222007 1170 raft_consensus.cc:515] T 0c0aeaabe14740718f75f723023aa087 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:51.224889 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_9c88c372f29d4f0980baa173608086f2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:51.225273 1172 tablet_bootstrap.cc:492] T f04473c3a5f74bdba196b91c2538537e P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
W20251029 02:51:51.225750 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9c88c372f29d4f0980baa173608086f2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:51.225977 1170 leader_election.cc:290] T 0c0aeaabe14740718f75f723023aa087 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Requested vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:51:51.226647 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c0aeaabe14740718f75f723023aa087" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:51:51.226691 1176 ts_tablet_manager.cc:1950] T 2dfa48aa89074a53bc13ed5ea199490e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:51:51.226935 776 raft_consensus.cc:3060] T 0c0aeaabe14740718f75f723023aa087 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:51:51.227558 1172 tablet_bootstrap.cc:654] T f04473c3a5f74bdba196b91c2538537e P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:51.229038 1175 ts_tablet_manager.cc:1916] T 355f26adb2b44b98bedcc97072d951ad P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:51.229259 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 2dfa48aa89074a53bc13ed5ea199490e (table default.Table_83ad3367766c49269223bd91b060a972 [id=3bd791c439904f8c9e66b9e4dd107a78]) successfully deleted
I20251029 02:51:51.229926 1176 tablet_replica.cc:333] T 94edf3f2e0854b07bbd348ecab82d3ca P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:51:51.231029 1176 raft_consensus.cc:2243] T 94edf3f2e0854b07bbd348ecab82d3ca P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:51.232167 1176 raft_consensus.cc:2272] T 94edf3f2e0854b07bbd348ecab82d3ca P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:51.233268 776 raft_consensus.cc:2468] T 0c0aeaabe14740718f75f723023aa087 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 1.
I20251029 02:51:51.233893 989 leader_election.cc:304] T 0c0aeaabe14740718f75f723023aa087 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters:
I20251029 02:51:51.234234 1170 raft_consensus.cc:2804] T 0c0aeaabe14740718f75f723023aa087 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:51:51.234637 517 master_service.cc:954] ReplaceTablet: received request to replace tablet a0cee0a6eb5a435b9bb3295cbe242b70 from {username='slave'} at 127.0.0.1:35314
I20251029 02:51:51.235463 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet deca0c4a114441079fce22e74b403050. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.238148 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d32532973b3f46dd8158a2ce42d26365. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:51:51.242728 518 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5886b8d39f384426bcbed7289ae72e51 consensus state has no leader
I20251029 02:51:51.243100 518 master_service.cc:954] ReplaceTablet: received request to replace tablet 94854145ac0d4be38780d4cdb721725e from {username='slave'} at 127.0.0.1:35314
I20251029 02:51:51.247356 1166 tablet_bootstrap.cc:492] T 494b058c819548dab52917d7690a03ca P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:51:51.247555 1166 ts_tablet_manager.cc:1403] T 494b058c819548dab52917d7690a03ca P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.101s user 0.005s sys 0.010s
I20251029 02:51:51.248523 1166 raft_consensus.cc:359] T 494b058c819548dab52917d7690a03ca P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:51.248718 1166 raft_consensus.cc:385] T 494b058c819548dab52917d7690a03ca P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:51.248800 1166 raft_consensus.cc:740] T 494b058c819548dab52917d7690a03ca P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:51:51.249018 1166 consensus_queue.cc:260] T 494b058c819548dab52917d7690a03ca P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:51.251622 1172 tablet_bootstrap.cc:492] T f04473c3a5f74bdba196b91c2538537e P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:51:51.248525 1170 raft_consensus.cc:697] T 0c0aeaabe14740718f75f723023aa087 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Becoming Leader. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Running, Role: LEADER
I20251029 02:51:51.251921 1172 ts_tablet_manager.cc:1403] T f04473c3a5f74bdba196b91c2538537e P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.003s
I20251029 02:51:51.252206 1170 consensus_queue.cc:237] T 0c0aeaabe14740718f75f723023aa087 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:51.252724 1172 raft_consensus.cc:359] T f04473c3a5f74bdba196b91c2538537e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:51.253087 1172 raft_consensus.cc:385] T f04473c3a5f74bdba196b91c2538537e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:51.253168 1172 raft_consensus.cc:740] T f04473c3a5f74bdba196b91c2538537e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:51:51.253469 1172 consensus_queue.cc:260] T f04473c3a5f74bdba196b91c2538537e P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
W20251029 02:51:51.256096 520 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9be9add0274d41b688edd6292a253a06 consensus state has no leader
I20251029 02:51:51.256136 1172 ts_tablet_manager.cc:1434] T f04473c3a5f74bdba196b91c2538537e P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251029 02:51:51.256170 895 tablet_service.cc:1505] Processing CreateTablet for tablet 11f4252b5e694312aaa142468f8e68b4 (DEFAULT_TABLE table=default.Table_a80bdae756454abe89c1b71249975f9e [id=d0cd3d1567c54463947443927e18169f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.256587 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11f4252b5e694312aaa142468f8e68b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.256875 1172 tablet_bootstrap.cc:492] T 55c1b243d8a94938a5caa58754092cec P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:51:51.257910 520 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_c60a78e1cea24e8e864803a5c6124df1" } modify_external_catalogs: true
I20251029 02:51:51.258095 520 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_c60a78e1cea24e8e864803a5c6124df1" } modify_external_catalogs: true
I20251029 02:51:51.258445 1172 tablet_bootstrap.cc:654] T 55c1b243d8a94938a5caa58754092cec P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:51.258774 1176 ts_tablet_manager.cc:1916] T 94edf3f2e0854b07bbd348ecab82d3ca P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 02:51:51.259012 519 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8f7c746f8b6948b58676298c2c2ea221 consensus state has no leader
I20251029 02:51:51.259155 894 tablet_service.cc:1505] Processing CreateTablet for tablet 9be9add0274d41b688edd6292a253a06 (DEFAULT_TABLE table=default.Table_cf38449ac74341ddb95386e510cc9683 [id=f45ce142f37f4febbceb4e568e35e0f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.259552 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9be9add0274d41b688edd6292a253a06. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.259608 521 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 9622d65e1e094057b39bc6b1bf84c69f
I20251029 02:51:51.263980 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c0aeaabe14740718f75f723023aa087" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
I20251029 02:51:51.264359 919 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0c0aeaabe14740718f75f723023aa087" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:51:51.265033 990 leader_election.cc:343] T 0c0aeaabe14740718f75f723023aa087 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
W20251029 02:51:51.265379 990 leader_election.cc:343] T 0c0aeaabe14740718f75f723023aa087 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:51.270555 893 tablet_service.cc:1552] Processing DeleteTablet for tablet 6a378024f1fb45d086297968ab504156 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:50 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:51.275925 1172 tablet_bootstrap.cc:492] T 55c1b243d8a94938a5caa58754092cec P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:51:51.276087 1172 ts_tablet_manager.cc:1403] T 55c1b243d8a94938a5caa58754092cec P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.001s
I20251029 02:51:51.276866 1172 raft_consensus.cc:359] T 55c1b243d8a94938a5caa58754092cec P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:51.277052 1172 raft_consensus.cc:385] T 55c1b243d8a94938a5caa58754092cec P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:51.277129 1172 raft_consensus.cc:740] T 55c1b243d8a94938a5caa58754092cec P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:51:51.277351 1172 consensus_queue.cc:260] T 55c1b243d8a94938a5caa58754092cec P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:51.279026 1174 ts_tablet_manager.cc:1929] T 2dfa48aa89074a53bc13ed5ea199490e P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:51.279520 1174 log.cc:1199] T 2dfa48aa89074a53bc13ed5ea199490e P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/2dfa48aa89074a53bc13ed5ea199490e
I20251029 02:51:51.279497 1179 raft_consensus.cc:493] T 55c1b243d8a94938a5caa58754092cec P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:51.279666 1179 raft_consensus.cc:515] T 55c1b243d8a94938a5caa58754092cec P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:51.280325 1179 leader_election.cc:290] T 55c1b243d8a94938a5caa58754092cec P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:51:51.281538 1062 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "55c1b243d8a94938a5caa58754092cec" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:51:51.282130 712 leader_election.cc:343] T 55c1b243d8a94938a5caa58754092cec P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:51.282232 1172 ts_tablet_manager.cc:1434] T 55c1b243d8a94938a5caa58754092cec P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251029 02:51:51.283213 919 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "55c1b243d8a94938a5caa58754092cec" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:51:51.283526 1176 ts_tablet_manager.cc:1929] T 94edf3f2e0854b07bbd348ecab82d3ca P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:51.283646 1176 log.cc:1199] T 94edf3f2e0854b07bbd348ecab82d3ca P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/94edf3f2e0854b07bbd348ecab82d3ca
I20251029 02:51:51.283684 1174 ts_tablet_manager.cc:1950] T 2dfa48aa89074a53bc13ed5ea199490e P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
W20251029 02:51:51.283690 711 leader_election.cc:343] T 55c1b243d8a94938a5caa58754092cec P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:51.283830 711 leader_election.cc:304] T 55c1b243d8a94938a5caa58754092cec P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:51:51.284011 1176 ts_tablet_manager.cc:1950] T 94edf3f2e0854b07bbd348ecab82d3ca P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:51:51.284174 1172 tablet_bootstrap.cc:492] T e6bcea41d5d4480eaf5d6986a93f5e45 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:51:51.284601 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 94edf3f2e0854b07bbd348ecab82d3ca (table default.Table_bd27b9a173a243b0aa50adcf36f2a06d [id=13b28fb78eb54543b973bfda2fb1aad8]) successfully deleted
I20251029 02:51:51.284644 1179 raft_consensus.cc:2749] T 55c1b243d8a94938a5caa58754092cec P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 02:51:51.284950 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 3caf6aa732e6460e96145c0f4105a826 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3caf6aa732e6460e96145c0f4105a826 already in progress: creating tablet
I20251029 02:51:51.285972 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 2dfa48aa89074a53bc13ed5ea199490e (table default.Table_83ad3367766c49269223bd91b060a972 [id=3bd791c439904f8c9e66b9e4dd107a78]) successfully deleted
I20251029 02:51:51.287097 1172 tablet_bootstrap.cc:654] T e6bcea41d5d4480eaf5d6986a93f5e45 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:51.287657 1175 ts_tablet_manager.cc:1929] T 355f26adb2b44b98bedcc97072d951ad P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:51.287791 1175 log.cc:1199] T 355f26adb2b44b98bedcc97072d951ad P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/355f26adb2b44b98bedcc97072d951ad
I20251029 02:51:51.288115 1175 ts_tablet_manager.cc:1950] T 355f26adb2b44b98bedcc97072d951ad P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:51:51.294345 1175 tablet_replica.cc:333] T 2dfa48aa89074a53bc13ed5ea199490e P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:51:51.294373 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 355f26adb2b44b98bedcc97072d951ad (table default.Table_a85023af366f423b8a637b058d6c65b4 [id=144a24e58314435d85f9c759ef79fa60]) successfully deleted
I20251029 02:51:51.294658 1175 raft_consensus.cc:2243] T 2dfa48aa89074a53bc13ed5ea199490e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:51.294690 1174 tablet_replica.cc:333] T 3caf6aa732e6460e96145c0f4105a826 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:51:51.294888 1175 raft_consensus.cc:2272] T 2dfa48aa89074a53bc13ed5ea199490e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:51.295055 1174 raft_consensus.cc:2243] T 3caf6aa732e6460e96145c0f4105a826 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:51.295316 1174 raft_consensus.cc:2272] T 3caf6aa732e6460e96145c0f4105a826 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:51.295492 1175 ts_tablet_manager.cc:1916] T 2dfa48aa89074a53bc13ed5ea199490e P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:51.296375 1174 ts_tablet_manager.cc:1916] T 3caf6aa732e6460e96145c0f4105a826 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:51.299294 1167 tablet_bootstrap.cc:492] T 94edf3f2e0854b07bbd348ecab82d3ca P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:51:51.299507 1167 ts_tablet_manager.cc:1403] T 94edf3f2e0854b07bbd348ecab82d3ca P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.173s user 0.032s sys 0.036s
I20251029 02:51:51.300112 1176 tablet_replica.cc:333] T 6a378024f1fb45d086297968ab504156 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:51:51.300213 1172 tablet_bootstrap.cc:492] T e6bcea41d5d4480eaf5d6986a93f5e45 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:51:51.300339 1176 raft_consensus.cc:2243] T 6a378024f1fb45d086297968ab504156 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:51.300374 1172 ts_tablet_manager.cc:1403] T e6bcea41d5d4480eaf5d6986a93f5e45 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.000s
I20251029 02:51:51.300547 1176 raft_consensus.cc:2272] T 6a378024f1fb45d086297968ab504156 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:51.300582 1167 raft_consensus.cc:359] T 94edf3f2e0854b07bbd348ecab82d3ca P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:51.300859 1167 raft_consensus.cc:385] T 94edf3f2e0854b07bbd348ecab82d3ca P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:51.301239 1172 raft_consensus.cc:359] T e6bcea41d5d4480eaf5d6986a93f5e45 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:51.301484 1172 raft_consensus.cc:385] T e6bcea41d5d4480eaf5d6986a93f5e45 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:51.301577 1172 raft_consensus.cc:740] T e6bcea41d5d4480eaf5d6986a93f5e45 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:51:51.301543 1167 raft_consensus.cc:740] T 94edf3f2e0854b07bbd348ecab82d3ca P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:51:51.301815 1172 consensus_queue.cc:260] T e6bcea41d5d4480eaf5d6986a93f5e45 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:51.301921 1167 consensus_queue.cc:260] T 94edf3f2e0854b07bbd348ecab82d3ca P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:51.304870 1172 ts_tablet_manager.cc:1434] T e6bcea41d5d4480eaf5d6986a93f5e45 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20251029 02:51:51.305369 1172 tablet_bootstrap.cc:492] T 8b89ee9017b244c5b866b1fc7154ffe3 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:51:51.305455 1176 ts_tablet_manager.cc:1916] T 6a378024f1fb45d086297968ab504156 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:51.306392 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_c784291ac23b4f86ad8bd33ebd8f4755"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:51.307367 1172 tablet_bootstrap.cc:654] T 8b89ee9017b244c5b866b1fc7154ffe3 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
W20251029 02:51:51.307852 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c784291ac23b4f86ad8bd33ebd8f4755 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:51.308395 1167 ts_tablet_manager.cc:1434] T 94edf3f2e0854b07bbd348ecab82d3ca P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20251029 02:51:51.309641 521 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_06e55ce591df4a9ba1d6abea1bf01c3a" } new_table_name: "default.Table_469f3d7be86340a2a24565b8a5058d64" modify_external_catalogs: true
I20251029 02:51:51.311368 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_c17409b349f548f9a5062085395a2aee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:51.312065 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c17409b349f548f9a5062085395a2aee in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:51.316188 526 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet a0cee0a6eb5a435b9bb3295cbe242b70 (table default.Table_9becb486767e4662a93114d2242f031d [id=c2475b97464f45da8990e004660c42b3]). Expected version 1 got 0
I20251029 02:51:51.318449 1167 tablet_bootstrap.cc:492] T 0c0aeaabe14740718f75f723023aa087 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:51:51.320220 1167 tablet_bootstrap.cc:654] T 0c0aeaabe14740718f75f723023aa087 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:51.321427 894 tablet_service.cc:1505] Processing CreateTablet for tablet 6c32c6f5e1cc40a0a12688201cac62a7 (DEFAULT_TABLE table=default.Table_e6ca0a151fc14c5085c6ea454f460242 [id=2a711e1172654147a58b09ee43cac845]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.322212 1166 ts_tablet_manager.cc:1434] T 494b058c819548dab52917d7690a03ca P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.075s user 0.006s sys 0.007s
I20251029 02:51:51.322499 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c32c6f5e1cc40a0a12688201cac62a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.322613 1166 tablet_bootstrap.cc:492] T 3caf6aa732e6460e96145c0f4105a826 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:51:51.322646 751 tablet_service.cc:1505] Processing CreateTablet for tablet 6c32c6f5e1cc40a0a12688201cac62a7 (DEFAULT_TABLE table=default.Table_e6ca0a151fc14c5085c6ea454f460242 [id=2a711e1172654147a58b09ee43cac845]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.322703 1034 tablet_service.cc:1505] Processing CreateTablet for tablet 6c32c6f5e1cc40a0a12688201cac62a7 (DEFAULT_TABLE table=default.Table_e6ca0a151fc14c5085c6ea454f460242 [id=2a711e1172654147a58b09ee43cac845]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.323032 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c32c6f5e1cc40a0a12688201cac62a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.323093 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c32c6f5e1cc40a0a12688201cac62a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.324434 1166 tablet_bootstrap.cc:654] T 3caf6aa732e6460e96145c0f4105a826 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:51.329355 1175 ts_tablet_manager.cc:1929] T 2dfa48aa89074a53bc13ed5ea199490e P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:51.329514 1175 log.cc:1199] T 2dfa48aa89074a53bc13ed5ea199490e P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/2dfa48aa89074a53bc13ed5ea199490e
I20251029 02:51:51.329874 1175 ts_tablet_manager.cc:1950] T 2dfa48aa89074a53bc13ed5ea199490e P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:51:51.331338 526 catalog_manager.cc:5649] T 0c0aeaabe14740718f75f723023aa087 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } health_report { overall_health: UNKNOWN } } }
I20251029 02:51:51.332317 526 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet 0c0aeaabe14740718f75f723023aa087 (table default.Table_e2703b6254854a0980e73e76f49db366 [id=feb1f56ae44249c6adce65637017bb43]). Expected version 1 got 0
I20251029 02:51:51.334126 520 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 0dd31dc5873d407d958a0f5199f78782
I20251029 02:51:51.335032 1166 tablet_bootstrap.cc:492] T 3caf6aa732e6460e96145c0f4105a826 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:51:51.335011 751 tablet_service.cc:1552] Processing DeleteTablet for tablet 0dd31dc5873d407d958a0f5199f78782 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:51 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:51.335201 1166 ts_tablet_manager.cc:1403] T 3caf6aa732e6460e96145c0f4105a826 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.003s
I20251029 02:51:51.336432 1032 tablet_service.cc:1552] Processing DeleteTablet for tablet 0dd31dc5873d407d958a0f5199f78782 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:51 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:51.336210 1166 raft_consensus.cc:359] T 3caf6aa732e6460e96145c0f4105a826 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:51.336592 1166 raft_consensus.cc:385] T 3caf6aa732e6460e96145c0f4105a826 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:51.336673 1166 raft_consensus.cc:740] T 3caf6aa732e6460e96145c0f4105a826 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:51:51.336714 892 tablet_service.cc:1552] Processing DeleteTablet for tablet 0dd31dc5873d407d958a0f5199f78782 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:51 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:51.336908 1166 consensus_queue.cc:260] T 3caf6aa732e6460e96145c0f4105a826 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:51.337409 1166 ts_tablet_manager.cc:1434] T 3caf6aa732e6460e96145c0f4105a826 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 02:51:51.337519 1174 ts_tablet_manager.cc:1929] T 3caf6aa732e6460e96145c0f4105a826 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:51.337697 1174 log.cc:1199] T 3caf6aa732e6460e96145c0f4105a826 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/3caf6aa732e6460e96145c0f4105a826
I20251029 02:51:51.337770 1166 tablet_bootstrap.cc:492] T 22be6bc81efb434ab7fc3242193789c3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:51:51.339437 1172 tablet_bootstrap.cc:492] T 8b89ee9017b244c5b866b1fc7154ffe3 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:51:51.339594 1166 tablet_bootstrap.cc:654] T 22be6bc81efb434ab7fc3242193789c3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:51.339613 1172 ts_tablet_manager.cc:1403] T 8b89ee9017b244c5b866b1fc7154ffe3 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.034s user 0.005s sys 0.001s
I20251029 02:51:51.339725 1174 ts_tablet_manager.cc:1950] T 3caf6aa732e6460e96145c0f4105a826 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:51:51.340410 1172 raft_consensus.cc:359] T 8b89ee9017b244c5b866b1fc7154ffe3 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:51.340603 1172 raft_consensus.cc:385] T 8b89ee9017b244c5b866b1fc7154ffe3 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:51.340688 1172 raft_consensus.cc:740] T 8b89ee9017b244c5b866b1fc7154ffe3 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:51:51.340906 1172 consensus_queue.cc:260] T 8b89ee9017b244c5b866b1fc7154ffe3 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:51.341310 1176 ts_tablet_manager.cc:1929] T 6a378024f1fb45d086297968ab504156 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:51.341459 1176 log.cc:1199] T 6a378024f1fb45d086297968ab504156 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/6a378024f1fb45d086297968ab504156
I20251029 02:51:51.341802 1176 ts_tablet_manager.cc:1950] T 6a378024f1fb45d086297968ab504156 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:51:51.342221 894 tablet_service.cc:1505] Processing CreateTablet for tablet d7ba1756532c4825a6c7024511dbae51 (DEFAULT_TABLE table=default.Table_4c5fd91d917e4f1a99baef291b2d77e0 [id=5587d5c964034bd7ad0efa338b74df65]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.342664 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7ba1756532c4825a6c7024511dbae51. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.342799 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 6a378024f1fb45d086297968ab504156 (table default.Table_14462d3ac1c24574963a67d66a1cb274 [id=2c98232622ff4370a069b00d8d787f18]) successfully deleted
I20251029 02:51:51.343734 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 2dfa48aa89074a53bc13ed5ea199490e (table default.Table_83ad3367766c49269223bd91b060a972 [id=3bd791c439904f8c9e66b9e4dd107a78]) successfully deleted
I20251029 02:51:51.342330 1172 ts_tablet_manager.cc:1434] T 8b89ee9017b244c5b866b1fc7154ffe3 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
W20251029 02:51:51.344041 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 0dd31dc5873d407d958a0f5199f78782 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0dd31dc5873d407d958a0f5199f78782 already in progress: creating tablet
I20251029 02:51:51.343473 1175 tablet_replica.cc:333] T 3caf6aa732e6460e96145c0f4105a826 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:51:51.344455 1175 raft_consensus.cc:2243] T 3caf6aa732e6460e96145c0f4105a826 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:51.344635 1175 raft_consensus.cc:2272] T 3caf6aa732e6460e96145c0f4105a826 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:51.345183 1175 ts_tablet_manager.cc:1916] T 3caf6aa732e6460e96145c0f4105a826 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:51.346724 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 3caf6aa732e6460e96145c0f4105a826 (table default.Table_4bdbcbb87e5b40aeb9f98eb1aa0e6fe3 [id=78615d8807a74b4496a3223ef6fe709d]) successfully deleted
I20251029 02:51:51.346858 1174 tablet_replica.cc:333] T 494b058c819548dab52917d7690a03ca P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:51:51.346940 1172 tablet_bootstrap.cc:492] T 42f463dc3cc1434b8ee5ba268cb9fc9a P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:51:51.347033 1174 raft_consensus.cc:2243] T 494b058c819548dab52917d7690a03ca P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:51.347227 1174 raft_consensus.cc:2272] T 494b058c819548dab52917d7690a03ca P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:51.348784 1172 tablet_bootstrap.cc:654] T 42f463dc3cc1434b8ee5ba268cb9fc9a P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:51.351001 1174 ts_tablet_manager.cc:1916] T 494b058c819548dab52917d7690a03ca P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:51.353180 1175 ts_tablet_manager.cc:1929] T 3caf6aa732e6460e96145c0f4105a826 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:51.353327 1175 log.cc:1199] T 3caf6aa732e6460e96145c0f4105a826 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/3caf6aa732e6460e96145c0f4105a826
I20251029 02:51:51.353477 1032 tablet_service.cc:1505] Processing CreateTablet for tablet d7ba1756532c4825a6c7024511dbae51 (DEFAULT_TABLE table=default.Table_4c5fd91d917e4f1a99baef291b2d77e0 [id=5587d5c964034bd7ad0efa338b74df65]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.353636 1175 ts_tablet_manager.cc:1950] T 3caf6aa732e6460e96145c0f4105a826 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:51:51.353945 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7ba1756532c4825a6c7024511dbae51. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.354984 522 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 2ebde39f813d46e898ad1754325ad1a0
I20251029 02:51:51.356936 522 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet 8f7c746f8b6948b58676298c2c2ea221 (table default.Table_c1952ce7217c4687bee6f2c2c4381b33 [id=669ae9dd0a5749b1aa3557ab8183a057]). Expected version 2 got 0
I20251029 02:51:51.357107 522 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet e6bcea41d5d4480eaf5d6986a93f5e45 (table default.Table_5b3a7640a9e54e7a9868985322ade0cb [id=1a1aeb237dad4464b717bbae5d4001b0]). Expected version 2 got 0
I20251029 02:51:51.357246 522 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet 55c1b243d8a94938a5caa58754092cec (table default.Table_823c69ddf86f442baf2cfc774320e6e4 [id=bd2335c28dd44b5584ab71694cebda2d]). Expected version 1 got 0
W20251029 02:51:51.358168 522 catalog_manager.cc:5841] Failed to send AlterTable RPC for tablet e6bcea41d5d4480eaf5d6986a93f5e45 (table default.Table_5b3a7640a9e54e7a9868985322ade0cb [id=1a1aeb237dad4464b717bbae5d4001b0], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet e6bcea41d5d4480eaf5d6986a93f5e45 consensus state has no leader
I20251029 02:51:51.358742 525 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 22be6bc81efb434ab7fc3242193789c3
I20251029 02:51:51.359426 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_194067b477f64d7397f6e8b2c30cf06e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:51.359541 751 tablet_service.cc:1552] Processing DeleteTablet for tablet 2ebde39f813d46e898ad1754325ad1a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:51 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:51.359768 752 tablet_service.cc:1552] Processing DeleteTablet for tablet 22be6bc81efb434ab7fc3242193789c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:51 UTC) from {username='slave'} at 127.0.0.1:44580
W20251029 02:51:51.360531 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_194067b477f64d7397f6e8b2c30cf06e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:51.362825 1166 tablet_bootstrap.cc:492] T 22be6bc81efb434ab7fc3242193789c3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:51:51.363440 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_0cb428bfceb5464985b45d16597c11cb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:51.363708 1166 ts_tablet_manager.cc:1403] T 22be6bc81efb434ab7fc3242193789c3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.026s user 0.008s sys 0.001s
I20251029 02:51:51.364274 1037 tablet_service.cc:1552] Processing DeleteTablet for tablet 22be6bc81efb434ab7fc3242193789c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:51 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:51.364284 895 tablet_service.cc:1552] Processing DeleteTablet for tablet 22be6bc81efb434ab7fc3242193789c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:51 UTC) from {username='slave'} at 127.0.0.1:35702
W20251029 02:51:51.364259 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0cb428bfceb5464985b45d16597c11cb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:51.364567 892 tablet_service.cc:1552] Processing DeleteTablet for tablet 2ebde39f813d46e898ad1754325ad1a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:51 UTC) from {username='slave'} at 127.0.0.1:35702
W20251029 02:51:51.364871 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 22be6bc81efb434ab7fc3242193789c3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 22be6bc81efb434ab7fc3242193789c3 already in progress: creating tablet
I20251029 02:51:51.365317 1032 tablet_service.cc:1552] Processing DeleteTablet for tablet 2ebde39f813d46e898ad1754325ad1a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:51 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:51.365260 1166 raft_consensus.cc:359] T 22be6bc81efb434ab7fc3242193789c3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
W20251029 02:51:51.365962 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 2ebde39f813d46e898ad1754325ad1a0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2ebde39f813d46e898ad1754325ad1a0 already in progress: creating tablet
I20251029 02:51:51.366142 1166 raft_consensus.cc:385] T 22be6bc81efb434ab7fc3242193789c3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:51.366299 1166 raft_consensus.cc:740] T 22be6bc81efb434ab7fc3242193789c3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:51:51.366583 1166 consensus_queue.cc:260] T 22be6bc81efb434ab7fc3242193789c3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:51.367580 1166 ts_tablet_manager.cc:1434] T 22be6bc81efb434ab7fc3242193789c3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251029 02:51:51.367952 1166 tablet_bootstrap.cc:492] T 55c1b243d8a94938a5caa58754092cec P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:51:51.369593 1167 tablet_bootstrap.cc:492] T 0c0aeaabe14740718f75f723023aa087 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:51:51.369776 1167 ts_tablet_manager.cc:1403] T 0c0aeaabe14740718f75f723023aa087 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.051s user 0.003s sys 0.001s
I20251029 02:51:51.370095 1166 tablet_bootstrap.cc:654] T 55c1b243d8a94938a5caa58754092cec P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:51.370656 1167 raft_consensus.cc:359] T 0c0aeaabe14740718f75f723023aa087 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:51.370853 1167 raft_consensus.cc:385] T 0c0aeaabe14740718f75f723023aa087 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:51.370934 1167 raft_consensus.cc:740] T 0c0aeaabe14740718f75f723023aa087 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:51:51.371212 1167 consensus_queue.cc:260] T 0c0aeaabe14740718f75f723023aa087 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:51.373222 1167 ts_tablet_manager.cc:1434] T 0c0aeaabe14740718f75f723023aa087 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251029 02:51:51.374269 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 3caf6aa732e6460e96145c0f4105a826 (table default.Table_4bdbcbb87e5b40aeb9f98eb1aa0e6fe3 [id=78615d8807a74b4496a3223ef6fe709d]) successfully deleted
I20251029 02:51:51.374580 1175 tablet_replica.cc:333] T 22be6bc81efb434ab7fc3242193789c3 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
W20251029 02:51:51.374606 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 0dd31dc5873d407d958a0f5199f78782 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0dd31dc5873d407d958a0f5199f78782 already in progress: creating tablet
W20251029 02:51:51.374845 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 2ebde39f813d46e898ad1754325ad1a0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2ebde39f813d46e898ad1754325ad1a0 already in progress: creating tablet
I20251029 02:51:51.374833 1175 raft_consensus.cc:2243] T 22be6bc81efb434ab7fc3242193789c3 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:51.375074 1175 raft_consensus.cc:2272] T 22be6bc81efb434ab7fc3242193789c3 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:51.375867 1175 ts_tablet_manager.cc:1916] T 22be6bc81efb434ab7fc3242193789c3 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:51.376155 1167 tablet_bootstrap.cc:492] T 22be6bc81efb434ab7fc3242193789c3 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:51:51.383802 752 tablet_service.cc:1505] Processing CreateTablet for tablet d7ba1756532c4825a6c7024511dbae51 (DEFAULT_TABLE table=default.Table_4c5fd91d917e4f1a99baef291b2d77e0 [id=5587d5c964034bd7ad0efa338b74df65]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.384235 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7ba1756532c4825a6c7024511dbae51. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.385082 1032 tablet_service.cc:1505] Processing CreateTablet for tablet cb55b3d5bc63405f96cdc122f5a612cb (DEFAULT_TABLE table=default.Table_47a1a59db0444fa49f7274509385ff0c [id=283c09ba60114ba9b165e2ff144a6c35]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.385519 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb55b3d5bc63405f96cdc122f5a612cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.385993 1167 tablet_bootstrap.cc:654] T 22be6bc81efb434ab7fc3242193789c3 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:51.386029 1037 tablet_service.cc:1505] Processing CreateTablet for tablet bf210435010d4dde8903fd116eb468e3 (DEFAULT_TABLE table=default.Table_4bdbcbb87e5b40aeb9f98eb1aa0e6fe3 [id=78615d8807a74b4496a3223ef6fe709d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.386425 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf210435010d4dde8903fd116eb468e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.388564 751 tablet_service.cc:1505] Processing CreateTablet for tablet 6dd4696ad4b44b6dab87a205f3f16ebd (DEFAULT_TABLE table=default.Table_3276fcafb63848adaa3184302586391c [id=90dfeabf0e8945c9bd86472d68884221]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.388928 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6dd4696ad4b44b6dab87a205f3f16ebd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.392513 1033 tablet_service.cc:1505] Processing CreateTablet for tablet 6dd4696ad4b44b6dab87a205f3f16ebd (DEFAULT_TABLE table=default.Table_3276fcafb63848adaa3184302586391c [id=90dfeabf0e8945c9bd86472d68884221]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.392939 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6dd4696ad4b44b6dab87a205f3f16ebd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.393069 754 tablet_service.cc:1505] Processing CreateTablet for tablet cb55b3d5bc63405f96cdc122f5a612cb (DEFAULT_TABLE table=default.Table_47a1a59db0444fa49f7274509385ff0c [id=283c09ba60114ba9b165e2ff144a6c35]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.393976 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb55b3d5bc63405f96cdc122f5a612cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.396143 749 tablet_service.cc:1505] Processing CreateTablet for tablet bf210435010d4dde8903fd116eb468e3 (DEFAULT_TABLE table=default.Table_4bdbcbb87e5b40aeb9f98eb1aa0e6fe3 [id=78615d8807a74b4496a3223ef6fe709d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.396531 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf210435010d4dde8903fd116eb468e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.396770 1174 ts_tablet_manager.cc:1929] T 494b058c819548dab52917d7690a03ca P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:51.396896 1174 log.cc:1199] T 494b058c819548dab52917d7690a03ca P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/494b058c819548dab52917d7690a03ca
I20251029 02:51:51.397266 1174 ts_tablet_manager.cc:1950] T 494b058c819548dab52917d7690a03ca P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:51:51.402065 1167 tablet_bootstrap.cc:492] T 22be6bc81efb434ab7fc3242193789c3 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:51:51.402210 1167 ts_tablet_manager.cc:1403] T 22be6bc81efb434ab7fc3242193789c3 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.002s
I20251029 02:51:51.402499 1166 tablet_bootstrap.cc:492] T 55c1b243d8a94938a5caa58754092cec P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:51:51.402684 1166 ts_tablet_manager.cc:1403] T 55c1b243d8a94938a5caa58754092cec P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.035s user 0.005s sys 0.003s
I20251029 02:51:51.403136 1167 raft_consensus.cc:359] T 22be6bc81efb434ab7fc3242193789c3 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:51.403317 1167 raft_consensus.cc:385] T 22be6bc81efb434ab7fc3242193789c3 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:51.403409 1167 raft_consensus.cc:740] T 22be6bc81efb434ab7fc3242193789c3 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:51:51.403414 1166 raft_consensus.cc:359] T 55c1b243d8a94938a5caa58754092cec P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:51.403651 1166 raft_consensus.cc:385] T 55c1b243d8a94938a5caa58754092cec P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:51.403714 1166 raft_consensus.cc:740] T 55c1b243d8a94938a5caa58754092cec P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:51:51.403683 1167 consensus_queue.cc:260] T 22be6bc81efb434ab7fc3242193789c3 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:51.403932 1166 consensus_queue.cc:260] T 55c1b243d8a94938a5caa58754092cec P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:51.404636 1166 ts_tablet_manager.cc:1434] T 55c1b243d8a94938a5caa58754092cec P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 02:51:51.404940 1166 tablet_bootstrap.cc:492] T ba014ecfec7f4602ae1c09d931bac824 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:51:51.406641 1166 tablet_bootstrap.cc:654] T ba014ecfec7f4602ae1c09d931bac824 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:51.408128 1167 ts_tablet_manager.cc:1434] T 22be6bc81efb434ab7fc3242193789c3 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20251029 02:51:51.408244 1175 ts_tablet_manager.cc:1929] T 22be6bc81efb434ab7fc3242193789c3 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:51.408381 1175 log.cc:1199] T 22be6bc81efb434ab7fc3242193789c3 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/22be6bc81efb434ab7fc3242193789c3
I20251029 02:51:51.408763 1175 ts_tablet_manager.cc:1950] T 22be6bc81efb434ab7fc3242193789c3 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:51:51.409987 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 22be6bc81efb434ab7fc3242193789c3 (table default.Table_61af959119eb4551940deeb613b069d9 [id=cf081cb5322d40c3bdd89cb9e0f58a99]) successfully deleted
I20251029 02:51:51.410849 1167 tablet_bootstrap.cc:492] T e6bcea41d5d4480eaf5d6986a93f5e45 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:51:51.413497 523 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 86f8776b5c4c4082bf276199feaaefea
I20251029 02:51:51.413833 1174 tablet_replica.cc:333] T 6a378024f1fb45d086297968ab504156 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:51:51.414032 1174 raft_consensus.cc:2243] T 6a378024f1fb45d086297968ab504156 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:51.414197 1174 raft_consensus.cc:2272] T 6a378024f1fb45d086297968ab504156 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:51.415531 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_4d5341f07b99483183d6aedc61c9ddcd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:51.416040 1174 ts_tablet_manager.cc:1916] T 6a378024f1fb45d086297968ab504156 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 02:51:51.416096 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4d5341f07b99483183d6aedc61c9ddcd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:51.417562 755 tablet_service.cc:1552] Processing DeleteTablet for tablet 86f8776b5c4c4082bf276199feaaefea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:51 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:51.418249 1033 tablet_service.cc:1552] Processing DeleteTablet for tablet 86f8776b5c4c4082bf276199feaaefea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:51 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:51.418459 1167 tablet_bootstrap.cc:654] T e6bcea41d5d4480eaf5d6986a93f5e45 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:51.418802 520 master_service.cc:954] ReplaceTablet: received request to replace tablet 8b89ee9017b244c5b866b1fc7154ffe3 from {username='slave'} at 127.0.0.1:35314
W20251029 02:51:51.420075 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 86f8776b5c4c4082bf276199feaaefea because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 86f8776b5c4c4082bf276199feaaefea already in progress: creating tablet
W20251029 02:51:51.420630 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 86f8776b5c4c4082bf276199feaaefea because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 86f8776b5c4c4082bf276199feaaefea already in progress: creating tablet
I20251029 02:51:51.421043 518 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 94854145ac0d4be38780d4cdb721725e
I20251029 02:51:51.421487 518 catalog_manager.cc:6719] ReplaceTablet: tablet 94854145ac0d4be38780d4cdb721725e deleted and replaced by tablet 82f75c134c1a4858a2154ee6b1d1d29f
I20251029 02:51:51.421686 518 master_service.cc:954] ReplaceTablet: received request to replace tablet 0c0aeaabe14740718f75f723023aa087 from {username='slave'} at 127.0.0.1:35314
I20251029 02:51:51.422619 1033 tablet_service.cc:1552] Processing DeleteTablet for tablet 94854145ac0d4be38780d4cdb721725e with delete_type TABLET_DATA_DELETED (replaced by tablet 82f75c134c1a4858a2154ee6b1d1d29f) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:51.422909 754 tablet_service.cc:1552] Processing DeleteTablet for tablet 94854145ac0d4be38780d4cdb721725e with delete_type TABLET_DATA_DELETED (replaced by tablet 82f75c134c1a4858a2154ee6b1d1d29f) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:51.423162 1175 tablet_replica.cc:333] T 94854145ac0d4be38780d4cdb721725e P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:51:51.423454 1175 raft_consensus.cc:2243] T 94854145ac0d4be38780d4cdb721725e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:51.423657 1175 raft_consensus.cc:2272] T 94854145ac0d4be38780d4cdb721725e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:51.424701 1175 ts_tablet_manager.cc:1916] T 94854145ac0d4be38780d4cdb721725e P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 02:51:51.425683 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 94854145ac0d4be38780d4cdb721725e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 94854145ac0d4be38780d4cdb721725e already in progress: creating tablet
I20251029 02:51:51.428503 896 tablet_service.cc:1552] Processing DeleteTablet for tablet 94edf3f2e0854b07bbd348ecab82d3ca with delete_type TABLET_DATA_DELETED (replaced by tablet 86f8776b5c4c4082bf276199feaaefea) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:51.428730 900 tablet_service.cc:1552] Processing DeleteTablet for tablet 94854145ac0d4be38780d4cdb721725e with delete_type TABLET_DATA_DELETED (replaced by tablet 82f75c134c1a4858a2154ee6b1d1d29f) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:51.429060 895 tablet_service.cc:1505] Processing CreateTablet for tablet bf210435010d4dde8903fd116eb468e3 (DEFAULT_TABLE table=default.Table_4bdbcbb87e5b40aeb9f98eb1aa0e6fe3 [id=78615d8807a74b4496a3223ef6fe709d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.429643 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf210435010d4dde8903fd116eb468e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.432809 1175 ts_tablet_manager.cc:1929] T 94854145ac0d4be38780d4cdb721725e P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:51.433003 1175 log.cc:1199] T 94854145ac0d4be38780d4cdb721725e P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/94854145ac0d4be38780d4cdb721725e
I20251029 02:51:51.433414 893 tablet_service.cc:1552] Processing DeleteTablet for tablet 86f8776b5c4c4082bf276199feaaefea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:51 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:51.433480 1175 ts_tablet_manager.cc:1950] T 94854145ac0d4be38780d4cdb721725e P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:51:51.433681 892 tablet_service.cc:1505] Processing CreateTablet for tablet cb55b3d5bc63405f96cdc122f5a612cb (DEFAULT_TABLE table=default.Table_47a1a59db0444fa49f7274509385ff0c [id=283c09ba60114ba9b165e2ff144a6c35]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.434100 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb55b3d5bc63405f96cdc122f5a612cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.434751 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 94854145ac0d4be38780d4cdb721725e (table default.Table_631cf79ec867461baecbc486773b88c4 [id=7a493d6672624c90a81a647d7785181c]) successfully deleted
I20251029 02:51:51.435287 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 494b058c819548dab52917d7690a03ca (table default.Table_9c3c143c7d464c32acb4176464f0fb64 [id=ded0c39ca7954a40933ccc61c5d32909]) successfully deleted
W20251029 02:51:51.440459 521 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5e31e061d74146009d24d5cd57487a5e has no consensus state
I20251029 02:51:51.440785 521 master_service.cc:954] ReplaceTablet: received request to replace tablet 3caf6aa732e6460e96145c0f4105a826 from {username='slave'} at 127.0.0.1:35314
I20251029 02:51:51.441327 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_7680a98fa637454984a5d4da44855a48"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:51.441902 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7680a98fa637454984a5d4da44855a48 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:51.442890 893 tablet_service.cc:1505] Processing CreateTablet for tablet b867feae24e741aca6902868c35c1428 (DEFAULT_TABLE table=default.Table_77c77066213c4d05bdf61bc81facdebd [id=be817eb6ec33406eb316d36575c134fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.443313 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b867feae24e741aca6902868c35c1428. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.446869 754 tablet_service.cc:1505] Processing CreateTablet for tablet b867feae24e741aca6902868c35c1428 (DEFAULT_TABLE table=default.Table_77c77066213c4d05bdf61bc81facdebd [id=be817eb6ec33406eb316d36575c134fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.447417 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b867feae24e741aca6902868c35c1428. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.451031 1172 tablet_bootstrap.cc:492] T 42f463dc3cc1434b8ee5ba268cb9fc9a P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:51:51.451303 1172 ts_tablet_manager.cc:1403] T 42f463dc3cc1434b8ee5ba268cb9fc9a P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.104s user 0.011s sys 0.018s
I20251029 02:51:51.452495 1172 raft_consensus.cc:359] T 42f463dc3cc1434b8ee5ba268cb9fc9a P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:51.453826 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_dd81d4b79bbc4d81b0ada1be7a68177a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:51.454253 1172 raft_consensus.cc:385] T 42f463dc3cc1434b8ee5ba268cb9fc9a P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:51.454409 1172 raft_consensus.cc:740] T 42f463dc3cc1434b8ee5ba268cb9fc9a P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
W20251029 02:51:51.454643 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dd81d4b79bbc4d81b0ada1be7a68177a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:51.454751 894 tablet_service.cc:1505] Processing CreateTablet for tablet 6dd4696ad4b44b6dab87a205f3f16ebd (DEFAULT_TABLE table=default.Table_3276fcafb63848adaa3184302586391c [id=90dfeabf0e8945c9bd86472d68884221]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.455181 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6dd4696ad4b44b6dab87a205f3f16ebd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.455502 1172 consensus_queue.cc:260] T 42f463dc3cc1434b8ee5ba268cb9fc9a P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:51.460417 1034 tablet_service.cc:1505] Processing CreateTablet for tablet b867feae24e741aca6902868c35c1428 (DEFAULT_TABLE table=default.Table_77c77066213c4d05bdf61bc81facdebd [id=be817eb6ec33406eb316d36575c134fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.460981 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b867feae24e741aca6902868c35c1428. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.461905 1172 ts_tablet_manager.cc:1434] T 42f463dc3cc1434b8ee5ba268cb9fc9a P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20251029 02:51:51.468124 1172 tablet_bootstrap.cc:492] T 86f8776b5c4c4082bf276199feaaefea P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:51:51.468914 517 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet a0cee0a6eb5a435b9bb3295cbe242b70
I20251029 02:51:51.469446 517 catalog_manager.cc:6719] ReplaceTablet: tablet a0cee0a6eb5a435b9bb3295cbe242b70 deleted and replaced by tablet 6c22497760274380a832feff7c5a2fae
I20251029 02:51:51.469874 1033 tablet_service.cc:1552] Processing DeleteTablet for tablet a0cee0a6eb5a435b9bb3295cbe242b70 with delete_type TABLET_DATA_DELETED (replaced by tablet 6c22497760274380a832feff7c5a2fae) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:51.470541 517 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet 42f463dc3cc1434b8ee5ba268cb9fc9a (table default.Table_03f7d6f02b064939ba7464c5173e02e3 [id=9898cffc2b964059ba3163ee4d1395c2]). Expected version 1 got 0
W20251029 02:51:51.470640 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet a0cee0a6eb5a435b9bb3295cbe242b70 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a0cee0a6eb5a435b9bb3295cbe242b70 already in progress: creating tablet
I20251029 02:51:51.471385 517 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_c1952ce7217c4687bee6f2c2c4381b33" } new_table_name: "default.Table_9aac534a15054a30b80745984c279e62" modify_external_catalogs: true
I20251029 02:51:51.471832 1172 tablet_bootstrap.cc:654] T 86f8776b5c4c4082bf276199feaaefea P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:51.473141 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_7b36c69036a641f289a5f927f3101f16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:51.473703 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7b36c69036a641f289a5f927f3101f16 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:51.475329 752 tablet_service.cc:1552] Processing DeleteTablet for tablet a0cee0a6eb5a435b9bb3295cbe242b70 with delete_type TABLET_DATA_DELETED (replaced by tablet 6c22497760274380a832feff7c5a2fae) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:51.475672 1175 tablet_replica.cc:333] T a0cee0a6eb5a435b9bb3295cbe242b70 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:51:51.475852 1175 raft_consensus.cc:2243] T a0cee0a6eb5a435b9bb3295cbe242b70 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:51.476038 1175 raft_consensus.cc:2272] T a0cee0a6eb5a435b9bb3295cbe242b70 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:51.476651 1175 ts_tablet_manager.cc:1916] T a0cee0a6eb5a435b9bb3295cbe242b70 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:51.479568 892 tablet_service.cc:1552] Processing DeleteTablet for tablet a0cee0a6eb5a435b9bb3295cbe242b70 with delete_type TABLET_DATA_DELETED (replaced by tablet 6c22497760274380a832feff7c5a2fae) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:51.484195 1172 tablet_bootstrap.cc:492] T 86f8776b5c4c4082bf276199feaaefea P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:51:51.484364 1172 ts_tablet_manager.cc:1403] T 86f8776b5c4c4082bf276199feaaefea P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.002s
I20251029 02:51:51.484799 1174 ts_tablet_manager.cc:1929] T 6a378024f1fb45d086297968ab504156 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:51.484964 1174 log.cc:1199] T 6a378024f1fb45d086297968ab504156 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/6a378024f1fb45d086297968ab504156
I20251029 02:51:51.485210 1172 raft_consensus.cc:359] T 86f8776b5c4c4082bf276199feaaefea P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:51.485337 1174 ts_tablet_manager.cc:1950] T 6a378024f1fb45d086297968ab504156 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:51:51.485352 1175 ts_tablet_manager.cc:1929] T a0cee0a6eb5a435b9bb3295cbe242b70 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:51.485419 1172 raft_consensus.cc:385] T 86f8776b5c4c4082bf276199feaaefea P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:51.485466 1175 log.cc:1199] T a0cee0a6eb5a435b9bb3295cbe242b70 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/a0cee0a6eb5a435b9bb3295cbe242b70
I20251029 02:51:51.485530 1172 raft_consensus.cc:740] T 86f8776b5c4c4082bf276199feaaefea P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:51:51.485802 1172 consensus_queue.cc:260] T 86f8776b5c4c4082bf276199feaaefea P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:51.486449 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 6a378024f1fb45d086297968ab504156 (table default.Table_14462d3ac1c24574963a67d66a1cb274 [id=2c98232622ff4370a069b00d8d787f18]) successfully deleted
I20251029 02:51:51.487983 525 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet 42f463dc3cc1434b8ee5ba268cb9fc9a (table default.Table_03f7d6f02b064939ba7464c5173e02e3 [id=9898cffc2b964059ba3163ee4d1395c2]). Expected version 1 got 0
I20251029 02:51:51.489378 1172 ts_tablet_manager.cc:1434] T 86f8776b5c4c4082bf276199feaaefea P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251029 02:51:51.489576 1167 tablet_bootstrap.cc:492] T e6bcea41d5d4480eaf5d6986a93f5e45 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:51:51.489899 525 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet 55c1b243d8a94938a5caa58754092cec (table default.Table_823c69ddf86f442baf2cfc774320e6e4 [id=bd2335c28dd44b5584ab71694cebda2d]). Expected version 1 got 0
I20251029 02:51:51.489980 1167 ts_tablet_manager.cc:1403] T e6bcea41d5d4480eaf5d6986a93f5e45 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.079s user 0.005s sys 0.000s
I20251029 02:51:51.490005 1175 ts_tablet_manager.cc:1950] T a0cee0a6eb5a435b9bb3295cbe242b70 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:51:51.490844 1167 raft_consensus.cc:359] T e6bcea41d5d4480eaf5d6986a93f5e45 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:51.491045 1167 raft_consensus.cc:385] T e6bcea41d5d4480eaf5d6986a93f5e45 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:51.491122 1167 raft_consensus.cc:740] T e6bcea41d5d4480eaf5d6986a93f5e45 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:51:51.491379 1167 consensus_queue.cc:260] T e6bcea41d5d4480eaf5d6986a93f5e45 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:51.492975 1172 tablet_bootstrap.cc:492] T bc490b1327e44059a57e6cd7b63350e3 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:51:51.494611 1167 ts_tablet_manager.cc:1434] T e6bcea41d5d4480eaf5d6986a93f5e45 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251029 02:51:51.495265 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_4d2c99159e924a06858977465efa38a6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:51.495885 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4d2c99159e924a06858977465efa38a6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:51.496814 1172 tablet_bootstrap.cc:654] T bc490b1327e44059a57e6cd7b63350e3 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
W20251029 02:51:51.497013 513 catalog_manager.cc:5101] could not convert tablet's 0c0aeaabe14740718f75f723023aa087 schema from PB: OK
I20251029 02:51:51.497367 523 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_3276fcafb63848adaa3184302586391c" } new_table_name: "default.Table_69af0d3b63c44157abd3263a9deb02da" modify_external_catalogs: true
I20251029 02:51:51.498755 1167 tablet_bootstrap.cc:492] T a0cee0a6eb5a435b9bb3295cbe242b70 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:51:51.500897 1167 tablet_bootstrap.cc:654] T a0cee0a6eb5a435b9bb3295cbe242b70 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:51.501037 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet a0cee0a6eb5a435b9bb3295cbe242b70 (table default.Table_9becb486767e4662a93114d2242f031d [id=c2475b97464f45da8990e004660c42b3]) successfully deleted
I20251029 02:51:51.501350 754 tablet_service.cc:1552] Processing DeleteTablet for tablet 86f8776b5c4c4082bf276199feaaefea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:51 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:51.501677 1175 tablet_replica.cc:333] T 86f8776b5c4c4082bf276199feaaefea P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:51:51.501935 1175 raft_consensus.cc:2243] T 86f8776b5c4c4082bf276199feaaefea P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:51.502161 1175 raft_consensus.cc:2272] T 86f8776b5c4c4082bf276199feaaefea P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:51.502732 1037 tablet_service.cc:1552] Processing DeleteTablet for tablet 494b058c819548dab52917d7690a03ca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:50 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:51.503039 1176 tablet_replica.cc:333] T 494b058c819548dab52917d7690a03ca P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:51:51.503147 1175 ts_tablet_manager.cc:1916] T 86f8776b5c4c4082bf276199feaaefea P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:51.503247 1176 raft_consensus.cc:2243] T 494b058c819548dab52917d7690a03ca P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:51.503437 1176 raft_consensus.cc:2272] T 494b058c819548dab52917d7690a03ca P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:51.503844 1037 tablet_service.cc:1552] Processing DeleteTablet for tablet 22be6bc81efb434ab7fc3242193789c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:51 UTC) from {username='slave'} at 127.0.0.1:37626
W20251029 02:51:51.504837 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 0dd31dc5873d407d958a0f5199f78782 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0dd31dc5873d407d958a0f5199f78782 already in progress: creating tablet
I20251029 02:51:51.504997 1176 ts_tablet_manager.cc:1916] T 494b058c819548dab52917d7690a03ca P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:51.505349 1174 tablet_replica.cc:333] T 22be6bc81efb434ab7fc3242193789c3 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:51:51.505580 1174 raft_consensus.cc:2243] T 22be6bc81efb434ab7fc3242193789c3 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:51.506675 1032 tablet_service.cc:1552] Processing DeleteTablet for tablet 3caf6aa732e6460e96145c0f4105a826 with delete_type TABLET_DATA_DELETED (replaced by tablet bf210435010d4dde8903fd116eb468e3) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:51.506958 1174 raft_consensus.cc:2272] T 22be6bc81efb434ab7fc3242193789c3 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:51.507720 1174 ts_tablet_manager.cc:1916] T 22be6bc81efb434ab7fc3242193789c3 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:51.512355 1175 ts_tablet_manager.cc:1929] T 86f8776b5c4c4082bf276199feaaefea P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:51.512578 1175 log.cc:1199] T 86f8776b5c4c4082bf276199feaaefea P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/86f8776b5c4c4082bf276199feaaefea
I20251029 02:51:51.513029 1175 ts_tablet_manager.cc:1950] T 86f8776b5c4c4082bf276199feaaefea P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:51:51.514865 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 86f8776b5c4c4082bf276199feaaefea (table default.Table_bd27b9a173a243b0aa50adcf36f2a06d [id=13b28fb78eb54543b973bfda2fb1aad8]) successfully deleted
I20251029 02:51:51.518800 525 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_a85023af366f423b8a637b058d6c65b4" } new_table_name: "default.Table_1bd19c9d52204fbf87ee3cdbc62984af" modify_external_catalogs: true
I20251029 02:51:51.518455 1176 ts_tablet_manager.cc:1929] T 494b058c819548dab52917d7690a03ca P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:51.522969 1176 log.cc:1199] T 494b058c819548dab52917d7690a03ca P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/494b058c819548dab52917d7690a03ca
I20251029 02:51:51.523837 1176 ts_tablet_manager.cc:1950] T 494b058c819548dab52917d7690a03ca P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:51:51.525276 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 494b058c819548dab52917d7690a03ca (table default.Table_9c3c143c7d464c32acb4176464f0fb64 [id=ded0c39ca7954a40933ccc61c5d32909]) successfully deleted
I20251029 02:51:51.527997 1174 ts_tablet_manager.cc:1929] T 22be6bc81efb434ab7fc3242193789c3 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:51.529127 1166 tablet_bootstrap.cc:492] T ba014ecfec7f4602ae1c09d931bac824 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:51:51.529321 1166 ts_tablet_manager.cc:1403] T ba014ecfec7f4602ae1c09d931bac824 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.124s user 0.016s sys 0.056s
I20251029 02:51:51.530366 1166 raft_consensus.cc:359] T ba014ecfec7f4602ae1c09d931bac824 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:51.530573 1166 raft_consensus.cc:385] T ba014ecfec7f4602ae1c09d931bac824 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:51.530668 1166 raft_consensus.cc:740] T ba014ecfec7f4602ae1c09d931bac824 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:51:51.530896 1174 log.cc:1199] T 22be6bc81efb434ab7fc3242193789c3 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/22be6bc81efb434ab7fc3242193789c3
I20251029 02:51:51.530921 1166 consensus_queue.cc:260] T ba014ecfec7f4602ae1c09d931bac824 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:51.532845 1174 ts_tablet_manager.cc:1950] T 22be6bc81efb434ab7fc3242193789c3 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:51:51.536841 1176 tablet_replica.cc:333] T 22be6bc81efb434ab7fc3242193789c3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:51:51.537125 1176 raft_consensus.cc:2243] T 22be6bc81efb434ab7fc3242193789c3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:51.539136 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 22be6bc81efb434ab7fc3242193789c3 (table default.Table_61af959119eb4551940deeb613b069d9 [id=cf081cb5322d40c3bdd89cb9e0f58a99]) successfully deleted
I20251029 02:51:51.539608 520 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 8b89ee9017b244c5b866b1fc7154ffe3
I20251029 02:51:51.540180 520 catalog_manager.cc:6719] ReplaceTablet: tablet 8b89ee9017b244c5b866b1fc7154ffe3 deleted and replaced by tablet dd90fb9eb167478cbc3ec9363af86295
I20251029 02:51:51.541196 520 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet ba014ecfec7f4602ae1c09d931bac824 (table default.Table_64e512f8ce034224aea9b887ed0d09dc [id=7bb8b246fbf1431bb71bbb7c5520e301]). Expected version 2 got 0
I20251029 02:51:51.541384 892 tablet_service.cc:1552] Processing DeleteTablet for tablet 8b89ee9017b244c5b866b1fc7154ffe3 with delete_type TABLET_DATA_DELETED (replaced by tablet dd90fb9eb167478cbc3ec9363af86295) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:51.542017 520 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_e6ca0a151fc14c5085c6ea454f460242" } modify_external_catalogs: true
I20251029 02:51:51.542220 520 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_e6ca0a151fc14c5085c6ea454f460242" } modify_external_catalogs: true
W20251029 02:51:51.542966 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 2ebde39f813d46e898ad1754325ad1a0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2ebde39f813d46e898ad1754325ad1a0 already in progress: creating tablet
I20251029 02:51:51.543136 754 tablet_service.cc:1552] Processing DeleteTablet for tablet 8b89ee9017b244c5b866b1fc7154ffe3 with delete_type TABLET_DATA_DELETED (replaced by tablet dd90fb9eb167478cbc3ec9363af86295) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:51.543498 1175 tablet_replica.cc:333] T 8b89ee9017b244c5b866b1fc7154ffe3 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:51:51.543624 1174 tablet_replica.cc:333] T 94edf3f2e0854b07bbd348ecab82d3ca P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:51:51.543752 1175 raft_consensus.cc:2243] T 8b89ee9017b244c5b866b1fc7154ffe3 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:51.543839 1174 raft_consensus.cc:2243] T 94edf3f2e0854b07bbd348ecab82d3ca P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:51.543987 1175 raft_consensus.cc:2272] T 8b89ee9017b244c5b866b1fc7154ffe3 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:51.544027 1174 raft_consensus.cc:2272] T 94edf3f2e0854b07bbd348ecab82d3ca P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:51.545527 1174 ts_tablet_manager.cc:1916] T 94edf3f2e0854b07bbd348ecab82d3ca P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:51.537333 1176 raft_consensus.cc:2272] T 22be6bc81efb434ab7fc3242193789c3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:51.547911 1175 ts_tablet_manager.cc:1916] T 8b89ee9017b244c5b866b1fc7154ffe3 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:51.548600 1176 ts_tablet_manager.cc:1916] T 22be6bc81efb434ab7fc3242193789c3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:51.549615 1166 ts_tablet_manager.cc:1434] T ba014ecfec7f4602ae1c09d931bac824 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.020s user 0.003s sys 0.000s
I20251029 02:51:51.550630 1166 tablet_bootstrap.cc:492] T a0cee0a6eb5a435b9bb3295cbe242b70 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:51:51.541314 1032 tablet_service.cc:1552] Processing DeleteTablet for tablet 8b89ee9017b244c5b866b1fc7154ffe3 with delete_type TABLET_DATA_DELETED (replaced by tablet dd90fb9eb167478cbc3ec9363af86295) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:51.552896 919 raft_consensus.cc:3060] T 0c0aeaabe14740718f75f723023aa087 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:51:51.553076 776 raft_consensus.cc:1275] T 0c0aeaabe14740718f75f723023aa087 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Refusing update from remote peer 0cfd4bee16f84b7c8b0ff3ce14db20d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251029 02:51:51.555438 1167 tablet_bootstrap.cc:492] T a0cee0a6eb5a435b9bb3295cbe242b70 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:51:51.555634 1167 ts_tablet_manager.cc:1403] T a0cee0a6eb5a435b9bb3295cbe242b70 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.057s user 0.011s sys 0.002s
I20251029 02:51:51.556109 1170 consensus_queue.cc:1048] T 0c0aeaabe14740718f75f723023aa087 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:51:51.556486 1167 raft_consensus.cc:359] T a0cee0a6eb5a435b9bb3295cbe242b70 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:51.556692 1167 raft_consensus.cc:385] T a0cee0a6eb5a435b9bb3295cbe242b70 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:51.556789 1167 raft_consensus.cc:740] T a0cee0a6eb5a435b9bb3295cbe242b70 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:51:51.557039 1167 consensus_queue.cc:260] T a0cee0a6eb5a435b9bb3295cbe242b70 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:51.559134 919 raft_consensus.cc:1275] T 0c0aeaabe14740718f75f723023aa087 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Refusing update from remote peer 0cfd4bee16f84b7c8b0ff3ce14db20d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251029 02:51:51.559517 1167 ts_tablet_manager.cc:1434] T a0cee0a6eb5a435b9bb3295cbe242b70 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251029 02:51:51.560061 1167 tablet_bootstrap.cc:492] T 94854145ac0d4be38780d4cdb721725e P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:51:51.559981 1170 consensus_queue.cc:1048] T 0c0aeaabe14740718f75f723023aa087 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:51:51.560550 1179 raft_consensus.cc:493] T ba014ecfec7f4602ae1c09d931bac824 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:51.560729 1179 raft_consensus.cc:515] T ba014ecfec7f4602ae1c09d931bac824 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:51.561378 1179 leader_election.cc:290] T ba014ecfec7f4602ae1c09d931bac824 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:51:51.561817 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba014ecfec7f4602ae1c09d931bac824" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:51:51.562101 1062 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba014ecfec7f4602ae1c09d931bac824" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:51:51.562327 1062 raft_consensus.cc:2468] T ba014ecfec7f4602ae1c09d931bac824 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b495a49349324f7d81908f2da700f0d9 in term 0.
W20251029 02:51:51.562625 711 leader_election.cc:343] T ba014ecfec7f4602ae1c09d931bac824 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:51.562876 712 leader_election.cc:304] T ba014ecfec7f4602ae1c09d931bac824 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters: c756528642a543eeacd93cef3e335a7d
I20251029 02:51:51.563326 1179 raft_consensus.cc:2804] T ba014ecfec7f4602ae1c09d931bac824 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:51:51.563428 1179 raft_consensus.cc:493] T ba014ecfec7f4602ae1c09d931bac824 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:51:51.563530 1179 raft_consensus.cc:3060] T ba014ecfec7f4602ae1c09d931bac824 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:51:51.566654 1167 tablet_bootstrap.cc:654] T 94854145ac0d4be38780d4cdb721725e P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:51.567709 521 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet ba014ecfec7f4602ae1c09d931bac824 (table default.Table_64e512f8ce034224aea9b887ed0d09dc [id=7bb8b246fbf1431bb71bbb7c5520e301]). Expected version 2 got 0
I20251029 02:51:51.568591 521 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_e2703b6254854a0980e73e76f49db366" } new_table_name: "default.Table_d84474a10a954e70afe50f3eb8e8c1ed" modify_external_catalogs: true
I20251029 02:51:51.569998 1176 ts_tablet_manager.cc:1929] T 22be6bc81efb434ab7fc3242193789c3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:51.570150 1176 log.cc:1199] T 22be6bc81efb434ab7fc3242193789c3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/22be6bc81efb434ab7fc3242193789c3
I20251029 02:51:51.571166 1179 raft_consensus.cc:515] T ba014ecfec7f4602ae1c09d931bac824 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:51.572124 1166 tablet_bootstrap.cc:654] T a0cee0a6eb5a435b9bb3295cbe242b70 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:51.574365 1268 mvcc.cc:204] Tried to move back new op lower bound from 7215949052095918080 to 7215949050893172736. Current Snapshot: MvccSnapshot[applied={T|T < 7215949052095918080}]
I20251029 02:51:51.576516 1176 ts_tablet_manager.cc:1950] T 22be6bc81efb434ab7fc3242193789c3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:51:51.577594 1176 tablet_replica.cc:333] T 3caf6aa732e6460e96145c0f4105a826 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:51:51.577807 1176 raft_consensus.cc:2243] T 3caf6aa732e6460e96145c0f4105a826 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:51.578001 1176 raft_consensus.cc:2272] T 3caf6aa732e6460e96145c0f4105a826 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:51.578089 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 22be6bc81efb434ab7fc3242193789c3 (table default.Table_61af959119eb4551940deeb613b069d9 [id=cf081cb5322d40c3bdd89cb9e0f58a99]) successfully deleted
I20251029 02:51:51.578920 918 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba014ecfec7f4602ae1c09d931bac824" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
I20251029 02:51:51.579384 1062 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba014ecfec7f4602ae1c09d931bac824" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:51:51.579666 1062 raft_consensus.cc:3060] T ba014ecfec7f4602ae1c09d931bac824 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
W20251029 02:51:51.583894 517 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8f7c746f8b6948b58676298c2c2ea221 consensus state has no leader
I20251029 02:51:51.584307 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_83d3fe1030e0461895fc2437732b4ff8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:51.584583 1179 leader_election.cc:290] T ba014ecfec7f4602ae1c09d931bac824 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 election: Requested vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:51:51.576063 1172 tablet_bootstrap.cc:492] T bc490b1327e44059a57e6cd7b63350e3 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
W20251029 02:51:51.584955 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_83d3fe1030e0461895fc2437732b4ff8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:51.584985 1172 ts_tablet_manager.cc:1403] T bc490b1327e44059a57e6cd7b63350e3 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.092s user 0.013s sys 0.021s
I20251029 02:51:51.585183 1062 raft_consensus.cc:2468] T ba014ecfec7f4602ae1c09d931bac824 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b495a49349324f7d81908f2da700f0d9 in term 1.
I20251029 02:51:51.585670 712 leader_election.cc:304] T ba014ecfec7f4602ae1c09d931bac824 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters:
I20251029 02:51:51.585992 1179 raft_consensus.cc:2804] T ba014ecfec7f4602ae1c09d931bac824 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:51:51.586145 1179 raft_consensus.cc:697] T ba014ecfec7f4602ae1c09d931bac824 P b495a49349324f7d81908f2da700f0d9 [term 1 LEADER]: Becoming Leader. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Running, Role: LEADER
I20251029 02:51:51.586413 1179 consensus_queue.cc:237] T ba014ecfec7f4602ae1c09d931bac824 P b495a49349324f7d81908f2da700f0d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:51.586686 526 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_e5a6d34625a6441fa8a50e020642ff79" } modify_external_catalogs: true
I20251029 02:51:51.586860 526 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_e5a6d34625a6441fa8a50e020642ff79" } modify_external_catalogs: true
I20251029 02:51:51.587924 1172 raft_consensus.cc:359] T bc490b1327e44059a57e6cd7b63350e3 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:51.588112 1172 raft_consensus.cc:385] T bc490b1327e44059a57e6cd7b63350e3 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:51.588125 518 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 0c0aeaabe14740718f75f723023aa087
I20251029 02:51:51.588188 1172 raft_consensus.cc:740] T bc490b1327e44059a57e6cd7b63350e3 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:51:51.588718 518 catalog_manager.cc:6719] ReplaceTablet: tablet 0c0aeaabe14740718f75f723023aa087 deleted and replaced by tablet 6f93812578a940a3b7378015e2a46f95
I20251029 02:51:51.589100 518 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_194067b477f64d7397f6e8b2c30cf06e" } modify_external_catalogs: true
I20251029 02:51:51.589221 1032 tablet_service.cc:1552] Processing DeleteTablet for tablet 0c0aeaabe14740718f75f723023aa087 with delete_type TABLET_DATA_DELETED (replaced by tablet 6f93812578a940a3b7378015e2a46f95) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:51.589282 518 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_194067b477f64d7397f6e8b2c30cf06e" } modify_external_catalogs: true
I20251029 02:51:51.590145 1172 consensus_queue.cc:260] T bc490b1327e44059a57e6cd7b63350e3 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:51.593848 1175 ts_tablet_manager.cc:1929] T 8b89ee9017b244c5b866b1fc7154ffe3 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:51.593884 754 tablet_service.cc:1552] Processing DeleteTablet for tablet 0c0aeaabe14740718f75f723023aa087 with delete_type TABLET_DATA_DELETED (replaced by tablet 6f93812578a940a3b7378015e2a46f95) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:51.593990 1175 log.cc:1199] T 8b89ee9017b244c5b866b1fc7154ffe3 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/8b89ee9017b244c5b866b1fc7154ffe3
I20251029 02:51:51.594378 1175 ts_tablet_manager.cc:1950] T 8b89ee9017b244c5b866b1fc7154ffe3 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
W20251029 02:51:51.595993 711 leader_election.cc:343] T ba014ecfec7f4602ae1c09d931bac824 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:51.597435 892 tablet_service.cc:1552] Processing DeleteTablet for tablet 0c0aeaabe14740718f75f723023aa087 with delete_type TABLET_DATA_DELETED (replaced by tablet 6f93812578a940a3b7378015e2a46f95) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:51.598754 1172 ts_tablet_manager.cc:1434] T bc490b1327e44059a57e6cd7b63350e3 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.014s user 0.001s sys 0.001s
I20251029 02:51:51.598765 1176 ts_tablet_manager.cc:1916] T 3caf6aa732e6460e96145c0f4105a826 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:51.599135 1172 tablet_bootstrap.cc:492] T 2ebde39f813d46e898ad1754325ad1a0 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:51:51.600514 1174 ts_tablet_manager.cc:1929] T 94edf3f2e0854b07bbd348ecab82d3ca P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:51.600696 1174 log.cc:1199] T 94edf3f2e0854b07bbd348ecab82d3ca P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/94edf3f2e0854b07bbd348ecab82d3ca
I20251029 02:51:51.601119 1174 ts_tablet_manager.cc:1950] T 94edf3f2e0854b07bbd348ecab82d3ca P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:51:51.601843 1172 tablet_bootstrap.cc:654] T 2ebde39f813d46e898ad1754325ad1a0 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:51.601847 522 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_7e818850675e4c5c973db094e7223348" } modify_external_catalogs: true
I20251029 02:51:51.602056 522 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_7e818850675e4c5c973db094e7223348" } modify_external_catalogs: true
W20251029 02:51:51.603879 523 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6dd4696ad4b44b6dab87a205f3f16ebd consensus state has no leader
W20251029 02:51:51.604382 525 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet deca0c4a114441079fce22e74b403050 consensus state has no leader
I20251029 02:51:51.604769 520 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 6c32c6f5e1cc40a0a12688201cac62a7
I20251029 02:51:51.605695 1032 tablet_service.cc:1552] Processing DeleteTablet for tablet 6c32c6f5e1cc40a0a12688201cac62a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:51 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:51.605753 1175 tablet_replica.cc:333] T 0c0aeaabe14740718f75f723023aa087 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:51:51.606839 892 tablet_service.cc:1552] Processing DeleteTablet for tablet 6c32c6f5e1cc40a0a12688201cac62a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:51 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:51.607685 754 tablet_service.cc:1552] Processing DeleteTablet for tablet 6c32c6f5e1cc40a0a12688201cac62a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:51 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:51.608064 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 8b89ee9017b244c5b866b1fc7154ffe3 (table default.Table_25688635c7a948f1b010aa30b88a3b6c [id=052abf3dcc904bdcb869d5b885723dab]) successfully deleted
I20251029 02:51:51.608484 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 94edf3f2e0854b07bbd348ecab82d3ca (table default.Table_bd27b9a173a243b0aa50adcf36f2a06d [id=13b28fb78eb54543b973bfda2fb1aad8]) successfully deleted
W20251029 02:51:51.611018 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 94854145ac0d4be38780d4cdb721725e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 94854145ac0d4be38780d4cdb721725e already in progress: creating tablet
I20251029 02:51:51.611004 1274 tablet.cc:1722] T 0c0aeaabe14740718f75f723023aa087 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 02:51:51.611816 1280 tablet.cc:1722] T 0c0aeaabe14740718f75f723023aa087 P b495a49349324f7d81908f2da700f0d9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 02:51:51.613253 1175 raft_consensus.cc:2243] T 0c0aeaabe14740718f75f723023aa087 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:51.614550 1175 raft_consensus.cc:2272] T 0c0aeaabe14740718f75f723023aa087 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251029 02:51:51.615651 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 86f8776b5c4c4082bf276199feaaefea because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 86f8776b5c4c4082bf276199feaaefea already in progress: creating tablet
I20251029 02:51:51.616201 1174 tablet_replica.cc:333] T a0cee0a6eb5a435b9bb3295cbe242b70 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:51:51.616534 1174 raft_consensus.cc:2243] T a0cee0a6eb5a435b9bb3295cbe242b70 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:51.616848 1174 raft_consensus.cc:2272] T a0cee0a6eb5a435b9bb3295cbe242b70 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:51.619096 525 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_631cf79ec867461baecbc486773b88c4" } modify_external_catalogs: true
I20251029 02:51:51.619208 1174 ts_tablet_manager.cc:1916] T a0cee0a6eb5a435b9bb3295cbe242b70 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:51.619333 525 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_631cf79ec867461baecbc486773b88c4" } modify_external_catalogs: true
I20251029 02:51:51.626953 1167 tablet_bootstrap.cc:492] T 94854145ac0d4be38780d4cdb721725e P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:51:51.627187 1167 ts_tablet_manager.cc:1403] T 94854145ac0d4be38780d4cdb721725e P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.067s user 0.001s sys 0.009s
I20251029 02:51:51.628034 1167 raft_consensus.cc:359] T 94854145ac0d4be38780d4cdb721725e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:51.628293 1167 raft_consensus.cc:385] T 94854145ac0d4be38780d4cdb721725e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:51.628369 1167 raft_consensus.cc:740] T 94854145ac0d4be38780d4cdb721725e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:51:51.629163 1276 raft_consensus.cc:493] T 8f7c746f8b6948b58676298c2c2ea221 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:51.629091 1167 consensus_queue.cc:260] T 94854145ac0d4be38780d4cdb721725e P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:51.629345 1276 raft_consensus.cc:515] T 8f7c746f8b6948b58676298c2c2ea221 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:51.630071 526 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 5d0b2599053142e5b86a0643fb251252
I20251029 02:51:51.630407 518 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 30c18dac17124a5a948fe786f9b7918f
I20251029 02:51:51.631361 919 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8f7c746f8b6948b58676298c2c2ea221" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:51:51.632072 711 leader_election.cc:343] T 8f7c746f8b6948b58676298c2c2ea221 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:51.633036 1167 ts_tablet_manager.cc:1434] T 94854145ac0d4be38780d4cdb721725e P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251029 02:51:51.634418 1167 tablet_bootstrap.cc:492] T 55c1b243d8a94938a5caa58754092cec P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:51:51.634565 518 master_service.cc:954] ReplaceTablet: received request to replace tablet 42f463dc3cc1434b8ee5ba268cb9fc9a from {username='slave'} at 127.0.0.1:35314
I20251029 02:51:51.636382 1167 tablet_bootstrap.cc:654] T 55c1b243d8a94938a5caa58754092cec P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:51.637838 522 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet f04473c3a5f74bdba196b91c2538537e
I20251029 02:51:51.638863 525 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 82f75c134c1a4858a2154ee6b1d1d29f
I20251029 02:51:51.638821 1032 tablet_service.cc:1552] Processing DeleteTablet for tablet f04473c3a5f74bdba196b91c2538537e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:51 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:51.639010 892 tablet_service.cc:1552] Processing DeleteTablet for tablet f04473c3a5f74bdba196b91c2538537e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:51 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:51.639022 754 tablet_service.cc:1552] Processing DeleteTablet for tablet f04473c3a5f74bdba196b91c2538537e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:51 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:51.641201 1174 ts_tablet_manager.cc:1929] T a0cee0a6eb5a435b9bb3295cbe242b70 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:51.641325 1174 log.cc:1199] T a0cee0a6eb5a435b9bb3295cbe242b70 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/a0cee0a6eb5a435b9bb3295cbe242b70
I20251029 02:51:51.641738 1174 ts_tablet_manager.cc:1950] T a0cee0a6eb5a435b9bb3295cbe242b70 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:51:51.642783 1250 raft_consensus.cc:493] T 55c1b243d8a94938a5caa58754092cec P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:51.642994 1250 raft_consensus.cc:515] T 55c1b243d8a94938a5caa58754092cec P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:51.644049 1250 leader_election.cc:290] T 55c1b243d8a94938a5caa58754092cec P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:51:51.644706 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "55c1b243d8a94938a5caa58754092cec" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:51:51.644959 776 raft_consensus.cc:2468] T 55c1b243d8a94938a5caa58754092cec P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 0.
I20251029 02:51:51.645355 919 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "55c1b243d8a94938a5caa58754092cec" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:51:51.645588 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet a0cee0a6eb5a435b9bb3295cbe242b70 (table default.Table_9becb486767e4662a93114d2242f031d [id=c2475b97464f45da8990e004660c42b3]) successfully deleted
W20251029 02:51:51.645869 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 8b89ee9017b244c5b866b1fc7154ffe3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8b89ee9017b244c5b866b1fc7154ffe3 already in progress: creating tablet
I20251029 02:51:51.646034 989 leader_election.cc:304] T 55c1b243d8a94938a5caa58754092cec P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters:
W20251029 02:51:51.646378 990 leader_election.cc:343] T 55c1b243d8a94938a5caa58754092cec P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:51.646450 1250 raft_consensus.cc:2804] T 55c1b243d8a94938a5caa58754092cec P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:51:51.646620 1250 raft_consensus.cc:493] T 55c1b243d8a94938a5caa58754092cec P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:51:51.646725 1250 raft_consensus.cc:3060] T 55c1b243d8a94938a5caa58754092cec P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:51:51.647115 1174 tablet_replica.cc:333] T 0c0aeaabe14740718f75f723023aa087 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:51:51.650112 1250 raft_consensus.cc:515] T 55c1b243d8a94938a5caa58754092cec P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:51.650139 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_4c16a32e4f364fecb17b3c79b3a5b98c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:51.650905 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4c16a32e4f364fecb17b3c79b3a5b98c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:51.652557 522 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_77c77066213c4d05bdf61bc81facdebd" } new_table_name: "default.Table_c70eb9f284cd49109fd09e430a1cea6c" modify_external_catalogs: true
I20251029 02:51:51.653637 1167 tablet_bootstrap.cc:492] T 55c1b243d8a94938a5caa58754092cec P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:51:51.653801 1167 ts_tablet_manager.cc:1403] T 55c1b243d8a94938a5caa58754092cec P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.001s
I20251029 02:51:51.654204 919 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "55c1b243d8a94938a5caa58754092cec" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
I20251029 02:51:51.654752 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "55c1b243d8a94938a5caa58754092cec" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:51:51.654891 1250 leader_election.cc:290] T 55c1b243d8a94938a5caa58754092cec P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Requested vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:51:51.654981 776 raft_consensus.cc:3060] T 55c1b243d8a94938a5caa58754092cec P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:51:51.656029 517 master_service.cc:954] ReplaceTablet: received request to replace tablet 8b89ee9017b244c5b866b1fc7154ffe3 from {username='slave'} at 127.0.0.1:35314
I20251029 02:51:51.655982 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_e95a0439ea534f538683392bbc8e5219"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:51.656586 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e95a0439ea534f538683392bbc8e5219 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:51.656447 1167 raft_consensus.cc:359] T 55c1b243d8a94938a5caa58754092cec P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:51.656689 1167 raft_consensus.cc:385] T 55c1b243d8a94938a5caa58754092cec P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:51.656783 1167 raft_consensus.cc:740] T 55c1b243d8a94938a5caa58754092cec P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:51:51.657042 1167 consensus_queue.cc:260] T 55c1b243d8a94938a5caa58754092cec P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:51.657881 1167 ts_tablet_manager.cc:1434] T 55c1b243d8a94938a5caa58754092cec P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251029 02:51:51.658191 1167 tablet_bootstrap.cc:492] T ba014ecfec7f4602ae1c09d931bac824 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
W20251029 02:51:51.659723 990 leader_election.cc:343] T 55c1b243d8a94938a5caa58754092cec P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:51.660588 1062 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8f7c746f8b6948b58676298c2c2ea221" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:51:51.660662 776 raft_consensus.cc:2468] T 55c1b243d8a94938a5caa58754092cec P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 1.
I20251029 02:51:51.661398 1167 tablet_bootstrap.cc:654] T ba014ecfec7f4602ae1c09d931bac824 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
W20251029 02:51:51.661885 712 leader_election.cc:343] T 8f7c746f8b6948b58676298c2c2ea221 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:51.662032 712 leader_election.cc:304] T 8f7c746f8b6948b58676298c2c2ea221 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:51:51.662362 1179 raft_consensus.cc:2749] T 8f7c746f8b6948b58676298c2c2ea221 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:51:51.662616 989 leader_election.cc:304] T 55c1b243d8a94938a5caa58754092cec P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters: c756528642a543eeacd93cef3e335a7d
I20251029 02:51:51.664247 1250 raft_consensus.cc:2804] T 55c1b243d8a94938a5caa58754092cec P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:51:51.664937 1250 raft_consensus.cc:697] T 55c1b243d8a94938a5caa58754092cec P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Becoming Leader. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Running, Role: LEADER
W20251029 02:51:51.666280 521 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6f93812578a940a3b7378015e2a46f95 has no consensus state
I20251029 02:51:51.668143 892 tablet_service.cc:1552] Processing DeleteTablet for tablet 94edf3f2e0854b07bbd348ecab82d3ca with delete_type TABLET_DATA_DELETED (replaced by tablet 86f8776b5c4c4082bf276199feaaefea) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:51.668479 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 22be6bc81efb434ab7fc3242193789c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:51 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:51.670008 1032 tablet_service.cc:1505] Processing CreateTablet for tablet 41a6139fd9b84707a8c7eebc0381e383 (DEFAULT_TABLE table=default.Table_9c88c372f29d4f0980baa173608086f2 [id=f45315b686594867aae0114f4ce2f26c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.670475 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41a6139fd9b84707a8c7eebc0381e383. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:51:51.672749 522 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b867feae24e741aca6902868c35c1428 consensus state has no leader
I20251029 02:51:51.675093 894 tablet_service.cc:1505] Processing CreateTablet for tablet 41a6139fd9b84707a8c7eebc0381e383 (DEFAULT_TABLE table=default.Table_9c88c372f29d4f0980baa173608086f2 [id=f45315b686594867aae0114f4ce2f26c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.675540 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41a6139fd9b84707a8c7eebc0381e383. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.680625 754 tablet_service.cc:1505] Processing CreateTablet for tablet 41a6139fd9b84707a8c7eebc0381e383 (DEFAULT_TABLE table=default.Table_9c88c372f29d4f0980baa173608086f2 [id=f45315b686594867aae0114f4ce2f26c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.680717 1250 consensus_queue.cc:237] T 55c1b243d8a94938a5caa58754092cec P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:51.681061 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41a6139fd9b84707a8c7eebc0381e383. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.681358 1033 tablet_service.cc:1505] Processing CreateTablet for tablet 5e31e061d74146009d24d5cd57487a5e (DEFAULT_TABLE table=default.Table_469f3d7be86340a2a24565b8a5058d64 [id=8e17d6b87b984d878787802bd6f627f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.681792 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e31e061d74146009d24d5cd57487a5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.686910 892 tablet_service.cc:1505] Processing CreateTablet for tablet 5e31e061d74146009d24d5cd57487a5e (DEFAULT_TABLE table=default.Table_469f3d7be86340a2a24565b8a5058d64 [id=8e17d6b87b984d878787802bd6f627f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.687340 895 tablet_service.cc:1505] Processing CreateTablet for tablet bc1a1cf274b64b52aba58811e9594330 (DEFAULT_TABLE table=default.Table_c17409b349f548f9a5062085395a2aee [id=efb8f381fea74cf48842af6b1a4d5a81]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.687491 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e31e061d74146009d24d5cd57487a5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.687784 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc1a1cf274b64b52aba58811e9594330. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.689018 752 tablet_service.cc:1505] Processing CreateTablet for tablet 5e31e061d74146009d24d5cd57487a5e (DEFAULT_TABLE table=default.Table_469f3d7be86340a2a24565b8a5058d64 [id=8e17d6b87b984d878787802bd6f627f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.689152 518 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 42f463dc3cc1434b8ee5ba268cb9fc9a
I20251029 02:51:51.689463 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e31e061d74146009d24d5cd57487a5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.689641 518 catalog_manager.cc:6719] ReplaceTablet: tablet 42f463dc3cc1434b8ee5ba268cb9fc9a deleted and replaced by tablet f71388036478404d878a52fc20ece216
I20251029 02:51:51.690377 900 tablet_service.cc:1552] Processing DeleteTablet for tablet 42f463dc3cc1434b8ee5ba268cb9fc9a with delete_type TABLET_DATA_DELETED (replaced by tablet f71388036478404d878a52fc20ece216) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:51.692765 1166 tablet_bootstrap.cc:492] T a0cee0a6eb5a435b9bb3295cbe242b70 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:51:51.692934 1166 ts_tablet_manager.cc:1403] T a0cee0a6eb5a435b9bb3295cbe242b70 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.142s user 0.015s sys 0.013s
I20251029 02:51:51.693706 1166 raft_consensus.cc:359] T a0cee0a6eb5a435b9bb3295cbe242b70 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:51.693985 1166 raft_consensus.cc:385] T a0cee0a6eb5a435b9bb3295cbe242b70 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:51.693989 1276 leader_election.cc:290] T 8f7c746f8b6948b58676298c2c2ea221 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:51:51.694063 1166 raft_consensus.cc:740] T a0cee0a6eb5a435b9bb3295cbe242b70 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:51:51.694247 749 tablet_service.cc:1505] Processing CreateTablet for tablet 70e8cfe5eaa9492c85874dc3214a1e0b (DEFAULT_TABLE table=default.Table_c784291ac23b4f86ad8bd33ebd8f4755 [id=82df85af001340efb570040e047fa136]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.686910 893 tablet_service.cc:1505] Processing CreateTablet for tablet 70e8cfe5eaa9492c85874dc3214a1e0b (DEFAULT_TABLE table=default.Table_c784291ac23b4f86ad8bd33ebd8f4755 [id=82df85af001340efb570040e047fa136]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.695266 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70e8cfe5eaa9492c85874dc3214a1e0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.695461 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_bc0bcc8e7fb34d05b925544848cbd7d2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:51.695616 900 tablet_service.cc:1505] Processing CreateTablet for tablet 61c1fec3ab4c402eb29f767b9bb7409f (DEFAULT_TABLE table=default.Table_42a71464d71d487baa7245b5c3bdbbcd [id=c124cfa700724f6db0f1a12fe495c2af]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.696053 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61c1fec3ab4c402eb29f767b9bb7409f. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:51:51.696281 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bc0bcc8e7fb34d05b925544848cbd7d2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:51.696714 525 catalog_manager.cc:5649] T ba014ecfec7f4602ae1c09d931bac824 P b495a49349324f7d81908f2da700f0d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to b495a49349324f7d81908f2da700f0d9 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "b495a49349324f7d81908f2da700f0d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } health_report { overall_health: HEALTHY } } }
I20251029 02:51:51.697137 525 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet ba014ecfec7f4602ae1c09d931bac824 (table default.Table_64e512f8ce034224aea9b887ed0d09dc [id=7bb8b246fbf1431bb71bbb7c5520e301]). Expected version 2 got 0
I20251029 02:51:51.699424 751 tablet_service.cc:1552] Processing DeleteTablet for tablet 42f463dc3cc1434b8ee5ba268cb9fc9a with delete_type TABLET_DATA_DELETED (replaced by tablet f71388036478404d878a52fc20ece216) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:51.699856 755 tablet_service.cc:1505] Processing CreateTablet for tablet bc1a1cf274b64b52aba58811e9594330 (DEFAULT_TABLE table=default.Table_c17409b349f548f9a5062085395a2aee [id=efb8f381fea74cf48842af6b1a4d5a81]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.699920 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_da0be17b8bdc4e579b6c3faff71d9c98"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:51.700657 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70e8cfe5eaa9492c85874dc3214a1e0b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.701140 1034 tablet_service.cc:1505] Processing CreateTablet for tablet 70e8cfe5eaa9492c85874dc3214a1e0b (DEFAULT_TABLE table=default.Table_c784291ac23b4f86ad8bd33ebd8f4755 [id=82df85af001340efb570040e047fa136]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.701115 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_62e57bedde8f4c6e80bcbdcc21580091"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:51.701537 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70e8cfe5eaa9492c85874dc3214a1e0b. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:51:51.701726 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_da0be17b8bdc4e579b6c3faff71d9c98 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:51.702817 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc1a1cf274b64b52aba58811e9594330. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:51:51.703032 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_62e57bedde8f4c6e80bcbdcc21580091 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:51.706883 1037 tablet_service.cc:1552] Processing DeleteTablet for tablet 42f463dc3cc1434b8ee5ba268cb9fc9a with delete_type TABLET_DATA_DELETED (replaced by tablet f71388036478404d878a52fc20ece216) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:51.707659 1037 tablet_service.cc:1505] Processing CreateTablet for tablet bc1a1cf274b64b52aba58811e9594330 (DEFAULT_TABLE table=default.Table_c17409b349f548f9a5062085395a2aee [id=efb8f381fea74cf48842af6b1a4d5a81]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.708024 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc1a1cf274b64b52aba58811e9594330. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.722488 751 tablet_service.cc:1505] Processing CreateTablet for tablet 61c1fec3ab4c402eb29f767b9bb7409f (DEFAULT_TABLE table=default.Table_42a71464d71d487baa7245b5c3bdbbcd [id=c124cfa700724f6db0f1a12fe495c2af]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.722584 1166 consensus_queue.cc:260] T a0cee0a6eb5a435b9bb3295cbe242b70 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:51.723035 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61c1fec3ab4c402eb29f767b9bb7409f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.723204 1176 ts_tablet_manager.cc:1929] T 3caf6aa732e6460e96145c0f4105a826 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:51.723340 1176 log.cc:1199] T 3caf6aa732e6460e96145c0f4105a826 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/3caf6aa732e6460e96145c0f4105a826
I20251029 02:51:51.723742 1176 ts_tablet_manager.cc:1950] T 3caf6aa732e6460e96145c0f4105a826 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:51:51.724324 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 3caf6aa732e6460e96145c0f4105a826 (table default.Table_4bdbcbb87e5b40aeb9f98eb1aa0e6fe3 [id=78615d8807a74b4496a3223ef6fe709d]) successfully deleted
I20251029 02:51:51.724764 1166 ts_tablet_manager.cc:1434] T a0cee0a6eb5a435b9bb3295cbe242b70 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.032s user 0.002s sys 0.000s
I20251029 02:51:51.725114 1166 tablet_bootstrap.cc:492] T f04473c3a5f74bdba196b91c2538537e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
W20251029 02:51:51.725587 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 8b89ee9017b244c5b866b1fc7154ffe3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8b89ee9017b244c5b866b1fc7154ffe3 already in progress: creating tablet
I20251029 02:51:51.724072 1283 tablet.cc:1722] T 0c0aeaabe14740718f75f723023aa087 P c756528642a543eeacd93cef3e335a7d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 02:51:51.726891 1176 tablet_replica.cc:333] T 0c0aeaabe14740718f75f723023aa087 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:51:51.727097 1166 tablet_bootstrap.cc:654] T f04473c3a5f74bdba196b91c2538537e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:51.727262 1174 raft_consensus.cc:2243] T 0c0aeaabe14740718f75f723023aa087 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:51.727732 1174 raft_consensus.cc:2272] T 0c0aeaabe14740718f75f723023aa087 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:51.732263 1176 raft_consensus.cc:2243] T 0c0aeaabe14740718f75f723023aa087 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:51:51.732869 1176 raft_consensus.cc:2272] T 0c0aeaabe14740718f75f723023aa087 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:51.738652 1037 tablet_service.cc:1505] Processing CreateTablet for tablet 61c1fec3ab4c402eb29f767b9bb7409f (DEFAULT_TABLE table=default.Table_42a71464d71d487baa7245b5c3bdbbcd [id=c124cfa700724f6db0f1a12fe495c2af]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.740546 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61c1fec3ab4c402eb29f767b9bb7409f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.740757 1166 tablet_bootstrap.cc:492] T f04473c3a5f74bdba196b91c2538537e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:51:51.740921 1166 ts_tablet_manager.cc:1403] T f04473c3a5f74bdba196b91c2538537e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.000s
I20251029 02:51:51.741890 1176 ts_tablet_manager.cc:1916] T 0c0aeaabe14740718f75f723023aa087 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:51.744963 1171 raft_consensus.cc:493] T 94854145ac0d4be38780d4cdb721725e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:51.745124 1171 raft_consensus.cc:515] T 94854145ac0d4be38780d4cdb721725e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:51.745821 1171 leader_election.cc:290] T 94854145ac0d4be38780d4cdb721725e P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:51:51.746583 1062 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "94854145ac0d4be38780d4cdb721725e" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:51:51.747695 849 leader_election.cc:343] T 94854145ac0d4be38780d4cdb721725e P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
W20251029 02:51:51.748078 847 leader_election.cc:343] T 94854145ac0d4be38780d4cdb721725e P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 94854145ac0d4be38780d4cdb721725e
I20251029 02:51:51.748231 847 leader_election.cc:304] T 94854145ac0d4be38780d4cdb721725e P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9
I20251029 02:51:51.748845 1171 raft_consensus.cc:2749] T 94854145ac0d4be38780d4cdb721725e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:51:51.749733 522 master_service.cc:954] ReplaceTablet: received request to replace tablet 2ebde39f813d46e898ad1754325ad1a0 from {username='slave'} at 127.0.0.1:35314
I20251029 02:51:51.749768 1276 raft_consensus.cc:493] T bc490b1327e44059a57e6cd7b63350e3 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:51.749955 1276 raft_consensus.cc:515] T bc490b1327e44059a57e6cd7b63350e3 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:51.750633 1276 leader_election.cc:290] T bc490b1327e44059a57e6cd7b63350e3 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:51:51.751108 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc490b1327e44059a57e6cd7b63350e3" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:51:51.751535 1062 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc490b1327e44059a57e6cd7b63350e3" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:51:51.751782 711 leader_election.cc:343] T bc490b1327e44059a57e6cd7b63350e3 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
W20251029 02:51:51.752104 712 leader_election.cc:343] T bc490b1327e44059a57e6cd7b63350e3 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:51.752230 712 leader_election.cc:304] T bc490b1327e44059a57e6cd7b63350e3 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:51:51.752712 1276 raft_consensus.cc:2749] T bc490b1327e44059a57e6cd7b63350e3 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:51:51.753492 1172 tablet_bootstrap.cc:492] T 2ebde39f813d46e898ad1754325ad1a0 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:51:51.753643 1172 ts_tablet_manager.cc:1403] T 2ebde39f813d46e898ad1754325ad1a0 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.155s user 0.005s sys 0.014s
I20251029 02:51:51.754498 1172 raft_consensus.cc:359] T 2ebde39f813d46e898ad1754325ad1a0 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:51.754716 1172 raft_consensus.cc:385] T 2ebde39f813d46e898ad1754325ad1a0 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:51.754808 1172 raft_consensus.cc:740] T 2ebde39f813d46e898ad1754325ad1a0 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:51:51.755033 1172 consensus_queue.cc:260] T 2ebde39f813d46e898ad1754325ad1a0 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:51.755558 1172 ts_tablet_manager.cc:1434] T 2ebde39f813d46e898ad1754325ad1a0 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 02:51:51.755962 1172 tablet_bootstrap.cc:492] T d32532973b3f46dd8158a2ce42d26365 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:51:51.758366 1172 tablet_bootstrap.cc:654] T d32532973b3f46dd8158a2ce42d26365 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:51.758409 1167 tablet_bootstrap.cc:492] T ba014ecfec7f4602ae1c09d931bac824 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:51:51.758627 1167 ts_tablet_manager.cc:1403] T ba014ecfec7f4602ae1c09d931bac824 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.100s user 0.008s sys 0.002s
I20251029 02:51:51.759434 1167 raft_consensus.cc:359] T ba014ecfec7f4602ae1c09d931bac824 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:51.759662 1167 raft_consensus.cc:385] T ba014ecfec7f4602ae1c09d931bac824 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:51.759789 1167 raft_consensus.cc:740] T ba014ecfec7f4602ae1c09d931bac824 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:51:51.761502 1167 consensus_queue.cc:260] T ba014ecfec7f4602ae1c09d931bac824 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:51.764308 1167 ts_tablet_manager.cc:1434] T ba014ecfec7f4602ae1c09d931bac824 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20251029 02:51:51.764444 522 master_service.cc:954] ReplaceTablet: received request to replace tablet deca0c4a114441079fce22e74b403050 from {username='slave'} at 127.0.0.1:35314
I20251029 02:51:51.766444 1167 tablet_bootstrap.cc:492] T 8b89ee9017b244c5b866b1fc7154ffe3 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:51:51.767102 749 tablet_service.cc:1552] Processing DeleteTablet for tablet 2ebde39f813d46e898ad1754325ad1a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:51 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:51.767969 1166 raft_consensus.cc:359] T f04473c3a5f74bdba196b91c2538537e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:51.768165 1166 raft_consensus.cc:385] T f04473c3a5f74bdba196b91c2538537e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:51.768277 1166 raft_consensus.cc:740] T f04473c3a5f74bdba196b91c2538537e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:51:51.768497 1166 consensus_queue.cc:260] T f04473c3a5f74bdba196b91c2538537e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:51.769645 1175 ts_tablet_manager.cc:1916] T 0c0aeaabe14740718f75f723023aa087 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:51.772228 525 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet e6bcea41d5d4480eaf5d6986a93f5e45 (table default.Table_5b3a7640a9e54e7a9868985322ade0cb [id=1a1aeb237dad4464b717bbae5d4001b0]). Expected version 2 got 0
I20251029 02:51:51.772440 525 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet 55c1b243d8a94938a5caa58754092cec (table default.Table_823c69ddf86f442baf2cfc774320e6e4 [id=bd2335c28dd44b5584ab71694cebda2d]). Expected version 1 got 0
I20251029 02:51:51.774390 893 tablet_service.cc:1552] Processing DeleteTablet for tablet 94854145ac0d4be38780d4cdb721725e with delete_type TABLET_DATA_DELETED (replaced by tablet 82f75c134c1a4858a2154ee6b1d1d29f) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:51.775795 522 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet deca0c4a114441079fce22e74b403050
I20251029 02:51:51.776310 522 catalog_manager.cc:6719] ReplaceTablet: tablet deca0c4a114441079fce22e74b403050 deleted and replaced by tablet e19c8b10f32849f5885fc00176c8bc12
I20251029 02:51:51.776261 893 tablet_service.cc:1505] Processing CreateTablet for tablet 4b76c4408a7340e59b8506171622175f (DEFAULT_TABLE table=default.Table_62e57bedde8f4c6e80bcbdcc21580091 [id=f3fa2dbbef7e47359492d5a5eef9b8fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.776695 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b76c4408a7340e59b8506171622175f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.776749 895 tablet_service.cc:1552] Processing DeleteTablet for tablet deca0c4a114441079fce22e74b403050 with delete_type TABLET_DATA_DELETED (replaced by tablet e19c8b10f32849f5885fc00176c8bc12) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:51.776849 1176 ts_tablet_manager.cc:1929] T 0c0aeaabe14740718f75f723023aa087 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251029 02:51:51.776983 1176 log.cc:1199] T 0c0aeaabe14740718f75f723023aa087 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/0c0aeaabe14740718f75f723023aa087
I20251029 02:51:51.777000 749 tablet_service.cc:1552] Processing DeleteTablet for tablet deca0c4a114441079fce22e74b403050 with delete_type TABLET_DATA_DELETED (replaced by tablet e19c8b10f32849f5885fc00176c8bc12) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:51.777678 1174 ts_tablet_manager.cc:1916] T 0c0aeaabe14740718f75f723023aa087 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:51.778853 1166 ts_tablet_manager.cc:1434] T f04473c3a5f74bdba196b91c2538537e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.038s user 0.003s sys 0.000s
I20251029 02:51:51.781075 522 catalog_manager.cc:5649] T 55c1b243d8a94938a5caa58754092cec P 0cfd4bee16f84b7c8b0ff3ce14db20d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } health_report { overall_health: UNKNOWN } } }
I20251029 02:51:51.781527 522 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet 55c1b243d8a94938a5caa58754092cec (table default.Table_823c69ddf86f442baf2cfc774320e6e4 [id=bd2335c28dd44b5584ab71694cebda2d]). Expected version 1 got 0
I20251029 02:51:51.784307 1175 ts_tablet_manager.cc:1929] T 0c0aeaabe14740718f75f723023aa087 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251029 02:51:51.784538 1175 log.cc:1199] T 0c0aeaabe14740718f75f723023aa087 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/0c0aeaabe14740718f75f723023aa087
I20251029 02:51:51.785045 1175 ts_tablet_manager.cc:1950] T 0c0aeaabe14740718f75f723023aa087 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:51:51.785861 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_69bda6d32a204cb6923a73ba4626e96b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:51.786165 1166 tablet_bootstrap.cc:492] T 8f7c746f8b6948b58676298c2c2ea221 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:51:51.786338 1175 tablet_replica.cc:333] T f04473c3a5f74bdba196b91c2538537e P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
W20251029 02:51:51.786425 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_69bda6d32a204cb6923a73ba4626e96b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:51.786568 1175 raft_consensus.cc:2243] T f04473c3a5f74bdba196b91c2538537e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:51.786752 1175 raft_consensus.cc:2272] T f04473c3a5f74bdba196b91c2538537e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:51.787279 1175 ts_tablet_manager.cc:1916] T f04473c3a5f74bdba196b91c2538537e P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:51.787772 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 0c0aeaabe14740718f75f723023aa087 (table default.Table_d84474a10a954e70afe50f3eb8e8c1ed [id=feb1f56ae44249c6adce65637017bb43]) successfully deleted
W20251029 02:51:51.788048 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 6c32c6f5e1cc40a0a12688201cac62a7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6c32c6f5e1cc40a0a12688201cac62a7 already in progress: creating tablet
I20251029 02:51:51.788244 1166 tablet_bootstrap.cc:654] T 8f7c746f8b6948b58676298c2c2ea221 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:51.788156 518 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_0cb428bfceb5464985b45d16597c11cb" } new_table_name: "default.Table_762d41caedac41b09d445c6bfea97e75" modify_external_catalogs: true
I20251029 02:51:51.789016 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_cb594ce4077d4d02aa70ef94d5f801cc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:51.789546 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cb594ce4077d4d02aa70ef94d5f801cc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:51.790158 1176 ts_tablet_manager.cc:1950] T 0c0aeaabe14740718f75f723023aa087 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:51:51.790788 1276 raft_consensus.cc:493] T 42f463dc3cc1434b8ee5ba268cb9fc9a P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:51.790962 1276 raft_consensus.cc:515] T 42f463dc3cc1434b8ee5ba268cb9fc9a P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:51.791641 1276 leader_election.cc:290] T 42f463dc3cc1434b8ee5ba268cb9fc9a P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:51:51.792548 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_e0e6e47b18d749e8ae80f822af1e23f0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:51.793051 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e0e6e47b18d749e8ae80f822af1e23f0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:51.793677 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42f463dc3cc1434b8ee5ba268cb9fc9a" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:51:51.794230 711 leader_election.cc:343] T 42f463dc3cc1434b8ee5ba268cb9fc9a P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:51.794430 1062 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42f463dc3cc1434b8ee5ba268cb9fc9a" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:51:51.795248 1034 tablet_service.cc:1505] Processing CreateTablet for tablet 4b76c4408a7340e59b8506171622175f (DEFAULT_TABLE table=default.Table_62e57bedde8f4c6e80bcbdcc21580091 [id=f3fa2dbbef7e47359492d5a5eef9b8fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.795672 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b76c4408a7340e59b8506171622175f. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:51:51.796022 712 leader_election.cc:343] T 42f463dc3cc1434b8ee5ba268cb9fc9a P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:51.796140 712 leader_election.cc:304] T 42f463dc3cc1434b8ee5ba268cb9fc9a P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:51:51.796226 749 tablet_service.cc:1505] Processing CreateTablet for tablet 4b76c4408a7340e59b8506171622175f (DEFAULT_TABLE table=default.Table_62e57bedde8f4c6e80bcbdcc21580091 [id=f3fa2dbbef7e47359492d5a5eef9b8fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.796589 1276 raft_consensus.cc:2749] T 42f463dc3cc1434b8ee5ba268cb9fc9a P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:51:51.796607 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b76c4408a7340e59b8506171622175f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.797961 893 tablet_service.cc:1505] Processing CreateTablet for tablet 24016ceaa84147eca0b94b3ad288e772 (DEFAULT_TABLE table=default.Table_7b36c69036a641f289a5f927f3101f16 [id=bd279b3ce623477ca20e8d35e3b7c0df]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.798784 751 tablet_service.cc:1505] Processing CreateTablet for tablet 24016ceaa84147eca0b94b3ad288e772 (DEFAULT_TABLE table=default.Table_7b36c69036a641f289a5f927f3101f16 [id=bd279b3ce623477ca20e8d35e3b7c0df]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.799180 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24016ceaa84147eca0b94b3ad288e772. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.800144 1033 tablet_service.cc:1552] Processing DeleteTablet for tablet deca0c4a114441079fce22e74b403050 with delete_type TABLET_DATA_DELETED (replaced by tablet e19c8b10f32849f5885fc00176c8bc12) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:51.802958 1062 raft_consensus.cc:1275] T ba014ecfec7f4602ae1c09d931bac824 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Refusing update from remote peer b495a49349324f7d81908f2da700f0d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251029 02:51:51.802973 752 tablet_service.cc:1505] Processing CreateTablet for tablet ab421dfaa3f84593842bb732a2693160 (DEFAULT_TABLE table=default.Table_7680a98fa637454984a5d4da44855a48 [id=03a25420ce6a46fc93a9fb753246bc7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.803377 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab421dfaa3f84593842bb732a2693160. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.803848 1179 consensus_queue.cc:1048] T ba014ecfec7f4602ae1c09d931bac824 P b495a49349324f7d81908f2da700f0d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:51:51.806173 524 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet ba014ecfec7f4602ae1c09d931bac824 (table default.Table_64e512f8ce034224aea9b887ed0d09dc [id=7bb8b246fbf1431bb71bbb7c5520e301]). Expected version 2 got 0
I20251029 02:51:51.799461 895 tablet_service.cc:1505] Processing CreateTablet for tablet ab421dfaa3f84593842bb732a2693160 (DEFAULT_TABLE table=default.Table_7680a98fa637454984a5d4da44855a48 [id=03a25420ce6a46fc93a9fb753246bc7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.805779 920 raft_consensus.cc:3060] T ba014ecfec7f4602ae1c09d931bac824 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:51:51.809801 1033 tablet_service.cc:1505] Processing CreateTablet for tablet 24016ceaa84147eca0b94b3ad288e772 (DEFAULT_TABLE table=default.Table_7b36c69036a641f289a5f927f3101f16 [id=bd279b3ce623477ca20e8d35e3b7c0df]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.810276 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24016ceaa84147eca0b94b3ad288e772. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.810338 520 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_823c69ddf86f442baf2cfc774320e6e4" } new_table_name: "default.Table_953f59050dc54b04b57c06364595c612" modify_external_catalogs: true
I20251029 02:51:51.811905 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_75949df235e044029e69b130e96b8d27"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:51.812094 920 raft_consensus.cc:1275] T ba014ecfec7f4602ae1c09d931bac824 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Refusing update from remote peer b495a49349324f7d81908f2da700f0d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251029 02:51:51.798446 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24016ceaa84147eca0b94b3ad288e772. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:51:51.812427 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_75949df235e044029e69b130e96b8d27 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:51.812600 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab421dfaa3f84593842bb732a2693160. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.813042 1276 consensus_queue.cc:1048] T ba014ecfec7f4602ae1c09d931bac824 P b495a49349324f7d81908f2da700f0d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:51:51.813576 524 master_service.cc:954] ReplaceTablet: received request to replace tablet cb55b3d5bc63405f96cdc122f5a612cb from {username='slave'} at 127.0.0.1:35314
I20251029 02:51:51.814852 519 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_bc0bcc8e7fb34d05b925544848cbd7d2" } new_table_name: "default.Table_2e8a7a51e2414926b5508b8a06d6a55d" modify_external_catalogs: true
I20251029 02:51:51.820365 1037 tablet_service.cc:1505] Processing CreateTablet for tablet ab421dfaa3f84593842bb732a2693160 (DEFAULT_TABLE table=default.Table_7680a98fa637454984a5d4da44855a48 [id=03a25420ce6a46fc93a9fb753246bc7b]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 02:51:51.822368 518 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fa5127bcfcda40a7b5ad4ef016af072a consensus state has no leader
I20251029 02:51:51.822402 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab421dfaa3f84593842bb732a2693160. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.824596 517 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_4c5fd91d917e4f1a99baef291b2d77e0" } modify_external_catalogs: true
I20251029 02:51:51.824815 517 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_4c5fd91d917e4f1a99baef291b2d77e0" } modify_external_catalogs: true
I20251029 02:51:51.825872 895 tablet_service.cc:1505] Processing CreateTablet for tablet fa5127bcfcda40a7b5ad4ef016af072a (DEFAULT_TABLE table=default.Table_0cb428bfceb5464985b45d16597c11cb [id=ae5d56d40fa941a68f2b10d5e6e9ac83]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.827183 1172 tablet_bootstrap.cc:492] T d32532973b3f46dd8158a2ce42d26365 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:51:51.827255 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa5127bcfcda40a7b5ad4ef016af072a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.827387 1172 ts_tablet_manager.cc:1403] T d32532973b3f46dd8158a2ce42d26365 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.072s user 0.004s sys 0.000s
I20251029 02:51:51.828655 1172 raft_consensus.cc:359] T d32532973b3f46dd8158a2ce42d26365 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:51.828881 1172 raft_consensus.cc:385] T d32532973b3f46dd8158a2ce42d26365 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:51.828955 1172 raft_consensus.cc:740] T d32532973b3f46dd8158a2ce42d26365 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:51:51.829191 1172 consensus_queue.cc:260] T d32532973b3f46dd8158a2ce42d26365 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:51.830873 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_0b84a2f666074fd18085ff4ce249a2fe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:51.831485 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0b84a2f666074fd18085ff4ce249a2fe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:51.832278 1172 ts_tablet_manager.cc:1434] T d32532973b3f46dd8158a2ce42d26365 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20251029 02:51:51.832317 752 tablet_service.cc:1505] Processing CreateTablet for tablet fa5127bcfcda40a7b5ad4ef016af072a (DEFAULT_TABLE table=default.Table_762d41caedac41b09d445c6bfea97e75 [id=ae5d56d40fa941a68f2b10d5e6e9ac83]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.832733 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa5127bcfcda40a7b5ad4ef016af072a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.833117 1172 tablet_bootstrap.cc:492] T 0dd31dc5873d407d958a0f5199f78782 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:51:51.835300 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 0c0aeaabe14740718f75f723023aa087 (table default.Table_d84474a10a954e70afe50f3eb8e8c1ed [id=feb1f56ae44249c6adce65637017bb43]) successfully deleted
I20251029 02:51:51.835398 1034 tablet_service.cc:1505] Processing CreateTablet for tablet fa5127bcfcda40a7b5ad4ef016af072a (DEFAULT_TABLE table=default.Table_762d41caedac41b09d445c6bfea97e75 [id=ae5d56d40fa941a68f2b10d5e6e9ac83]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.835911 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa5127bcfcda40a7b5ad4ef016af072a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.837169 1174 ts_tablet_manager.cc:1929] T 0c0aeaabe14740718f75f723023aa087 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251029 02:51:51.837246 1172 tablet_bootstrap.cc:654] T 0dd31dc5873d407d958a0f5199f78782 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:51.839107 892 tablet_service.cc:1505] Processing CreateTablet for tablet 64b0fb400e944ce2b2c7a49d12135e9c (DEFAULT_TABLE table=default.Table_bc0bcc8e7fb34d05b925544848cbd7d2 [id=e71aea48b9884bacb250412624ec4615]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.839527 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64b0fb400e944ce2b2c7a49d12135e9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.842576 518 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_4bdbcbb87e5b40aeb9f98eb1aa0e6fe3" } modify_external_catalogs: true
I20251029 02:51:51.842798 518 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_4bdbcbb87e5b40aeb9f98eb1aa0e6fe3" } modify_external_catalogs: true
I20251029 02:51:51.843091 525 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet ba014ecfec7f4602ae1c09d931bac824 (table default.Table_64e512f8ce034224aea9b887ed0d09dc [id=7bb8b246fbf1431bb71bbb7c5520e301]). Expected version 2 got 0
I20251029 02:51:51.844606 1176 tablet_replica.cc:333] T f04473c3a5f74bdba196b91c2538537e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:51:51.846154 1280 tablet.cc:1722] T ba014ecfec7f4602ae1c09d931bac824 P b495a49349324f7d81908f2da700f0d9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 02:51:51.846354 894 tablet_service.cc:1505] Processing CreateTablet for tablet c4884e22cf2044a4a1070c1b4b759d9e (DEFAULT_TABLE table=default.Table_da0be17b8bdc4e579b6c3faff71d9c98 [id=6e06ac70c6814e7d89a008bbaddccc7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.846807 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4884e22cf2044a4a1070c1b4b759d9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.847797 1274 tablet.cc:1722] T ba014ecfec7f4602ae1c09d931bac824 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 02:51:51.848621 1176 raft_consensus.cc:2243] T f04473c3a5f74bdba196b91c2538537e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:51.848830 1176 raft_consensus.cc:2272] T f04473c3a5f74bdba196b91c2538537e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:51.849402 525 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet ba014ecfec7f4602ae1c09d931bac824 (table default.Table_64e512f8ce034224aea9b887ed0d09dc [id=7bb8b246fbf1431bb71bbb7c5520e301]). Expected version 2 got 0
I20251029 02:51:51.849603 1176 ts_tablet_manager.cc:1916] T f04473c3a5f74bdba196b91c2538537e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:51.853106 900 tablet_service.cc:1505] Processing CreateTablet for tablet 7c041f4909274cd5a03b9fd78353faf9 (DEFAULT_TABLE table=default.Table_83d3fe1030e0461895fc2437732b4ff8 [id=e11618f2bd094526ad359c65bee6a9ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.853583 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c041f4909274cd5a03b9fd78353faf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.859545 751 tablet_service.cc:1505] Processing CreateTablet for tablet c4884e22cf2044a4a1070c1b4b759d9e (DEFAULT_TABLE table=default.Table_da0be17b8bdc4e579b6c3faff71d9c98 [id=6e06ac70c6814e7d89a008bbaddccc7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.860131 754 tablet_service.cc:1505] Processing CreateTablet for tablet 7c041f4909274cd5a03b9fd78353faf9 (DEFAULT_TABLE table=default.Table_83d3fe1030e0461895fc2437732b4ff8 [id=e11618f2bd094526ad359c65bee6a9ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.860455 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4884e22cf2044a4a1070c1b4b759d9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.860707 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c041f4909274cd5a03b9fd78353faf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.861284 1166 tablet_bootstrap.cc:492] T 8f7c746f8b6948b58676298c2c2ea221 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:51:51.861444 1166 ts_tablet_manager.cc:1403] T 8f7c746f8b6948b58676298c2c2ea221 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.075s user 0.005s sys 0.001s
I20251029 02:51:51.862175 1166 raft_consensus.cc:359] T 8f7c746f8b6948b58676298c2c2ea221 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:51.862392 1166 raft_consensus.cc:385] T 8f7c746f8b6948b58676298c2c2ea221 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:51.862447 1166 raft_consensus.cc:740] T 8f7c746f8b6948b58676298c2c2ea221 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:51:51.862650 1166 consensus_queue.cc:260] T 8f7c746f8b6948b58676298c2c2ea221 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:51.864126 1166 ts_tablet_manager.cc:1434] T 8f7c746f8b6948b58676298c2c2ea221 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251029 02:51:51.865391 1166 tablet_bootstrap.cc:492] T 42f463dc3cc1434b8ee5ba268cb9fc9a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:51:51.864077 749 tablet_service.cc:1505] Processing CreateTablet for tablet 64b0fb400e944ce2b2c7a49d12135e9c (DEFAULT_TABLE table=default.Table_bc0bcc8e7fb34d05b925544848cbd7d2 [id=e71aea48b9884bacb250412624ec4615]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.866343 1033 tablet_service.cc:1505] Processing CreateTablet for tablet 64b0fb400e944ce2b2c7a49d12135e9c (DEFAULT_TABLE table=default.Table_bc0bcc8e7fb34d05b925544848cbd7d2 [id=e71aea48b9884bacb250412624ec4615]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.866745 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64b0fb400e944ce2b2c7a49d12135e9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.867154 521 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_839fd4e3c7874233afc8cea6d4c17026" } modify_external_catalogs: true
I20251029 02:51:51.867337 521 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_839fd4e3c7874233afc8cea6d4c17026" } modify_external_catalogs: true
I20251029 02:51:51.868825 523 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_1bd19c9d52204fbf87ee3cdbc62984af" } modify_external_catalogs: true
I20251029 02:51:51.868992 523 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_1bd19c9d52204fbf87ee3cdbc62984af" } modify_external_catalogs: true
I20251029 02:51:51.871596 1032 tablet_service.cc:1552] Processing DeleteTablet for tablet a0cee0a6eb5a435b9bb3295cbe242b70 with delete_type TABLET_DATA_DELETED (replaced by tablet 6c22497760274380a832feff7c5a2fae) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:51.874110 1283 tablet.cc:1722] T ba014ecfec7f4602ae1c09d931bac824 P c756528642a543eeacd93cef3e335a7d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 02:51:51.875208 1032 tablet_service.cc:1505] Processing CreateTablet for tablet c4884e22cf2044a4a1070c1b4b759d9e (DEFAULT_TABLE table=default.Table_da0be17b8bdc4e579b6c3faff71d9c98 [id=6e06ac70c6814e7d89a008bbaddccc7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.875628 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4884e22cf2044a4a1070c1b4b759d9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.876101 755 tablet_service.cc:1505] Processing CreateTablet for tablet aacb6f4b9284457bb6607e3a85d5c809 (DEFAULT_TABLE table=default.Table_4c16a32e4f364fecb17b3c79b3a5b98c [id=9f95160639ad40e995a6c89e297a0fad]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.877941 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64b0fb400e944ce2b2c7a49d12135e9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.878364 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aacb6f4b9284457bb6607e3a85d5c809. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.882740 892 tablet_service.cc:1505] Processing CreateTablet for tablet 59af906061d049c99ae2153945597007 (DEFAULT_TABLE table=default.Table_dd81d4b79bbc4d81b0ada1be7a68177a [id=25f16d95843d41c981d6a07ec0d4b48a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.883368 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59af906061d049c99ae2153945597007. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.837340 1174 log.cc:1199] T 0c0aeaabe14740718f75f723023aa087 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/0c0aeaabe14740718f75f723023aa087
I20251029 02:51:51.884577 1174 ts_tablet_manager.cc:1950] T 0c0aeaabe14740718f75f723023aa087 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:51:51.885703 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 0c0aeaabe14740718f75f723023aa087 (table default.Table_d84474a10a954e70afe50f3eb8e8c1ed [id=feb1f56ae44249c6adce65637017bb43]) successfully deleted
W20251029 02:51:51.886351 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 6c32c6f5e1cc40a0a12688201cac62a7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6c32c6f5e1cc40a0a12688201cac62a7 already in progress: creating tablet
W20251029 02:51:51.886932 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet f04473c3a5f74bdba196b91c2538537e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f04473c3a5f74bdba196b91c2538537e already in progress: creating tablet
W20251029 02:51:51.887466 512 catalog_manager.cc:4952] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 94edf3f2e0854b07bbd348ecab82d3ca because the tablet was not found. No further retry: Not found: Tablet not found: 94edf3f2e0854b07bbd348ecab82d3ca
W20251029 02:51:51.888010 512 catalog_manager.cc:4952] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 22be6bc81efb434ab7fc3242193789c3 because the tablet was not found. No further retry: Not found: Tablet not found: 22be6bc81efb434ab7fc3242193789c3
W20251029 02:51:51.888566 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 42f463dc3cc1434b8ee5ba268cb9fc9a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 42f463dc3cc1434b8ee5ba268cb9fc9a already in progress: creating tablet
I20251029 02:51:51.888890 1174 tablet_replica.cc:333] T 94854145ac0d4be38780d4cdb721725e P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:51:51.889093 1174 raft_consensus.cc:2243] T 94854145ac0d4be38780d4cdb721725e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:51.889283 1174 raft_consensus.cc:2272] T 94854145ac0d4be38780d4cdb721725e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:51.890138 900 tablet_service.cc:1505] Processing CreateTablet for tablet aacb6f4b9284457bb6607e3a85d5c809 (DEFAULT_TABLE table=default.Table_4c16a32e4f364fecb17b3c79b3a5b98c [id=9f95160639ad40e995a6c89e297a0fad]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.890372 1166 tablet_bootstrap.cc:654] T 42f463dc3cc1434b8ee5ba268cb9fc9a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
W20251029 02:51:51.890843 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 6c32c6f5e1cc40a0a12688201cac62a7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6c32c6f5e1cc40a0a12688201cac62a7 already in progress: creating tablet
W20251029 02:51:51.892170 510 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 94edf3f2e0854b07bbd348ecab82d3ca (table default.Table_bd27b9a173a243b0aa50adcf36f2a06d [id=13b28fb78eb54543b973bfda2fb1aad8], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 94edf3f2e0854b07bbd348ecab82d3ca consensus state has no leader
I20251029 02:51:51.893168 751 tablet_service.cc:1505] Processing CreateTablet for tablet 59af906061d049c99ae2153945597007 (DEFAULT_TABLE table=default.Table_dd81d4b79bbc4d81b0ada1be7a68177a [id=25f16d95843d41c981d6a07ec0d4b48a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.893344 1033 tablet_service.cc:1505] Processing CreateTablet for tablet 59af906061d049c99ae2153945597007 (DEFAULT_TABLE table=default.Table_dd81d4b79bbc4d81b0ada1be7a68177a [id=25f16d95843d41c981d6a07ec0d4b48a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.893603 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59af906061d049c99ae2153945597007. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.893759 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59af906061d049c99ae2153945597007. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.894243 1176 ts_tablet_manager.cc:1929] T f04473c3a5f74bdba196b91c2538537e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:51.894734 1176 log.cc:1199] T f04473c3a5f74bdba196b91c2538537e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/f04473c3a5f74bdba196b91c2538537e
I20251029 02:51:51.895180 1176 ts_tablet_manager.cc:1950] T f04473c3a5f74bdba196b91c2538537e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:51:51.896323 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet f04473c3a5f74bdba196b91c2538537e (table default.Table_7e818850675e4c5c973db094e7223348 [id=ec8cdbf19343401da05dec0c8c060ae6]) successfully deleted
I20251029 02:51:51.897548 1032 tablet_service.cc:1505] Processing CreateTablet for tablet dd90fb9eb167478cbc3ec9363af86295 (DEFAULT_TABLE table=default.Table_25688635c7a948f1b010aa30b88a3b6c [id=052abf3dcc904bdcb869d5b885723dab]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.897794 755 tablet_service.cc:1505] Processing CreateTablet for tablet dd90fb9eb167478cbc3ec9363af86295 (DEFAULT_TABLE table=default.Table_25688635c7a948f1b010aa30b88a3b6c [id=052abf3dcc904bdcb869d5b885723dab]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.898180 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd90fb9eb167478cbc3ec9363af86295. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.898183 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd90fb9eb167478cbc3ec9363af86295. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.898838 1034 tablet_service.cc:1505] Processing CreateTablet for tablet aacb6f4b9284457bb6607e3a85d5c809 (DEFAULT_TABLE table=default.Table_4c16a32e4f364fecb17b3c79b3a5b98c [id=9f95160639ad40e995a6c89e297a0fad]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.899209 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aacb6f4b9284457bb6607e3a85d5c809. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.900460 1175 ts_tablet_manager.cc:1929] T f04473c3a5f74bdba196b91c2538537e P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:51.900595 1175 log.cc:1199] T f04473c3a5f74bdba196b91c2538537e P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/f04473c3a5f74bdba196b91c2538537e
I20251029 02:51:51.900979 1175 ts_tablet_manager.cc:1950] T f04473c3a5f74bdba196b91c2538537e P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:51:51.906888 892 tablet_service.cc:1505] Processing CreateTablet for tablet c519832030884a049839d309cc738c28 (DEFAULT_TABLE table=default.Table_e95a0439ea534f538683392bbc8e5219 [id=8ad554168c7e4b86a373a3c4579e6211]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.907979 1176 tablet_replica.cc:333] T a0cee0a6eb5a435b9bb3295cbe242b70 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:51:51.908507 1176 raft_consensus.cc:2243] T a0cee0a6eb5a435b9bb3295cbe242b70 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:51.910810 896 tablet_service.cc:1505] Processing CreateTablet for tablet 6f93812578a940a3b7378015e2a46f95 (DEFAULT_TABLE table=default.Table_d84474a10a954e70afe50f3eb8e8c1ed [id=feb1f56ae44249c6adce65637017bb43]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.910889 1176 raft_consensus.cc:2272] T a0cee0a6eb5a435b9bb3295cbe242b70 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:51.912034 1176 ts_tablet_manager.cc:1916] T a0cee0a6eb5a435b9bb3295cbe242b70 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:51.913138 754 tablet_service.cc:1505] Processing CreateTablet for tablet 6f93812578a940a3b7378015e2a46f95 (DEFAULT_TABLE table=default.Table_d84474a10a954e70afe50f3eb8e8c1ed [id=feb1f56ae44249c6adce65637017bb43]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.913530 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f93812578a940a3b7378015e2a46f95. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.918058 1037 tablet_service.cc:1505] Processing CreateTablet for tablet 7c041f4909274cd5a03b9fd78353faf9 (DEFAULT_TABLE table=default.Table_83d3fe1030e0461895fc2437732b4ff8 [id=e11618f2bd094526ad359c65bee6a9ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.921221 1176 ts_tablet_manager.cc:1929] T a0cee0a6eb5a435b9bb3295cbe242b70 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:51.921380 1176 log.cc:1199] T a0cee0a6eb5a435b9bb3295cbe242b70 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/a0cee0a6eb5a435b9bb3295cbe242b70
W20251029 02:51:51.921634 519 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 64b0fb400e944ce2b2c7a49d12135e9c consensus state has no leader
I20251029 02:51:51.921715 1176 ts_tablet_manager.cc:1950] T a0cee0a6eb5a435b9bb3295cbe242b70 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:51:51.923259 750 tablet_service.cc:1505] Processing CreateTablet for tablet d6e1170dbb594a5696a3899cde8da7c9 (DEFAULT_TABLE table=default.Table_4d2c99159e924a06858977465efa38a6 [id=ad583c111f524bb0a40dcf1eb4d5c527]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.923688 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6e1170dbb594a5696a3899cde8da7c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.924345 524 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet cb55b3d5bc63405f96cdc122f5a612cb
I20251029 02:51:51.925284 524 catalog_manager.cc:6719] ReplaceTablet: tablet cb55b3d5bc63405f96cdc122f5a612cb deleted and replaced by tablet aa776cde2f494fa5810539edf9f289a1
I20251029 02:51:51.930109 753 tablet_service.cc:1552] Processing DeleteTablet for tablet cb55b3d5bc63405f96cdc122f5a612cb with delete_type TABLET_DATA_DELETED (replaced by tablet aa776cde2f494fa5810539edf9f289a1) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:51.931145 525 catalog_manager.cc:6194] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: default.Table_64e512f8ce034224aea9b887ed0d09dc [id=7bb8b246fbf1431bb71bbb7c5520e301] alter complete (version 2)
I20251029 02:51:51.931186 1172 tablet_bootstrap.cc:492] T 0dd31dc5873d407d958a0f5199f78782 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:51:51.931272 521 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 5886b8d39f384426bcbed7289ae72e51
I20251029 02:51:51.931317 1172 ts_tablet_manager.cc:1403] T 0dd31dc5873d407d958a0f5199f78782 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.099s user 0.004s sys 0.000s
I20251029 02:51:51.931586 525 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_69bda6d32a204cb6923a73ba4626e96b" } new_table_name: "default.Table_d20d0c82c12a4bba9b0749e62a17c2e5" modify_external_catalogs: true
I20251029 02:51:51.932366 521 master_service.cc:954] ReplaceTablet: received request to replace tablet e6bcea41d5d4480eaf5d6986a93f5e45 from {username='slave'} at 127.0.0.1:35314
I20251029 02:51:51.932266 1172 raft_consensus.cc:359] T 0dd31dc5873d407d958a0f5199f78782 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:51.932463 1172 raft_consensus.cc:385] T 0dd31dc5873d407d958a0f5199f78782 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:51.932577 1172 raft_consensus.cc:740] T 0dd31dc5873d407d958a0f5199f78782 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:51:51.932823 1172 consensus_queue.cc:260] T 0dd31dc5873d407d958a0f5199f78782 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:51.933449 1172 ts_tablet_manager.cc:1434] T 0dd31dc5873d407d958a0f5199f78782 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 02:51:51.933751 1172 tablet_bootstrap.cc:492] T 5886b8d39f384426bcbed7289ae72e51 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:51:51.934088 519 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet ba014ecfec7f4602ae1c09d931bac824 (table default.Table_64e512f8ce034224aea9b887ed0d09dc [id=7bb8b246fbf1431bb71bbb7c5520e301]). Expected version 2 got 0
I20251029 02:51:51.934147 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_8b39acd343e94119804478e17baaa785"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:51.934707 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8b39acd343e94119804478e17baaa785 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:51.934823 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_7df0e7c7f61d486891adb26e68972a1c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:51.935285 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7df0e7c7f61d486891adb26e68972a1c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:51.935467 1172 tablet_bootstrap.cc:654] T 5886b8d39f384426bcbed7289ae72e51 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:51.936206 753 tablet_service.cc:1552] Processing DeleteTablet for tablet 5886b8d39f384426bcbed7289ae72e51 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:51 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:51.937307 518 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet bf210435010d4dde8903fd116eb468e3
I20251029 02:51:51.937711 522 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_469f3d7be86340a2a24565b8a5058d64" } new_table_name: "default.Table_f2bc74933428449197f8c28182c8698f" modify_external_catalogs: true
I20251029 02:51:51.938786 517 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet d7ba1756532c4825a6c7024511dbae51
I20251029 02:51:51.939376 753 tablet_service.cc:1552] Processing DeleteTablet for tablet bf210435010d4dde8903fd116eb468e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:51 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:51.939711 753 tablet_service.cc:1552] Processing DeleteTablet for tablet d7ba1756532c4825a6c7024511dbae51 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:51 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:51.939877 1175 tablet_replica.cc:333] T 42f463dc3cc1434b8ee5ba268cb9fc9a P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:51:51.939905 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet f04473c3a5f74bdba196b91c2538537e (table default.Table_7e818850675e4c5c973db094e7223348 [id=ec8cdbf19343401da05dec0c8c060ae6]) successfully deleted
I20251029 02:51:51.940130 1175 raft_consensus.cc:2243] T 42f463dc3cc1434b8ee5ba268cb9fc9a P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:51.940337 1175 raft_consensus.cc:2272] T 42f463dc3cc1434b8ee5ba268cb9fc9a P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:51.940909 524 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_5b3a7640a9e54e7a9868985322ade0cb" } modify_external_catalogs: true
I20251029 02:51:51.941263 524 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_5b3a7640a9e54e7a9868985322ade0cb" } modify_external_catalogs: true
I20251029 02:51:51.945240 752 tablet_service.cc:1505] Processing CreateTablet for tablet 8d9722c764ac4b27a540595e9c6aafcd (DEFAULT_TABLE table=default.Table_4d5341f07b99483183d6aedc61c9ddcd [id=09fbf31759f047be924a5ef1f0030765]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.945739 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d9722c764ac4b27a540595e9c6aafcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.946420 1172 tablet_bootstrap.cc:492] T 5886b8d39f384426bcbed7289ae72e51 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:51:51.946619 1172 ts_tablet_manager.cc:1403] T 5886b8d39f384426bcbed7289ae72e51 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.004s
I20251029 02:51:51.947469 1172 raft_consensus.cc:359] T 5886b8d39f384426bcbed7289ae72e51 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:51.947706 1172 raft_consensus.cc:385] T 5886b8d39f384426bcbed7289ae72e51 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:51.947793 1172 raft_consensus.cc:740] T 5886b8d39f384426bcbed7289ae72e51 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:51:51.948067 1172 consensus_queue.cc:260] T 5886b8d39f384426bcbed7289ae72e51 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:51.948640 523 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet e19c8b10f32849f5885fc00176c8bc12
I20251029 02:51:51.950078 894 tablet_service.cc:1552] Processing DeleteTablet for tablet cb55b3d5bc63405f96cdc122f5a612cb with delete_type TABLET_DATA_DELETED (replaced by tablet aa776cde2f494fa5810539edf9f289a1) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:51.950206 1172 ts_tablet_manager.cc:1434] T 5886b8d39f384426bcbed7289ae72e51 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251029 02:51:51.950559 1172 tablet_bootstrap.cc:492] T 9be9add0274d41b688edd6292a253a06 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:51:51.950578 898 tablet_service.cc:1505] Processing CreateTablet for tablet d6e1170dbb594a5696a3899cde8da7c9 (DEFAULT_TABLE table=default.Table_4d2c99159e924a06858977465efa38a6 [id=ad583c111f524bb0a40dcf1eb4d5c527]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.950934 895 tablet_service.cc:1505] Processing CreateTablet for tablet dd90fb9eb167478cbc3ec9363af86295 (DEFAULT_TABLE table=default.Table_25688635c7a948f1b010aa30b88a3b6c [id=052abf3dcc904bdcb869d5b885723dab]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 02:51:51.951161 525 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 14698af6348f41e384f10569392d9b23 has no consensus state
I20251029 02:51:51.953310 893 tablet_service.cc:1505] Processing CreateTablet for tablet 8d9722c764ac4b27a540595e9c6aafcd (DEFAULT_TABLE table=default.Table_4d5341f07b99483183d6aedc61c9ddcd [id=09fbf31759f047be924a5ef1f0030765]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.954236 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aacb6f4b9284457bb6607e3a85d5c809. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.954352 1172 tablet_bootstrap.cc:654] T 9be9add0274d41b688edd6292a253a06 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:51.954737 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6e1170dbb594a5696a3899cde8da7c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.956159 754 tablet_service.cc:1505] Processing CreateTablet for tablet 6c22497760274380a832feff7c5a2fae (DEFAULT_TABLE table=default.Table_9becb486767e4662a93114d2242f031d [id=c2475b97464f45da8990e004660c42b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.956629 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c22497760274380a832feff7c5a2fae. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.957485 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c519832030884a049839d309cc738c28. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.959010 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f93812578a940a3b7378015e2a46f95. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.959254 753 tablet_service.cc:1505] Processing CreateTablet for tablet f71388036478404d878a52fc20ece216 (DEFAULT_TABLE table=default.Table_03f7d6f02b064939ba7464c5173e02e3 [id=9898cffc2b964059ba3163ee4d1395c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.959369 1174 ts_tablet_manager.cc:1916] T 94854145ac0d4be38780d4cdb721725e P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:51.959661 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f71388036478404d878a52fc20ece216. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.962810 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd90fb9eb167478cbc3ec9363af86295. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.953359 1167 tablet_bootstrap.cc:654] T 8b89ee9017b244c5b866b1fc7154ffe3 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:51.962996 1175 ts_tablet_manager.cc:1916] T 42f463dc3cc1434b8ee5ba268cb9fc9a P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:51.967492 1172 tablet_bootstrap.cc:492] T 9be9add0274d41b688edd6292a253a06 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:51:51.967658 1172 ts_tablet_manager.cc:1403] T 9be9add0274d41b688edd6292a253a06 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.002s
I20251029 02:51:51.968469 1172 raft_consensus.cc:359] T 9be9add0274d41b688edd6292a253a06 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:51.968672 1172 raft_consensus.cc:385] T 9be9add0274d41b688edd6292a253a06 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:51.968760 1172 raft_consensus.cc:740] T 9be9add0274d41b688edd6292a253a06 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:51:51.970463 521 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet e6bcea41d5d4480eaf5d6986a93f5e45
I20251029 02:51:51.970773 1172 consensus_queue.cc:260] T 9be9add0274d41b688edd6292a253a06 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:51.970968 521 catalog_manager.cc:6719] ReplaceTablet: tablet e6bcea41d5d4480eaf5d6986a93f5e45 deleted and replaced by tablet 96b5703bc52c41af9296ab96faa5e507
I20251029 02:51:51.970925 755 tablet_service.cc:1505] Processing CreateTablet for tablet c519832030884a049839d309cc738c28 (DEFAULT_TABLE table=default.Table_e95a0439ea534f538683392bbc8e5219 [id=8ad554168c7e4b86a373a3c4579e6211]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:51.972210 749 tablet_service.cc:1552] Processing DeleteTablet for tablet e6bcea41d5d4480eaf5d6986a93f5e45 with delete_type TABLET_DATA_DELETED (replaced by tablet 96b5703bc52c41af9296ab96faa5e507) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:51.973021 1172 ts_tablet_manager.cc:1434] T 9be9add0274d41b688edd6292a253a06 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251029 02:51:51.973383 1172 tablet_bootstrap.cc:492] T deca0c4a114441079fce22e74b403050 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:51:51.975219 1172 tablet_bootstrap.cc:654] T deca0c4a114441079fce22e74b403050 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:51.977475 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d9722c764ac4b27a540595e9c6aafcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.980984 1172 tablet_bootstrap.cc:492] T deca0c4a114441079fce22e74b403050 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:51:51.981155 1172 ts_tablet_manager.cc:1403] T deca0c4a114441079fce22e74b403050 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20251029 02:51:51.981937 1172 raft_consensus.cc:359] T deca0c4a114441079fce22e74b403050 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:51.982141 1172 raft_consensus.cc:385] T deca0c4a114441079fce22e74b403050 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:51.982219 1172 raft_consensus.cc:740] T deca0c4a114441079fce22e74b403050 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:51:51.984279 1172 consensus_queue.cc:260] T deca0c4a114441079fce22e74b403050 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:51.985939 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c519832030884a049839d309cc738c28. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:51.986596 1172 ts_tablet_manager.cc:1434] T deca0c4a114441079fce22e74b403050 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251029 02:51:51.986910 1172 tablet_bootstrap.cc:492] T 11f4252b5e694312aaa142468f8e68b4 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:51:51.988684 1172 tablet_bootstrap.cc:654] T 11f4252b5e694312aaa142468f8e68b4 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:51.994590 1167 tablet_bootstrap.cc:492] T 8b89ee9017b244c5b866b1fc7154ffe3 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:51:51.994767 1167 ts_tablet_manager.cc:1403] T 8b89ee9017b244c5b866b1fc7154ffe3 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.228s user 0.005s sys 0.000s
I20251029 02:51:51.995723 1167 raft_consensus.cc:359] T 8b89ee9017b244c5b866b1fc7154ffe3 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:51.995944 1167 raft_consensus.cc:385] T 8b89ee9017b244c5b866b1fc7154ffe3 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:51.996055 1167 raft_consensus.cc:740] T 8b89ee9017b244c5b866b1fc7154ffe3 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:51:51.996306 1167 consensus_queue.cc:260] T 8b89ee9017b244c5b866b1fc7154ffe3 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
W20251029 02:51:51.997234 512 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.60:41233, user_credentials={real_user=slave}} blocked reactor thread for 67271us
I20251029 02:51:52.000969 894 tablet_service.cc:1505] Processing CreateTablet for tablet f71388036478404d878a52fc20ece216 (DEFAULT_TABLE table=default.Table_03f7d6f02b064939ba7464c5173e02e3 [id=9898cffc2b964059ba3163ee4d1395c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.001475 898 tablet_service.cc:1505] Processing CreateTablet for tablet 6c22497760274380a832feff7c5a2fae (DEFAULT_TABLE table=default.Table_9becb486767e4662a93114d2242f031d [id=c2475b97464f45da8990e004660c42b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.001852 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f71388036478404d878a52fc20ece216. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.001856 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c041f4909274cd5a03b9fd78353faf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.007225 1032 tablet_service.cc:1505] Processing CreateTablet for tablet c519832030884a049839d309cc738c28 (DEFAULT_TABLE table=default.Table_e95a0439ea534f538683392bbc8e5219 [id=8ad554168c7e4b86a373a3c4579e6211]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.007341 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c22497760274380a832feff7c5a2fae. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.007629 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c519832030884a049839d309cc738c28. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.009903 896 tablet_service.cc:1552] Processing DeleteTablet for tablet d7ba1756532c4825a6c7024511dbae51 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:51 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:52.009950 1175 ts_tablet_manager.cc:1929] T 42f463dc3cc1434b8ee5ba268cb9fc9a P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:52.010100 1175 log.cc:1199] T 42f463dc3cc1434b8ee5ba268cb9fc9a P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/42f463dc3cc1434b8ee5ba268cb9fc9a
I20251029 02:51:52.010432 1175 ts_tablet_manager.cc:1950] T 42f463dc3cc1434b8ee5ba268cb9fc9a P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:51:52.010848 1039 tablet_service.cc:1505] Processing CreateTablet for tablet 6f93812578a940a3b7378015e2a46f95 (DEFAULT_TABLE table=default.Table_d84474a10a954e70afe50f3eb8e8c1ed [id=feb1f56ae44249c6adce65637017bb43]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.011193 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f93812578a940a3b7378015e2a46f95. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.012022 895 tablet_service.cc:1552] Processing DeleteTablet for tablet bf210435010d4dde8903fd116eb468e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:51 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:52.012357 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet cb55b3d5bc63405f96cdc122f5a612cb with delete_type TABLET_DATA_DELETED (replaced by tablet aa776cde2f494fa5810539edf9f289a1) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:52.013192 1040 tablet_service.cc:1552] Processing DeleteTablet for tablet 5886b8d39f384426bcbed7289ae72e51 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:51 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:52.013579 1035 tablet_service.cc:1505] Processing CreateTablet for tablet 8d9722c764ac4b27a540595e9c6aafcd (DEFAULT_TABLE table=default.Table_4d5341f07b99483183d6aedc61c9ddcd [id=09fbf31759f047be924a5ef1f0030765]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.013921 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d9722c764ac4b27a540595e9c6aafcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.014808 1031 tablet_service.cc:1505] Processing CreateTablet for tablet d6e1170dbb594a5696a3899cde8da7c9 (DEFAULT_TABLE table=default.Table_4d2c99159e924a06858977465efa38a6 [id=ad583c111f524bb0a40dcf1eb4d5c527]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.015239 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6e1170dbb594a5696a3899cde8da7c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.016227 895 tablet_service.cc:1552] Processing DeleteTablet for tablet e6bcea41d5d4480eaf5d6986a93f5e45 with delete_type TABLET_DATA_DELETED (replaced by tablet 96b5703bc52c41af9296ab96faa5e507) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:52.017212 1040 tablet_service.cc:1552] Processing DeleteTablet for tablet bf210435010d4dde8903fd116eb468e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:51 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:52.018009 1040 tablet_service.cc:1552] Processing DeleteTablet for tablet d7ba1756532c4825a6c7024511dbae51 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:51 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:52.019830 776 raft_consensus.cc:1275] T 55c1b243d8a94938a5caa58754092cec P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Refusing update from remote peer 0cfd4bee16f84b7c8b0ff3ce14db20d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251029 02:51:52.019953 920 raft_consensus.cc:3060] T 55c1b243d8a94938a5caa58754092cec P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:51:52.019923 1038 tablet_service.cc:1505] Processing CreateTablet for tablet f71388036478404d878a52fc20ece216 (DEFAULT_TABLE table=default.Table_03f7d6f02b064939ba7464c5173e02e3 [id=9898cffc2b964059ba3163ee4d1395c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.020305 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f71388036478404d878a52fc20ece216. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.020320 1040 tablet_service.cc:1505] Processing CreateTablet for tablet 6c22497760274380a832feff7c5a2fae (DEFAULT_TABLE table=default.Table_9becb486767e4662a93114d2242f031d [id=c2475b97464f45da8990e004660c42b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.022301 1250 consensus_queue.cc:1048] T 55c1b243d8a94938a5caa58754092cec P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251029 02:51:52.022392 1030 tablet_service.cc:1552] Processing DeleteTablet for tablet e6bcea41d5d4480eaf5d6986a93f5e45 with delete_type TABLET_DATA_DELETED (replaced by tablet 96b5703bc52c41af9296ab96faa5e507) from {username='slave'} at 127.0.0.1:37626
W20251029 02:51:52.023519 522 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5e31e061d74146009d24d5cd57487a5e consensus state has no leader
I20251029 02:51:52.023875 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c22497760274380a832feff7c5a2fae. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.025166 920 raft_consensus.cc:1275] T 55c1b243d8a94938a5caa58754092cec P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Refusing update from remote peer 0cfd4bee16f84b7c8b0ff3ce14db20d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20251029 02:51:52.026206 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 42f463dc3cc1434b8ee5ba268cb9fc9a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 42f463dc3cc1434b8ee5ba268cb9fc9a already in progress: creating tablet
I20251029 02:51:52.026669 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet a0cee0a6eb5a435b9bb3295cbe242b70 (table default.Table_9becb486767e4662a93114d2242f031d [id=c2475b97464f45da8990e004660c42b3]) successfully deleted
W20251029 02:51:52.027287 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet cb55b3d5bc63405f96cdc122f5a612cb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cb55b3d5bc63405f96cdc122f5a612cb already in progress: creating tablet
W20251029 02:51:52.027515 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 5886b8d39f384426bcbed7289ae72e51 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5886b8d39f384426bcbed7289ae72e51 already in progress: creating tablet
W20251029 02:51:52.027740 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet bf210435010d4dde8903fd116eb468e3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bf210435010d4dde8903fd116eb468e3 already in progress: creating tablet
I20251029 02:51:52.027843 1250 consensus_queue.cc:1048] T 55c1b243d8a94938a5caa58754092cec P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20251029 02:51:52.028405 892 tablet_service.cc:1552] Processing DeleteTablet for tablet 5886b8d39f384426bcbed7289ae72e51 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:51 UTC) from {username='slave'} at 127.0.0.1:35702
W20251029 02:51:52.027926 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet d7ba1756532c4825a6c7024511dbae51 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d7ba1756532c4825a6c7024511dbae51 already in progress: creating tablet
W20251029 02:51:52.032526 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet e6bcea41d5d4480eaf5d6986a93f5e45 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e6bcea41d5d4480eaf5d6986a93f5e45 already in progress: creating tablet
I20251029 02:51:52.033227 524 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 96b5703bc52c41af9296ab96faa5e507
I20251029 02:51:52.033303 522 master_service.cc:954] ReplaceTablet: received request to replace tablet c4884e22cf2044a4a1070c1b4b759d9e from {username='slave'} at 127.0.0.1:35314
I20251029 02:51:52.039543 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_a5eff5a49d424fa89b57d375abc4d155"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:52.040170 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a5eff5a49d424fa89b57d375abc4d155 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:52.042800 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_8d5ec842f38f4e6f9c387ff7126ce94d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:52.043334 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8d5ec842f38f4e6f9c387ff7126ce94d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:52.043937 1167 ts_tablet_manager.cc:1434] T 8b89ee9017b244c5b866b1fc7154ffe3 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.049s user 0.002s sys 0.000s
I20251029 02:51:52.045464 1167 tablet_bootstrap.cc:492] T 8f7c746f8b6948b58676298c2c2ea221 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:51:52.047917 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_fd85e1b0ff874cdc985600a3fa32085c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:52.048475 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fd85e1b0ff874cdc985600a3fa32085c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:52.048882 1174 ts_tablet_manager.cc:1929] T 94854145ac0d4be38780d4cdb721725e P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:52.049022 1174 log.cc:1199] T 94854145ac0d4be38780d4cdb721725e P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/94854145ac0d4be38780d4cdb721725e
I20251029 02:51:52.049459 1174 ts_tablet_manager.cc:1950] T 94854145ac0d4be38780d4cdb721725e P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:51:52.050153 525 master_service.cc:954] ReplaceTablet: received request to replace tablet 64b0fb400e944ce2b2c7a49d12135e9c from {username='slave'} at 127.0.0.1:35314
I20251029 02:51:52.050441 523 master_service.cc:954] ReplaceTablet: received request to replace tablet d32532973b3f46dd8158a2ce42d26365 from {username='slave'} at 127.0.0.1:35314
I20251029 02:51:52.050459 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_50252b3b3cc4403da5e259f9f91be915"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:52.050858 517 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_a80bdae756454abe89c1b71249975f9e" } new_table_name: "default.Table_14156dbda95043dcb54dd316874be455" modify_external_catalogs: true
W20251029 02:51:52.051257 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_50252b3b3cc4403da5e259f9f91be915 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:52.052646 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_3fae9f42aaba465bb13918661650e73a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:52.053129 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3fae9f42aaba465bb13918661650e73a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:52.057014 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 42f463dc3cc1434b8ee5ba268cb9fc9a (table default.Table_03f7d6f02b064939ba7464c5173e02e3 [id=9898cffc2b964059ba3163ee4d1395c2]) successfully deleted
I20251029 02:51:52.057108 1175 tablet_replica.cc:333] T 2ebde39f813d46e898ad1754325ad1a0 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:51:52.057085 1283 tablet.cc:1722] T 55c1b243d8a94938a5caa58754092cec P c756528642a543eeacd93cef3e335a7d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 02:51:52.057286 1175 raft_consensus.cc:2243] T 2ebde39f813d46e898ad1754325ad1a0 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:52.057475 1175 raft_consensus.cc:2272] T 2ebde39f813d46e898ad1754325ad1a0 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:52.058338 1274 tablet.cc:1722] T 55c1b243d8a94938a5caa58754092cec P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 02:51:52.062727 519 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet 8f7c746f8b6948b58676298c2c2ea221 (table default.Table_9aac534a15054a30b80745984c279e62 [id=669ae9dd0a5749b1aa3557ab8183a057]). Expected version 3 got 0
I20251029 02:51:52.063125 1280 tablet.cc:1722] T 55c1b243d8a94938a5caa58754092cec P b495a49349324f7d81908f2da700f0d9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 02:51:52.064034 1175 ts_tablet_manager.cc:1916] T 2ebde39f813d46e898ad1754325ad1a0 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:52.064316 1167 tablet_bootstrap.cc:654] T 8f7c746f8b6948b58676298c2c2ea221 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:52.066128 894 tablet_service.cc:1505] Processing CreateTablet for tablet fd900eec3f0c43bf965c148ea90870a3 (DEFAULT_TABLE table=default.Table_e0e6e47b18d749e8ae80f822af1e23f0 [id=125ae5c7b5074378bcecb546e70f0c13]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.066962 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd900eec3f0c43bf965c148ea90870a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.071621 898 tablet_service.cc:1505] Processing CreateTablet for tablet 14698af6348f41e384f10569392d9b23 (DEFAULT_TABLE table=default.Table_d20d0c82c12a4bba9b0749e62a17c2e5 [id=7de93d6d48ab4d58b5aed9319d7299a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.072095 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14698af6348f41e384f10569392d9b23. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.074834 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_40421c4fdf1d42be8ed6114e0b77f1f9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:52.075505 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_40421c4fdf1d42be8ed6114e0b77f1f9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:52.075768 519 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet 9be9add0274d41b688edd6292a253a06 (table default.Table_c7daf9eda79e470186f5774de2b824d3 [id=f45ce142f37f4febbceb4e568e35e0f2]). Expected version 1 got 0
W20251029 02:51:52.079011 517 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 11f4252b5e694312aaa142468f8e68b4 consensus state has no leader
I20251029 02:51:52.089490 752 tablet_service.cc:1552] Processing DeleteTablet for tablet 0dd31dc5873d407d958a0f5199f78782 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:51 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:52.090046 754 tablet_service.cc:1505] Processing CreateTablet for tablet fd900eec3f0c43bf965c148ea90870a3 (DEFAULT_TABLE table=default.Table_e0e6e47b18d749e8ae80f822af1e23f0 [id=125ae5c7b5074378bcecb546e70f0c13]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.090570 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd900eec3f0c43bf965c148ea90870a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.092473 1172 tablet_bootstrap.cc:492] T 11f4252b5e694312aaa142468f8e68b4 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:51:52.092660 1172 ts_tablet_manager.cc:1403] T 11f4252b5e694312aaa142468f8e68b4 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.106s user 0.005s sys 0.005s
I20251029 02:51:52.093469 1172 raft_consensus.cc:359] T 11f4252b5e694312aaa142468f8e68b4 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:52.093694 1172 raft_consensus.cc:385] T 11f4252b5e694312aaa142468f8e68b4 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:52.093793 1172 raft_consensus.cc:740] T 11f4252b5e694312aaa142468f8e68b4 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:51:52.094017 1172 consensus_queue.cc:260] T 11f4252b5e694312aaa142468f8e68b4 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:52.100809 752 tablet_service.cc:1505] Processing CreateTablet for tablet 14698af6348f41e384f10569392d9b23 (DEFAULT_TABLE table=default.Table_d20d0c82c12a4bba9b0749e62a17c2e5 [id=7de93d6d48ab4d58b5aed9319d7299a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.101339 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14698af6348f41e384f10569392d9b23. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.102084 900 tablet_service.cc:1505] Processing CreateTablet for tablet 012ce243fd07466ab5301d1c9ce4b413 (DEFAULT_TABLE table=default.Table_cb594ce4077d4d02aa70ef94d5f801cc [id=b792f15b5ea34e44b32b3087b983567c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.102581 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 012ce243fd07466ab5301d1c9ce4b413. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.110687 751 tablet_service.cc:1505] Processing CreateTablet for tablet 012ce243fd07466ab5301d1c9ce4b413 (DEFAULT_TABLE table=default.Table_cb594ce4077d4d02aa70ef94d5f801cc [id=b792f15b5ea34e44b32b3087b983567c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.111040 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 94854145ac0d4be38780d4cdb721725e (table default.Table_631cf79ec867461baecbc486773b88c4 [id=7a493d6672624c90a81a647d7785181c]) successfully deleted
I20251029 02:51:52.111266 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 012ce243fd07466ab5301d1c9ce4b413. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:51:52.112211 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet cb55b3d5bc63405f96cdc122f5a612cb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cb55b3d5bc63405f96cdc122f5a612cb already in progress: creating tablet
I20251029 02:51:52.116767 1038 tablet_service.cc:1505] Processing CreateTablet for tablet 012ce243fd07466ab5301d1c9ce4b413 (DEFAULT_TABLE table=default.Table_cb594ce4077d4d02aa70ef94d5f801cc [id=b792f15b5ea34e44b32b3087b983567c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.117345 1172 ts_tablet_manager.cc:1434] T 11f4252b5e694312aaa142468f8e68b4 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.023s user 0.001s sys 0.001s
I20251029 02:51:52.113471 1037 tablet_service.cc:1505] Processing CreateTablet for tablet 14698af6348f41e384f10569392d9b23 (DEFAULT_TABLE table=default.Table_d20d0c82c12a4bba9b0749e62a17c2e5 [id=7de93d6d48ab4d58b5aed9319d7299a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.117517 1040 tablet_service.cc:1505] Processing CreateTablet for tablet 75236b2fa82f4f28b82efae9a2c5160d (DEFAULT_TABLE table=default.Table_0b84a2f666074fd18085ff4ce249a2fe [id=715a1df4e483412da9a63eca19b8f02d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.117728 1172 tablet_bootstrap.cc:492] T 6c32c6f5e1cc40a0a12688201cac62a7 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:51:52.117988 1031 tablet_service.cc:1505] Processing CreateTablet for tablet aa776cde2f494fa5810539edf9f289a1 (DEFAULT_TABLE table=default.Table_47a1a59db0444fa49f7274509385ff0c [id=283c09ba60114ba9b165e2ff144a6c35]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.119149 1033 tablet_service.cc:1505] Processing CreateTablet for tablet fd900eec3f0c43bf965c148ea90870a3 (DEFAULT_TABLE table=default.Table_e0e6e47b18d749e8ae80f822af1e23f0 [id=125ae5c7b5074378bcecb546e70f0c13]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.119670 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 012ce243fd07466ab5301d1c9ce4b413. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.119935 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75236b2fa82f4f28b82efae9a2c5160d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.120213 900 tablet_service.cc:1505] Processing CreateTablet for tablet 75236b2fa82f4f28b82efae9a2c5160d (DEFAULT_TABLE table=default.Table_0b84a2f666074fd18085ff4ce249a2fe [id=715a1df4e483412da9a63eca19b8f02d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.120522 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa776cde2f494fa5810539edf9f289a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.120641 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75236b2fa82f4f28b82efae9a2c5160d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.121676 755 tablet_service.cc:1505] Processing CreateTablet for tablet aa776cde2f494fa5810539edf9f289a1 (DEFAULT_TABLE table=default.Table_47a1a59db0444fa49f7274509385ff0c [id=283c09ba60114ba9b165e2ff144a6c35]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.122283 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa776cde2f494fa5810539edf9f289a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.124024 749 tablet_service.cc:1505] Processing CreateTablet for tablet 75236b2fa82f4f28b82efae9a2c5160d (DEFAULT_TABLE table=default.Table_0b84a2f666074fd18085ff4ce249a2fe [id=715a1df4e483412da9a63eca19b8f02d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.124176 1167 tablet_bootstrap.cc:492] T 8f7c746f8b6948b58676298c2c2ea221 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:51:52.124342 1167 ts_tablet_manager.cc:1403] T 8f7c746f8b6948b58676298c2c2ea221 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.079s user 0.002s sys 0.001s
I20251029 02:51:52.124534 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75236b2fa82f4f28b82efae9a2c5160d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.125447 523 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet d32532973b3f46dd8158a2ce42d26365
I20251029 02:51:52.125903 523 catalog_manager.cc:6719] ReplaceTablet: tablet d32532973b3f46dd8158a2ce42d26365 deleted and replaced by tablet 65fdec7cf5d143208d78d4c3806b1352
I20251029 02:51:52.126116 893 tablet_service.cc:1552] Processing DeleteTablet for tablet d32532973b3f46dd8158a2ce42d26365 with delete_type TABLET_DATA_DELETED (replaced by tablet 65fdec7cf5d143208d78d4c3806b1352) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:52.126410 1167 raft_consensus.cc:359] T 8f7c746f8b6948b58676298c2c2ea221 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:52.126652 1167 raft_consensus.cc:385] T 8f7c746f8b6948b58676298c2c2ea221 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:52.126741 1167 raft_consensus.cc:740] T 8f7c746f8b6948b58676298c2c2ea221 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:51:52.127002 1167 consensus_queue.cc:260] T 8f7c746f8b6948b58676298c2c2ea221 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:52.127183 753 tablet_service.cc:1552] Processing DeleteTablet for tablet d32532973b3f46dd8158a2ce42d26365 with delete_type TABLET_DATA_DELETED (replaced by tablet 65fdec7cf5d143208d78d4c3806b1352) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:52.129035 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet d32532973b3f46dd8158a2ce42d26365 with delete_type TABLET_DATA_DELETED (replaced by tablet 65fdec7cf5d143208d78d4c3806b1352) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:52.129670 1167 ts_tablet_manager.cc:1434] T 8f7c746f8b6948b58676298c2c2ea221 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
W20251029 02:51:52.130146 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet d32532973b3f46dd8158a2ce42d26365 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d32532973b3f46dd8158a2ce42d26365 already in progress: creating tablet
I20251029 02:51:52.133443 893 tablet_service.cc:1505] Processing CreateTablet for tablet d2990e9f626c40618c118266bc595684 (DEFAULT_TABLE table=default.Table_75949df235e044029e69b130e96b8d27 [id=f733948e92d8470da9d8050bd26a06b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.133916 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2990e9f626c40618c118266bc595684. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.136124 525 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 64b0fb400e944ce2b2c7a49d12135e9c
I20251029 02:51:52.136572 525 catalog_manager.cc:6719] ReplaceTablet: tablet 64b0fb400e944ce2b2c7a49d12135e9c deleted and replaced by tablet 9e07291d6b5645feb4d5890f58e1ef74
I20251029 02:51:52.137773 753 tablet_service.cc:1552] Processing DeleteTablet for tablet 64b0fb400e944ce2b2c7a49d12135e9c with delete_type TABLET_DATA_DELETED (replaced by tablet 9e07291d6b5645feb4d5890f58e1ef74) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:52.138152 1174 tablet_replica.cc:333] T e6bcea41d5d4480eaf5d6986a93f5e45 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:51:52.138545 1174 raft_consensus.cc:2243] T e6bcea41d5d4480eaf5d6986a93f5e45 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:52.138868 1174 raft_consensus.cc:2272] T e6bcea41d5d4480eaf5d6986a93f5e45 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus is shut down!
W20251029 02:51:52.139235 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet d7ba1756532c4825a6c7024511dbae51 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d7ba1756532c4825a6c7024511dbae51 already in progress: creating tablet
I20251029 02:51:52.139397 1166 tablet_bootstrap.cc:492] T 42f463dc3cc1434b8ee5ba268cb9fc9a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
W20251029 02:51:52.139523 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet bf210435010d4dde8903fd116eb468e3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bf210435010d4dde8903fd116eb468e3 already in progress: creating tablet
I20251029 02:51:52.139855 1174 ts_tablet_manager.cc:1916] T e6bcea41d5d4480eaf5d6986a93f5e45 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:52.140223 892 tablet_service.cc:1552] Processing DeleteTablet for tablet 64b0fb400e944ce2b2c7a49d12135e9c with delete_type TABLET_DATA_DELETED (replaced by tablet 9e07291d6b5645feb4d5890f58e1ef74) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:52.142642 1167 tablet_bootstrap.cc:492] T f04473c3a5f74bdba196b91c2538537e P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:51:52.142643 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd900eec3f0c43bf965c148ea90870a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.142972 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14698af6348f41e384f10569392d9b23. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.142454 1166 ts_tablet_manager.cc:1403] T 42f463dc3cc1434b8ee5ba268cb9fc9a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.274s user 0.009s sys 0.031s
I20251029 02:51:52.144029 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet 64b0fb400e944ce2b2c7a49d12135e9c with delete_type TABLET_DATA_DELETED (replaced by tablet 9e07291d6b5645feb4d5890f58e1ef74) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:52.145376 522 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet c4884e22cf2044a4a1070c1b4b759d9e
I20251029 02:51:52.145892 522 catalog_manager.cc:6719] ReplaceTablet: tablet c4884e22cf2044a4a1070c1b4b759d9e deleted and replaced by tablet 9964ad774890422f8e14f6b99f28cb55
I20251029 02:51:52.146970 892 tablet_service.cc:1552] Processing DeleteTablet for tablet c4884e22cf2044a4a1070c1b4b759d9e with delete_type TABLET_DATA_DELETED (replaced by tablet 9964ad774890422f8e14f6b99f28cb55) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:52.147861 898 tablet_service.cc:1505] Processing CreateTablet for tablet aa776cde2f494fa5810539edf9f289a1 (DEFAULT_TABLE table=default.Table_47a1a59db0444fa49f7274509385ff0c [id=283c09ba60114ba9b165e2ff144a6c35]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.148329 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa776cde2f494fa5810539edf9f289a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.155326 1167 tablet_bootstrap.cc:654] T f04473c3a5f74bdba196b91c2538537e P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:52.155649 1175 ts_tablet_manager.cc:1929] T 2ebde39f813d46e898ad1754325ad1a0 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:52.155730 749 tablet_service.cc:1552] Processing DeleteTablet for tablet c4884e22cf2044a4a1070c1b4b759d9e with delete_type TABLET_DATA_DELETED (replaced by tablet 9964ad774890422f8e14f6b99f28cb55) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:52.156139 753 tablet_service.cc:1505] Processing CreateTablet for tablet d2990e9f626c40618c118266bc595684 (DEFAULT_TABLE table=default.Table_75949df235e044029e69b130e96b8d27 [id=f733948e92d8470da9d8050bd26a06b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.155823 1175 log.cc:1199] T 2ebde39f813d46e898ad1754325ad1a0 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/2ebde39f813d46e898ad1754325ad1a0
I20251029 02:51:52.157047 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2990e9f626c40618c118266bc595684. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.159363 1166 raft_consensus.cc:359] T 42f463dc3cc1434b8ee5ba268cb9fc9a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:52.159566 1166 raft_consensus.cc:385] T 42f463dc3cc1434b8ee5ba268cb9fc9a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:52.159570 1179 raft_consensus.cc:493] T e6bcea41d5d4480eaf5d6986a93f5e45 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:52.159704 1179 raft_consensus.cc:515] T e6bcea41d5d4480eaf5d6986a93f5e45 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:52.159962 1166 raft_consensus.cc:740] T 42f463dc3cc1434b8ee5ba268cb9fc9a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:51:52.159677 1038 tablet_service.cc:1505] Processing CreateTablet for tablet d2990e9f626c40618c118266bc595684 (DEFAULT_TABLE table=default.Table_75949df235e044029e69b130e96b8d27 [id=f733948e92d8470da9d8050bd26a06b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.160471 1179 leader_election.cc:290] T e6bcea41d5d4480eaf5d6986a93f5e45 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:51:52.160905 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e6bcea41d5d4480eaf5d6986a93f5e45" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:51:52.162001 711 leader_election.cc:343] T e6bcea41d5d4480eaf5d6986a93f5e45 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: Tablet not RUNNING: STOPPED
I20251029 02:51:52.162333 1166 consensus_queue.cc:260] T 42f463dc3cc1434b8ee5ba268cb9fc9a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:52.166728 1175 ts_tablet_manager.cc:1950] T 2ebde39f813d46e898ad1754325ad1a0 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:51:52.166721 1062 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e6bcea41d5d4480eaf5d6986a93f5e45" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:51:52.167677 1175 tablet_replica.cc:333] T deca0c4a114441079fce22e74b403050 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:51:52.167907 1175 raft_consensus.cc:2243] T deca0c4a114441079fce22e74b403050 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:52.168076 1175 raft_consensus.cc:2272] T deca0c4a114441079fce22e74b403050 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:52.168941 1175 ts_tablet_manager.cc:1916] T deca0c4a114441079fce22e74b403050 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:52.169754 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 2ebde39f813d46e898ad1754325ad1a0 (table default.Table_1a524bb5f82a4041b8008ff91a0b9d27 [id=5db5ea0ab3e249f4acea117bb07187f4]) successfully deleted
W20251029 02:51:52.174783 712 leader_election.cc:343] T e6bcea41d5d4480eaf5d6986a93f5e45 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:52.174940 712 leader_election.cc:304] T e6bcea41d5d4480eaf5d6986a93f5e45 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:51:52.175278 1166 ts_tablet_manager.cc:1434] T 42f463dc3cc1434b8ee5ba268cb9fc9a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.020s user 0.002s sys 0.000s
I20251029 02:51:52.175436 1179 raft_consensus.cc:2749] T e6bcea41d5d4480eaf5d6986a93f5e45 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:51:52.175879 1172 tablet_bootstrap.cc:654] T 6c32c6f5e1cc40a0a12688201cac62a7 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:52.178385 1175 ts_tablet_manager.cc:1929] T deca0c4a114441079fce22e74b403050 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:52.179040 1166 tablet_bootstrap.cc:492] T e6bcea41d5d4480eaf5d6986a93f5e45 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:51:52.179205 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2990e9f626c40618c118266bc595684. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.181238 1174 ts_tablet_manager.cc:1929] T e6bcea41d5d4480eaf5d6986a93f5e45 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:52.181428 1174 log.cc:1199] T e6bcea41d5d4480eaf5d6986a93f5e45 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/e6bcea41d5d4480eaf5d6986a93f5e45
I20251029 02:51:52.181828 1174 ts_tablet_manager.cc:1950] T e6bcea41d5d4480eaf5d6986a93f5e45 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:51:52.183348 1179 raft_consensus.cc:493] T d32532973b3f46dd8158a2ce42d26365 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:52.186698 1175 log.cc:1199] T deca0c4a114441079fce22e74b403050 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/deca0c4a114441079fce22e74b403050
I20251029 02:51:52.186698 1179 raft_consensus.cc:515] T d32532973b3f46dd8158a2ce42d26365 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:52.187173 1175 ts_tablet_manager.cc:1950] T deca0c4a114441079fce22e74b403050 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:51:52.189550 1062 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d32532973b3f46dd8158a2ce42d26365" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:51:52.190806 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d32532973b3f46dd8158a2ce42d26365" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:51:52.191382 519 catalog_manager.cc:6194] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: default.Table_953f59050dc54b04b57c06364595c612 [id=bd2335c28dd44b5584ab71694cebda2d] alter complete (version 2)
I20251029 02:51:52.191542 1179 leader_election.cc:290] T d32532973b3f46dd8158a2ce42d26365 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:51:52.198041 1166 tablet_bootstrap.cc:654] T e6bcea41d5d4480eaf5d6986a93f5e45 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:52.198750 898 tablet_service.cc:1552] Processing DeleteTablet for tablet 8b89ee9017b244c5b866b1fc7154ffe3 with delete_type TABLET_DATA_DELETED (replaced by tablet dd90fb9eb167478cbc3ec9363af86295) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:52.199798 1040 tablet_service.cc:1552] Processing DeleteTablet for tablet c4884e22cf2044a4a1070c1b4b759d9e with delete_type TABLET_DATA_DELETED (replaced by tablet 9964ad774890422f8e14f6b99f28cb55) from {username='slave'} at 127.0.0.1:37626
W20251029 02:51:52.202051 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 64b0fb400e944ce2b2c7a49d12135e9c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 64b0fb400e944ce2b2c7a49d12135e9c already in progress: creating tablet
W20251029 02:51:52.202715 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet c4884e22cf2044a4a1070c1b4b759d9e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c4884e22cf2044a4a1070c1b4b759d9e already in progress: creating tablet
I20251029 02:51:52.207326 524 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet 8f7c746f8b6948b58676298c2c2ea221 (table default.Table_9aac534a15054a30b80745984c279e62 [id=669ae9dd0a5749b1aa3557ab8183a057]). Expected version 3 got 0
I20251029 02:51:52.209213 1040 tablet_service.cc:1552] Processing DeleteTablet for tablet 42f463dc3cc1434b8ee5ba268cb9fc9a with delete_type TABLET_DATA_DELETED (replaced by tablet f71388036478404d878a52fc20ece216) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:52.209609 1176 tablet_replica.cc:333] T 42f463dc3cc1434b8ee5ba268cb9fc9a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:51:52.209784 1176 raft_consensus.cc:2243] T 42f463dc3cc1434b8ee5ba268cb9fc9a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:52.209964 1176 raft_consensus.cc:2272] T 42f463dc3cc1434b8ee5ba268cb9fc9a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:52.210836 1176 ts_tablet_manager.cc:1916] T 42f463dc3cc1434b8ee5ba268cb9fc9a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:52.210906 898 tablet_service.cc:1505] Processing CreateTablet for tablet 078ace5eada044c88d2462efb2b57aa8 (DEFAULT_TABLE table=default.Table_3fae9f42aaba465bb13918661650e73a [id=9173815e55cc4e7a95b8b663e30c622c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.211344 900 tablet_service.cc:1505] Processing CreateTablet for tablet b5d5b737c86844d58c51b2388c0b1705 (DEFAULT_TABLE table=default.Table_8b39acd343e94119804478e17baaa785 [id=0b43261c80a5422092e96d77c0d2ee9e]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 02:51:52.213213 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 5886b8d39f384426bcbed7289ae72e51 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5886b8d39f384426bcbed7289ae72e51 already in progress: creating tablet
W20251029 02:51:52.213554 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet d32532973b3f46dd8158a2ce42d26365 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d32532973b3f46dd8158a2ce42d26365 already in progress: creating tablet
I20251029 02:51:52.213768 1174 tablet_replica.cc:333] T 8b89ee9017b244c5b866b1fc7154ffe3 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
W20251029 02:51:52.213779 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 64b0fb400e944ce2b2c7a49d12135e9c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 64b0fb400e944ce2b2c7a49d12135e9c already in progress: creating tablet
I20251029 02:51:52.213951 1174 raft_consensus.cc:2243] T 8b89ee9017b244c5b866b1fc7154ffe3 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:52.214136 1174 raft_consensus.cc:2272] T 8b89ee9017b244c5b866b1fc7154ffe3 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus is shut down!
W20251029 02:51:52.214912 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet c4884e22cf2044a4a1070c1b4b759d9e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c4884e22cf2044a4a1070c1b4b759d9e already in progress: creating tablet
I20251029 02:51:52.211366 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 078ace5eada044c88d2462efb2b57aa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.217836 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5d5b737c86844d58c51b2388c0b1705. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.218788 1176 ts_tablet_manager.cc:1929] T 42f463dc3cc1434b8ee5ba268cb9fc9a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:52.218950 1176 log.cc:1199] T 42f463dc3cc1434b8ee5ba268cb9fc9a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/42f463dc3cc1434b8ee5ba268cb9fc9a
I20251029 02:51:52.219399 1176 ts_tablet_manager.cc:1950] T 42f463dc3cc1434b8ee5ba268cb9fc9a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:51:52.221752 1040 tablet_service.cc:1505] Processing CreateTablet for tablet 078ace5eada044c88d2462efb2b57aa8 (DEFAULT_TABLE table=default.Table_3fae9f42aaba465bb13918661650e73a [id=9173815e55cc4e7a95b8b663e30c622c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.222219 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 078ace5eada044c88d2462efb2b57aa8. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:51:52.227707 712 leader_election.cc:343] T d32532973b3f46dd8158a2ce42d26365 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:52.228579 1172 tablet_bootstrap.cc:492] T 6c32c6f5e1cc40a0a12688201cac62a7 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:51:52.234748 1172 ts_tablet_manager.cc:1403] T 6c32c6f5e1cc40a0a12688201cac62a7 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.111s user 0.004s sys 0.000s
I20251029 02:51:52.235299 1175 tablet_replica.cc:333] T 5886b8d39f384426bcbed7289ae72e51 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:51:52.235497 1175 raft_consensus.cc:2243] T 5886b8d39f384426bcbed7289ae72e51 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:52.235679 1175 raft_consensus.cc:2272] T 5886b8d39f384426bcbed7289ae72e51 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:52.235704 1172 raft_consensus.cc:359] T 6c32c6f5e1cc40a0a12688201cac62a7 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:52.235934 1172 raft_consensus.cc:385] T 6c32c6f5e1cc40a0a12688201cac62a7 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:52.236017 1172 raft_consensus.cc:740] T 6c32c6f5e1cc40a0a12688201cac62a7 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:51:52.236296 1172 consensus_queue.cc:260] T 6c32c6f5e1cc40a0a12688201cac62a7 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:52.236466 1175 ts_tablet_manager.cc:1916] T 5886b8d39f384426bcbed7289ae72e51 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:52.236749 1172 ts_tablet_manager.cc:1434] T 6c32c6f5e1cc40a0a12688201cac62a7 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251029 02:51:52.237103 1172 tablet_bootstrap.cc:492] T d7ba1756532c4825a6c7024511dbae51 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
W20251029 02:51:52.238543 711 leader_election.cc:343] T d32532973b3f46dd8158a2ce42d26365 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:52.238996 711 leader_election.cc:304] T d32532973b3f46dd8158a2ce42d26365 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:51:52.239264 1179 raft_consensus.cc:2749] T d32532973b3f46dd8158a2ce42d26365 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 02:51:52.239395 711 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:41701, user_credentials={real_user=slave}} blocked reactor thread for 47942.6us
W20251029 02:51:52.240020 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet cb55b3d5bc63405f96cdc122f5a612cb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cb55b3d5bc63405f96cdc122f5a612cb already in progress: creating tablet
I20251029 02:51:52.240675 753 tablet_service.cc:1505] Processing CreateTablet for tablet 078ace5eada044c88d2462efb2b57aa8 (DEFAULT_TABLE table=default.Table_3fae9f42aaba465bb13918661650e73a [id=9173815e55cc4e7a95b8b663e30c622c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.241103 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 078ace5eada044c88d2462efb2b57aa8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.241178 755 tablet_service.cc:1505] Processing CreateTablet for tablet b5d5b737c86844d58c51b2388c0b1705 (DEFAULT_TABLE table=default.Table_8b39acd343e94119804478e17baaa785 [id=0b43261c80a5422092e96d77c0d2ee9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.241569 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5d5b737c86844d58c51b2388c0b1705. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.242336 1172 tablet_bootstrap.cc:654] T d7ba1756532c4825a6c7024511dbae51 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:52.244679 524 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_62e57bedde8f4c6e80bcbdcc21580091" } new_table_name: "default.Table_c8ea18626a024fcdbdbf869395f44e94" modify_external_catalogs: true
I20251029 02:51:52.244943 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_198644e9ee1f45ecb6ad95a759b54607"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:52.245532 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_198644e9ee1f45ecb6ad95a759b54607 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:52.245636 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 42f463dc3cc1434b8ee5ba268cb9fc9a (table default.Table_03f7d6f02b064939ba7464c5173e02e3 [id=9898cffc2b964059ba3163ee4d1395c2]) successfully deleted
I20251029 02:51:52.246153 517 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_2e8a7a51e2414926b5508b8a06d6a55d" } new_table_name: "default.Table_8b68c930049e4c159e83b15be8126f03" modify_external_catalogs: true
I20251029 02:51:52.246424 1037 tablet_service.cc:1505] Processing CreateTablet for tablet 4bf06476c87a4549a73f5994314bb75f (DEFAULT_TABLE table=default.Table_8d5ec842f38f4e6f9c387ff7126ce94d [id=046010a03d774e91b0ad10df97848c56]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.246855 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4bf06476c87a4549a73f5994314bb75f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.246791 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_22184a76d5034feea912425485a578a4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:52.246863 894 tablet_service.cc:1505] Processing CreateTablet for tablet 4bf06476c87a4549a73f5994314bb75f (DEFAULT_TABLE table=default.Table_8d5ec842f38f4e6f9c387ff7126ce94d [id=046010a03d774e91b0ad10df97848c56]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.247272 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4bf06476c87a4549a73f5994314bb75f. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:51:52.247272 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_22184a76d5034feea912425485a578a4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:52.248427 525 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_d1a375b674704cddb1c3bb13c566729e" } modify_external_catalogs: true
I20251029 02:51:52.248590 525 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_d1a375b674704cddb1c3bb13c566729e" } modify_external_catalogs: true
I20251029 02:51:52.249922 523 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_4d2c99159e924a06858977465efa38a6" } modify_external_catalogs: true
I20251029 02:51:52.250077 523 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_4d2c99159e924a06858977465efa38a6" } modify_external_catalogs: true
I20251029 02:51:52.250871 1031 tablet_service.cc:1505] Processing CreateTablet for tablet b5d5b737c86844d58c51b2388c0b1705 (DEFAULT_TABLE table=default.Table_8b39acd343e94119804478e17baaa785 [id=0b43261c80a5422092e96d77c0d2ee9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.251330 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5d5b737c86844d58c51b2388c0b1705. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.251322 518 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_dd81d4b79bbc4d81b0ada1be7a68177a" } new_table_name: "default.Table_3bc6a1815ce646fb9a89e91fdda46ae0" modify_external_catalogs: true
I20251029 02:51:52.252704 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_e523efa604564452a1f56dada6e21daf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:52.253190 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e523efa604564452a1f56dada6e21daf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:52.259210 520 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_03f7d6f02b064939ba7464c5173e02e3" } modify_external_catalogs: true
I20251029 02:51:52.259521 520 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_03f7d6f02b064939ba7464c5173e02e3" } modify_external_catalogs: true
I20251029 02:51:52.244709 519 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_0b84a2f666074fd18085ff4ce249a2fe" } modify_external_catalogs: true
I20251029 02:51:52.261379 519 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_0b84a2f666074fd18085ff4ce249a2fe" } modify_external_catalogs: true
I20251029 02:51:52.261897 1175 ts_tablet_manager.cc:1929] T 5886b8d39f384426bcbed7289ae72e51 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:52.262037 1175 log.cc:1199] T 5886b8d39f384426bcbed7289ae72e51 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/5886b8d39f384426bcbed7289ae72e51
I20251029 02:51:52.262395 1175 ts_tablet_manager.cc:1950] T 5886b8d39f384426bcbed7289ae72e51 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:51:52.263634 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 5886b8d39f384426bcbed7289ae72e51 (table default.Table_839fd4e3c7874233afc8cea6d4c17026 [id=ace1554749d24504aad642b368b1781f]) successfully deleted
I20251029 02:51:52.263836 1175 tablet_replica.cc:333] T e6bcea41d5d4480eaf5d6986a93f5e45 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
W20251029 02:51:52.263931 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet bf210435010d4dde8903fd116eb468e3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bf210435010d4dde8903fd116eb468e3 already in progress: creating tablet
I20251029 02:51:52.264055 1175 raft_consensus.cc:2243] T e6bcea41d5d4480eaf5d6986a93f5e45 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
W20251029 02:51:52.264184 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet d7ba1756532c4825a6c7024511dbae51 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d7ba1756532c4825a6c7024511dbae51 already in progress: creating tablet
I20251029 02:51:52.264215 1175 raft_consensus.cc:2272] T e6bcea41d5d4480eaf5d6986a93f5e45 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:52.264778 1175 ts_tablet_manager.cc:1916] T e6bcea41d5d4480eaf5d6986a93f5e45 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 02:51:52.268447 524 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4b76c4408a7340e59b8506171622175f consensus state has no leader
I20251029 02:51:52.268796 755 tablet_service.cc:1505] Processing CreateTablet for tablet 4bf06476c87a4549a73f5994314bb75f (DEFAULT_TABLE table=default.Table_8d5ec842f38f4e6f9c387ff7126ce94d [id=046010a03d774e91b0ad10df97848c56]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.269637 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4bf06476c87a4549a73f5994314bb75f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.269889 894 tablet_service.cc:1505] Processing CreateTablet for tablet 9e07291d6b5645feb4d5890f58e1ef74 (DEFAULT_TABLE table=default.Table_2e8a7a51e2414926b5508b8a06d6a55d [id=e71aea48b9884bacb250412624ec4615]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.270991 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e07291d6b5645feb4d5890f58e1ef74. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.272377 753 tablet_service.cc:1505] Processing CreateTablet for tablet 9e07291d6b5645feb4d5890f58e1ef74 (DEFAULT_TABLE table=default.Table_2e8a7a51e2414926b5508b8a06d6a55d [id=e71aea48b9884bacb250412624ec4615]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.272871 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e07291d6b5645feb4d5890f58e1ef74. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.275219 526 master_service.cc:954] ReplaceTablet: received request to replace tablet 61c1fec3ab4c402eb29f767b9bb7409f from {username='slave'} at 127.0.0.1:35314
I20251029 02:51:52.276286 1166 tablet_bootstrap.cc:492] T e6bcea41d5d4480eaf5d6986a93f5e45 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:51:52.276510 1166 ts_tablet_manager.cc:1403] T e6bcea41d5d4480eaf5d6986a93f5e45 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.098s user 0.004s sys 0.000s
I20251029 02:51:52.278057 751 tablet_service.cc:1505] Processing CreateTablet for tablet 9964ad774890422f8e14f6b99f28cb55 (DEFAULT_TABLE table=default.Table_da0be17b8bdc4e579b6c3faff71d9c98 [id=6e06ac70c6814e7d89a008bbaddccc7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.278641 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9964ad774890422f8e14f6b99f28cb55. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.278896 1166 raft_consensus.cc:359] T e6bcea41d5d4480eaf5d6986a93f5e45 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:52.279119 1166 raft_consensus.cc:385] T e6bcea41d5d4480eaf5d6986a93f5e45 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:52.279203 1166 raft_consensus.cc:740] T e6bcea41d5d4480eaf5d6986a93f5e45 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:51:52.279471 1166 consensus_queue.cc:260] T e6bcea41d5d4480eaf5d6986a93f5e45 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:52.281304 1034 tablet_service.cc:1505] Processing CreateTablet for tablet 9e07291d6b5645feb4d5890f58e1ef74 (DEFAULT_TABLE table=default.Table_2e8a7a51e2414926b5508b8a06d6a55d [id=e71aea48b9884bacb250412624ec4615]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.281739 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e07291d6b5645feb4d5890f58e1ef74. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.285835 1175 ts_tablet_manager.cc:1929] T e6bcea41d5d4480eaf5d6986a93f5e45 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:52.286010 1175 log.cc:1199] T e6bcea41d5d4480eaf5d6986a93f5e45 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/e6bcea41d5d4480eaf5d6986a93f5e45
I20251029 02:51:52.286413 1175 ts_tablet_manager.cc:1950] T e6bcea41d5d4480eaf5d6986a93f5e45 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:51:52.290992 900 tablet_service.cc:1505] Processing CreateTablet for tablet 9964ad774890422f8e14f6b99f28cb55 (DEFAULT_TABLE table=default.Table_da0be17b8bdc4e579b6c3faff71d9c98 [id=6e06ac70c6814e7d89a008bbaddccc7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.291589 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9964ad774890422f8e14f6b99f28cb55. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.295615 1166 ts_tablet_manager.cc:1434] T e6bcea41d5d4480eaf5d6986a93f5e45 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.019s user 0.003s sys 0.000s
I20251029 02:51:52.296061 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_f7c46eb111f04bcc8824f60ec45a3a6b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:52.296710 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f7c46eb111f04bcc8824f60ec45a3a6b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:52.298751 524 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet 11f4252b5e694312aaa142468f8e68b4 (table default.Table_14156dbda95043dcb54dd316874be455 [id=d0cd3d1567c54463947443927e18169f]). Expected version 4 got 0
I20251029 02:51:52.299429 1175 tablet_replica.cc:333] T 0dd31dc5873d407d958a0f5199f78782 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:51:52.299635 1175 raft_consensus.cc:2243] T 0dd31dc5873d407d958a0f5199f78782 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:52.299829 1175 raft_consensus.cc:2272] T 0dd31dc5873d407d958a0f5199f78782 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:52.300607 524 master_service.cc:954] ReplaceTablet: received request to replace tablet 14698af6348f41e384f10569392d9b23 from {username='slave'} at 127.0.0.1:35314
I20251029 02:51:52.301699 1166 tablet_bootstrap.cc:492] T 8b89ee9017b244c5b866b1fc7154ffe3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:51:52.301779 1175 ts_tablet_manager.cc:1916] T 0dd31dc5873d407d958a0f5199f78782 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:52.308017 1171 raft_consensus.cc:493] T 8f7c746f8b6948b58676298c2c2ea221 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:52.308234 1171 raft_consensus.cc:515] T 8f7c746f8b6948b58676298c2c2ea221 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:52.308986 1171 leader_election.cc:290] T 8f7c746f8b6948b58676298c2c2ea221 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:51:52.309891 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8f7c746f8b6948b58676298c2c2ea221" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:51:52.310132 776 raft_consensus.cc:2468] T 8f7c746f8b6948b58676298c2c2ea221 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:51:52.311070 1062 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8f7c746f8b6948b58676298c2c2ea221" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:51:52.311298 1062 raft_consensus.cc:2468] T 8f7c746f8b6948b58676298c2c2ea221 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:51:52.311335 1167 tablet_bootstrap.cc:492] T f04473c3a5f74bdba196b91c2538537e P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:51:52.311509 1167 ts_tablet_manager.cc:1403] T f04473c3a5f74bdba196b91c2538537e P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.169s user 0.010s sys 0.006s
I20251029 02:51:52.312233 849 leader_election.cc:304] T 8f7c746f8b6948b58676298c2c2ea221 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:51:52.312525 1167 raft_consensus.cc:359] T f04473c3a5f74bdba196b91c2538537e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:52.312777 1167 raft_consensus.cc:385] T f04473c3a5f74bdba196b91c2538537e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:52.312891 1167 raft_consensus.cc:740] T f04473c3a5f74bdba196b91c2538537e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:51:52.313167 1171 raft_consensus.cc:2804] T 8f7c746f8b6948b58676298c2c2ea221 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:51:52.313162 1167 consensus_queue.cc:260] T f04473c3a5f74bdba196b91c2538537e P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:52.313344 1171 raft_consensus.cc:493] T 8f7c746f8b6948b58676298c2c2ea221 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:51:52.313426 1171 raft_consensus.cc:3060] T 8f7c746f8b6948b58676298c2c2ea221 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:51:52.315979 893 tablet_service.cc:1505] Processing CreateTablet for tablet 0a47781d036149569e1164dd938a45e9 (DEFAULT_TABLE table=default.Table_a5eff5a49d424fa89b57d375abc4d155 [id=28bda0a81e3e4b65a3912c7ec0801fb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.316193 1171 raft_consensus.cc:515] T 8f7c746f8b6948b58676298c2c2ea221 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:52.316452 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a47781d036149569e1164dd938a45e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.317626 1062 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8f7c746f8b6948b58676298c2c2ea221" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:51:52.317875 1062 raft_consensus.cc:3060] T 8f7c746f8b6948b58676298c2c2ea221 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:51:52.318663 1175 ts_tablet_manager.cc:1929] T 0dd31dc5873d407d958a0f5199f78782 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:52.318809 1175 log.cc:1199] T 0dd31dc5873d407d958a0f5199f78782 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/0dd31dc5873d407d958a0f5199f78782
I20251029 02:51:52.319193 1175 ts_tablet_manager.cc:1950] T 0dd31dc5873d407d958a0f5199f78782 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:51:52.320169 1175 tablet_replica.cc:333] T d32532973b3f46dd8158a2ce42d26365 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:51:52.320258 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 0dd31dc5873d407d958a0f5199f78782 (table default.Table_c60a78e1cea24e8e864803a5c6124df1 [id=99445ef970c644f8a39388b2015da927]) successfully deleted
I20251029 02:51:52.320374 1175 raft_consensus.cc:2243] T d32532973b3f46dd8158a2ce42d26365 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:52.320533 1175 raft_consensus.cc:2272] T d32532973b3f46dd8158a2ce42d26365 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:52.322032 1062 raft_consensus.cc:2468] T 8f7c746f8b6948b58676298c2c2ea221 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:51:52.323642 849 leader_election.cc:304] T 8f7c746f8b6948b58676298c2c2ea221 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:51:52.325358 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8f7c746f8b6948b58676298c2c2ea221" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:51:52.325604 776 raft_consensus.cc:3060] T 8f7c746f8b6948b58676298c2c2ea221 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:51:52.216612 1174 ts_tablet_manager.cc:1916] T 8b89ee9017b244c5b866b1fc7154ffe3 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:52.328572 776 raft_consensus.cc:2468] T 8f7c746f8b6948b58676298c2c2ea221 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:51:52.329339 1175 ts_tablet_manager.cc:1916] T d32532973b3f46dd8158a2ce42d26365 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:52.329347 1171 leader_election.cc:290] T 8f7c746f8b6948b58676298c2c2ea221 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:51:52.330775 1038 tablet_service.cc:1552] Processing DeleteTablet for tablet e6bcea41d5d4480eaf5d6986a93f5e45 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:51 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:52.331177 1176 tablet_replica.cc:333] T e6bcea41d5d4480eaf5d6986a93f5e45 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:51:52.331990 1176 raft_consensus.cc:2243] T e6bcea41d5d4480eaf5d6986a93f5e45 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:52.332204 1176 raft_consensus.cc:2272] T e6bcea41d5d4480eaf5d6986a93f5e45 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:52.332451 893 tablet_service.cc:1505] Processing CreateTablet for tablet fd0f89180aba43eabb20f926abccbaa6 (DEFAULT_TABLE table=default.Table_50252b3b3cc4403da5e259f9f91be915 [id=ff64a323de354792997beb99bae4a52c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.333176 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd0f89180aba43eabb20f926abccbaa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.333302 753 tablet_service.cc:1505] Processing CreateTablet for tablet 0a47781d036149569e1164dd938a45e9 (DEFAULT_TABLE table=default.Table_a5eff5a49d424fa89b57d375abc4d155 [id=28bda0a81e3e4b65a3912c7ec0801fb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.333384 1176 ts_tablet_manager.cc:1916] T e6bcea41d5d4480eaf5d6986a93f5e45 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:52.333740 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a47781d036149569e1164dd938a45e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.341013 1176 ts_tablet_manager.cc:1929] T e6bcea41d5d4480eaf5d6986a93f5e45 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:52.341396 1176 log.cc:1199] T e6bcea41d5d4480eaf5d6986a93f5e45 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/e6bcea41d5d4480eaf5d6986a93f5e45
I20251029 02:51:52.341861 1176 ts_tablet_manager.cc:1950] T e6bcea41d5d4480eaf5d6986a93f5e45 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:51:52.343743 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet e6bcea41d5d4480eaf5d6986a93f5e45 (table default.Table_5b3a7640a9e54e7a9868985322ade0cb [id=1a1aeb237dad4464b717bbae5d4001b0]) successfully deleted
I20251029 02:51:52.329632 1308 raft_consensus.cc:2804] T 8f7c746f8b6948b58676298c2c2ea221 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:51:52.346415 755 tablet_service.cc:1505] Processing CreateTablet for tablet fd0f89180aba43eabb20f926abccbaa6 (DEFAULT_TABLE table=default.Table_50252b3b3cc4403da5e259f9f91be915 [id=ff64a323de354792997beb99bae4a52c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.346889 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd0f89180aba43eabb20f926abccbaa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.347636 1038 tablet_service.cc:1505] Processing CreateTablet for tablet 9964ad774890422f8e14f6b99f28cb55 (DEFAULT_TABLE table=default.Table_da0be17b8bdc4e579b6c3faff71d9c98 [id=6e06ac70c6814e7d89a008bbaddccc7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.348090 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9964ad774890422f8e14f6b99f28cb55. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.348871 754 tablet_service.cc:1505] Processing CreateTablet for tablet 65fdec7cf5d143208d78d4c3806b1352 (DEFAULT_TABLE table=default.Table_d1a375b674704cddb1c3bb13c566729e [id=909ecde47c7245b2b84dcd665e779b1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.349246 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65fdec7cf5d143208d78d4c3806b1352. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.349372 892 tablet_service.cc:1505] Processing CreateTablet for tablet 44784a3a760946e0be24743bc2fc8138 (DEFAULT_TABLE table=default.Table_7df0e7c7f61d486891adb26e68972a1c [id=915b76276e0940618b7180ebc06cd464]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.349745 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44784a3a760946e0be24743bc2fc8138. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.350852 1040 tablet_service.cc:1505] Processing CreateTablet for tablet 0a47781d036149569e1164dd938a45e9 (DEFAULT_TABLE table=default.Table_a5eff5a49d424fa89b57d375abc4d155 [id=28bda0a81e3e4b65a3912c7ec0801fb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.351084 1308 raft_consensus.cc:697] T 8f7c746f8b6948b58676298c2c2ea221 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:51:52.351320 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a47781d036149569e1164dd938a45e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.351483 1308 consensus_queue.cc:237] T 8f7c746f8b6948b58676298c2c2ea221 P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:52.352986 1031 tablet_service.cc:1505] Processing CreateTablet for tablet fd0f89180aba43eabb20f926abccbaa6 (DEFAULT_TABLE table=default.Table_50252b3b3cc4403da5e259f9f91be915 [id=ff64a323de354792997beb99bae4a52c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.353389 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd0f89180aba43eabb20f926abccbaa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.359077 1172 tablet_bootstrap.cc:492] T d7ba1756532c4825a6c7024511dbae51 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:51:52.359347 1172 ts_tablet_manager.cc:1403] T d7ba1756532c4825a6c7024511dbae51 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.122s user 0.011s sys 0.019s
I20251029 02:51:52.359490 1179 raft_consensus.cc:493] T 11f4252b5e694312aaa142468f8e68b4 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:52.359584 1034 tablet_service.cc:1505] Processing CreateTablet for tablet 65fdec7cf5d143208d78d4c3806b1352 (DEFAULT_TABLE table=default.Table_d1a375b674704cddb1c3bb13c566729e [id=909ecde47c7245b2b84dcd665e779b1e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.359624 1179 raft_consensus.cc:515] T 11f4252b5e694312aaa142468f8e68b4 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:52.360006 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65fdec7cf5d143208d78d4c3806b1352. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.360774 1179 leader_election.cc:290] T 11f4252b5e694312aaa142468f8e68b4 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:51:52.360857 1172 raft_consensus.cc:359] T d7ba1756532c4825a6c7024511dbae51 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:52.361027 1172 raft_consensus.cc:385] T d7ba1756532c4825a6c7024511dbae51 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:52.361137 1172 raft_consensus.cc:740] T d7ba1756532c4825a6c7024511dbae51 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:51:52.361424 1172 consensus_queue.cc:260] T d7ba1756532c4825a6c7024511dbae51 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:52.361600 1062 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11f4252b5e694312aaa142468f8e68b4" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:51:52.361878 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11f4252b5e694312aaa142468f8e68b4" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:51:52.361969 1172 ts_tablet_manager.cc:1434] T d7ba1756532c4825a6c7024511dbae51 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20251029 02:51:52.362649 712 leader_election.cc:343] T 11f4252b5e694312aaa142468f8e68b4 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
W20251029 02:51:52.362645 510 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.62:42343, user_credentials={real_user=slave}} blocked reactor thread for 50542us
W20251029 02:51:52.363206 711 leader_election.cc:343] T 11f4252b5e694312aaa142468f8e68b4 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:52.363354 711 leader_election.cc:304] T 11f4252b5e694312aaa142468f8e68b4 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:51:52.364099 1276 raft_consensus.cc:2749] T 11f4252b5e694312aaa142468f8e68b4 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:51:52.366885 1166 tablet_bootstrap.cc:654] T 8b89ee9017b244c5b866b1fc7154ffe3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:52.367568 1172 tablet_bootstrap.cc:492] T 6dd4696ad4b44b6dab87a205f3f16ebd P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:51:52.368151 525 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 65fdec7cf5d143208d78d4c3806b1352
I20251029 02:51:52.369045 1179 raft_consensus.cc:493] T d7ba1756532c4825a6c7024511dbae51 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:52.369241 1179 raft_consensus.cc:515] T d7ba1756532c4825a6c7024511dbae51 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:52.369552 1172 tablet_bootstrap.cc:654] T 6dd4696ad4b44b6dab87a205f3f16ebd P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:52.369696 1033 tablet_service.cc:1552] Processing DeleteTablet for tablet 65fdec7cf5d143208d78d4c3806b1352 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:52 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:52.370018 1179 leader_election.cc:290] T d7ba1756532c4825a6c7024511dbae51 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:51:52.371698 1062 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d7ba1756532c4825a6c7024511dbae51" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:51:52.372058 512 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.60:41233: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20251029 02:51:52.373369 512 consensus_peers.cc:597] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0 -> Peer e5e5222b26fe4145948620e51edf53fb (127.0.104.60:41233): Couldn't send request to peer e5e5222b26fe4145948620e51edf53fb. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.60:41233: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20251029 02:51:52.375450 892 tablet_service.cc:1552] Processing DeleteTablet for tablet f04473c3a5f74bdba196b91c2538537e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:51 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:52.375703 525 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_9becb486767e4662a93114d2242f031d" } new_table_name: "default.Table_0fa5f3e932c846168255e7540c35a515" modify_external_catalogs: true
I20251029 02:51:52.376056 1037 tablet_service.cc:1505] Processing CreateTablet for tablet 44784a3a760946e0be24743bc2fc8138 (DEFAULT_TABLE table=default.Table_7df0e7c7f61d486891adb26e68972a1c [id=915b76276e0940618b7180ebc06cd464]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.376475 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44784a3a760946e0be24743bc2fc8138. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.378352 894 tablet_service.cc:1505] Processing CreateTablet for tablet 65fdec7cf5d143208d78d4c3806b1352 (DEFAULT_TABLE table=default.Table_d1a375b674704cddb1c3bb13c566729e [id=909ecde47c7245b2b84dcd665e779b1e]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 02:51:52.378702 712 leader_election.cc:343] T d7ba1756532c4825a6c7024511dbae51 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:52.379499 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65fdec7cf5d143208d78d4c3806b1352. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.380664 1033 tablet_service.cc:1552] Processing DeleteTablet for tablet e6bcea41d5d4480eaf5d6986a93f5e45 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:51 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:52.380702 752 tablet_service.cc:1505] Processing CreateTablet for tablet 44784a3a760946e0be24743bc2fc8138 (DEFAULT_TABLE table=default.Table_7df0e7c7f61d486891adb26e68972a1c [id=915b76276e0940618b7180ebc06cd464]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.381134 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44784a3a760946e0be24743bc2fc8138. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.383049 751 tablet_service.cc:1552] Processing DeleteTablet for tablet 65fdec7cf5d143208d78d4c3806b1352 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:52 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:52.383266 1175 ts_tablet_manager.cc:1929] T d32532973b3f46dd8158a2ce42d26365 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:52.383411 1175 log.cc:1199] T d32532973b3f46dd8158a2ce42d26365 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/d32532973b3f46dd8158a2ce42d26365
I20251029 02:51:52.383778 1175 ts_tablet_manager.cc:1950] T d32532973b3f46dd8158a2ce42d26365 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:51:52.386138 1172 tablet_bootstrap.cc:492] T 6dd4696ad4b44b6dab87a205f3f16ebd P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:51:52.386305 1172 ts_tablet_manager.cc:1403] T 6dd4696ad4b44b6dab87a205f3f16ebd P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.001s
I20251029 02:51:52.382750 898 tablet_service.cc:1505] Processing CreateTablet for tablet 69597d48631d43b6a5b92abbf60d5ed3 (DEFAULT_TABLE table=default.Table_fd85e1b0ff874cdc985600a3fa32085c [id=ffce392867b14b6ea9db44f4bdee57c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.386742 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d7ba1756532c4825a6c7024511dbae51" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:51:52.386864 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69597d48631d43b6a5b92abbf60d5ed3. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:51:52.387253 513 catalog_manager.cc:4952] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 65fdec7cf5d143208d78d4c3806b1352 because the tablet was not found. No further retry: Not found: Tablet not found: 65fdec7cf5d143208d78d4c3806b1352
I20251029 02:51:52.387215 1172 raft_consensus.cc:359] T 6dd4696ad4b44b6dab87a205f3f16ebd P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:52.387403 1172 raft_consensus.cc:385] T 6dd4696ad4b44b6dab87a205f3f16ebd P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:52.387483 1172 raft_consensus.cc:740] T 6dd4696ad4b44b6dab87a205f3f16ebd P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
W20251029 02:51:52.387537 513 catalog_manager.cc:4952] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet e6bcea41d5d4480eaf5d6986a93f5e45 because the tablet was not found. No further retry: Not found: Tablet not found: e6bcea41d5d4480eaf5d6986a93f5e45
I20251029 02:51:52.387696 1172 consensus_queue.cc:260] T 6dd4696ad4b44b6dab87a205f3f16ebd P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:52.388464 1033 tablet_service.cc:1505] Processing CreateTablet for tablet 69597d48631d43b6a5b92abbf60d5ed3 (DEFAULT_TABLE table=default.Table_fd85e1b0ff874cdc985600a3fa32085c [id=ffce392867b14b6ea9db44f4bdee57c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.388909 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69597d48631d43b6a5b92abbf60d5ed3. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:51:52.389153 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 64b0fb400e944ce2b2c7a49d12135e9c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 64b0fb400e944ce2b2c7a49d12135e9c already in progress: creating tablet
W20251029 02:51:52.389379 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet c4884e22cf2044a4a1070c1b4b759d9e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c4884e22cf2044a4a1070c1b4b759d9e already in progress: creating tablet
W20251029 02:51:52.389503 711 leader_election.cc:343] T d7ba1756532c4825a6c7024511dbae51 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
W20251029 02:51:52.389601 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 65fdec7cf5d143208d78d4c3806b1352 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 65fdec7cf5d143208d78d4c3806b1352 already in progress: creating tablet
I20251029 02:51:52.389657 711 leader_election.cc:304] T d7ba1756532c4825a6c7024511dbae51 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:51:52.390383 1179 raft_consensus.cc:2749] T d7ba1756532c4825a6c7024511dbae51 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:51:52.390434 754 tablet_service.cc:1552] Processing DeleteTablet for tablet 6c32c6f5e1cc40a0a12688201cac62a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:51 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:52.390928 1172 ts_tablet_manager.cc:1434] T 6dd4696ad4b44b6dab87a205f3f16ebd P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251029 02:51:52.392989 523 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet d6e1170dbb594a5696a3899cde8da7c9
I20251029 02:51:52.393465 1172 tablet_bootstrap.cc:492] T cb55b3d5bc63405f96cdc122f5a612cb P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:51:52.393461 1174 ts_tablet_manager.cc:1929] T 8b89ee9017b244c5b866b1fc7154ffe3 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:52.393586 1174 log.cc:1199] T 8b89ee9017b244c5b866b1fc7154ffe3 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/8b89ee9017b244c5b866b1fc7154ffe3
I20251029 02:51:52.393795 523 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_e95a0439ea534f538683392bbc8e5219" } modify_external_catalogs: true
I20251029 02:51:52.394002 1174 ts_tablet_manager.cc:1950] T 8b89ee9017b244c5b866b1fc7154ffe3 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:51:52.393986 523 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_e95a0439ea534f538683392bbc8e5219" } modify_external_catalogs: true
W20251029 02:51:52.394456 517 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9e07291d6b5645feb4d5890f58e1ef74 consensus state has no leader
I20251029 02:51:52.394825 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_b8bb03c0242d4262b78640fa8c88dc27"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:52.395200 1172 tablet_bootstrap.cc:654] T cb55b3d5bc63405f96cdc122f5a612cb P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
W20251029 02:51:52.395355 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b8bb03c0242d4262b78640fa8c88dc27 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:51:52.395470 518 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 59af906061d049c99ae2153945597007 consensus state has no leader
I20251029 02:51:52.395838 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_e60149a6638a4b49843367e80eebda79"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:52.396337 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e60149a6638a4b49843367e80eebda79 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:52.396873 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_73790f318ab641a4bdb26c2b76b57e7e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:52.397322 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_73790f318ab641a4bdb26c2b76b57e7e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:52.397461 521 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_40421c4fdf1d42be8ed6114e0b77f1f9" } new_table_name: "default.Table_e43a2189b18a43f9beb9f3ce25848374" modify_external_catalogs: true
I20251029 02:51:52.398541 524 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 14698af6348f41e384f10569392d9b23
I20251029 02:51:52.398684 1175 tablet_replica.cc:333] T 6c32c6f5e1cc40a0a12688201cac62a7 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:51:52.398701 520 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet f71388036478404d878a52fc20ece216
I20251029 02:51:52.398877 1175 raft_consensus.cc:2243] T 6c32c6f5e1cc40a0a12688201cac62a7 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:52.399000 524 catalog_manager.cc:6719] ReplaceTablet: tablet 14698af6348f41e384f10569392d9b23 deleted and replaced by tablet 7309d3190cfa415281c5253a634fa7f3
I20251029 02:51:52.399077 1175 raft_consensus.cc:2272] T 6c32c6f5e1cc40a0a12688201cac62a7 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:52.399318 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_8b952165b0ae4603b38a50cfa0e121e0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:52.399389 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_cb3d27722cc74fa18ddc8740590639a9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:52.399797 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8b952165b0ae4603b38a50cfa0e121e0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:51:52.399863 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cb3d27722cc74fa18ddc8740590639a9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:52.400954 519 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 75236b2fa82f4f28b82efae9a2c5160d
I20251029 02:51:52.401528 519 master_service.cc:954] ReplaceTablet: received request to replace tablet b867feae24e741aca6902868c35c1428 from {username='slave'} at 127.0.0.1:35314
I20251029 02:51:52.401718 1175 ts_tablet_manager.cc:1916] T 6c32c6f5e1cc40a0a12688201cac62a7 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:52.402683 526 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 61c1fec3ab4c402eb29f767b9bb7409f
I20251029 02:51:52.403081 526 catalog_manager.cc:6719] ReplaceTablet: tablet 61c1fec3ab4c402eb29f767b9bb7409f deleted and replaced by tablet a5c3a15510cb4321bbb80d61fabd0c0a
I20251029 02:51:52.403286 526 master_service.cc:954] ReplaceTablet: received request to replace tablet 2ebde39f813d46e898ad1754325ad1a0 from {username='slave'} at 127.0.0.1:35314
I20251029 02:51:52.405329 749 tablet_service.cc:1552] Processing DeleteTablet for tablet 14698af6348f41e384f10569392d9b23 with delete_type TABLET_DATA_DELETED (replaced by tablet 7309d3190cfa415281c5253a634fa7f3) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:52.405735 749 tablet_service.cc:1552] Processing DeleteTablet for tablet f71388036478404d878a52fc20ece216 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:52 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:52.406446 749 tablet_service.cc:1552] Processing DeleteTablet for tablet 75236b2fa82f4f28b82efae9a2c5160d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:52 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:52.407389 754 tablet_service.cc:1552] Processing DeleteTablet for tablet d6e1170dbb594a5696a3899cde8da7c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:52 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:52.408550 750 tablet_service.cc:1552] Processing DeleteTablet for tablet 61c1fec3ab4c402eb29f767b9bb7409f with delete_type TABLET_DATA_DELETED (replaced by tablet a5c3a15510cb4321bbb80d61fabd0c0a) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:52.410456 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet f71388036478404d878a52fc20ece216 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:52 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:52.410771 1039 tablet_service.cc:1552] Processing DeleteTablet for tablet 75236b2fa82f4f28b82efae9a2c5160d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:52 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:52.411167 1039 tablet_service.cc:1552] Processing DeleteTablet for tablet 61c1fec3ab4c402eb29f767b9bb7409f with delete_type TABLET_DATA_DELETED (replaced by tablet a5c3a15510cb4321bbb80d61fabd0c0a) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:52.413511 1175 ts_tablet_manager.cc:1929] T 6c32c6f5e1cc40a0a12688201cac62a7 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:52.413641 1175 log.cc:1199] T 6c32c6f5e1cc40a0a12688201cac62a7 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/6c32c6f5e1cc40a0a12688201cac62a7
I20251029 02:51:52.414032 1175 ts_tablet_manager.cc:1950] T 6c32c6f5e1cc40a0a12688201cac62a7 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:51:52.415140 1033 tablet_service.cc:1552] Processing DeleteTablet for tablet 14698af6348f41e384f10569392d9b23 with delete_type TABLET_DATA_DELETED (replaced by tablet 7309d3190cfa415281c5253a634fa7f3) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:52.416502 892 tablet_service.cc:1552] Processing DeleteTablet for tablet 14698af6348f41e384f10569392d9b23 with delete_type TABLET_DATA_DELETED (replaced by tablet 7309d3190cfa415281c5253a634fa7f3) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:52.416608 526 catalog_manager.cc:5649] T 8f7c746f8b6948b58676298c2c2ea221 P c756528642a543eeacd93cef3e335a7d reported cstate change: term changed from 0 to 1, leader changed from <none> to c756528642a543eeacd93cef3e335a7d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "c756528642a543eeacd93cef3e335a7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } health_report { overall_health: UNKNOWN } } }
I20251029 02:51:52.416924 892 tablet_service.cc:1552] Processing DeleteTablet for tablet 61c1fec3ab4c402eb29f767b9bb7409f with delete_type TABLET_DATA_DELETED (replaced by tablet a5c3a15510cb4321bbb80d61fabd0c0a) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:52.417021 526 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet 8f7c746f8b6948b58676298c2c2ea221 (table default.Table_9aac534a15054a30b80745984c279e62 [id=669ae9dd0a5749b1aa3557ab8183a057]). Expected version 3 got 0
I20251029 02:51:52.417450 895 tablet_service.cc:1552] Processing DeleteTablet for tablet f71388036478404d878a52fc20ece216 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:52 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:52.417657 896 tablet_service.cc:1552] Processing DeleteTablet for tablet 75236b2fa82f4f28b82efae9a2c5160d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:52 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:52.417881 893 tablet_service.cc:1552] Processing DeleteTablet for tablet d6e1170dbb594a5696a3899cde8da7c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:52 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:52.419701 1172 tablet_bootstrap.cc:492] T cb55b3d5bc63405f96cdc122f5a612cb P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:51:52.420219 1172 ts_tablet_manager.cc:1403] T cb55b3d5bc63405f96cdc122f5a612cb P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.027s user 0.001s sys 0.004s
W20251029 02:51:52.422819 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet f71388036478404d878a52fc20ece216 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f71388036478404d878a52fc20ece216 already in progress: creating tablet
I20251029 02:51:52.422966 1172 raft_consensus.cc:359] T cb55b3d5bc63405f96cdc122f5a612cb P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
W20251029 02:51:52.423116 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 75236b2fa82f4f28b82efae9a2c5160d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 75236b2fa82f4f28b82efae9a2c5160d already in progress: creating tablet
I20251029 02:51:52.423163 1172 raft_consensus.cc:385] T cb55b3d5bc63405f96cdc122f5a612cb P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:52.423264 1172 raft_consensus.cc:740] T cb55b3d5bc63405f96cdc122f5a612cb P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
W20251029 02:51:52.423350 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 61c1fec3ab4c402eb29f767b9bb7409f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 61c1fec3ab4c402eb29f767b9bb7409f already in progress: creating tablet
W20251029 02:51:52.423580 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 14698af6348f41e384f10569392d9b23 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 14698af6348f41e384f10569392d9b23 already in progress: creating tablet
I20251029 02:51:52.423535 1172 consensus_queue.cc:260] T cb55b3d5bc63405f96cdc122f5a612cb P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:52.424377 751 tablet_service.cc:1505] Processing CreateTablet for tablet 69597d48631d43b6a5b92abbf60d5ed3 (DEFAULT_TABLE table=default.Table_fd85e1b0ff874cdc985600a3fa32085c [id=ffce392867b14b6ea9db44f4bdee57c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.425026 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69597d48631d43b6a5b92abbf60d5ed3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.426548 1172 ts_tablet_manager.cc:1434] T cb55b3d5bc63405f96cdc122f5a612cb P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251029 02:51:52.428196 1172 tablet_bootstrap.cc:492] T bf210435010d4dde8903fd116eb468e3 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:51:52.429049 1034 tablet_service.cc:1552] Processing DeleteTablet for tablet d6e1170dbb594a5696a3899cde8da7c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:52 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:52.429783 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_06e7c4a46c50490196ec56306b13a2bb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:52.430372 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_06e7c4a46c50490196ec56306b13a2bb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:52.431922 518 master_service.cc:954] ReplaceTablet: received request to replace tablet 8d9722c764ac4b27a540595e9c6aafcd from {username='slave'} at 127.0.0.1:35314
I20251029 02:51:52.431929 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 8b89ee9017b244c5b866b1fc7154ffe3 (table default.Table_25688635c7a948f1b010aa30b88a3b6c [id=052abf3dcc904bdcb869d5b885723dab]) successfully deleted
W20251029 02:51:52.433770 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet f04473c3a5f74bdba196b91c2538537e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f04473c3a5f74bdba196b91c2538537e already in progress: creating tablet
I20251029 02:51:52.433877 523 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet c519832030884a049839d309cc738c28
I20251029 02:51:52.435184 893 tablet_service.cc:1552] Processing DeleteTablet for tablet c519832030884a049839d309cc738c28 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:52 UTC) from {username='slave'} at 127.0.0.1:35702
W20251029 02:51:52.435344 525 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6c22497760274380a832feff7c5a2fae consensus state has no leader
W20251029 02:51:52.436018 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 14698af6348f41e384f10569392d9b23 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 14698af6348f41e384f10569392d9b23 already in progress: creating tablet
I20251029 02:51:52.436337 1166 tablet_bootstrap.cc:492] T 8b89ee9017b244c5b866b1fc7154ffe3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:51:52.436517 1166 ts_tablet_manager.cc:1403] T 8b89ee9017b244c5b866b1fc7154ffe3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.135s user 0.003s sys 0.001s
W20251029 02:51:52.436864 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 61c1fec3ab4c402eb29f767b9bb7409f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 61c1fec3ab4c402eb29f767b9bb7409f already in progress: creating tablet
I20251029 02:51:52.437390 1166 raft_consensus.cc:359] T 8b89ee9017b244c5b866b1fc7154ffe3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
W20251029 02:51:52.437495 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet f71388036478404d878a52fc20ece216 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f71388036478404d878a52fc20ece216 already in progress: creating tablet
I20251029 02:51:52.437601 1166 raft_consensus.cc:385] T 8b89ee9017b244c5b866b1fc7154ffe3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:52.437691 1166 raft_consensus.cc:740] T 8b89ee9017b244c5b866b1fc7154ffe3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
W20251029 02:51:52.439026 521 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5695b5a7b1854c3983e2d4ab684898b0 has no consensus state
I20251029 02:51:52.439240 1166 consensus_queue.cc:260] T 8b89ee9017b244c5b866b1fc7154ffe3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:52.440265 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 6c32c6f5e1cc40a0a12688201cac62a7 (table default.Table_e6ca0a151fc14c5085c6ea454f460242 [id=2a711e1172654147a58b09ee43cac845]) successfully deleted
W20251029 02:51:52.440774 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 14698af6348f41e384f10569392d9b23 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 14698af6348f41e384f10569392d9b23 already in progress: creating tablet
W20251029 02:51:52.441210 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet f71388036478404d878a52fc20ece216 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f71388036478404d878a52fc20ece216 already in progress: creating tablet
W20251029 02:51:52.441766 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 75236b2fa82f4f28b82efae9a2c5160d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 75236b2fa82f4f28b82efae9a2c5160d already in progress: creating tablet
W20251029 02:51:52.442130 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 75236b2fa82f4f28b82efae9a2c5160d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 75236b2fa82f4f28b82efae9a2c5160d already in progress: creating tablet
W20251029 02:51:52.442751 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet d6e1170dbb594a5696a3899cde8da7c9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d6e1170dbb594a5696a3899cde8da7c9 already in progress: creating tablet
W20251029 02:51:52.443439 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet c519832030884a049839d309cc738c28 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c519832030884a049839d309cc738c28 already in progress: creating tablet
W20251029 02:51:52.443436 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet d6e1170dbb594a5696a3899cde8da7c9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d6e1170dbb594a5696a3899cde8da7c9 already in progress: creating tablet
W20251029 02:51:52.443894 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 61c1fec3ab4c402eb29f767b9bb7409f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 61c1fec3ab4c402eb29f767b9bb7409f already in progress: creating tablet
I20251029 02:51:52.444015 750 tablet_service.cc:1552] Processing DeleteTablet for tablet c519832030884a049839d309cc738c28 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:52 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:52.444213 752 tablet_service.cc:1552] Processing DeleteTablet for tablet d7ba1756532c4825a6c7024511dbae51 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:51 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:52.444641 1175 tablet_replica.cc:333] T d7ba1756532c4825a6c7024511dbae51 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:51:52.444846 1175 raft_consensus.cc:2243] T d7ba1756532c4825a6c7024511dbae51 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:52.445019 1175 raft_consensus.cc:2272] T d7ba1756532c4825a6c7024511dbae51 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251029 02:51:52.445111 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet c519832030884a049839d309cc738c28 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c519832030884a049839d309cc738c28 already in progress: creating tablet
I20251029 02:51:52.446879 1166 ts_tablet_manager.cc:1434] T 8b89ee9017b244c5b866b1fc7154ffe3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20251029 02:51:52.447125 1172 tablet_bootstrap.cc:654] T bf210435010d4dde8903fd116eb468e3 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:52.448630 1175 ts_tablet_manager.cc:1916] T d7ba1756532c4825a6c7024511dbae51 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:52.451035 519 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet b867feae24e741aca6902868c35c1428
I20251029 02:51:52.451107 1166 tablet_bootstrap.cc:492] T 94854145ac0d4be38780d4cdb721725e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:51:52.451561 519 catalog_manager.cc:6719] ReplaceTablet: tablet b867feae24e741aca6902868c35c1428 deleted and replaced by tablet 1bab90fe904944af948dab3db06b7e34
I20251029 02:51:52.454015 893 tablet_service.cc:1552] Processing DeleteTablet for tablet b867feae24e741aca6902868c35c1428 with delete_type TABLET_DATA_DELETED (replaced by tablet 1bab90fe904944af948dab3db06b7e34) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:52.455137 752 tablet_service.cc:1552] Processing DeleteTablet for tablet b867feae24e741aca6902868c35c1428 with delete_type TABLET_DATA_DELETED (replaced by tablet 1bab90fe904944af948dab3db06b7e34) from {username='slave'} at 127.0.0.1:44580
W20251029 02:51:52.455828 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet b867feae24e741aca6902868c35c1428 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b867feae24e741aca6902868c35c1428 already in progress: creating tablet
I20251029 02:51:52.455935 900 tablet_service.cc:1552] Processing DeleteTablet for tablet 65fdec7cf5d143208d78d4c3806b1352 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:52 UTC) from {username='slave'} at 127.0.0.1:35702
W20251029 02:51:52.456585 512 catalog_manager.cc:4952] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 65fdec7cf5d143208d78d4c3806b1352 because the tablet was not found. No further retry: Not found: Tablet not found: 65fdec7cf5d143208d78d4c3806b1352
I20251029 02:51:52.457777 1175 ts_tablet_manager.cc:1929] T d7ba1756532c4825a6c7024511dbae51 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:52.457934 1175 log.cc:1199] T d7ba1756532c4825a6c7024511dbae51 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/d7ba1756532c4825a6c7024511dbae51
I20251029 02:51:52.458318 1175 ts_tablet_manager.cc:1950] T d7ba1756532c4825a6c7024511dbae51 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:51:52.458750 1167 ts_tablet_manager.cc:1434] T f04473c3a5f74bdba196b91c2538537e P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.147s user 0.005s sys 0.000s
I20251029 02:51:52.459863 1167 tablet_bootstrap.cc:492] T 42f463dc3cc1434b8ee5ba268cb9fc9a P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:51:52.461989 1167 tablet_bootstrap.cc:654] T 42f463dc3cc1434b8ee5ba268cb9fc9a P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:52.464138 1166 tablet_bootstrap.cc:654] T 94854145ac0d4be38780d4cdb721725e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:52.468695 1031 tablet_service.cc:1552] Processing DeleteTablet for tablet c519832030884a049839d309cc738c28 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:52 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:52.468685 898 tablet_service.cc:1505] Processing CreateTablet for tablet 6cd7009e06a646acad0802918c2b425e (DEFAULT_TABLE table=default.Table_198644e9ee1f45ecb6ad95a759b54607 [id=7810f54ddc4849a8b13bb2928eb761a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.469118 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6cd7009e06a646acad0802918c2b425e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.469091 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet d7ba1756532c4825a6c7024511dbae51 (table default.Table_4c5fd91d917e4f1a99baef291b2d77e0 [id=5587d5c964034bd7ad0efa338b74df65]) successfully deleted
W20251029 02:51:52.470678 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet b867feae24e741aca6902868c35c1428 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b867feae24e741aca6902868c35c1428 already in progress: creating tablet
I20251029 02:51:52.471958 1031 tablet_service.cc:1552] Processing DeleteTablet for tablet 8b89ee9017b244c5b866b1fc7154ffe3 with delete_type TABLET_DATA_DELETED (replaced by tablet dd90fb9eb167478cbc3ec9363af86295) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:52.472219 1176 tablet_replica.cc:333] T 8b89ee9017b244c5b866b1fc7154ffe3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:51:52.472416 1176 raft_consensus.cc:2243] T 8b89ee9017b244c5b866b1fc7154ffe3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:52.472642 1176 raft_consensus.cc:2272] T 8b89ee9017b244c5b866b1fc7154ffe3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:52.472934 1308 raft_consensus.cc:493] T f04473c3a5f74bdba196b91c2538537e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:52.473101 1308 raft_consensus.cc:515] T f04473c3a5f74bdba196b91c2538537e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:52.473471 751 tablet_service.cc:1505] Processing CreateTablet for tablet 6cd7009e06a646acad0802918c2b425e (DEFAULT_TABLE table=default.Table_198644e9ee1f45ecb6ad95a759b54607 [id=7810f54ddc4849a8b13bb2928eb761a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.473780 1308 leader_election.cc:290] T f04473c3a5f74bdba196b91c2538537e P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:51:52.474025 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6cd7009e06a646acad0802918c2b425e. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:51:52.475183 849 leader_election.cc:343] T f04473c3a5f74bdba196b91c2538537e P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Not found: Tablet not found: f04473c3a5f74bdba196b91c2538537e
W20251029 02:51:52.475330 847 leader_election.cc:343] T f04473c3a5f74bdba196b91c2538537e P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: f04473c3a5f74bdba196b91c2538537e
I20251029 02:51:52.475485 847 leader_election.cc:304] T f04473c3a5f74bdba196b91c2538537e P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9
I20251029 02:51:52.476066 893 tablet_service.cc:1505] Processing CreateTablet for tablet 5695b5a7b1854c3983e2d4ab684898b0 (DEFAULT_TABLE table=default.Table_e43a2189b18a43f9beb9f3ce25848374 [id=d151d895aac64712b2eeee989d19b878]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.476217 1308 raft_consensus.cc:2749] T f04473c3a5f74bdba196b91c2538537e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:51:52.476482 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5695b5a7b1854c3983e2d4ab684898b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.477698 900 tablet_service.cc:1505] Processing CreateTablet for tablet 8afc01d52d784f2fa559cab7ce07c8be (DEFAULT_TABLE table=default.Table_22184a76d5034feea912425485a578a4 [id=ed7295f3f21848b79af75785d3983b67]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.478060 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8afc01d52d784f2fa559cab7ce07c8be. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.479737 1172 tablet_bootstrap.cc:492] T bf210435010d4dde8903fd116eb468e3 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:51:52.479815 1176 ts_tablet_manager.cc:1916] T 8b89ee9017b244c5b866b1fc7154ffe3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:52.479914 1172 ts_tablet_manager.cc:1403] T bf210435010d4dde8903fd116eb468e3 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.052s user 0.004s sys 0.008s
I20251029 02:51:52.480695 1172 raft_consensus.cc:359] T bf210435010d4dde8903fd116eb468e3 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:52.480897 1172 raft_consensus.cc:385] T bf210435010d4dde8903fd116eb468e3 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:52.480983 1172 raft_consensus.cc:740] T bf210435010d4dde8903fd116eb468e3 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:51:52.481283 753 tablet_service.cc:1505] Processing CreateTablet for tablet 5695b5a7b1854c3983e2d4ab684898b0 (DEFAULT_TABLE table=default.Table_e43a2189b18a43f9beb9f3ce25848374 [id=d151d895aac64712b2eeee989d19b878]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.481657 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5695b5a7b1854c3983e2d4ab684898b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.483903 518 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 8d9722c764ac4b27a540595e9c6aafcd
I20251029 02:51:52.484349 1179 raft_consensus.cc:493] T cb55b3d5bc63405f96cdc122f5a612cb P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:52.484400 518 catalog_manager.cc:6719] ReplaceTablet: tablet 8d9722c764ac4b27a540595e9c6aafcd deleted and replaced by tablet 51effc9324034ae79e2b25cb57a996c1
I20251029 02:51:52.484545 1179 raft_consensus.cc:515] T cb55b3d5bc63405f96cdc122f5a612cb P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:52.484884 752 tablet_service.cc:1552] Processing DeleteTablet for tablet 8d9722c764ac4b27a540595e9c6aafcd with delete_type TABLET_DATA_DELETED (replaced by tablet 51effc9324034ae79e2b25cb57a996c1) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:52.481288 1172 consensus_queue.cc:260] T bf210435010d4dde8903fd116eb468e3 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:52.485208 896 tablet_service.cc:1552] Processing DeleteTablet for tablet 8d9722c764ac4b27a540595e9c6aafcd with delete_type TABLET_DATA_DELETED (replaced by tablet 51effc9324034ae79e2b25cb57a996c1) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:52.485251 1179 leader_election.cc:290] T cb55b3d5bc63405f96cdc122f5a612cb P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:51:52.485836 1062 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb55b3d5bc63405f96cdc122f5a612cb" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:51:52.486060 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 8d9722c764ac4b27a540595e9c6aafcd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8d9722c764ac4b27a540595e9c6aafcd already in progress: creating tablet
I20251029 02:51:52.486645 1172 ts_tablet_manager.cc:1434] T bf210435010d4dde8903fd116eb468e3 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.007s user 0.002s sys 0.001s
W20251029 02:51:52.486690 712 leader_election.cc:343] T cb55b3d5bc63405f96cdc122f5a612cb P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:52.486990 1172 tablet_bootstrap.cc:492] T b867feae24e741aca6902868c35c1428 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:51:52.489120 755 tablet_service.cc:1505] Processing CreateTablet for tablet 8afc01d52d784f2fa559cab7ce07c8be (DEFAULT_TABLE table=default.Table_22184a76d5034feea912425485a578a4 [id=ed7295f3f21848b79af75785d3983b67]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.489490 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8afc01d52d784f2fa559cab7ce07c8be. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.491115 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb55b3d5bc63405f96cdc122f5a612cb" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:51:52.491809 711 leader_election.cc:343] T cb55b3d5bc63405f96cdc122f5a612cb P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:52.491945 711 leader_election.cc:304] T cb55b3d5bc63405f96cdc122f5a612cb P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:51:52.492167 1276 raft_consensus.cc:2749] T cb55b3d5bc63405f96cdc122f5a612cb P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 02:51:52.493917 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 8d9722c764ac4b27a540595e9c6aafcd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8d9722c764ac4b27a540595e9c6aafcd already in progress: creating tablet
I20251029 02:51:52.494266 1172 tablet_bootstrap.cc:654] T b867feae24e741aca6902868c35c1428 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:52.494771 1176 ts_tablet_manager.cc:1929] T 8b89ee9017b244c5b866b1fc7154ffe3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:52.494954 1176 log.cc:1199] T 8b89ee9017b244c5b866b1fc7154ffe3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/8b89ee9017b244c5b866b1fc7154ffe3
I20251029 02:51:52.495739 1176 ts_tablet_manager.cc:1950] T 8b89ee9017b244c5b866b1fc7154ffe3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:51:52.497152 1031 tablet_service.cc:1552] Processing DeleteTablet for tablet b867feae24e741aca6902868c35c1428 with delete_type TABLET_DATA_DELETED (replaced by tablet 1bab90fe904944af948dab3db06b7e34) from {username='slave'} at 127.0.0.1:37626
W20251029 02:51:52.500113 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet d6e1170dbb594a5696a3899cde8da7c9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d6e1170dbb594a5696a3899cde8da7c9 already in progress: creating tablet
I20251029 02:51:52.503425 1167 tablet_bootstrap.cc:492] T 42f463dc3cc1434b8ee5ba268cb9fc9a P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:51:52.503603 1167 ts_tablet_manager.cc:1403] T 42f463dc3cc1434b8ee5ba268cb9fc9a P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.044s user 0.004s sys 0.000s
I20251029 02:51:52.504531 1167 raft_consensus.cc:359] T 42f463dc3cc1434b8ee5ba268cb9fc9a P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:52.504740 1167 raft_consensus.cc:385] T 42f463dc3cc1434b8ee5ba268cb9fc9a P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:52.504805 1167 raft_consensus.cc:740] T 42f463dc3cc1434b8ee5ba268cb9fc9a P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:51:52.505055 1167 consensus_queue.cc:260] T 42f463dc3cc1434b8ee5ba268cb9fc9a P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:52.506970 1167 ts_tablet_manager.cc:1434] T 42f463dc3cc1434b8ee5ba268cb9fc9a P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20251029 02:51:52.507905 1167 tablet_bootstrap.cc:492] T bc490b1327e44059a57e6cd7b63350e3 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:51:52.509194 894 tablet_service.cc:1505] Processing CreateTablet for tablet 6feec2c8e030447fae9c292a626dcd1d (DEFAULT_TABLE table=default.Table_cb3d27722cc74fa18ddc8740590639a9 [id=37499c7664774cb9bd0a80c7c5bef158]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.509622 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6feec2c8e030447fae9c292a626dcd1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.509738 750 tablet_service.cc:1505] Processing CreateTablet for tablet 6feec2c8e030447fae9c292a626dcd1d (DEFAULT_TABLE table=default.Table_cb3d27722cc74fa18ddc8740590639a9 [id=37499c7664774cb9bd0a80c7c5bef158]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.510265 752 tablet_service.cc:1505] Processing CreateTablet for tablet c65469173762446e9725262ae998dbf3 (DEFAULT_TABLE table=default.Table_06e7c4a46c50490196ec56306b13a2bb [id=859e6643e304437c8dc9c14973b21b88]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 02:51:52.510732 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet c519832030884a049839d309cc738c28 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c519832030884a049839d309cc738c28 already in progress: creating tablet
I20251029 02:51:52.510977 754 tablet_service.cc:1505] Processing CreateTablet for tablet 3362e95063af41c8a24e935f547e3089 (DEFAULT_TABLE table=default.Table_b8bb03c0242d4262b78640fa8c88dc27 [id=fb42af08ec1f48498ee5e91cce62333e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.511045 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 8b89ee9017b244c5b866b1fc7154ffe3 (table default.Table_25688635c7a948f1b010aa30b88a3b6c [id=052abf3dcc904bdcb869d5b885723dab]) successfully deleted
I20251029 02:51:52.511610 1167 tablet_bootstrap.cc:654] T bc490b1327e44059a57e6cd7b63350e3 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:52.513252 1031 tablet_service.cc:1505] Processing CreateTablet for tablet 6cd7009e06a646acad0802918c2b425e (DEFAULT_TABLE table=default.Table_198644e9ee1f45ecb6ad95a759b54607 [id=7810f54ddc4849a8b13bb2928eb761a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.513738 1172 tablet_bootstrap.cc:492] T b867feae24e741aca6902868c35c1428 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:51:52.513742 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6cd7009e06a646acad0802918c2b425e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.513900 1172 ts_tablet_manager.cc:1403] T b867feae24e741aca6902868c35c1428 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.000s
I20251029 02:51:52.514681 1172 raft_consensus.cc:359] T b867feae24e741aca6902868c35c1428 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:52.514935 1172 raft_consensus.cc:385] T b867feae24e741aca6902868c35c1428 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:52.515017 1172 raft_consensus.cc:740] T b867feae24e741aca6902868c35c1428 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:51:52.515129 518 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_c8ea18626a024fcdbdbf869395f44e94" } new_table_name: "default.Table_1281926bf1224c0cb4541bf561102021" modify_external_catalogs: true
I20251029 02:51:52.515262 1172 consensus_queue.cc:260] T b867feae24e741aca6902868c35c1428 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:52.517189 1172 ts_tablet_manager.cc:1434] T b867feae24e741aca6902868c35c1428 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251029 02:51:52.520118 749 tablet_service.cc:1505] Processing CreateTablet for tablet faa835de84ff45b19525d1d603678987 (DEFAULT_TABLE table=default.Table_73790f318ab641a4bdb26c2b76b57e7e [id=a184f139921d4864b1627057a7359810]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.520237 896 tablet_service.cc:1552] Processing DeleteTablet for tablet f04473c3a5f74bdba196b91c2538537e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:51 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:52.520463 756 tablet_service.cc:1505] Processing CreateTablet for tablet 1bab90fe904944af948dab3db06b7e34 (DEFAULT_TABLE table=default.Table_c70eb9f284cd49109fd09e430a1cea6c [id=be817eb6ec33406eb316d36575c134fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.520509 893 tablet_service.cc:1505] Processing CreateTablet for tablet faa835de84ff45b19525d1d603678987 (DEFAULT_TABLE table=default.Table_73790f318ab641a4bdb26c2b76b57e7e [id=a184f139921d4864b1627057a7359810]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.520956 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet faa835de84ff45b19525d1d603678987. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.520936 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_1eac587418124e4b9aa52669873936f8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:52.521243 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet b867feae24e741aca6902868c35c1428 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b867feae24e741aca6902868c35c1428 already in progress: creating tablet
W20251029 02:51:52.521499 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1eac587418124e4b9aa52669873936f8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:52.522899 895 tablet_service.cc:1505] Processing CreateTablet for tablet 1bab90fe904944af948dab3db06b7e34 (DEFAULT_TABLE table=default.Table_c70eb9f284cd49109fd09e430a1cea6c [id=be817eb6ec33406eb316d36575c134fb]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 02:51:52.523326 518 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4b76c4408a7340e59b8506171622175f consensus state has no leader
I20251029 02:51:52.523350 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bab90fe904944af948dab3db06b7e34. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.523590 526 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet 6dd4696ad4b44b6dab87a205f3f16ebd (table default.Table_69af0d3b63c44157abd3263a9deb02da [id=90dfeabf0e8945c9bd86472d68884221]). Expected version 1 got 0
I20251029 02:51:52.526582 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_8ef360f4c77f422c805d7b24f4ef9ddb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:52.526981 1172 tablet_bootstrap.cc:492] T 5e31e061d74146009d24d5cd57487a5e P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
W20251029 02:51:52.527045 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8ef360f4c77f422c805d7b24f4ef9ddb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:52.528107 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6feec2c8e030447fae9c292a626dcd1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.520629 1174 tablet_replica.cc:333] T f04473c3a5f74bdba196b91c2538537e P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:51:52.528995 1174 raft_consensus.cc:2243] T f04473c3a5f74bdba196b91c2538537e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:52.529197 1174 raft_consensus.cc:2272] T f04473c3a5f74bdba196b91c2538537e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:52.533445 900 tablet_service.cc:1505] Processing CreateTablet for tablet c65469173762446e9725262ae998dbf3 (DEFAULT_TABLE table=default.Table_06e7c4a46c50490196ec56306b13a2bb [id=859e6643e304437c8dc9c14973b21b88]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.536525 1174 ts_tablet_manager.cc:1916] T f04473c3a5f74bdba196b91c2538537e P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:52.536761 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c65469173762446e9725262ae998dbf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.537428 896 tablet_service.cc:1505] Processing CreateTablet for tablet 3cded156380f45cc95893825c71ba0e9 (DEFAULT_TABLE table=default.Table_8b952165b0ae4603b38a50cfa0e121e0 [id=46182aaaeecc4a4d9ec60cefb55ff50d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.537606 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c65469173762446e9725262ae998dbf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.537844 892 tablet_service.cc:1505] Processing CreateTablet for tablet 1fc87e2d7b4240e08f2b4ac7277293ff (DEFAULT_TABLE table=default.Table_f7c46eb111f04bcc8824f60ec45a3a6b [id=130ea33221324d7388a014abd549b2ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.538084 891 tablet_service.cc:1505] Processing CreateTablet for tablet a5c3a15510cb4321bbb80d61fabd0c0a (DEFAULT_TABLE table=default.Table_42a71464d71d487baa7245b5c3bdbbcd [id=c124cfa700724f6db0f1a12fe495c2af]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.538138 888 tablet_service.cc:1552] Processing DeleteTablet for tablet 42f463dc3cc1434b8ee5ba268cb9fc9a with delete_type TABLET_DATA_DELETED (replaced by tablet f71388036478404d878a52fc20ece216) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:52.538800 890 tablet_service.cc:1505] Processing CreateTablet for tablet 392a36a6b59a42f78e3ac7c8fb63214b (DEFAULT_TABLE table=default.Table_e523efa604564452a1f56dada6e21daf [id=2d40c37ccbd541b9a1ed21c7f6bfb6c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.539559 899 tablet_service.cc:1505] Processing CreateTablet for tablet 2242f5b2c20346d9b753d7cd29242441 (DEFAULT_TABLE table=default.Table_e60149a6638a4b49843367e80eebda79 [id=a0e161a5d9d94a3b936669e6dfe6fb4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.539942 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet faa835de84ff45b19525d1d603678987. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.540642 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3362e95063af41c8a24e935f547e3089. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.541837 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1fc87e2d7b4240e08f2b4ac7277293ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.542459 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2242f5b2c20346d9b753d7cd29242441. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.543454 747 tablet_service.cc:1505] Processing CreateTablet for tablet 1fc87e2d7b4240e08f2b4ac7277293ff (DEFAULT_TABLE table=default.Table_f7c46eb111f04bcc8824f60ec45a3a6b [id=130ea33221324d7388a014abd549b2ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.541564 748 tablet_service.cc:1505] Processing CreateTablet for tablet 3cded156380f45cc95893825c71ba0e9 (DEFAULT_TABLE table=default.Table_8b952165b0ae4603b38a50cfa0e121e0 [id=46182aaaeecc4a4d9ec60cefb55ff50d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.544387 745 tablet_service.cc:1505] Processing CreateTablet for tablet a5c3a15510cb4321bbb80d61fabd0c0a (DEFAULT_TABLE table=default.Table_42a71464d71d487baa7245b5c3bdbbcd [id=c124cfa700724f6db0f1a12fe495c2af]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.541715 746 tablet_service.cc:1552] Processing DeleteTablet for tablet cb55b3d5bc63405f96cdc122f5a612cb with delete_type TABLET_DATA_DELETED (replaced by tablet aa776cde2f494fa5810539edf9f289a1) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:52.544782 744 tablet_service.cc:1505] Processing CreateTablet for tablet 392a36a6b59a42f78e3ac7c8fb63214b (DEFAULT_TABLE table=default.Table_e523efa604564452a1f56dada6e21daf [id=2d40c37ccbd541b9a1ed21c7f6bfb6c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.545267 1175 tablet_replica.cc:333] T cb55b3d5bc63405f96cdc122f5a612cb P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:51:52.545450 1175 raft_consensus.cc:2243] T cb55b3d5bc63405f96cdc122f5a612cb P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:52.545779 1175 raft_consensus.cc:2272] T cb55b3d5bc63405f96cdc122f5a612cb P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:52.546587 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bab90fe904944af948dab3db06b7e34. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.546753 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3cded156380f45cc95893825c71ba0e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.546948 1175 ts_tablet_manager.cc:1916] T cb55b3d5bc63405f96cdc122f5a612cb P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:52.547238 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5c3a15510cb4321bbb80d61fabd0c0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.547513 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 392a36a6b59a42f78e3ac7c8fb63214b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.547735 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 392a36a6b59a42f78e3ac7c8fb63214b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.547924 898 tablet_service.cc:1505] Processing CreateTablet for tablet 3362e95063af41c8a24e935f547e3089 (DEFAULT_TABLE table=default.Table_b8bb03c0242d4262b78640fa8c88dc27 [id=fb42af08ec1f48498ee5e91cce62333e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.548259 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5c3a15510cb4321bbb80d61fabd0c0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.548987 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3cded156380f45cc95893825c71ba0e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.549073 1038 tablet_service.cc:1505] Processing CreateTablet for tablet 5695b5a7b1854c3983e2d4ab684898b0 (DEFAULT_TABLE table=default.Table_e43a2189b18a43f9beb9f3ce25848374 [id=d151d895aac64712b2eeee989d19b878]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.549490 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5695b5a7b1854c3983e2d4ab684898b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.550805 1276 raft_consensus.cc:493] T 6dd4696ad4b44b6dab87a205f3f16ebd P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:52.550941 1276 raft_consensus.cc:515] T 6dd4696ad4b44b6dab87a205f3f16ebd P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:52.551095 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1fc87e2d7b4240e08f2b4ac7277293ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.551419 1033 tablet_service.cc:1505] Processing CreateTablet for tablet 6feec2c8e030447fae9c292a626dcd1d (DEFAULT_TABLE table=default.Table_cb3d27722cc74fa18ddc8740590639a9 [id=37499c7664774cb9bd0a80c7c5bef158]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.551677 1276 leader_election.cc:290] T 6dd4696ad4b44b6dab87a205f3f16ebd P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:51:52.551786 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6feec2c8e030447fae9c292a626dcd1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.552421 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dd4696ad4b44b6dab87a205f3f16ebd" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:51:52.552937 1032 tablet_service.cc:1505] Processing CreateTablet for tablet 1bab90fe904944af948dab3db06b7e34 (DEFAULT_TABLE table=default.Table_c70eb9f284cd49109fd09e430a1cea6c [id=be817eb6ec33406eb316d36575c134fb]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 02:51:52.552969 711 leader_election.cc:343] T 6dd4696ad4b44b6dab87a205f3f16ebd P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:52.553263 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bab90fe904944af948dab3db06b7e34. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.554312 1040 tablet_service.cc:1505] Processing CreateTablet for tablet 8afc01d52d784f2fa559cab7ce07c8be (DEFAULT_TABLE table=default.Table_22184a76d5034feea912425485a578a4 [id=ed7295f3f21848b79af75785d3983b67]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.554558 1037 tablet_service.cc:1552] Processing DeleteTablet for tablet 8d9722c764ac4b27a540595e9c6aafcd with delete_type TABLET_DATA_DELETED (replaced by tablet 51effc9324034ae79e2b25cb57a996c1) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:52.554862 1034 tablet_service.cc:1505] Processing CreateTablet for tablet c65469173762446e9725262ae998dbf3 (DEFAULT_TABLE table=default.Table_06e7c4a46c50490196ec56306b13a2bb [id=859e6643e304437c8dc9c14973b21b88]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.555163 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c65469173762446e9725262ae998dbf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.556010 1172 tablet_bootstrap.cc:654] T 5e31e061d74146009d24d5cd57487a5e P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:52.558684 1035 tablet_service.cc:1505] Processing CreateTablet for tablet faa835de84ff45b19525d1d603678987 (DEFAULT_TABLE table=default.Table_73790f318ab641a4bdb26c2b76b57e7e [id=a184f139921d4864b1627057a7359810]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.559094 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet faa835de84ff45b19525d1d603678987. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.559095 1039 tablet_service.cc:1505] Processing CreateTablet for tablet 3362e95063af41c8a24e935f547e3089 (DEFAULT_TABLE table=default.Table_b8bb03c0242d4262b78640fa8c88dc27 [id=fb42af08ec1f48498ee5e91cce62333e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.560552 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8afc01d52d784f2fa559cab7ce07c8be. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.560796 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3362e95063af41c8a24e935f547e3089. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.562575 1036 tablet_service.cc:1505] Processing CreateTablet for tablet a5c3a15510cb4321bbb80d61fabd0c0a (DEFAULT_TABLE table=default.Table_42a71464d71d487baa7245b5c3bdbbcd [id=c124cfa700724f6db0f1a12fe495c2af]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.563251 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5c3a15510cb4321bbb80d61fabd0c0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.565181 1041 tablet_service.cc:1505] Processing CreateTablet for tablet 392a36a6b59a42f78e3ac7c8fb63214b (DEFAULT_TABLE table=default.Table_e523efa604564452a1f56dada6e21daf [id=2d40c37ccbd541b9a1ed21c7f6bfb6c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.565510 1037 tablet_service.cc:1505] Processing CreateTablet for tablet 3cded156380f45cc95893825c71ba0e9 (DEFAULT_TABLE table=default.Table_8b952165b0ae4603b38a50cfa0e121e0 [id=46182aaaeecc4a4d9ec60cefb55ff50d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.566571 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3362e95063af41c8a24e935f547e3089. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.571947 1062 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dd4696ad4b44b6dab87a205f3f16ebd" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:51:52.588486 712 leader_election.cc:343] T 6dd4696ad4b44b6dab87a205f3f16ebd P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:52.588658 712 leader_election.cc:304] T 6dd4696ad4b44b6dab87a205f3f16ebd P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:51:52.589342 1276 raft_consensus.cc:2749] T 6dd4696ad4b44b6dab87a205f3f16ebd P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:51:52.593890 1174 ts_tablet_manager.cc:1929] T f04473c3a5f74bdba196b91c2538537e P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:52.595744 520 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_83d3fe1030e0461895fc2437732b4ff8" } modify_external_catalogs: true
I20251029 02:51:52.595947 520 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_83d3fe1030e0461895fc2437732b4ff8" } modify_external_catalogs: true
I20251029 02:51:52.564920 1030 tablet_service.cc:1505] Processing CreateTablet for tablet 1fc87e2d7b4240e08f2b4ac7277293ff (DEFAULT_TABLE table=default.Table_f7c46eb111f04bcc8824f60ec45a3a6b [id=130ea33221324d7388a014abd549b2ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.598351 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 392a36a6b59a42f78e3ac7c8fb63214b. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:51:52.600698 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 8d9722c764ac4b27a540595e9c6aafcd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8d9722c764ac4b27a540595e9c6aafcd already in progress: creating tablet
I20251029 02:51:52.571349 1042 tablet_service.cc:1505] Processing CreateTablet for tablet 2242f5b2c20346d9b753d7cd29242441 (DEFAULT_TABLE table=default.Table_e60149a6638a4b49843367e80eebda79 [id=a0e161a5d9d94a3b936669e6dfe6fb4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.602828 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1fc87e2d7b4240e08f2b4ac7277293ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.604182 756 tablet_service.cc:1552] Processing DeleteTablet for tablet b867feae24e741aca6902868c35c1428 with delete_type TABLET_DATA_DELETED (replaced by tablet 1bab90fe904944af948dab3db06b7e34) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:52.606671 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2242f5b2c20346d9b753d7cd29242441. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.607477 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3cded156380f45cc95893825c71ba0e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.608398 1167 tablet_bootstrap.cc:492] T bc490b1327e44059a57e6cd7b63350e3 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:51:52.608566 1167 ts_tablet_manager.cc:1403] T bc490b1327e44059a57e6cd7b63350e3 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.101s user 0.000s sys 0.008s
I20251029 02:51:52.609336 520 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 7c041f4909274cd5a03b9fd78353faf9
I20251029 02:51:52.609382 1167 raft_consensus.cc:359] T bc490b1327e44059a57e6cd7b63350e3 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:52.609572 1167 raft_consensus.cc:385] T bc490b1327e44059a57e6cd7b63350e3 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:52.609654 1167 raft_consensus.cc:740] T bc490b1327e44059a57e6cd7b63350e3 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:51:52.609897 1167 consensus_queue.cc:260] T bc490b1327e44059a57e6cd7b63350e3 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:52.611359 1167 ts_tablet_manager.cc:1434] T bc490b1327e44059a57e6cd7b63350e3 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20251029 02:51:52.611688 1167 tablet_bootstrap.cc:492] T 2ebde39f813d46e898ad1754325ad1a0 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:51:52.613761 756 tablet_service.cc:1552] Processing DeleteTablet for tablet 7c041f4909274cd5a03b9fd78353faf9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:52 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:52.614928 1167 tablet_bootstrap.cc:654] T 2ebde39f813d46e898ad1754325ad1a0 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:52.616539 900 tablet_service.cc:1552] Processing DeleteTablet for tablet 7c041f4909274cd5a03b9fd78353faf9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:52 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:52.616729 1174 log.cc:1199] T f04473c3a5f74bdba196b91c2538537e P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/f04473c3a5f74bdba196b91c2538537e
I20251029 02:51:52.617435 522 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_73790f318ab641a4bdb26c2b76b57e7e" } new_table_name: "default.Table_f44737ca5bae4763878b58e5ab45495a" modify_external_catalogs: true
I20251029 02:51:52.617590 518 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_cb3d27722cc74fa18ddc8740590639a9" } modify_external_catalogs: true
I20251029 02:51:52.617726 518 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_cb3d27722cc74fa18ddc8740590639a9" } modify_external_catalogs: true
I20251029 02:51:52.618065 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_4350e92f14de4ba8a04afbe12ea7422a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:52.618675 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4350e92f14de4ba8a04afbe12ea7422a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:52.618628 526 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_8ef360f4c77f422c805d7b24f4ef9ddb" } modify_external_catalogs: true
I20251029 02:51:52.618813 526 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_8ef360f4c77f422c805d7b24f4ef9ddb" } modify_external_catalogs: true
I20251029 02:51:52.619117 519 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_d20d0c82c12a4bba9b0749e62a17c2e5" } new_table_name: "default.Table_ef563eb489f947d4a4772678806a0777" modify_external_catalogs: true
I20251029 02:51:52.617127 1174 ts_tablet_manager.cc:1950] T f04473c3a5f74bdba196b91c2538537e P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:51:52.623363 1174 tablet_replica.cc:333] T 42f463dc3cc1434b8ee5ba268cb9fc9a P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:51:52.624138 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet f04473c3a5f74bdba196b91c2538537e (table default.Table_7e818850675e4c5c973db094e7223348 [id=ec8cdbf19343401da05dec0c8c060ae6]) successfully deleted
I20251029 02:51:52.625170 1174 raft_consensus.cc:2243] T 42f463dc3cc1434b8ee5ba268cb9fc9a P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:52.625380 1174 raft_consensus.cc:2272] T 42f463dc3cc1434b8ee5ba268cb9fc9a P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:52.630498 518 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 6feec2c8e030447fae9c292a626dcd1d
I20251029 02:51:52.634312 900 tablet_service.cc:1552] Processing DeleteTablet for tablet 6feec2c8e030447fae9c292a626dcd1d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:52 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:52.636096 1175 ts_tablet_manager.cc:1929] T cb55b3d5bc63405f96cdc122f5a612cb P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:52.636253 1175 log.cc:1199] T cb55b3d5bc63405f96cdc122f5a612cb P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/cb55b3d5bc63405f96cdc122f5a612cb
I20251029 02:51:52.636684 1175 ts_tablet_manager.cc:1950] T cb55b3d5bc63405f96cdc122f5a612cb P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:51:52.637154 526 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 57f0da6b219e4e7d8e0d5d5a691ec209
W20251029 02:51:52.637264 519 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7309d3190cfa415281c5253a634fa7f3 consensus state has no leader
W20251029 02:51:52.637661 522 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet faa835de84ff45b19525d1d603678987 consensus state has no leader
I20251029 02:51:52.638136 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet cb55b3d5bc63405f96cdc122f5a612cb (table default.Table_47a1a59db0444fa49f7274509385ff0c [id=283c09ba60114ba9b165e2ff144a6c35]) successfully deleted
I20251029 02:51:52.640389 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_a873546f69334d12b0cbecce2cd489fb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:52.641001 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a873546f69334d12b0cbecce2cd489fb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:52.644372 522 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_69af0d3b63c44157abd3263a9deb02da" } modify_external_catalogs: true
I20251029 02:51:52.644563 522 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_69af0d3b63c44157abd3263a9deb02da" } modify_external_catalogs: true
I20251029 02:51:52.657873 1033 tablet_service.cc:1552] Processing DeleteTablet for tablet 6feec2c8e030447fae9c292a626dcd1d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:52 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:52.658696 748 tablet_service.cc:1552] Processing DeleteTablet for tablet bf210435010d4dde8903fd116eb468e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:51 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:52.659082 744 tablet_service.cc:1552] Processing DeleteTablet for tablet 6feec2c8e030447fae9c292a626dcd1d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:52 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:52.662815 1039 tablet_service.cc:1552] Processing DeleteTablet for tablet 7c041f4909274cd5a03b9fd78353faf9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:52 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:52.668051 519 master_service.cc:954] ReplaceTablet: received request to replace tablet ab421dfaa3f84593842bb732a2693160 from {username='slave'} at 127.0.0.1:35314
I20251029 02:51:52.670890 526 master_service.cc:954] ReplaceTablet: received request to replace tablet fd900eec3f0c43bf965c148ea90870a3 from {username='slave'} at 127.0.0.1:35314
I20251029 02:51:52.673684 424 master_runner.cc:186] Error getting master registration for 127.0.104.60:41233: Network error: Client connection negotiation failed: client connection to 127.0.104.60:41233: connect: Connection refused (error 111),
I20251029 02:51:52.673897 424 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251029 02:51:52.674798 1175 tablet_replica.cc:333] T b867feae24e741aca6902868c35c1428 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:51:52.674449 518 master_service.cc:954] ReplaceTablet: received request to replace tablet bc1a1cf274b64b52aba58811e9594330 from {username='slave'} at 127.0.0.1:35314
I20251029 02:51:52.675082 1175 raft_consensus.cc:2243] T b867feae24e741aca6902868c35c1428 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:52.675309 1175 raft_consensus.cc:2272] T b867feae24e741aca6902868c35c1428 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:52.676047 1174 ts_tablet_manager.cc:1916] T 42f463dc3cc1434b8ee5ba268cb9fc9a P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:52.676306 525 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_fd85e1b0ff874cdc985600a3fa32085c" } new_table_name: "default.Table_f2172770f9a94bb98d7813da6e1848fe" modify_external_catalogs: true
I20251029 02:51:52.676491 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_ed3714840df249bfb7cd3f3d9abaa578"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:52.678678 1175 ts_tablet_manager.cc:1916] T b867feae24e741aca6902868c35c1428 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:52.679387 517 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_06e7c4a46c50490196ec56306b13a2bb" } modify_external_catalogs: true
I20251029 02:51:52.681892 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_282e9d0ae9384d3284e662fa864925a9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:52.682435 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_282e9d0ae9384d3284e662fa864925a9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:52.682715 517 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_06e7c4a46c50490196ec56306b13a2bb" } modify_external_catalogs: true
W20251029 02:51:52.685220 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ed3714840df249bfb7cd3f3d9abaa578 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:52.686039 753 tablet_service.cc:1505] Processing CreateTablet for tablet 2242f5b2c20346d9b753d7cd29242441 (DEFAULT_TABLE table=default.Table_e60149a6638a4b49843367e80eebda79 [id=a0e161a5d9d94a3b936669e6dfe6fb4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.686691 893 tablet_service.cc:1505] Processing CreateTablet for tablet 7309d3190cfa415281c5253a634fa7f3 (DEFAULT_TABLE table=default.Table_ef563eb489f947d4a4772678806a0777 [id=7de93d6d48ab4d58b5aed9319d7299a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.686905 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2242f5b2c20346d9b753d7cd29242441. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.687088 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7309d3190cfa415281c5253a634fa7f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.687404 1167 tablet_bootstrap.cc:492] T 2ebde39f813d46e898ad1754325ad1a0 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:51:52.687613 1167 ts_tablet_manager.cc:1403] T 2ebde39f813d46e898ad1754325ad1a0 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.076s user 0.018s sys 0.007s
W20251029 02:51:52.688037 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 6feec2c8e030447fae9c292a626dcd1d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6feec2c8e030447fae9c292a626dcd1d already in progress: creating tablet
W20251029 02:51:52.688302 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 7c041f4909274cd5a03b9fd78353faf9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7c041f4909274cd5a03b9fd78353faf9 already in progress: creating tablet
I20251029 02:51:52.688536 1167 raft_consensus.cc:359] T 2ebde39f813d46e898ad1754325ad1a0 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:52.688771 1167 raft_consensus.cc:385] T 2ebde39f813d46e898ad1754325ad1a0 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:52.688637 755 tablet_service.cc:1505] Processing CreateTablet for tablet 7309d3190cfa415281c5253a634fa7f3 (DEFAULT_TABLE table=default.Table_ef563eb489f947d4a4772678806a0777 [id=7de93d6d48ab4d58b5aed9319d7299a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.688855 1167 raft_consensus.cc:740] T 2ebde39f813d46e898ad1754325ad1a0 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:51:52.689188 1167 consensus_queue.cc:260] T 2ebde39f813d46e898ad1754325ad1a0 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:52.689662 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7309d3190cfa415281c5253a634fa7f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.689579 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_f178e528491d416a99b0191e864f53d8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:52.690096 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f178e528491d416a99b0191e864f53d8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:52.691071 1167 ts_tablet_manager.cc:1434] T 2ebde39f813d46e898ad1754325ad1a0 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251029 02:51:52.691401 1167 tablet_bootstrap.cc:492] T 86f8776b5c4c4082bf276199feaaefea P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:51:52.695359 1167 tablet_bootstrap.cc:654] T 86f8776b5c4c4082bf276199feaaefea P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:52.695616 524 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_25688635c7a948f1b010aa30b88a3b6c" } new_table_name: "default.Table_eb28ff62f787444189cb7dfe9e735992" modify_external_catalogs: true
I20251029 02:51:52.696553 1175 ts_tablet_manager.cc:1929] T b867feae24e741aca6902868c35c1428 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:52.696692 1175 log.cc:1199] T b867feae24e741aca6902868c35c1428 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/b867feae24e741aca6902868c35c1428
I20251029 02:51:52.697048 1175 ts_tablet_manager.cc:1950] T b867feae24e741aca6902868c35c1428 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:51:52.698100 1175 tablet_replica.cc:333] T bf210435010d4dde8903fd116eb468e3 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:51:52.698103 522 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 6dd4696ad4b44b6dab87a205f3f16ebd
I20251029 02:51:52.698273 1175 raft_consensus.cc:2243] T bf210435010d4dde8903fd116eb468e3 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:52.698462 1175 raft_consensus.cc:2272] T bf210435010d4dde8903fd116eb468e3 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:52.699222 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet b867feae24e741aca6902868c35c1428 (table default.Table_c70eb9f284cd49109fd09e430a1cea6c [id=be817eb6ec33406eb316d36575c134fb]) successfully deleted
W20251029 02:51:52.699463 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 7c041f4909274cd5a03b9fd78353faf9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7c041f4909274cd5a03b9fd78353faf9 already in progress: creating tablet
I20251029 02:51:52.700035 1172 tablet_bootstrap.cc:492] T 5e31e061d74146009d24d5cd57487a5e P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:51:52.700194 1172 ts_tablet_manager.cc:1403] T 5e31e061d74146009d24d5cd57487a5e P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.173s user 0.019s sys 0.014s
I20251029 02:51:52.700830 744 tablet_service.cc:1552] Processing DeleteTablet for tablet 6dd4696ad4b44b6dab87a205f3f16ebd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:52 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:52.701013 1172 raft_consensus.cc:359] T 5e31e061d74146009d24d5cd57487a5e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:52.701212 1172 raft_consensus.cc:385] T 5e31e061d74146009d24d5cd57487a5e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:52.701301 1172 raft_consensus.cc:740] T 5e31e061d74146009d24d5cd57487a5e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:51:52.701606 1172 consensus_queue.cc:260] T 5e31e061d74146009d24d5cd57487a5e P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:52.703194 1276 raft_consensus.cc:493] T 9be9add0274d41b688edd6292a253a06 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:52.703352 1276 raft_consensus.cc:515] T 9be9add0274d41b688edd6292a253a06 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:52.704043 1276 leader_election.cc:290] T 9be9add0274d41b688edd6292a253a06 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:51:52.704576 1172 ts_tablet_manager.cc:1434] T 5e31e061d74146009d24d5cd57487a5e P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251029 02:51:52.704706 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9be9add0274d41b688edd6292a253a06" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:51:52.705319 711 leader_election.cc:343] T 9be9add0274d41b688edd6292a253a06 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:52.706738 1062 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9be9add0274d41b688edd6292a253a06" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:51:52.707422 712 leader_election.cc:343] T 9be9add0274d41b688edd6292a253a06 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:52.707567 712 leader_election.cc:304] T 9be9add0274d41b688edd6292a253a06 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:51:52.707597 893 tablet_service.cc:1552] Processing DeleteTablet for tablet 6dd4696ad4b44b6dab87a205f3f16ebd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:52 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:52.709911 1276 raft_consensus.cc:2749] T 9be9add0274d41b688edd6292a253a06 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:51:52.711217 1037 tablet_service.cc:1505] Processing CreateTablet for tablet 7309d3190cfa415281c5253a634fa7f3 (DEFAULT_TABLE table=default.Table_ef563eb489f947d4a4772678806a0777 [id=7de93d6d48ab4d58b5aed9319d7299a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.711670 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7309d3190cfa415281c5253a634fa7f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.713447 1172 tablet_bootstrap.cc:492] T 61c1fec3ab4c402eb29f767b9bb7409f P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:51:52.715423 1172 tablet_bootstrap.cc:654] T 61c1fec3ab4c402eb29f767b9bb7409f P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:52.716593 1174 ts_tablet_manager.cc:1929] T 42f463dc3cc1434b8ee5ba268cb9fc9a P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:52.716727 1174 log.cc:1199] T 42f463dc3cc1434b8ee5ba268cb9fc9a P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/42f463dc3cc1434b8ee5ba268cb9fc9a
I20251029 02:51:52.717120 1174 ts_tablet_manager.cc:1950] T 42f463dc3cc1434b8ee5ba268cb9fc9a P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:51:52.718331 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 42f463dc3cc1434b8ee5ba268cb9fc9a (table default.Table_03f7d6f02b064939ba7464c5173e02e3 [id=9898cffc2b964059ba3163ee4d1395c2]) successfully deleted
I20251029 02:51:52.721586 1175 ts_tablet_manager.cc:1916] T bf210435010d4dde8903fd116eb468e3 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 02:51:52.724233 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 7c041f4909274cd5a03b9fd78353faf9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7c041f4909274cd5a03b9fd78353faf9 already in progress: creating tablet
I20251029 02:51:52.727748 1037 tablet_service.cc:1552] Processing DeleteTablet for tablet 6dd4696ad4b44b6dab87a205f3f16ebd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:52 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:52.728077 1166 tablet_bootstrap.cc:492] T 94854145ac0d4be38780d4cdb721725e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
W20251029 02:51:52.728843 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 6feec2c8e030447fae9c292a626dcd1d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6feec2c8e030447fae9c292a626dcd1d already in progress: creating tablet
W20251029 02:51:52.729591 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 6dd4696ad4b44b6dab87a205f3f16ebd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6dd4696ad4b44b6dab87a205f3f16ebd already in progress: creating tablet
I20251029 02:51:52.730926 526 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet fd900eec3f0c43bf965c148ea90870a3
I20251029 02:51:52.731369 526 catalog_manager.cc:6719] ReplaceTablet: tablet fd900eec3f0c43bf965c148ea90870a3 deleted and replaced by tablet b2b2d64dcf964da8be4b544dbbe31977
W20251029 02:51:52.731760 525 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 69597d48631d43b6a5b92abbf60d5ed3 consensus state has no leader
I20251029 02:51:52.732499 893 tablet_service.cc:1552] Processing DeleteTablet for tablet fd900eec3f0c43bf965c148ea90870a3 with delete_type TABLET_DATA_DELETED (replaced by tablet b2b2d64dcf964da8be4b544dbbe31977) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:52.732913 519 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet ab421dfaa3f84593842bb732a2693160
I20251029 02:51:52.733265 519 catalog_manager.cc:6719] ReplaceTablet: tablet ab421dfaa3f84593842bb732a2693160 deleted and replaced by tablet c9e923724ca9402682f13b252dbc37aa
I20251029 02:51:52.733918 893 tablet_service.cc:1552] Processing DeleteTablet for tablet ab421dfaa3f84593842bb732a2693160 with delete_type TABLET_DATA_DELETED (replaced by tablet c9e923724ca9402682f13b252dbc37aa) from {username='slave'} at 127.0.0.1:35702
W20251029 02:51:52.734846 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet fd900eec3f0c43bf965c148ea90870a3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fd900eec3f0c43bf965c148ea90870a3 already in progress: creating tablet
I20251029 02:51:52.735033 1167 tablet_bootstrap.cc:492] T 86f8776b5c4c4082bf276199feaaefea P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:51:52.735185 1167 ts_tablet_manager.cc:1403] T 86f8776b5c4c4082bf276199feaaefea P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.044s user 0.004s sys 0.004s
W20251029 02:51:52.735638 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet ab421dfaa3f84593842bb732a2693160 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ab421dfaa3f84593842bb732a2693160 already in progress: creating tablet
I20251029 02:51:52.735747 1166 ts_tablet_manager.cc:1403] T 94854145ac0d4be38780d4cdb721725e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.285s user 0.012s sys 0.006s
I20251029 02:51:52.736054 1167 raft_consensus.cc:359] T 86f8776b5c4c4082bf276199feaaefea P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:52.736264 1167 raft_consensus.cc:385] T 86f8776b5c4c4082bf276199feaaefea P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:52.736380 1167 raft_consensus.cc:740] T 86f8776b5c4c4082bf276199feaaefea P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:51:52.736806 1166 raft_consensus.cc:359] T 94854145ac0d4be38780d4cdb721725e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:52.737498 526 master_service.cc:954] ReplaceTablet: received request to replace tablet 3362e95063af41c8a24e935f547e3089 from {username='slave'} at 127.0.0.1:35314
I20251029 02:51:52.737718 1166 raft_consensus.cc:385] T 94854145ac0d4be38780d4cdb721725e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:52.738330 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_7ab08f1eb370432b985cd5c0f9988017"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:52.738436 753 tablet_service.cc:1552] Processing DeleteTablet for tablet ab421dfaa3f84593842bb732a2693160 with delete_type TABLET_DATA_DELETED (replaced by tablet c9e923724ca9402682f13b252dbc37aa) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:52.738744 1166 raft_consensus.cc:740] T 94854145ac0d4be38780d4cdb721725e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
W20251029 02:51:52.738940 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7ab08f1eb370432b985cd5c0f9988017 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:52.739239 755 tablet_service.cc:1552] Processing DeleteTablet for tablet fd900eec3f0c43bf965c148ea90870a3 with delete_type TABLET_DATA_DELETED (replaced by tablet b2b2d64dcf964da8be4b544dbbe31977) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:52.739248 1166 consensus_queue.cc:260] T 94854145ac0d4be38780d4cdb721725e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:52.740062 1172 tablet_bootstrap.cc:492] T 61c1fec3ab4c402eb29f767b9bb7409f P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:51:52.740223 1172 ts_tablet_manager.cc:1403] T 61c1fec3ab4c402eb29f767b9bb7409f P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.027s user 0.006s sys 0.001s
I20251029 02:51:52.740243 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_99592a0735024ba7a0d8d6145b61f2e4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:52.740775 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_99592a0735024ba7a0d8d6145b61f2e4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:52.740999 1172 raft_consensus.cc:359] T 61c1fec3ab4c402eb29f767b9bb7409f P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:52.741196 1172 raft_consensus.cc:385] T 61c1fec3ab4c402eb29f767b9bb7409f P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:52.741290 1172 raft_consensus.cc:740] T 61c1fec3ab4c402eb29f767b9bb7409f P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:51:52.741232 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_41af15055e73444a9fbe64c4c4f63dba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:52.741686 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_41af15055e73444a9fbe64c4c4f63dba in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:52.743402 517 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet c65469173762446e9725262ae998dbf3
I20251029 02:51:52.744017 517 master_service.cc:954] ReplaceTablet: received request to replace tablet 2242f5b2c20346d9b753d7cd29242441 from {username='slave'} at 127.0.0.1:35314
I20251029 02:51:52.746372 521 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_1eac587418124e4b9aa52669873936f8" } new_table_name: "default.Table_72f8739f5f654172b10d619a9ac306bd" modify_external_catalogs: true
I20251029 02:51:52.747910 893 tablet_service.cc:1552] Processing DeleteTablet for tablet c65469173762446e9725262ae998dbf3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:52 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:52.748476 1166 ts_tablet_manager.cc:1434] T 94854145ac0d4be38780d4cdb721725e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20251029 02:51:52.748662 518 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet bc1a1cf274b64b52aba58811e9594330
I20251029 02:51:52.748653 755 tablet_service.cc:1552] Processing DeleteTablet for tablet c65469173762446e9725262ae998dbf3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:52 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:52.749194 518 catalog_manager.cc:6719] ReplaceTablet: tablet bc1a1cf274b64b52aba58811e9594330 deleted and replaced by tablet f81aecb20d6e498bb9b498698886d65e
I20251029 02:51:52.749136 1172 consensus_queue.cc:260] T 61c1fec3ab4c402eb29f767b9bb7409f P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
W20251029 02:51:52.749578 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet c65469173762446e9725262ae998dbf3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c65469173762446e9725262ae998dbf3 already in progress: creating tablet
I20251029 02:51:52.750176 893 tablet_service.cc:1552] Processing DeleteTablet for tablet bc1a1cf274b64b52aba58811e9594330 with delete_type TABLET_DATA_DELETED (replaced by tablet f81aecb20d6e498bb9b498698886d65e) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:52.751195 755 tablet_service.cc:1552] Processing DeleteTablet for tablet bc1a1cf274b64b52aba58811e9594330 with delete_type TABLET_DATA_DELETED (replaced by tablet f81aecb20d6e498bb9b498698886d65e) from {username='slave'} at 127.0.0.1:44580
W20251029 02:51:52.751616 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet bc1a1cf274b64b52aba58811e9594330 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bc1a1cf274b64b52aba58811e9594330 already in progress: creating tablet
I20251029 02:51:52.754316 1037 tablet_service.cc:1552] Processing DeleteTablet for tablet fd900eec3f0c43bf965c148ea90870a3 with delete_type TABLET_DATA_DELETED (replaced by tablet b2b2d64dcf964da8be4b544dbbe31977) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:52.755672 1166 tablet_bootstrap.cc:492] T 86f8776b5c4c4082bf276199feaaefea P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:51:52.756124 1172 ts_tablet_manager.cc:1434] T 61c1fec3ab4c402eb29f767b9bb7409f P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.016s user 0.001s sys 0.001s
I20251029 02:51:52.756440 1172 tablet_bootstrap.cc:492] T bc1a1cf274b64b52aba58811e9594330 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:51:52.757751 1037 tablet_service.cc:1552] Processing DeleteTablet for tablet ab421dfaa3f84593842bb732a2693160 with delete_type TABLET_DATA_DELETED (replaced by tablet c9e923724ca9402682f13b252dbc37aa) from {username='slave'} at 127.0.0.1:37626
W20251029 02:51:52.758924 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 6dd4696ad4b44b6dab87a205f3f16ebd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6dd4696ad4b44b6dab87a205f3f16ebd already in progress: creating tablet
I20251029 02:51:52.758947 1167 consensus_queue.cc:260] T 86f8776b5c4c4082bf276199feaaefea P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:52.760208 1166 tablet_bootstrap.cc:654] T 86f8776b5c4c4082bf276199feaaefea P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:52.761364 1167 ts_tablet_manager.cc:1434] T 86f8776b5c4c4082bf276199feaaefea P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.026s user 0.003s sys 0.000s
W20251029 02:51:52.762995 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet fd900eec3f0c43bf965c148ea90870a3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fd900eec3f0c43bf965c148ea90870a3 already in progress: creating tablet
W20251029 02:51:52.763336 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet ab421dfaa3f84593842bb732a2693160 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ab421dfaa3f84593842bb732a2693160 already in progress: creating tablet
I20251029 02:51:52.766414 1037 tablet_service.cc:1552] Processing DeleteTablet for tablet c65469173762446e9725262ae998dbf3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:52 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:52.768757 1037 tablet_service.cc:1552] Processing DeleteTablet for tablet 94854145ac0d4be38780d4cdb721725e with delete_type TABLET_DATA_DELETED (replaced by tablet 82f75c134c1a4858a2154ee6b1d1d29f) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:52.769047 1176 tablet_replica.cc:333] T 94854145ac0d4be38780d4cdb721725e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:51:52.769239 1176 raft_consensus.cc:2243] T 94854145ac0d4be38780d4cdb721725e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:52.769423 1176 raft_consensus.cc:2272] T 94854145ac0d4be38780d4cdb721725e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:52.770232 1176 ts_tablet_manager.cc:1916] T 94854145ac0d4be38780d4cdb721725e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:52.772258 1172 tablet_bootstrap.cc:654] T bc1a1cf274b64b52aba58811e9594330 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:52.772910 893 tablet_service.cc:1552] Processing DeleteTablet for tablet 2ebde39f813d46e898ad1754325ad1a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:51 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:52.773149 1174 tablet_replica.cc:333] T 2ebde39f813d46e898ad1754325ad1a0 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:51:52.773355 1174 raft_consensus.cc:2243] T 2ebde39f813d46e898ad1754325ad1a0 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:52.773425 1167 tablet_bootstrap.cc:492] T deca0c4a114441079fce22e74b403050 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:51:52.773543 1174 raft_consensus.cc:2272] T 2ebde39f813d46e898ad1754325ad1a0 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:52.774338 1174 ts_tablet_manager.cc:1916] T 2ebde39f813d46e898ad1754325ad1a0 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:52.777603 1176 ts_tablet_manager.cc:1929] T 94854145ac0d4be38780d4cdb721725e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:52.777840 1176 log.cc:1199] T 94854145ac0d4be38780d4cdb721725e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/94854145ac0d4be38780d4cdb721725e
I20251029 02:51:52.778436 1176 ts_tablet_manager.cc:1950] T 94854145ac0d4be38780d4cdb721725e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
W20251029 02:51:52.779495 524 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dd90fb9eb167478cbc3ec9363af86295 consensus state has no leader
I20251029 02:51:52.780788 1037 tablet_service.cc:1552] Processing DeleteTablet for tablet bc1a1cf274b64b52aba58811e9594330 with delete_type TABLET_DATA_DELETED (replaced by tablet f81aecb20d6e498bb9b498698886d65e) from {username='slave'} at 127.0.0.1:37626
W20251029 02:51:52.783560 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet c65469173762446e9725262ae998dbf3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c65469173762446e9725262ae998dbf3 already in progress: creating tablet
I20251029 02:51:52.783897 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 94854145ac0d4be38780d4cdb721725e (table default.Table_631cf79ec867461baecbc486773b88c4 [id=7a493d6672624c90a81a647d7785181c]) successfully deleted
W20251029 02:51:52.784149 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet bc1a1cf274b64b52aba58811e9594330 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bc1a1cf274b64b52aba58811e9594330 already in progress: creating tablet
I20251029 02:51:52.785020 1174 ts_tablet_manager.cc:1929] T 2ebde39f813d46e898ad1754325ad1a0 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:52.785269 1174 log.cc:1199] T 2ebde39f813d46e898ad1754325ad1a0 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/2ebde39f813d46e898ad1754325ad1a0
I20251029 02:51:52.785749 1174 ts_tablet_manager.cc:1950] T 2ebde39f813d46e898ad1754325ad1a0 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:51:52.786584 893 tablet_service.cc:1552] Processing DeleteTablet for tablet 86f8776b5c4c4082bf276199feaaefea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:51 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:52.786916 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 2ebde39f813d46e898ad1754325ad1a0 (table default.Table_1a524bb5f82a4041b8008ff91a0b9d27 [id=5db5ea0ab3e249f4acea117bb07187f4]) successfully deleted
I20251029 02:51:52.789517 1167 tablet_bootstrap.cc:654] T deca0c4a114441079fce22e74b403050 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:52.791049 1174 tablet_replica.cc:333] T 86f8776b5c4c4082bf276199feaaefea P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:51:52.791503 1166 tablet_bootstrap.cc:492] T 86f8776b5c4c4082bf276199feaaefea P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:51:52.791707 1166 ts_tablet_manager.cc:1403] T 86f8776b5c4c4082bf276199feaaefea P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.036s user 0.002s sys 0.003s
I20251029 02:51:52.792557 1166 raft_consensus.cc:359] T 86f8776b5c4c4082bf276199feaaefea P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:52.792881 1166 raft_consensus.cc:385] T 86f8776b5c4c4082bf276199feaaefea P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:52.793006 1166 raft_consensus.cc:740] T 86f8776b5c4c4082bf276199feaaefea P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:51:52.793326 1166 consensus_queue.cc:260] T 86f8776b5c4c4082bf276199feaaefea P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:52.796447 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_dea0766540d24b5e9bd6cc8b6338b47c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:52.797082 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dea0766540d24b5e9bd6cc8b6338b47c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:52.798398 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_4013d5c712574aa69143efea710b9b39"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:52.798972 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4013d5c712574aa69143efea710b9b39 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:52.800227 1037 tablet_service.cc:1552] Processing DeleteTablet for tablet 86f8776b5c4c4082bf276199feaaefea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:51 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:52.800534 1174 raft_consensus.cc:2243] T 86f8776b5c4c4082bf276199feaaefea P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:52.800976 1174 raft_consensus.cc:2272] T 86f8776b5c4c4082bf276199feaaefea P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus is shut down!
W20251029 02:51:52.801015 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 86f8776b5c4c4082bf276199feaaefea because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 86f8776b5c4c4082bf276199feaaefea already in progress: creating tablet
I20251029 02:51:52.801678 1166 ts_tablet_manager.cc:1434] T 86f8776b5c4c4082bf276199feaaefea P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20251029 02:51:52.803994 1308 raft_consensus.cc:493] T bc490b1327e44059a57e6cd7b63350e3 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:52.804191 1308 raft_consensus.cc:515] T bc490b1327e44059a57e6cd7b63350e3 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:52.804915 1308 leader_election.cc:290] T bc490b1327e44059a57e6cd7b63350e3 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:51:52.806165 1166 tablet_bootstrap.cc:492] T 2ebde39f813d46e898ad1754325ad1a0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:51:52.808327 1062 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc490b1327e44059a57e6cd7b63350e3" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:51:52.808992 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc490b1327e44059a57e6cd7b63350e3" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
W20251029 02:51:52.809064 849 leader_election.cc:343] T bc490b1327e44059a57e6cd7b63350e3 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:52.809218 776 raft_consensus.cc:2468] T bc490b1327e44059a57e6cd7b63350e3 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:51:52.809350 1166 tablet_bootstrap.cc:654] T 2ebde39f813d46e898ad1754325ad1a0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:52.810106 847 leader_election.cc:304] T bc490b1327e44059a57e6cd7b63350e3 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5
I20251029 02:51:52.811133 1308 raft_consensus.cc:2804] T bc490b1327e44059a57e6cd7b63350e3 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:51:52.811180 1174 ts_tablet_manager.cc:1916] T 86f8776b5c4c4082bf276199feaaefea P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:52.811247 1308 raft_consensus.cc:493] T bc490b1327e44059a57e6cd7b63350e3 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:51:52.811383 1308 raft_consensus.cc:3060] T bc490b1327e44059a57e6cd7b63350e3 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:51:52.814482 1308 raft_consensus.cc:515] T bc490b1327e44059a57e6cd7b63350e3 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:52.816358 1062 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc490b1327e44059a57e6cd7b63350e3" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
W20251029 02:51:52.816910 849 leader_election.cc:343] T bc490b1327e44059a57e6cd7b63350e3 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:52.818428 1174 ts_tablet_manager.cc:1929] T 86f8776b5c4c4082bf276199feaaefea P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:52.818576 1174 log.cc:1199] T 86f8776b5c4c4082bf276199feaaefea P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/86f8776b5c4c4082bf276199feaaefea
I20251029 02:51:52.818953 1174 ts_tablet_manager.cc:1950] T 86f8776b5c4c4082bf276199feaaefea P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:51:52.819365 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc490b1327e44059a57e6cd7b63350e3" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:51:52.819584 776 raft_consensus.cc:3060] T bc490b1327e44059a57e6cd7b63350e3 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:51:52.826308 1166 tablet_bootstrap.cc:492] T 2ebde39f813d46e898ad1754325ad1a0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:51:52.826498 1166 ts_tablet_manager.cc:1403] T 2ebde39f813d46e898ad1754325ad1a0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.020s user 0.011s sys 0.006s
I20251029 02:51:52.827280 1166 raft_consensus.cc:359] T 2ebde39f813d46e898ad1754325ad1a0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:52.827654 1166 raft_consensus.cc:385] T 2ebde39f813d46e898ad1754325ad1a0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:52.827744 1166 raft_consensus.cc:740] T 2ebde39f813d46e898ad1754325ad1a0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:51:52.828261 776 raft_consensus.cc:2468] T bc490b1327e44059a57e6cd7b63350e3 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:51:52.828716 1166 consensus_queue.cc:260] T 2ebde39f813d46e898ad1754325ad1a0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:52.828810 847 leader_election.cc:304] T bc490b1327e44059a57e6cd7b63350e3 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5
I20251029 02:51:52.829201 1307 raft_consensus.cc:2804] T bc490b1327e44059a57e6cd7b63350e3 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:51:52.829384 1307 raft_consensus.cc:697] T bc490b1327e44059a57e6cd7b63350e3 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:51:52.829430 1308 leader_election.cc:290] T bc490b1327e44059a57e6cd7b63350e3 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:51:52.829704 1307 consensus_queue.cc:237] T bc490b1327e44059a57e6cd7b63350e3 P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:52.831674 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 86f8776b5c4c4082bf276199feaaefea (table default.Table_bd27b9a173a243b0aa50adcf36f2a06d [id=13b28fb78eb54543b973bfda2fb1aad8]) successfully deleted
I20251029 02:51:52.834638 524 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_6add80f3d25b4701b9a0fa5639dec66c" } modify_external_catalogs: true
I20251029 02:51:52.834810 1037 tablet_service.cc:1552] Processing DeleteTablet for tablet 86f8776b5c4c4082bf276199feaaefea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:51 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:52.834852 524 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_6add80f3d25b4701b9a0fa5639dec66c" } modify_external_catalogs: true
I20251029 02:51:52.835065 1176 tablet_replica.cc:333] T 86f8776b5c4c4082bf276199feaaefea P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:51:52.835232 1176 raft_consensus.cc:2243] T 86f8776b5c4c4082bf276199feaaefea P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:52.835398 1176 raft_consensus.cc:2272] T 86f8776b5c4c4082bf276199feaaefea P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:52.836534 1176 ts_tablet_manager.cc:1916] T 86f8776b5c4c4082bf276199feaaefea P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 02:51:52.836947 521 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1f5b40c0403d4a49a4bfde222d8c947a has no consensus state
I20251029 02:51:52.838716 492 master_runner.cc:186] Error getting master registration for 127.0.104.60:41233: Network error: Client connection negotiation failed: client connection to 127.0.104.60:41233: connect: Connection refused (error 111),
I20251029 02:51:52.838892 492 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251029 02:51:52.841607 1037 tablet_service.cc:1552] Processing DeleteTablet for tablet 2ebde39f813d46e898ad1754325ad1a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:51 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:52.842746 1175 ts_tablet_manager.cc:1929] T bf210435010d4dde8903fd116eb468e3 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:52.842904 1175 log.cc:1199] T bf210435010d4dde8903fd116eb468e3 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/bf210435010d4dde8903fd116eb468e3
I20251029 02:51:52.843331 1175 ts_tablet_manager.cc:1950] T bf210435010d4dde8903fd116eb468e3 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:51:52.844421 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet bf210435010d4dde8903fd116eb468e3 (table default.Table_4bdbcbb87e5b40aeb9f98eb1aa0e6fe3 [id=78615d8807a74b4496a3223ef6fe709d]) successfully deleted
I20251029 02:51:52.847461 1176 ts_tablet_manager.cc:1929] T 86f8776b5c4c4082bf276199feaaefea P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:52.849817 1176 log.cc:1199] T 86f8776b5c4c4082bf276199feaaefea P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/86f8776b5c4c4082bf276199feaaefea
I20251029 02:51:52.850420 1176 ts_tablet_manager.cc:1950] T 86f8776b5c4c4082bf276199feaaefea P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:51:52.851902 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 86f8776b5c4c4082bf276199feaaefea (table default.Table_bd27b9a173a243b0aa50adcf36f2a06d [id=13b28fb78eb54543b973bfda2fb1aad8]) successfully deleted
I20251029 02:51:52.852814 1166 ts_tablet_manager.cc:1434] T 2ebde39f813d46e898ad1754325ad1a0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.026s user 0.000s sys 0.002s
I20251029 02:51:52.855862 1176 tablet_replica.cc:333] T 2ebde39f813d46e898ad1754325ad1a0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:51:52.856058 1176 raft_consensus.cc:2243] T 2ebde39f813d46e898ad1754325ad1a0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:52.856220 1176 raft_consensus.cc:2272] T 2ebde39f813d46e898ad1754325ad1a0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:52.856652 1166 tablet_bootstrap.cc:492] T bc490b1327e44059a57e6cd7b63350e3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:51:52.860139 1176 ts_tablet_manager.cc:1916] T 2ebde39f813d46e898ad1754325ad1a0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:52.863411 1166 tablet_bootstrap.cc:654] T bc490b1327e44059a57e6cd7b63350e3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:52.863490 1167 tablet_bootstrap.cc:492] T deca0c4a114441079fce22e74b403050 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:51:52.863693 1167 ts_tablet_manager.cc:1403] T deca0c4a114441079fce22e74b403050 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.090s user 0.002s sys 0.001s
I20251029 02:51:52.864571 1167 raft_consensus.cc:359] T deca0c4a114441079fce22e74b403050 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:52.864779 1167 raft_consensus.cc:385] T deca0c4a114441079fce22e74b403050 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:52.864853 1167 raft_consensus.cc:740] T deca0c4a114441079fce22e74b403050 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:51:52.865087 1167 consensus_queue.cc:260] T deca0c4a114441079fce22e74b403050 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:52.865576 1167 ts_tablet_manager.cc:1434] T deca0c4a114441079fce22e74b403050 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251029 02:51:52.872543 523 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_ef563eb489f947d4a4772678806a0777" } modify_external_catalogs: true
I20251029 02:51:52.872756 523 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_ef563eb489f947d4a4772678806a0777" } modify_external_catalogs: true
I20251029 02:51:52.873546 1167 tablet_bootstrap.cc:492] T 0dd31dc5873d407d958a0f5199f78782 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:51:52.873509 521 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_e43a2189b18a43f9beb9f3ce25848374" } modify_external_catalogs: true
I20251029 02:51:52.873735 521 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_e43a2189b18a43f9beb9f3ce25848374" } modify_external_catalogs: true
I20251029 02:51:52.875448 1167 tablet_bootstrap.cc:654] T 0dd31dc5873d407d958a0f5199f78782 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:52.875567 519 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_c17409b349f548f9a5062085395a2aee" } new_table_name: "default.Table_98034c2933664cefab94c26cb8349dc4" modify_external_catalogs: true
I20251029 02:51:52.878150 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_bce83bae13a94cd5bf6733b53e6a1e9e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:52.878948 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bce83bae13a94cd5bf6733b53e6a1e9e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:52.885986 1176 ts_tablet_manager.cc:1929] T 2ebde39f813d46e898ad1754325ad1a0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:52.886278 1176 log.cc:1199] T 2ebde39f813d46e898ad1754325ad1a0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/2ebde39f813d46e898ad1754325ad1a0
I20251029 02:51:52.886842 1176 ts_tablet_manager.cc:1950] T 2ebde39f813d46e898ad1754325ad1a0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:51:52.886989 1167 tablet_bootstrap.cc:492] T 0dd31dc5873d407d958a0f5199f78782 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:51:52.887167 1167 ts_tablet_manager.cc:1403] T 0dd31dc5873d407d958a0f5199f78782 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.005s
I20251029 02:51:52.887966 1167 raft_consensus.cc:359] T 0dd31dc5873d407d958a0f5199f78782 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:52.888154 1167 raft_consensus.cc:385] T 0dd31dc5873d407d958a0f5199f78782 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:52.888279 1167 raft_consensus.cc:740] T 0dd31dc5873d407d958a0f5199f78782 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:51:52.888520 1167 consensus_queue.cc:260] T 0dd31dc5873d407d958a0f5199f78782 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:52.889065 1167 ts_tablet_manager.cc:1434] T 0dd31dc5873d407d958a0f5199f78782 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251029 02:51:52.889364 1167 tablet_bootstrap.cc:492] T 5886b8d39f384426bcbed7289ae72e51 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:51:52.893057 1175 tablet_replica.cc:333] T 6dd4696ad4b44b6dab87a205f3f16ebd P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:51:52.893294 1175 raft_consensus.cc:2243] T 6dd4696ad4b44b6dab87a205f3f16ebd P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:52.893261 518 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet 5e31e061d74146009d24d5cd57487a5e (table default.Table_f2bc74933428449197f8c28182c8698f [id=8e17d6b87b984d878787802bd6f627f2]). Expected version 2 got 0
I20251029 02:51:52.893451 1175 raft_consensus.cc:2272] T 6dd4696ad4b44b6dab87a205f3f16ebd P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:52.895226 518 catalog_manager.cc:5649] T bc490b1327e44059a57e6cd7b63350e3 P c756528642a543eeacd93cef3e335a7d reported cstate change: term changed from 0 to 1, leader changed from <none> to c756528642a543eeacd93cef3e335a7d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "c756528642a543eeacd93cef3e335a7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } health_report { overall_health: UNKNOWN } } }
I20251029 02:51:52.897606 893 tablet_service.cc:1505] Processing CreateTablet for tablet dbc2092b63e0417997565c8fadb53ae0 (DEFAULT_TABLE table=default.Table_a873546f69334d12b0cbecce2cd489fb [id=1358e2055c254030a51e2de446eb742f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.898093 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dbc2092b63e0417997565c8fadb53ae0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.899258 895 tablet_service.cc:1505] Processing CreateTablet for tablet 24cb2a71ba3841d4a5e3e5c6fe99ae2f (DEFAULT_TABLE table=default.Table_4350e92f14de4ba8a04afbe12ea7422a [id=226a2d412a7b40e59b6f5340b5a9b739]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.899701 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24cb2a71ba3841d4a5e3e5c6fe99ae2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.901301 522 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_8b952165b0ae4603b38a50cfa0e121e0" } new_table_name: "default.Table_6254452e8df24c21ab7d8ae32a06bdfb" modify_external_catalogs: true
W20251029 02:51:52.903131 511 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 4b76c4408a7340e59b8506171622175f (table default.Table_1281926bf1224c0cb4541bf561102021 [id=f3fa2dbbef7e47359492d5a5eef9b8fd], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 4b76c4408a7340e59b8506171622175f consensus state has no leader
W20251029 02:51:52.903390 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 6feec2c8e030447fae9c292a626dcd1d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6feec2c8e030447fae9c292a626dcd1d already in progress: creating tablet
I20251029 02:51:52.903661 755 tablet_service.cc:1505] Processing CreateTablet for tablet dbc2092b63e0417997565c8fadb53ae0 (DEFAULT_TABLE table=default.Table_a873546f69334d12b0cbecce2cd489fb [id=1358e2055c254030a51e2de446eb742f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.904155 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dbc2092b63e0417997565c8fadb53ae0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.905402 1037 tablet_service.cc:1505] Processing CreateTablet for tablet dbc2092b63e0417997565c8fadb53ae0 (DEFAULT_TABLE table=default.Table_a873546f69334d12b0cbecce2cd489fb [id=1358e2055c254030a51e2de446eb742f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.905948 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dbc2092b63e0417997565c8fadb53ae0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.906602 1167 tablet_bootstrap.cc:654] T 5886b8d39f384426bcbed7289ae72e51 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:52.909586 1031 tablet_service.cc:1505] Processing CreateTablet for tablet 24cb2a71ba3841d4a5e3e5c6fe99ae2f (DEFAULT_TABLE table=default.Table_4350e92f14de4ba8a04afbe12ea7422a [id=226a2d412a7b40e59b6f5340b5a9b739]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.910104 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24cb2a71ba3841d4a5e3e5c6fe99ae2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.914482 1172 tablet_bootstrap.cc:492] T bc1a1cf274b64b52aba58811e9594330 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:51:52.914716 1172 ts_tablet_manager.cc:1403] T bc1a1cf274b64b52aba58811e9594330 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.158s user 0.000s sys 0.004s
I20251029 02:51:52.916435 1172 raft_consensus.cc:359] T bc1a1cf274b64b52aba58811e9594330 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:52.916707 1172 raft_consensus.cc:385] T bc1a1cf274b64b52aba58811e9594330 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:52.916783 1172 raft_consensus.cc:740] T bc1a1cf274b64b52aba58811e9594330 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:51:52.917078 1172 consensus_queue.cc:260] T bc1a1cf274b64b52aba58811e9594330 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:52.919703 1062 raft_consensus.cc:1275] T 8f7c746f8b6948b58676298c2c2ea221 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Refusing update from remote peer c756528642a543eeacd93cef3e335a7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251029 02:51:52.919656 900 tablet_service.cc:1505] Processing CreateTablet for tablet 1f5b40c0403d4a49a4bfde222d8c947a (DEFAULT_TABLE table=default.Table_72f8739f5f654172b10d619a9ac306bd [id=1565d03ae28649e48c2a614a8d067227]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.920078 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f5b40c0403d4a49a4bfde222d8c947a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.923211 776 raft_consensus.cc:1275] T 8f7c746f8b6948b58676298c2c2ea221 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Refusing update from remote peer c756528642a543eeacd93cef3e335a7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251029 02:51:52.923138 753 tablet_service.cc:1505] Processing CreateTablet for tablet 1f5b40c0403d4a49a4bfde222d8c947a (DEFAULT_TABLE table=default.Table_72f8739f5f654172b10d619a9ac306bd [id=1565d03ae28649e48c2a614a8d067227]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.923334 1041 tablet_service.cc:1505] Processing CreateTablet for tablet 1f5b40c0403d4a49a4bfde222d8c947a (DEFAULT_TABLE table=default.Table_72f8739f5f654172b10d619a9ac306bd [id=1565d03ae28649e48c2a614a8d067227]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.923573 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f5b40c0403d4a49a4bfde222d8c947a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.923754 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f5b40c0403d4a49a4bfde222d8c947a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.923969 1308 consensus_queue.cc:1048] T 8f7c746f8b6948b58676298c2c2ea221 P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:51:52.924667 1308 consensus_queue.cc:1048] T 8f7c746f8b6948b58676298c2c2ea221 P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:51:52.926831 894 tablet_service.cc:1505] Processing CreateTablet for tablet 51effc9324034ae79e2b25cb57a996c1 (DEFAULT_TABLE table=default.Table_4d5341f07b99483183d6aedc61c9ddcd [id=09fbf31759f047be924a5ef1f0030765]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.926847 744 tablet_service.cc:1505] Processing CreateTablet for tablet 51effc9324034ae79e2b25cb57a996c1 (DEFAULT_TABLE table=default.Table_4d5341f07b99483183d6aedc61c9ddcd [id=09fbf31759f047be924a5ef1f0030765]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.927201 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51effc9324034ae79e2b25cb57a996c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.927202 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51effc9324034ae79e2b25cb57a996c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.933792 755 tablet_service.cc:1505] Processing CreateTablet for tablet 24cb2a71ba3841d4a5e3e5c6fe99ae2f (DEFAULT_TABLE table=default.Table_4350e92f14de4ba8a04afbe12ea7422a [id=226a2d412a7b40e59b6f5340b5a9b739]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.934316 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24cb2a71ba3841d4a5e3e5c6fe99ae2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.939508 1039 tablet_service.cc:1505] Processing CreateTablet for tablet 51effc9324034ae79e2b25cb57a996c1 (DEFAULT_TABLE table=default.Table_4d5341f07b99483183d6aedc61c9ddcd [id=09fbf31759f047be924a5ef1f0030765]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:52.940101 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51effc9324034ae79e2b25cb57a996c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:52.969138 1172 ts_tablet_manager.cc:1434] T bc1a1cf274b64b52aba58811e9594330 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.054s user 0.000s sys 0.003s
I20251029 02:51:52.971249 1172 tablet_bootstrap.cc:492] T 41a6139fd9b84707a8c7eebc0381e383 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
W20251029 02:51:52.973495 519 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f81aecb20d6e498bb9b498698886d65e has no consensus state
I20251029 02:51:52.973984 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_262db98ea44648baab95fc2b3081dc4e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:52.974812 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_262db98ea44648baab95fc2b3081dc4e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:52.976647 520 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet 5e31e061d74146009d24d5cd57487a5e (table default.Table_f2bc74933428449197f8c28182c8698f [id=8e17d6b87b984d878787802bd6f627f2]). Expected version 2 got 0
I20251029 02:51:52.978152 520 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_953f59050dc54b04b57c06364595c612" } modify_external_catalogs: true
I20251029 02:51:52.978379 520 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_953f59050dc54b04b57c06364595c612" } modify_external_catalogs: true
I20251029 02:51:52.978652 1175 ts_tablet_manager.cc:1916] T 6dd4696ad4b44b6dab87a205f3f16ebd P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:52.980541 755 tablet_service.cc:1552] Processing DeleteTablet for tablet 61c1fec3ab4c402eb29f767b9bb7409f with delete_type TABLET_DATA_DELETED (replaced by tablet a5c3a15510cb4321bbb80d61fabd0c0a) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:52.985472 1166 tablet_bootstrap.cc:492] T bc490b1327e44059a57e6cd7b63350e3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:51:52.985625 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 2ebde39f813d46e898ad1754325ad1a0 (table default.Table_1a524bb5f82a4041b8008ff91a0b9d27 [id=5db5ea0ab3e249f4acea117bb07187f4]) successfully deleted
I20251029 02:51:52.985743 1166 ts_tablet_manager.cc:1403] T bc490b1327e44059a57e6cd7b63350e3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.129s user 0.033s sys 0.049s
I20251029 02:51:52.986627 1166 raft_consensus.cc:359] T bc490b1327e44059a57e6cd7b63350e3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:52.986860 1166 raft_consensus.cc:385] T bc490b1327e44059a57e6cd7b63350e3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:52.986929 1167 tablet_bootstrap.cc:492] T 5886b8d39f384426bcbed7289ae72e51 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:51:52.986974 1166 raft_consensus.cc:740] T bc490b1327e44059a57e6cd7b63350e3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:51:52.987124 1167 ts_tablet_manager.cc:1403] T 5886b8d39f384426bcbed7289ae72e51 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.098s user 0.008s sys 0.018s
I20251029 02:51:52.987205 1166 consensus_queue.cc:260] T bc490b1327e44059a57e6cd7b63350e3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:52.987396 1307 raft_consensus.cc:493] T 0dd31dc5873d407d958a0f5199f78782 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:52.987581 1307 raft_consensus.cc:515] T 0dd31dc5873d407d958a0f5199f78782 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:52.988080 1167 raft_consensus.cc:359] T 5886b8d39f384426bcbed7289ae72e51 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:52.988284 1307 leader_election.cc:290] T 0dd31dc5873d407d958a0f5199f78782 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:51:52.988312 1167 raft_consensus.cc:385] T 5886b8d39f384426bcbed7289ae72e51 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:52.988518 1167 raft_consensus.cc:740] T 5886b8d39f384426bcbed7289ae72e51 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:51:52.988742 1061 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0dd31dc5873d407d958a0f5199f78782" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:51:52.988792 1167 consensus_queue.cc:260] T 5886b8d39f384426bcbed7289ae72e51 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:52.989017 1166 ts_tablet_manager.cc:1434] T bc490b1327e44059a57e6cd7b63350e3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
W20251029 02:51:52.989425 847 leader_election.cc:343] T 0dd31dc5873d407d958a0f5199f78782 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 0dd31dc5873d407d958a0f5199f78782
I20251029 02:51:52.989418 1167 ts_tablet_manager.cc:1434] T 5886b8d39f384426bcbed7289ae72e51 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 02:51:52.989970 1167 tablet_bootstrap.cc:492] T 11f4252b5e694312aaa142468f8e68b4 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:51:52.990227 1166 tablet_bootstrap.cc:492] T 5886b8d39f384426bcbed7289ae72e51 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
W20251029 02:51:52.990432 849 leader_election.cc:343] T 0dd31dc5873d407d958a0f5199f78782 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:52.990566 849 leader_election.cc:304] T 0dd31dc5873d407d958a0f5199f78782 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9
I20251029 02:51:52.990912 1307 raft_consensus.cc:2749] T 0dd31dc5873d407d958a0f5199f78782 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:51:52.993194 1167 tablet_bootstrap.cc:654] T 11f4252b5e694312aaa142468f8e68b4 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:53.005395 1172 tablet_bootstrap.cc:654] T 41a6139fd9b84707a8c7eebc0381e383 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:53.012082 1340 tablet.cc:1722] T 8f7c746f8b6948b58676298c2c2ea221 P b495a49349324f7d81908f2da700f0d9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251029 02:51:53.015465 1339 tablet.cc:1722] T 8f7c746f8b6948b58676298c2c2ea221 P c756528642a543eeacd93cef3e335a7d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251029 02:51:53.018319 1166 tablet_bootstrap.cc:654] T 5886b8d39f384426bcbed7289ae72e51 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:53.030553 1342 tablet.cc:1722] T 8f7c746f8b6948b58676298c2c2ea221 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251029 02:51:53.034456 525 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet 8f7c746f8b6948b58676298c2c2ea221 (table default.Table_9aac534a15054a30b80745984c279e62 [id=669ae9dd0a5749b1aa3557ab8183a057]). Expected version 3 got 0
I20251029 02:51:53.035876 525 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet 8f7c746f8b6948b58676298c2c2ea221 (table default.Table_9aac534a15054a30b80745984c279e62 [id=669ae9dd0a5749b1aa3557ab8183a057]). Expected version 3 got 0
I20251029 02:51:53.036739 525 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_9aac534a15054a30b80745984c279e62" } new_table_name: "default.Table_da4961f316764505bb9e0f191a62a327" modify_external_catalogs: true
I20251029 02:51:53.038897 1332 mvcc.cc:204] Tried to move back new op lower bound from 7215949057705975808 to 7215949055503839232. Current Snapshot: MvccSnapshot[applied={T|T < 7215949057705975808}]
I20251029 02:51:53.043511 1166 tablet_bootstrap.cc:492] T 5886b8d39f384426bcbed7289ae72e51 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:51:53.043746 1166 ts_tablet_manager.cc:1403] T 5886b8d39f384426bcbed7289ae72e51 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.054s user 0.008s sys 0.005s
I20251029 02:51:53.044644 1166 raft_consensus.cc:359] T 5886b8d39f384426bcbed7289ae72e51 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:53.044934 1166 raft_consensus.cc:385] T 5886b8d39f384426bcbed7289ae72e51 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:53.045061 1166 raft_consensus.cc:740] T 5886b8d39f384426bcbed7289ae72e51 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:51:53.045332 1166 consensus_queue.cc:260] T 5886b8d39f384426bcbed7289ae72e51 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:53.045924 1166 ts_tablet_manager.cc:1434] T 5886b8d39f384426bcbed7289ae72e51 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 02:51:53.048038 1166 tablet_bootstrap.cc:492] T 9be9add0274d41b688edd6292a253a06 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:51:53.050346 1166 tablet_bootstrap.cc:654] T 9be9add0274d41b688edd6292a253a06 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:53.052228 517 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 2242f5b2c20346d9b753d7cd29242441
I20251029 02:51:53.052934 517 catalog_manager.cc:6719] ReplaceTablet: tablet 2242f5b2c20346d9b753d7cd29242441 deleted and replaced by tablet e1fa4980f2624321b617b84c68b90113
I20251029 02:51:53.054631 1039 tablet_service.cc:1552] Processing DeleteTablet for tablet 2242f5b2c20346d9b753d7cd29242441 with delete_type TABLET_DATA_DELETED (replaced by tablet e1fa4980f2624321b617b84c68b90113) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:53.054935 755 tablet_service.cc:1552] Processing DeleteTablet for tablet 2242f5b2c20346d9b753d7cd29242441 with delete_type TABLET_DATA_DELETED (replaced by tablet e1fa4980f2624321b617b84c68b90113) from {username='slave'} at 127.0.0.1:44580
W20251029 02:51:53.055647 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 2242f5b2c20346d9b753d7cd29242441 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2242f5b2c20346d9b753d7cd29242441 already in progress: creating tablet
I20251029 02:51:53.058072 895 tablet_service.cc:1552] Processing DeleteTablet for tablet 2242f5b2c20346d9b753d7cd29242441 with delete_type TABLET_DATA_DELETED (replaced by tablet e1fa4980f2624321b617b84c68b90113) from {username='slave'} at 127.0.0.1:35702
W20251029 02:51:53.058852 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 2242f5b2c20346d9b753d7cd29242441 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2242f5b2c20346d9b753d7cd29242441 already in progress: creating tablet
I20251029 02:51:53.062825 1339 tablet.cc:1717] T 8f7c746f8b6948b58676298c2c2ea221 P c756528642a543eeacd93cef3e335a7d: Skipping requested alter to schema version 3, tablet already version 3
I20251029 02:51:53.064041 1342 tablet.cc:1717] T 8f7c746f8b6948b58676298c2c2ea221 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Skipping requested alter to schema version 3, tablet already version 3
I20251029 02:51:53.074777 1166 tablet_bootstrap.cc:492] T 9be9add0274d41b688edd6292a253a06 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:51:53.074965 1166 ts_tablet_manager.cc:1403] T 9be9add0274d41b688edd6292a253a06 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.027s user 0.017s sys 0.004s
I20251029 02:51:53.073911 1340 tablet.cc:1717] T 8f7c746f8b6948b58676298c2c2ea221 P b495a49349324f7d81908f2da700f0d9: Skipping requested alter to schema version 3, tablet already version 3
I20251029 02:51:53.076135 1166 raft_consensus.cc:359] T 9be9add0274d41b688edd6292a253a06 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:53.076601 1166 raft_consensus.cc:385] T 9be9add0274d41b688edd6292a253a06 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:53.076711 1166 raft_consensus.cc:740] T 9be9add0274d41b688edd6292a253a06 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:51:53.077013 1166 consensus_queue.cc:260] T 9be9add0274d41b688edd6292a253a06 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:53.079403 1166 ts_tablet_manager.cc:1434] T 9be9add0274d41b688edd6292a253a06 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251029 02:51:53.080524 1166 tablet_bootstrap.cc:492] T deca0c4a114441079fce22e74b403050 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:51:53.082835 1166 tablet_bootstrap.cc:654] T deca0c4a114441079fce22e74b403050 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:53.084158 523 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 7309d3190cfa415281c5253a634fa7f3
I20251029 02:51:53.086097 755 tablet_service.cc:1552] Processing DeleteTablet for tablet 7309d3190cfa415281c5253a634fa7f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:52 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:53.086162 1039 tablet_service.cc:1552] Processing DeleteTablet for tablet 7309d3190cfa415281c5253a634fa7f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:52 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:53.086725 895 tablet_service.cc:1552] Processing DeleteTablet for tablet 7309d3190cfa415281c5253a634fa7f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:52 UTC) from {username='slave'} at 127.0.0.1:35702
W20251029 02:51:53.087122 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 7309d3190cfa415281c5253a634fa7f3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7309d3190cfa415281c5253a634fa7f3 already in progress: creating tablet
W20251029 02:51:53.087422 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 7309d3190cfa415281c5253a634fa7f3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7309d3190cfa415281c5253a634fa7f3 already in progress: creating tablet
I20251029 02:51:53.091949 1166 tablet_bootstrap.cc:492] T deca0c4a114441079fce22e74b403050 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:51:53.092201 1166 ts_tablet_manager.cc:1403] T deca0c4a114441079fce22e74b403050 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.006s
I20251029 02:51:53.093097 1166 raft_consensus.cc:359] T deca0c4a114441079fce22e74b403050 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:53.093394 1166 raft_consensus.cc:385] T deca0c4a114441079fce22e74b403050 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:53.093516 1166 raft_consensus.cc:740] T deca0c4a114441079fce22e74b403050 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:51:53.093828 1166 consensus_queue.cc:260] T deca0c4a114441079fce22e74b403050 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:53.095584 1166 ts_tablet_manager.cc:1434] T deca0c4a114441079fce22e74b403050 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251029 02:51:53.096408 526 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 3362e95063af41c8a24e935f547e3089
I20251029 02:51:53.096953 526 catalog_manager.cc:6719] ReplaceTablet: tablet 3362e95063af41c8a24e935f547e3089 deleted and replaced by tablet da2a21c3236a4bd99872b89b12a36508
I20251029 02:51:53.097368 1166 tablet_bootstrap.cc:492] T d32532973b3f46dd8158a2ce42d26365 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:51:53.097759 895 tablet_service.cc:1552] Processing DeleteTablet for tablet 3362e95063af41c8a24e935f547e3089 with delete_type TABLET_DATA_DELETED (replaced by tablet da2a21c3236a4bd99872b89b12a36508) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:53.098009 1039 tablet_service.cc:1552] Processing DeleteTablet for tablet 3362e95063af41c8a24e935f547e3089 with delete_type TABLET_DATA_DELETED (replaced by tablet da2a21c3236a4bd99872b89b12a36508) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:53.098289 755 tablet_service.cc:1552] Processing DeleteTablet for tablet 3362e95063af41c8a24e935f547e3089 with delete_type TABLET_DATA_DELETED (replaced by tablet da2a21c3236a4bd99872b89b12a36508) from {username='slave'} at 127.0.0.1:44580
W20251029 02:51:53.098620 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 3362e95063af41c8a24e935f547e3089 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3362e95063af41c8a24e935f547e3089 already in progress: creating tablet
W20251029 02:51:53.098675 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 3362e95063af41c8a24e935f547e3089 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3362e95063af41c8a24e935f547e3089 already in progress: creating tablet
I20251029 02:51:53.101274 1166 tablet_bootstrap.cc:654] T d32532973b3f46dd8158a2ce42d26365 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:53.103380 526 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_7ab08f1eb370432b985cd5c0f9988017" } modify_external_catalogs: true
I20251029 02:51:53.103591 526 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_7ab08f1eb370432b985cd5c0f9988017" } modify_external_catalogs: true
I20251029 02:51:53.107157 1175 ts_tablet_manager.cc:1929] T 6dd4696ad4b44b6dab87a205f3f16ebd P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:53.112049 521 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 5695b5a7b1854c3983e2d4ab684898b0
I20251029 02:51:53.113391 1039 tablet_service.cc:1552] Processing DeleteTablet for tablet 5695b5a7b1854c3983e2d4ab684898b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:52 UTC) from {username='slave'} at 127.0.0.1:37626
W20251029 02:51:53.114737 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 5695b5a7b1854c3983e2d4ab684898b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5695b5a7b1854c3983e2d4ab684898b0 already in progress: creating tablet
I20251029 02:51:53.114913 895 tablet_service.cc:1552] Processing DeleteTablet for tablet 5695b5a7b1854c3983e2d4ab684898b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:52 UTC) from {username='slave'} at 127.0.0.1:35702
W20251029 02:51:53.115756 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 5695b5a7b1854c3983e2d4ab684898b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5695b5a7b1854c3983e2d4ab684898b0 already in progress: creating tablet
I20251029 02:51:53.119441 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_febb0d5740c847ce8aafa84d39ccbe91"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:53.120061 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_febb0d5740c847ce8aafa84d39ccbe91 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:53.127871 1251 raft_consensus.cc:493] T 9be9add0274d41b688edd6292a253a06 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:53.128104 1251 raft_consensus.cc:515] T 9be9add0274d41b688edd6292a253a06 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:53.129066 1251 leader_election.cc:290] T 9be9add0274d41b688edd6292a253a06 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:51:53.130034 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9be9add0274d41b688edd6292a253a06" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:51:53.130848 990 leader_election.cc:343] T 9be9add0274d41b688edd6292a253a06 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:53.116050 755 tablet_service.cc:1552] Processing DeleteTablet for tablet 5695b5a7b1854c3983e2d4ab684898b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:52 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:53.123734 1172 tablet_bootstrap.cc:492] T 41a6139fd9b84707a8c7eebc0381e383 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:51:53.130280 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9be9add0274d41b688edd6292a253a06" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:51:53.133574 1172 ts_tablet_manager.cc:1403] T 41a6139fd9b84707a8c7eebc0381e383 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.163s user 0.001s sys 0.004s
I20251029 02:51:53.133584 775 raft_consensus.cc:2468] T 9be9add0274d41b688edd6292a253a06 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 0.
I20251029 02:51:53.134114 989 leader_election.cc:304] T 9be9add0274d41b688edd6292a253a06 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters: c756528642a543eeacd93cef3e335a7d
I20251029 02:51:53.134698 1251 raft_consensus.cc:2804] T 9be9add0274d41b688edd6292a253a06 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:51:53.134846 1251 raft_consensus.cc:493] T 9be9add0274d41b688edd6292a253a06 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:51:53.134953 1251 raft_consensus.cc:3060] T 9be9add0274d41b688edd6292a253a06 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:51:53.134888 1172 raft_consensus.cc:359] T 41a6139fd9b84707a8c7eebc0381e383 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:53.135125 1172 raft_consensus.cc:385] T 41a6139fd9b84707a8c7eebc0381e383 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:53.135226 1172 raft_consensus.cc:740] T 41a6139fd9b84707a8c7eebc0381e383 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:51:53.135546 1172 consensus_queue.cc:260] T 41a6139fd9b84707a8c7eebc0381e383 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:53.140753 1251 raft_consensus.cc:515] T 9be9add0274d41b688edd6292a253a06 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:53.140919 1172 ts_tablet_manager.cc:1434] T 41a6139fd9b84707a8c7eebc0381e383 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20251029 02:51:53.142313 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9be9add0274d41b688edd6292a253a06" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
I20251029 02:51:53.142490 1251 leader_election.cc:290] T 9be9add0274d41b688edd6292a253a06 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Requested vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
W20251029 02:51:53.142872 990 leader_election.cc:343] T 9be9add0274d41b688edd6292a253a06 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:53.143906 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9be9add0274d41b688edd6292a253a06" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:51:53.144133 775 raft_consensus.cc:3060] T 9be9add0274d41b688edd6292a253a06 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:51:53.144933 1172 tablet_bootstrap.cc:492] T 70e8cfe5eaa9492c85874dc3214a1e0b P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:51:53.147316 1172 tablet_bootstrap.cc:654] T 70e8cfe5eaa9492c85874dc3214a1e0b P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:53.148442 775 raft_consensus.cc:2468] T 9be9add0274d41b688edd6292a253a06 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 1.
I20251029 02:51:53.148592 1175 log.cc:1199] T 6dd4696ad4b44b6dab87a205f3f16ebd P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/6dd4696ad4b44b6dab87a205f3f16ebd
I20251029 02:51:53.149369 989 leader_election.cc:304] T 9be9add0274d41b688edd6292a253a06 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters: c756528642a543eeacd93cef3e335a7d
I20251029 02:51:53.149510 1175 ts_tablet_manager.cc:1950] T 6dd4696ad4b44b6dab87a205f3f16ebd P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:51:53.149904 1251 raft_consensus.cc:2804] T 9be9add0274d41b688edd6292a253a06 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:51:53.151242 1172 tablet_bootstrap.cc:492] T 70e8cfe5eaa9492c85874dc3214a1e0b P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:51:53.151427 1172 ts_tablet_manager.cc:1403] T 70e8cfe5eaa9492c85874dc3214a1e0b P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20251029 02:51:53.152156 1172 raft_consensus.cc:359] T 70e8cfe5eaa9492c85874dc3214a1e0b P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:53.152369 1172 raft_consensus.cc:385] T 70e8cfe5eaa9492c85874dc3214a1e0b P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:53.152453 1172 raft_consensus.cc:740] T 70e8cfe5eaa9492c85874dc3214a1e0b P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:51:53.152699 1172 consensus_queue.cc:260] T 70e8cfe5eaa9492c85874dc3214a1e0b P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:53.153199 1172 ts_tablet_manager.cc:1434] T 70e8cfe5eaa9492c85874dc3214a1e0b P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251029 02:51:53.154677 1166 tablet_bootstrap.cc:492] T d32532973b3f46dd8158a2ce42d26365 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:51:53.154922 1166 ts_tablet_manager.cc:1403] T d32532973b3f46dd8158a2ce42d26365 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.058s user 0.015s sys 0.036s
I20251029 02:51:53.155984 1166 raft_consensus.cc:359] T d32532973b3f46dd8158a2ce42d26365 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
W20251029 02:51:53.156123 522 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3cded156380f45cc95893825c71ba0e9 consensus state has no leader
I20251029 02:51:53.156224 1166 raft_consensus.cc:385] T d32532973b3f46dd8158a2ce42d26365 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:53.156311 1166 raft_consensus.cc:740] T d32532973b3f46dd8158a2ce42d26365 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:51:53.156565 1166 consensus_queue.cc:260] T d32532973b3f46dd8158a2ce42d26365 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:53.156635 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_e4bba3a0c06a4ff5bd8829e1081b9c68"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:53.157166 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e4bba3a0c06a4ff5bd8829e1081b9c68 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:53.158581 1167 tablet_bootstrap.cc:492] T 11f4252b5e694312aaa142468f8e68b4 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:51:53.158735 1167 ts_tablet_manager.cc:1403] T 11f4252b5e694312aaa142468f8e68b4 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.169s user 0.058s sys 0.029s
I20251029 02:51:53.159557 1167 raft_consensus.cc:359] T 11f4252b5e694312aaa142468f8e68b4 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:53.159746 1167 raft_consensus.cc:385] T 11f4252b5e694312aaa142468f8e68b4 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:53.159843 1167 raft_consensus.cc:740] T 11f4252b5e694312aaa142468f8e68b4 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:51:53.160084 1167 consensus_queue.cc:260] T 11f4252b5e694312aaa142468f8e68b4 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:53.161563 1167 ts_tablet_manager.cc:1434] T 11f4252b5e694312aaa142468f8e68b4 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20251029 02:51:53.161895 1167 tablet_bootstrap.cc:492] T d32532973b3f46dd8158a2ce42d26365 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:51:53.163923 1167 tablet_bootstrap.cc:654] T d32532973b3f46dd8158a2ce42d26365 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:53.164595 1166 ts_tablet_manager.cc:1434] T d32532973b3f46dd8158a2ce42d26365 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20251029 02:51:53.165187 1166 tablet_bootstrap.cc:492] T 11f4252b5e694312aaa142468f8e68b4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:51:53.166006 1251 raft_consensus.cc:697] T 9be9add0274d41b688edd6292a253a06 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Becoming Leader. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Running, Role: LEADER
I20251029 02:51:53.166476 1251 consensus_queue.cc:237] T 9be9add0274d41b688edd6292a253a06 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:53.168731 1166 tablet_bootstrap.cc:654] T 11f4252b5e694312aaa142468f8e68b4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:53.169203 1167 tablet_bootstrap.cc:492] T d32532973b3f46dd8158a2ce42d26365 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:51:53.169385 1167 ts_tablet_manager.cc:1403] T d32532973b3f46dd8158a2ce42d26365 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20251029 02:51:53.170384 1167 raft_consensus.cc:359] T d32532973b3f46dd8158a2ce42d26365 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:53.170678 1167 raft_consensus.cc:385] T d32532973b3f46dd8158a2ce42d26365 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:53.170778 1167 raft_consensus.cc:740] T d32532973b3f46dd8158a2ce42d26365 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:51:53.171123 1167 consensus_queue.cc:260] T d32532973b3f46dd8158a2ce42d26365 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:53.172863 526 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 8e7da9d6b0d7464880e474202610da67
I20251029 02:51:53.174554 520 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 55c1b243d8a94938a5caa58754092cec
I20251029 02:51:53.175614 1039 tablet_service.cc:1552] Processing DeleteTablet for tablet 55c1b243d8a94938a5caa58754092cec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:52 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:53.175614 1351 raft_consensus.cc:493] T 5886b8d39f384426bcbed7289ae72e51 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:53.175748 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 6dd4696ad4b44b6dab87a205f3f16ebd (table default.Table_69af0d3b63c44157abd3263a9deb02da [id=90dfeabf0e8945c9bd86472d68884221]) successfully deleted
I20251029 02:51:53.175834 1351 raft_consensus.cc:515] T 5886b8d39f384426bcbed7289ae72e51 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:53.176092 1176 tablet_replica.cc:333] T 55c1b243d8a94938a5caa58754092cec P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
W20251029 02:51:53.176061 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet ab421dfaa3f84593842bb732a2693160 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ab421dfaa3f84593842bb732a2693160 already in progress: creating tablet
I20251029 02:51:53.176302 1176 raft_consensus.cc:2243] T 55c1b243d8a94938a5caa58754092cec P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:51:53.176362 755 tablet_service.cc:1552] Processing DeleteTablet for tablet 55c1b243d8a94938a5caa58754092cec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:52 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:53.176636 1176 raft_consensus.cc:2272] T 55c1b243d8a94938a5caa58754092cec P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:53.176673 1172 tablet_bootstrap.cc:492] T 24016ceaa84147eca0b94b3ad288e772 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:51:53.176992 1351 leader_election.cc:290] T 5886b8d39f384426bcbed7289ae72e51 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:51:53.177301 1176 ts_tablet_manager.cc:1916] T 55c1b243d8a94938a5caa58754092cec P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:53.177875 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5886b8d39f384426bcbed7289ae72e51" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:51:53.178063 920 raft_consensus.cc:2468] T 5886b8d39f384426bcbed7289ae72e51 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 0.
W20251029 02:51:53.178179 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet fd900eec3f0c43bf965c148ea90870a3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fd900eec3f0c43bf965c148ea90870a3 already in progress: creating tablet
I20251029 02:51:53.178314 895 tablet_service.cc:1552] Processing DeleteTablet for tablet 55c1b243d8a94938a5caa58754092cec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:52 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:53.178609 1174 tablet_replica.cc:333] T 55c1b243d8a94938a5caa58754092cec P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:51:53.178647 990 leader_election.cc:304] T 5886b8d39f384426bcbed7289ae72e51 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:51:53.178802 1174 raft_consensus.cc:2243] T 55c1b243d8a94938a5caa58754092cec P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:53.178994 1174 raft_consensus.cc:2272] T 55c1b243d8a94938a5caa58754092cec P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:53.179589 1174 ts_tablet_manager.cc:1916] T 55c1b243d8a94938a5caa58754092cec P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:53.180708 1175 tablet_replica.cc:333] T bc1a1cf274b64b52aba58811e9594330 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:51:53.180895 1175 raft_consensus.cc:2243] T bc1a1cf274b64b52aba58811e9594330 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:53.181099 1175 raft_consensus.cc:2272] T bc1a1cf274b64b52aba58811e9594330 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251029 02:51:53.182781 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet c65469173762446e9725262ae998dbf3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c65469173762446e9725262ae998dbf3 already in progress: creating tablet
W20251029 02:51:53.182828 989 leader_election.cc:343] T 5886b8d39f384426bcbed7289ae72e51 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 5886b8d39f384426bcbed7289ae72e51
I20251029 02:51:53.183295 1172 tablet_bootstrap.cc:654] T 24016ceaa84147eca0b94b3ad288e772 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:53.185493 1351 raft_consensus.cc:2804] T 5886b8d39f384426bcbed7289ae72e51 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:51:53.185662 1351 raft_consensus.cc:493] T 5886b8d39f384426bcbed7289ae72e51 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:51:53.185773 1351 raft_consensus.cc:3060] T 5886b8d39f384426bcbed7289ae72e51 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:51:53.187624 1176 ts_tablet_manager.cc:1929] T 55c1b243d8a94938a5caa58754092cec P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251029 02:51:53.187732 1176 log.cc:1199] T 55c1b243d8a94938a5caa58754092cec P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/55c1b243d8a94938a5caa58754092cec
I20251029 02:51:53.188156 1176 ts_tablet_manager.cc:1950] T 55c1b243d8a94938a5caa58754092cec P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:51:53.189344 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 55c1b243d8a94938a5caa58754092cec (table default.Table_953f59050dc54b04b57c06364595c612 [id=bd2335c28dd44b5584ab71694cebda2d]) successfully deleted
I20251029 02:51:53.192768 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_218b1046ed7c4b0d8b9c55a9542a6a2e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:53.193403 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_218b1046ed7c4b0d8b9c55a9542a6a2e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:53.193547 525 master_service.cc:954] ReplaceTablet: received request to replace tablet aa776cde2f494fa5810539edf9f289a1 from {username='slave'} at 127.0.0.1:35314
I20251029 02:51:53.193807 524 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet bc490b1327e44059a57e6cd7b63350e3
I20251029 02:51:53.194561 524 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_4013d5c712574aa69143efea710b9b39" } new_table_name: "default.Table_4e899dcc04fe4cb1b96ced1fa1b03585" modify_external_catalogs: true
I20251029 02:51:53.194660 1039 tablet_service.cc:1552] Processing DeleteTablet for tablet bc490b1327e44059a57e6cd7b63350e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:52 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:53.195183 1176 tablet_replica.cc:333] T bc490b1327e44059a57e6cd7b63350e3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:51:53.195405 1176 raft_consensus.cc:2243] T bc490b1327e44059a57e6cd7b63350e3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:53.195573 1176 raft_consensus.cc:2272] T bc490b1327e44059a57e6cd7b63350e3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:53.195902 518 master_service.cc:954] ReplaceTablet: received request to replace tablet 3cded156380f45cc95893825c71ba0e9 from {username='slave'} at 127.0.0.1:35314
I20251029 02:51:53.197273 1174 ts_tablet_manager.cc:1929] T 55c1b243d8a94938a5caa58754092cec P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251029 02:51:53.197778 1172 tablet_bootstrap.cc:492] T 24016ceaa84147eca0b94b3ad288e772 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:51:53.197947 1172 ts_tablet_manager.cc:1403] T 24016ceaa84147eca0b94b3ad288e772 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.001s
I20251029 02:51:53.198922 1172 raft_consensus.cc:359] T 24016ceaa84147eca0b94b3ad288e772 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:53.199131 1172 raft_consensus.cc:385] T 24016ceaa84147eca0b94b3ad288e772 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:53.198947 1176 ts_tablet_manager.cc:1916] T bc490b1327e44059a57e6cd7b63350e3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:53.199229 1172 raft_consensus.cc:740] T 24016ceaa84147eca0b94b3ad288e772 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:51:53.199481 1172 consensus_queue.cc:260] T 24016ceaa84147eca0b94b3ad288e772 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:53.200191 1172 ts_tablet_manager.cc:1434] T 24016ceaa84147eca0b94b3ad288e772 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 02:51:53.200395 1174 log.cc:1199] T 55c1b243d8a94938a5caa58754092cec P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/55c1b243d8a94938a5caa58754092cec
I20251029 02:51:53.200716 1172 tablet_bootstrap.cc:492] T 4b76c4408a7340e59b8506171622175f P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:51:53.201220 1174 ts_tablet_manager.cc:1950] T 55c1b243d8a94938a5caa58754092cec P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:51:53.201606 517 master_service.cc:954] ReplaceTablet: received request to replace tablet 7309d3190cfa415281c5253a634fa7f3 from {username='slave'} at 127.0.0.1:35314
I20251029 02:51:53.202489 1172 tablet_bootstrap.cc:654] T 4b76c4408a7340e59b8506171622175f P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:53.198403 1167 ts_tablet_manager.cc:1434] T d32532973b3f46dd8158a2ce42d26365 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.029s user 0.003s sys 0.000s
I20251029 02:51:53.204313 1167 tablet_bootstrap.cc:492] T 9be9add0274d41b688edd6292a253a06 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:51:53.204361 1166 tablet_bootstrap.cc:492] T 11f4252b5e694312aaa142468f8e68b4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:51:53.204528 1166 ts_tablet_manager.cc:1403] T 11f4252b5e694312aaa142468f8e68b4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.039s user 0.007s sys 0.006s
I20251029 02:51:53.205367 1166 raft_consensus.cc:359] T 11f4252b5e694312aaa142468f8e68b4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:53.205559 1166 raft_consensus.cc:385] T 11f4252b5e694312aaa142468f8e68b4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:53.205629 1166 raft_consensus.cc:740] T 11f4252b5e694312aaa142468f8e68b4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:51:53.205960 1167 tablet_bootstrap.cc:654] T 9be9add0274d41b688edd6292a253a06 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:53.205873 1166 consensus_queue.cc:260] T 11f4252b5e694312aaa142468f8e68b4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:53.206907 526 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_7b36c69036a641f289a5f927f3101f16" } new_table_name: "default.Table_f06003da6e324504ade193bbbb11d36a" modify_external_catalogs: true
I20251029 02:51:53.207108 1166 ts_tablet_manager.cc:1434] T 11f4252b5e694312aaa142468f8e68b4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 02:51:53.207495 1166 tablet_bootstrap.cc:492] T 0dd31dc5873d407d958a0f5199f78782 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:51:53.208417 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_42bbf44edb134c16b76604d387907461"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:53.210422 1166 tablet_bootstrap.cc:654] T 0dd31dc5873d407d958a0f5199f78782 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
W20251029 02:51:53.210763 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_42bbf44edb134c16b76604d387907461 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:53.213011 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 55c1b243d8a94938a5caa58754092cec (table default.Table_953f59050dc54b04b57c06364595c612 [id=bd2335c28dd44b5584ab71694cebda2d]) successfully deleted
I20251029 02:51:53.213198 894 tablet_service.cc:1552] Processing DeleteTablet for tablet bc490b1327e44059a57e6cd7b63350e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:52 UTC) from {username='slave'} at 127.0.0.1:35702
W20251029 02:51:53.213186 1215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251029 02:51:53.213601 1215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251029 02:51:53.213711 1215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20251029 02:51:53.213930 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_4e97f5e298434c16913bf7ca93ca30f5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:53.215454 1172 tablet_bootstrap.cc:492] T 4b76c4408a7340e59b8506171622175f P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:51:53.215651 1172 ts_tablet_manager.cc:1403] T 4b76c4408a7340e59b8506171622175f P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.003s
I20251029 02:51:53.216033 755 tablet_service.cc:1552] Processing DeleteTablet for tablet bc490b1327e44059a57e6cd7b63350e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:52 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:53.216835 1172 raft_consensus.cc:359] T 4b76c4408a7340e59b8506171622175f P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
W20251029 02:51:53.217011 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4e97f5e298434c16913bf7ca93ca30f5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:53.217075 1172 raft_consensus.cc:385] T 4b76c4408a7340e59b8506171622175f P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:53.217147 1172 raft_consensus.cc:740] T 4b76c4408a7340e59b8506171622175f P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:51:53.217407 1172 consensus_queue.cc:260] T 4b76c4408a7340e59b8506171622175f P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:53.217963 1172 ts_tablet_manager.cc:1434] T 4b76c4408a7340e59b8506171622175f P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251029 02:51:53.218385 1339 tablet.cc:1722] T 8f7c746f8b6948b58676298c2c2ea221 P c756528642a543eeacd93cef3e335a7d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20251029 02:51:53.219792 1172 tablet_bootstrap.cc:492] T ab421dfaa3f84593842bb732a2693160 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:51:53.221624 1172 tablet_bootstrap.cc:654] T ab421dfaa3f84593842bb732a2693160 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:53.224610 1342 tablet.cc:1722] T 8f7c746f8b6948b58676298c2c2ea221 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20251029 02:51:53.224764 1340 tablet.cc:1722] T 8f7c746f8b6948b58676298c2c2ea221 P b495a49349324f7d81908f2da700f0d9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
W20251029 02:51:53.229031 1215 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251029 02:51:53.229135 1215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251029 02:51:53.229209 1215 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251029 02:51:53.229269 1215 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251029 02:51:53.231694 1174 tablet_replica.cc:333] T bc490b1327e44059a57e6cd7b63350e3 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:51:53.231971 1174 raft_consensus.cc:2243] T bc490b1327e44059a57e6cd7b63350e3 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:51:53.232290 1174 pending_rounds.cc:70] T bc490b1327e44059a57e6cd7b63350e3 P c756528642a543eeacd93cef3e335a7d: Trying to abort 1 pending ops.
I20251029 02:51:53.232400 1174 pending_rounds.cc:77] T bc490b1327e44059a57e6cd7b63350e3 P c756528642a543eeacd93cef3e335a7d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215949057353981952 op_type: NO_OP noop_request { }
I20251029 02:51:53.232607 1174 raft_consensus.cc:2889] T bc490b1327e44059a57e6cd7b63350e3 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 02:51:53.232765 1174 raft_consensus.cc:2272] T bc490b1327e44059a57e6cd7b63350e3 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:53.236143 1174 ts_tablet_manager.cc:1916] T bc490b1327e44059a57e6cd7b63350e3 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:53.245796 1172 tablet_bootstrap.cc:492] T ab421dfaa3f84593842bb732a2693160 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:51:53.246044 1172 ts_tablet_manager.cc:1403] T ab421dfaa3f84593842bb732a2693160 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.028s user 0.000s sys 0.010s
I20251029 02:51:53.247231 1172 raft_consensus.cc:359] T ab421dfaa3f84593842bb732a2693160 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:53.247567 1172 raft_consensus.cc:385] T ab421dfaa3f84593842bb732a2693160 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:53.247709 1172 raft_consensus.cc:740] T ab421dfaa3f84593842bb732a2693160 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:51:53.247901 1215 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:44691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/master-2/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:42343,127.0.104.61:34239,127.0.104.60:41233
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.60:41233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.104.60
--webserver_port=42893
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 60f5e5267b92c39485a66121d3ce3cc7ef57b0e0
build type FASTDEBUG
built by None at 29 Oct 2025 02:43:25 UTC on 24a791456cd2
build id 8768
ASAN enabled
I20251029 02:51:53.248039 1172 consensus_queue.cc:260] T ab421dfaa3f84593842bb732a2693160 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:53.248621 1172 ts_tablet_manager.cc:1434] T ab421dfaa3f84593842bb732a2693160 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251029 02:51:53.248950 1172 tablet_bootstrap.cc:492] T 7c041f4909274cd5a03b9fd78353faf9 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:51:53.251338 1215 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251029 02:51:53.254667 1167 tablet_bootstrap.cc:492] T 9be9add0274d41b688edd6292a253a06 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:51:53.254819 1167 ts_tablet_manager.cc:1403] T 9be9add0274d41b688edd6292a253a06 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.051s user 0.001s sys 0.002s
I20251029 02:51:53.255770 1215 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251029 02:51:53.255901 1167 raft_consensus.cc:359] T 9be9add0274d41b688edd6292a253a06 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:53.256088 1167 raft_consensus.cc:385] T 9be9add0274d41b688edd6292a253a06 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:53.256183 1167 raft_consensus.cc:740] T 9be9add0274d41b688edd6292a253a06 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:51:53.256475 1167 consensus_queue.cc:260] T 9be9add0274d41b688edd6292a253a06 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:53.257308 1167 ts_tablet_manager.cc:1434] T 9be9add0274d41b688edd6292a253a06 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 02:51:53.258630 1167 tablet_bootstrap.cc:492] T 6c32c6f5e1cc40a0a12688201cac62a7 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:51:53.259259 1176 ts_tablet_manager.cc:1929] T bc490b1327e44059a57e6cd7b63350e3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:53.259440 1176 log.cc:1199] T bc490b1327e44059a57e6cd7b63350e3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/bc490b1327e44059a57e6cd7b63350e3
I20251029 02:51:53.259824 1176 ts_tablet_manager.cc:1950] T bc490b1327e44059a57e6cd7b63350e3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:51:53.260888 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet bc490b1327e44059a57e6cd7b63350e3 (table default.Table_6add80f3d25b4701b9a0fa5639dec66c [id=14bb2451b1c6422bb5fe048cd378d1b0]) successfully deleted
I20251029 02:51:53.261945 1351 raft_consensus.cc:515] T 5886b8d39f384426bcbed7289ae72e51 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:53.262683 1351 leader_election.cc:290] T 5886b8d39f384426bcbed7289ae72e51 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Requested vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:51:53.263361 1175 ts_tablet_manager.cc:1916] T bc1a1cf274b64b52aba58811e9594330 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:53.266036 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_33eacbf00740465185d70b46a3782893"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:53.266728 1166 tablet_bootstrap.cc:492] T 0dd31dc5873d407d958a0f5199f78782 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
W20251029 02:51:53.266718 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_33eacbf00740465185d70b46a3782893 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:53.266870 1166 ts_tablet_manager.cc:1403] T 0dd31dc5873d407d958a0f5199f78782 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.059s user 0.016s sys 0.009s
I20251029 02:51:53.269506 1167 tablet_bootstrap.cc:654] T 6c32c6f5e1cc40a0a12688201cac62a7 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
W20251029 02:51:53.270637 524 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 71068739b4384badbf305313d902f0f5 has no consensus state
I20251029 02:51:53.272912 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_b313ce7369b24372b396d8e9a257a369"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:53.273512 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b313ce7369b24372b396d8e9a257a369 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:53.273746 1174 ts_tablet_manager.cc:1929] T bc490b1327e44059a57e6cd7b63350e3 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:51:53.273890 1174 log.cc:1199] T bc490b1327e44059a57e6cd7b63350e3 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/bc490b1327e44059a57e6cd7b63350e3
I20251029 02:51:53.273883 1175 ts_tablet_manager.cc:1929] T bc1a1cf274b64b52aba58811e9594330 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:53.274019 1175 log.cc:1199] T bc1a1cf274b64b52aba58811e9594330 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/bc1a1cf274b64b52aba58811e9594330
I20251029 02:51:53.274396 1175 ts_tablet_manager.cc:1950] T bc1a1cf274b64b52aba58811e9594330 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:51:53.274394 1174 ts_tablet_manager.cc:1950] T bc490b1327e44059a57e6cd7b63350e3 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
W20251029 02:51:53.276137 526 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 24016ceaa84147eca0b94b3ad288e772 consensus state has no leader
I20251029 02:51:53.276254 1172 tablet_bootstrap.cc:654] T 7c041f4909274cd5a03b9fd78353faf9 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
W20251029 02:51:53.270105 989 leader_election.cc:343] T 5886b8d39f384426bcbed7289ae72e51 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 5886b8d39f384426bcbed7289ae72e51
I20251029 02:51:53.267741 1166 raft_consensus.cc:359] T 0dd31dc5873d407d958a0f5199f78782 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:53.276818 1166 raft_consensus.cc:385] T 0dd31dc5873d407d958a0f5199f78782 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:53.276922 1166 raft_consensus.cc:740] T 0dd31dc5873d407d958a0f5199f78782 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:51:53.276937 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_e4de02c7c1f04704bc39e1f0efd2c9d9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:53.277210 1166 consensus_queue.cc:260] T 0dd31dc5873d407d958a0f5199f78782 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:53.277753 1166 ts_tablet_manager.cc:1434] T 0dd31dc5873d407d958a0f5199f78782 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20251029 02:51:53.278123 1166 tablet_bootstrap.cc:492] T d7ba1756532c4825a6c7024511dbae51 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
W20251029 02:51:53.278101 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e4de02c7c1f04704bc39e1f0efd2c9d9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:53.280983 1172 tablet_bootstrap.cc:492] T 7c041f4909274cd5a03b9fd78353faf9 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:51:53.281147 1172 ts_tablet_manager.cc:1403] T 7c041f4909274cd5a03b9fd78353faf9 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.032s user 0.000s sys 0.003s
I20251029 02:51:53.281209 1166 tablet_bootstrap.cc:654] T d7ba1756532c4825a6c7024511dbae51 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:53.281833 1172 raft_consensus.cc:359] T 7c041f4909274cd5a03b9fd78353faf9 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:53.282045 1172 raft_consensus.cc:385] T 7c041f4909274cd5a03b9fd78353faf9 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:53.282105 1172 raft_consensus.cc:740] T 7c041f4909274cd5a03b9fd78353faf9 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:51:53.282361 1172 consensus_queue.cc:260] T 7c041f4909274cd5a03b9fd78353faf9 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:53.282560 1175 tablet_replica.cc:333] T 61c1fec3ab4c402eb29f767b9bb7409f P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:51:53.282826 1175 raft_consensus.cc:2243] T 61c1fec3ab4c402eb29f767b9bb7409f P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:53.282835 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet bc1a1cf274b64b52aba58811e9594330 (table default.Table_98034c2933664cefab94c26cb8349dc4 [id=efb8f381fea74cf48842af6b1a4d5a81]) successfully deleted
I20251029 02:51:53.283530 1172 ts_tablet_manager.cc:1434] T 7c041f4909274cd5a03b9fd78353faf9 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 02:51:53.283766 1175 raft_consensus.cc:2272] T 61c1fec3ab4c402eb29f767b9bb7409f P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:53.284456 1175 ts_tablet_manager.cc:1916] T 61c1fec3ab4c402eb29f767b9bb7409f P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:53.284510 1172 tablet_bootstrap.cc:492] T aacb6f4b9284457bb6607e3a85d5c809 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:51:53.286398 1308 raft_consensus.cc:493] T d32532973b3f46dd8158a2ce42d26365 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:53.286613 1308 raft_consensus.cc:515] T d32532973b3f46dd8158a2ce42d26365 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:53.287464 1308 leader_election.cc:290] T d32532973b3f46dd8158a2ce42d26365 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:51:53.288561 1172 tablet_bootstrap.cc:654] T aacb6f4b9284457bb6607e3a85d5c809 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:53.288717 1062 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d32532973b3f46dd8158a2ce42d26365" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:51:53.288929 1062 raft_consensus.cc:2468] T d32532973b3f46dd8158a2ce42d26365 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:51:53.289642 849 leader_election.cc:304] T d32532973b3f46dd8158a2ce42d26365 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters:
W20251029 02:51:53.290117 847 leader_election.cc:343] T d32532973b3f46dd8158a2ce42d26365 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: d32532973b3f46dd8158a2ce42d26365
I20251029 02:51:53.291047 1308 raft_consensus.cc:2804] T d32532973b3f46dd8158a2ce42d26365 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:51:53.291173 1308 raft_consensus.cc:493] T d32532973b3f46dd8158a2ce42d26365 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:51:53.291303 1308 raft_consensus.cc:3060] T d32532973b3f46dd8158a2ce42d26365 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:51:53.293587 1166 tablet_bootstrap.cc:492] T d7ba1756532c4825a6c7024511dbae51 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:51:53.293756 1166 ts_tablet_manager.cc:1403] T d7ba1756532c4825a6c7024511dbae51 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.001s
I20251029 02:51:53.294579 1166 raft_consensus.cc:359] T d7ba1756532c4825a6c7024511dbae51 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:53.294718 1308 raft_consensus.cc:515] T d32532973b3f46dd8158a2ce42d26365 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:53.295331 1166 raft_consensus.cc:385] T d7ba1756532c4825a6c7024511dbae51 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:53.295428 1166 raft_consensus.cc:740] T d7ba1756532c4825a6c7024511dbae51 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:51:53.295691 1166 consensus_queue.cc:260] T d7ba1756532c4825a6c7024511dbae51 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:53.295982 1062 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d32532973b3f46dd8158a2ce42d26365" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:51:53.296149 1062 raft_consensus.cc:3060] T d32532973b3f46dd8158a2ce42d26365 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:51:53.296444 1166 ts_tablet_manager.cc:1434] T d7ba1756532c4825a6c7024511dbae51 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20251029 02:51:53.296736 1166 tablet_bootstrap.cc:492] T cb55b3d5bc63405f96cdc122f5a612cb P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
W20251029 02:51:53.300544 847 leader_election.cc:343] T d32532973b3f46dd8158a2ce42d26365 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: d32532973b3f46dd8158a2ce42d26365
I20251029 02:51:53.300844 1308 leader_election.cc:290] T d32532973b3f46dd8158a2ce42d26365 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:51:53.300943 1276 raft_consensus.cc:493] T 24016ceaa84147eca0b94b3ad288e772 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:53.301102 1276 raft_consensus.cc:515] T 24016ceaa84147eca0b94b3ad288e772 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:53.301808 1276 leader_election.cc:290] T 24016ceaa84147eca0b94b3ad288e772 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:51:53.302758 1061 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24016ceaa84147eca0b94b3ad288e772" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:51:53.302419 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24016ceaa84147eca0b94b3ad288e772" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:51:53.303493 1172 tablet_bootstrap.cc:492] T aacb6f4b9284457bb6607e3a85d5c809 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:51:53.303638 1166 tablet_bootstrap.cc:654] T cb55b3d5bc63405f96cdc122f5a612cb P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
W20251029 02:51:53.303602 712 leader_election.cc:343] T 24016ceaa84147eca0b94b3ad288e772 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:53.303689 1172 ts_tablet_manager.cc:1403] T aacb6f4b9284457bb6607e3a85d5c809 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.004s
W20251029 02:51:53.304140 711 leader_election.cc:343] T 24016ceaa84147eca0b94b3ad288e772 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:53.304294 711 leader_election.cc:304] T 24016ceaa84147eca0b94b3ad288e772 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:51:53.304788 1172 raft_consensus.cc:359] T aacb6f4b9284457bb6607e3a85d5c809 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:53.304936 1276 raft_consensus.cc:2749] T 24016ceaa84147eca0b94b3ad288e772 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:51:53.304997 1172 raft_consensus.cc:385] T aacb6f4b9284457bb6607e3a85d5c809 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:53.305081 1172 raft_consensus.cc:740] T aacb6f4b9284457bb6607e3a85d5c809 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:51:53.305318 1172 consensus_queue.cc:260] T aacb6f4b9284457bb6607e3a85d5c809 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:53.305682 1175 ts_tablet_manager.cc:1929] T 61c1fec3ab4c402eb29f767b9bb7409f P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:53.305821 1175 log.cc:1199] T 61c1fec3ab4c402eb29f767b9bb7409f P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/61c1fec3ab4c402eb29f767b9bb7409f
I20251029 02:51:53.306134 1175 ts_tablet_manager.cc:1950] T 61c1fec3ab4c402eb29f767b9bb7409f P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:51:53.306110 1062 raft_consensus.cc:2468] T d32532973b3f46dd8158a2ce42d26365 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:51:53.306635 849 leader_election.cc:304] T d32532973b3f46dd8158a2ce42d26365 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters: b495a49349324f7d81908f2da700f0d9
I20251029 02:51:53.307760 1175 tablet_replica.cc:333] T 55c1b243d8a94938a5caa58754092cec P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:51:53.307760 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 61c1fec3ab4c402eb29f767b9bb7409f (table default.Table_42a71464d71d487baa7245b5c3bdbbcd [id=c124cfa700724f6db0f1a12fe495c2af]) successfully deleted
I20251029 02:51:53.307945 1175 raft_consensus.cc:2243] T 55c1b243d8a94938a5caa58754092cec P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus shutting down.
W20251029 02:51:53.308027 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 2242f5b2c20346d9b753d7cd29242441 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2242f5b2c20346d9b753d7cd29242441 already in progress: creating tablet
I20251029 02:51:53.308146 1175 raft_consensus.cc:2272] T 55c1b243d8a94938a5caa58754092cec P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251029 02:51:53.308234 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 7309d3190cfa415281c5253a634fa7f3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7309d3190cfa415281c5253a634fa7f3 already in progress: creating tablet
W20251029 02:51:53.308480 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 3362e95063af41c8a24e935f547e3089 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3362e95063af41c8a24e935f547e3089 already in progress: creating tablet
I20251029 02:51:53.308739 1175 ts_tablet_manager.cc:1916] T 55c1b243d8a94938a5caa58754092cec P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 02:51:53.309597 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 5695b5a7b1854c3983e2d4ab684898b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5695b5a7b1854c3983e2d4ab684898b0 already in progress: creating tablet
I20251029 02:51:53.311439 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5886b8d39f384426bcbed7289ae72e51" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
I20251029 02:51:53.311668 920 raft_consensus.cc:3060] T 5886b8d39f384426bcbed7289ae72e51 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:51:53.315711 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet bc490b1327e44059a57e6cd7b63350e3 (table default.Table_6add80f3d25b4701b9a0fa5639dec66c [id=14bb2451b1c6422bb5fe048cd378d1b0]) successfully deleted
I20251029 02:51:53.316381 920 raft_consensus.cc:2468] T 5886b8d39f384426bcbed7289ae72e51 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 1.
I20251029 02:51:53.317307 520 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_8b68c930049e4c159e83b15be8126f03" } new_table_name: "default.Table_9e6e0bf44516404ab570d4f843bd542f" modify_external_catalogs: true
I20251029 02:51:53.319460 990 leader_election.cc:304] T 5886b8d39f384426bcbed7289ae72e51 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters: b495a49349324f7d81908f2da700f0d9
I20251029 02:51:53.319830 1351 raft_consensus.cc:2804] T 5886b8d39f384426bcbed7289ae72e51 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:51:53.319993 1351 raft_consensus.cc:697] T 5886b8d39f384426bcbed7289ae72e51 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Becoming Leader. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Running, Role: LEADER
I20251029 02:51:53.320278 1351 consensus_queue.cc:237] T 5886b8d39f384426bcbed7289ae72e51 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:53.322160 1308 raft_consensus.cc:2804] T d32532973b3f46dd8158a2ce42d26365 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:51:53.322537 1308 raft_consensus.cc:697] T d32532973b3f46dd8158a2ce42d26365 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:51:53.323943 1308 consensus_queue.cc:237] T d32532973b3f46dd8158a2ce42d26365 P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:53.334705 1276 raft_consensus.cc:493] T 41a6139fd9b84707a8c7eebc0381e383 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:53.334909 1276 raft_consensus.cc:515] T 41a6139fd9b84707a8c7eebc0381e383 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:53.335668 1276 leader_election.cc:290] T 41a6139fd9b84707a8c7eebc0381e383 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:51:53.335812 1172 ts_tablet_manager.cc:1434] T aacb6f4b9284457bb6607e3a85d5c809 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251029 02:51:53.336428 1172 tablet_bootstrap.cc:492] T c4884e22cf2044a4a1070c1b4b759d9e P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:51:53.337989 1062 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "41a6139fd9b84707a8c7eebc0381e383" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:51:53.338925 712 leader_election.cc:343] T 41a6139fd9b84707a8c7eebc0381e383 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:53.339025 1172 tablet_bootstrap.cc:654] T c4884e22cf2044a4a1070c1b4b759d9e P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:53.339679 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "41a6139fd9b84707a8c7eebc0381e383" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:51:53.340325 711 leader_election.cc:343] T 41a6139fd9b84707a8c7eebc0381e383 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:53.340481 711 leader_election.cc:304] T 41a6139fd9b84707a8c7eebc0381e383 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:51:53.340978 1276 raft_consensus.cc:2749] T 41a6139fd9b84707a8c7eebc0381e383 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:51:53.343335 1172 tablet_bootstrap.cc:492] T c4884e22cf2044a4a1070c1b4b759d9e P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:51:53.343637 1172 ts_tablet_manager.cc:1403] T c4884e22cf2044a4a1070c1b4b759d9e P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.003s
I20251029 02:51:53.345088 1172 raft_consensus.cc:359] T c4884e22cf2044a4a1070c1b4b759d9e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:53.345497 1172 raft_consensus.cc:385] T c4884e22cf2044a4a1070c1b4b759d9e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:53.345635 1172 raft_consensus.cc:740] T c4884e22cf2044a4a1070c1b4b759d9e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:51:53.346043 1172 consensus_queue.cc:260] T c4884e22cf2044a4a1070c1b4b759d9e P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:53.346982 1172 ts_tablet_manager.cc:1434] T c4884e22cf2044a4a1070c1b4b759d9e P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20251029 02:51:53.347500 1172 tablet_bootstrap.cc:492] T 64b0fb400e944ce2b2c7a49d12135e9c P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:51:53.347853 1351 raft_consensus.cc:493] T d7ba1756532c4825a6c7024511dbae51 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:53.348021 1351 raft_consensus.cc:515] T d7ba1756532c4825a6c7024511dbae51 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:53.348659 1351 leader_election.cc:290] T d7ba1756532c4825a6c7024511dbae51 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:51:53.348874 1166 tablet_bootstrap.cc:492] T cb55b3d5bc63405f96cdc122f5a612cb P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:51:53.349011 1166 ts_tablet_manager.cc:1403] T cb55b3d5bc63405f96cdc122f5a612cb P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.052s user 0.001s sys 0.005s
W20251029 02:51:53.349592 989 leader_election.cc:343] T d7ba1756532c4825a6c7024511dbae51 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: d7ba1756532c4825a6c7024511dbae51
I20251029 02:51:53.349874 1166 raft_consensus.cc:359] T cb55b3d5bc63405f96cdc122f5a612cb P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:53.350049 1166 raft_consensus.cc:385] T cb55b3d5bc63405f96cdc122f5a612cb P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:53.350147 1166 raft_consensus.cc:740] T cb55b3d5bc63405f96cdc122f5a612cb P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:51:53.350375 1166 consensus_queue.cc:260] T cb55b3d5bc63405f96cdc122f5a612cb P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:53.350797 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d7ba1756532c4825a6c7024511dbae51" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:51:53.351025 1166 ts_tablet_manager.cc:1434] T cb55b3d5bc63405f96cdc122f5a612cb P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 02:51:53.351329 1166 tablet_bootstrap.cc:492] T bf210435010d4dde8903fd116eb468e3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:51:53.351578 1172 tablet_bootstrap.cc:654] T 64b0fb400e944ce2b2c7a49d12135e9c P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:53.353068 1276 raft_consensus.cc:493] T aacb6f4b9284457bb6607e3a85d5c809 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:53.353224 1276 raft_consensus.cc:515] T aacb6f4b9284457bb6607e3a85d5c809 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:53.353680 1166 tablet_bootstrap.cc:654] T bf210435010d4dde8903fd116eb468e3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:53.353940 1276 leader_election.cc:290] T aacb6f4b9284457bb6607e3a85d5c809 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:51:53.354910 1175 ts_tablet_manager.cc:1929] T 55c1b243d8a94938a5caa58754092cec P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251029 02:51:53.355038 1175 log.cc:1199] T 55c1b243d8a94938a5caa58754092cec P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/55c1b243d8a94938a5caa58754092cec
I20251029 02:51:53.355123 1062 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aacb6f4b9284457bb6607e3a85d5c809" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:51:53.355700 1175 ts_tablet_manager.cc:1950] T 55c1b243d8a94938a5caa58754092cec P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
W20251029 02:51:53.355855 712 leader_election.cc:343] T aacb6f4b9284457bb6607e3a85d5c809 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
W20251029 02:51:53.356590 990 leader_election.cc:343] T d7ba1756532c4825a6c7024511dbae51 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:53.356689 1175 tablet_replica.cc:333] T bc490b1327e44059a57e6cd7b63350e3 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:51:53.356767 990 leader_election.cc:304] T d7ba1756532c4825a6c7024511dbae51 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5; no voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d
I20251029 02:51:53.356909 1175 raft_consensus.cc:2243] T bc490b1327e44059a57e6cd7b63350e3 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:53.357050 1351 raft_consensus.cc:2749] T d7ba1756532c4825a6c7024511dbae51 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:51:53.357095 1175 raft_consensus.cc:2272] T bc490b1327e44059a57e6cd7b63350e3 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:53.357676 1175 ts_tablet_manager.cc:1916] T bc490b1327e44059a57e6cd7b63350e3 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:53.358426 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 55c1b243d8a94938a5caa58754092cec (table default.Table_953f59050dc54b04b57c06364595c612 [id=bd2335c28dd44b5584ab71694cebda2d]) successfully deleted
I20251029 02:51:53.358960 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aacb6f4b9284457bb6607e3a85d5c809" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:51:53.359812 711 leader_election.cc:343] T aacb6f4b9284457bb6607e3a85d5c809 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:53.360015 711 leader_election.cc:304] T aacb6f4b9284457bb6607e3a85d5c809 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:51:53.360602 1276 raft_consensus.cc:2749] T aacb6f4b9284457bb6607e3a85d5c809 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:51:53.363878 1172 tablet_bootstrap.cc:492] T 64b0fb400e944ce2b2c7a49d12135e9c P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:51:53.364080 1172 ts_tablet_manager.cc:1403] T 64b0fb400e944ce2b2c7a49d12135e9c P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.000s
I20251029 02:51:53.365150 1172 raft_consensus.cc:359] T 64b0fb400e944ce2b2c7a49d12135e9c P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:53.365404 1172 raft_consensus.cc:385] T 64b0fb400e944ce2b2c7a49d12135e9c P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:53.365543 1172 raft_consensus.cc:740] T 64b0fb400e944ce2b2c7a49d12135e9c P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:51:53.365877 1172 consensus_queue.cc:260] T 64b0fb400e944ce2b2c7a49d12135e9c P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:53.366542 1172 ts_tablet_manager.cc:1434] T 64b0fb400e944ce2b2c7a49d12135e9c P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251029 02:51:53.366830 1172 tablet_bootstrap.cc:492] T dd90fb9eb167478cbc3ec9363af86295 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:51:53.372243 1167 tablet_bootstrap.cc:492] T 6c32c6f5e1cc40a0a12688201cac62a7 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:51:53.372589 1167 ts_tablet_manager.cc:1403] T 6c32c6f5e1cc40a0a12688201cac62a7 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.115s user 0.007s sys 0.029s
I20251029 02:51:53.373567 1167 raft_consensus.cc:359] T 6c32c6f5e1cc40a0a12688201cac62a7 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:53.373893 1167 raft_consensus.cc:385] T 6c32c6f5e1cc40a0a12688201cac62a7 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:53.374017 1167 raft_consensus.cc:740] T 6c32c6f5e1cc40a0a12688201cac62a7 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:51:53.374338 1167 consensus_queue.cc:260] T 6c32c6f5e1cc40a0a12688201cac62a7 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:53.375543 1167 ts_tablet_manager.cc:1434] T 6c32c6f5e1cc40a0a12688201cac62a7 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251029 02:51:53.376030 1167 tablet_bootstrap.cc:492] T d7ba1756532c4825a6c7024511dbae51 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:51:53.376047 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_0fc48a708ecc446990ad491afab2efdd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:53.376631 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0fc48a708ecc446990ad491afab2efdd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:53.379853 744 tablet_service.cc:1505] Processing CreateTablet for tablet 576c4e6e70e24b0399ac9597a30f0569 (DEFAULT_TABLE table=default.Table_f178e528491d416a99b0191e864f53d8 [id=7774a12db22f414f9266236d92b3b511]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.380302 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 576c4e6e70e24b0399ac9597a30f0569. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.380792 894 tablet_service.cc:1505] Processing CreateTablet for tablet 42cfdf88472b44979d86836561005ed1 (DEFAULT_TABLE table=default.Table_41af15055e73444a9fbe64c4c4f63dba [id=5e0c72957b9a491d960b33d3fffd8f70]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.380792 900 tablet_service.cc:1505] Processing CreateTablet for tablet c9dff4dff82f49e7ad49caad5e404f09 (DEFAULT_TABLE table=default.Table_282e9d0ae9384d3284e662fa864925a9 [id=e3b3113e72b747f2a2b55f3b8fae96c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.381090 755 tablet_service.cc:1505] Processing CreateTablet for tablet 42cfdf88472b44979d86836561005ed1 (DEFAULT_TABLE table=default.Table_41af15055e73444a9fbe64c4c4f63dba [id=5e0c72957b9a491d960b33d3fffd8f70]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.381253 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9dff4dff82f49e7ad49caad5e404f09. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.381482 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42cfdf88472b44979d86836561005ed1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.381460 895 tablet_service.cc:1505] Processing CreateTablet for tablet 576c4e6e70e24b0399ac9597a30f0569 (DEFAULT_TABLE table=default.Table_f178e528491d416a99b0191e864f53d8 [id=7774a12db22f414f9266236d92b3b511]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.381778 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 576c4e6e70e24b0399ac9597a30f0569. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.382791 893 tablet_service.cc:1505] Processing CreateTablet for tablet 7e6c4f30be624769961c9ec17c5d5dcb (DEFAULT_TABLE table=default.Table_dea0766540d24b5e9bd6cc8b6338b47c [id=1dfd5b42973c4d5d91e0d9014e34269f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.383149 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e6c4f30be624769961c9ec17c5d5dcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.384294 753 tablet_service.cc:1505] Processing CreateTablet for tablet c9dff4dff82f49e7ad49caad5e404f09 (DEFAULT_TABLE table=default.Table_282e9d0ae9384d3284e662fa864925a9 [id=e3b3113e72b747f2a2b55f3b8fae96c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.384683 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9dff4dff82f49e7ad49caad5e404f09. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.385020 1039 tablet_service.cc:1505] Processing CreateTablet for tablet 42cfdf88472b44979d86836561005ed1 (DEFAULT_TABLE table=default.Table_41af15055e73444a9fbe64c4c4f63dba [id=5e0c72957b9a491d960b33d3fffd8f70]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.385445 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42cfdf88472b44979d86836561005ed1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.386801 1041 tablet_service.cc:1505] Processing CreateTablet for tablet 7e6c4f30be624769961c9ec17c5d5dcb (DEFAULT_TABLE table=default.Table_dea0766540d24b5e9bd6cc8b6338b47c [id=1dfd5b42973c4d5d91e0d9014e34269f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.387195 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e6c4f30be624769961c9ec17c5d5dcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.387156 748 tablet_service.cc:1505] Processing CreateTablet for tablet 7e6c4f30be624769961c9ec17c5d5dcb (DEFAULT_TABLE table=default.Table_dea0766540d24b5e9bd6cc8b6338b47c [id=1dfd5b42973c4d5d91e0d9014e34269f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.386953 524 catalog_manager.cc:5649] T 9be9add0274d41b688edd6292a253a06 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } health_report { overall_health: UNKNOWN } } }
I20251029 02:51:53.387382 524 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet 9be9add0274d41b688edd6292a253a06 (table default.Table_c7daf9eda79e470186f5774de2b824d3 [id=f45ce142f37f4febbceb4e568e35e0f2]). Expected version 1 got 0
I20251029 02:51:53.387738 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e6c4f30be624769961c9ec17c5d5dcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.391414 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42cfdf88472b44979d86836561005ed1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.395238 1033 tablet_service.cc:1505] Processing CreateTablet for tablet cedfb53078d94c81b5e294ce7438325e (DEFAULT_TABLE table=default.Table_ed3714840df249bfb7cd3f3d9abaa578 [id=86195563fb4a4d7abd746f4ac7cd241c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.395699 1037 tablet_service.cc:1505] Processing CreateTablet for tablet 576c4e6e70e24b0399ac9597a30f0569 (DEFAULT_TABLE table=default.Table_f178e528491d416a99b0191e864f53d8 [id=7774a12db22f414f9266236d92b3b511]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.396009 1031 tablet_service.cc:1505] Processing CreateTablet for tablet c9dff4dff82f49e7ad49caad5e404f09 (DEFAULT_TABLE table=default.Table_282e9d0ae9384d3284e662fa864925a9 [id=e3b3113e72b747f2a2b55f3b8fae96c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.396368 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cedfb53078d94c81b5e294ce7438325e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.399147 519 master_service.cc:954] ReplaceTablet: received request to replace tablet 5886b8d39f384426bcbed7289ae72e51 from {username='slave'} at 127.0.0.1:35314
I20251029 02:51:53.399825 521 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_6254452e8df24c21ab7d8ae32a06bdfb" } modify_external_catalogs: true
I20251029 02:51:53.400022 521 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_6254452e8df24c21ab7d8ae32a06bdfb" } modify_external_catalogs: true
I20251029 02:51:53.400753 526 master_service.cc:954] ReplaceTablet: received request to replace tablet 24cb2a71ba3841d4a5e3e5c6fe99ae2f from {username='slave'} at 127.0.0.1:35314
I20251029 02:51:53.401310 1166 tablet_bootstrap.cc:492] T bf210435010d4dde8903fd116eb468e3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:51:53.401650 1166 ts_tablet_manager.cc:1403] T bf210435010d4dde8903fd116eb468e3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.050s user 0.004s sys 0.002s
W20251029 02:51:53.402295 520 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9e07291d6b5645feb4d5890f58e1ef74 consensus state has no leader
I20251029 02:51:53.402678 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 576c4e6e70e24b0399ac9597a30f0569. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.402674 520 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_72f8739f5f654172b10d619a9ac306bd" } modify_external_catalogs: true
I20251029 02:51:53.402647 1166 raft_consensus.cc:359] T bf210435010d4dde8903fd116eb468e3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:53.402873 1166 raft_consensus.cc:385] T bf210435010d4dde8903fd116eb468e3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:53.402853 520 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_72f8739f5f654172b10d619a9ac306bd" } modify_external_catalogs: true
I20251029 02:51:53.402948 1166 raft_consensus.cc:740] T bf210435010d4dde8903fd116eb468e3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:51:53.403297 1166 consensus_queue.cc:260] T bf210435010d4dde8903fd116eb468e3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:53.405735 1166 ts_tablet_manager.cc:1434] T bf210435010d4dde8903fd116eb468e3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20251029 02:51:53.406103 1166 tablet_bootstrap.cc:492] T 6c32c6f5e1cc40a0a12688201cac62a7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:51:53.406838 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9dff4dff82f49e7ad49caad5e404f09. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.413867 1172 tablet_bootstrap.cc:654] T dd90fb9eb167478cbc3ec9363af86295 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:53.421504 895 tablet_service.cc:1505] Processing CreateTablet for tablet f81aecb20d6e498bb9b498698886d65e (DEFAULT_TABLE table=default.Table_98034c2933664cefab94c26cb8349dc4 [id=efb8f381fea74cf48842af6b1a4d5a81]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.422046 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f81aecb20d6e498bb9b498698886d65e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.422407 1038 tablet_service.cc:1505] Processing CreateTablet for tablet 8df97119d0654d7d92bb5bd17c5d8562 (DEFAULT_TABLE table=default.Table_99592a0735024ba7a0d8d6145b61f2e4 [id=ced78d857c314dc7802545a7909bdf46]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.423426 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8df97119d0654d7d92bb5bd17c5d8562. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.423547 893 tablet_service.cc:1505] Processing CreateTablet for tablet 8df97119d0654d7d92bb5bd17c5d8562 (DEFAULT_TABLE table=default.Table_99592a0735024ba7a0d8d6145b61f2e4 [id=ced78d857c314dc7802545a7909bdf46]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.424124 1166 tablet_bootstrap.cc:654] T 6c32c6f5e1cc40a0a12688201cac62a7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:53.425683 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8df97119d0654d7d92bb5bd17c5d8562. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.426636 894 tablet_service.cc:1505] Processing CreateTablet for tablet cedfb53078d94c81b5e294ce7438325e (DEFAULT_TABLE table=default.Table_ed3714840df249bfb7cd3f3d9abaa578 [id=86195563fb4a4d7abd746f4ac7cd241c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.427301 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cedfb53078d94c81b5e294ce7438325e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.431217 754 tablet_service.cc:1505] Processing CreateTablet for tablet 8df97119d0654d7d92bb5bd17c5d8562 (DEFAULT_TABLE table=default.Table_99592a0735024ba7a0d8d6145b61f2e4 [id=ced78d857c314dc7802545a7909bdf46]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.431588 1167 tablet_bootstrap.cc:654] T d7ba1756532c4825a6c7024511dbae51 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:53.431696 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8df97119d0654d7d92bb5bd17c5d8562. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.431728 900 tablet_service.cc:1505] Processing CreateTablet for tablet c9e923724ca9402682f13b252dbc37aa (DEFAULT_TABLE table=default.Table_7680a98fa637454984a5d4da44855a48 [id=03a25420ce6a46fc93a9fb753246bc7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.432080 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9e923724ca9402682f13b252dbc37aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.434620 1166 tablet_bootstrap.cc:492] T 6c32c6f5e1cc40a0a12688201cac62a7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:51:53.434760 1276 raft_consensus.cc:493] T 5e31e061d74146009d24d5cd57487a5e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:53.434798 1166 ts_tablet_manager.cc:1403] T 6c32c6f5e1cc40a0a12688201cac62a7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.029s user 0.008s sys 0.000s
I20251029 02:51:53.434913 1276 raft_consensus.cc:515] T 5e31e061d74146009d24d5cd57487a5e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:53.435645 1276 leader_election.cc:290] T 5e31e061d74146009d24d5cd57487a5e P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:51:53.435634 1166 raft_consensus.cc:359] T 6c32c6f5e1cc40a0a12688201cac62a7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:53.435817 1166 raft_consensus.cc:385] T 6c32c6f5e1cc40a0a12688201cac62a7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:53.435895 1166 raft_consensus.cc:740] T 6c32c6f5e1cc40a0a12688201cac62a7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:51:53.436383 1062 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e31e061d74146009d24d5cd57487a5e" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:51:53.437235 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e31e061d74146009d24d5cd57487a5e" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:51:53.437214 1166 consensus_queue.cc:260] T 6c32c6f5e1cc40a0a12688201cac62a7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
W20251029 02:51:53.437969 712 leader_election.cc:343] T 5e31e061d74146009d24d5cd57487a5e P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
W20251029 02:51:53.438201 711 leader_election.cc:343] T 5e31e061d74146009d24d5cd57487a5e P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:53.438319 711 leader_election.cc:304] T 5e31e061d74146009d24d5cd57487a5e P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:51:53.438773 1276 raft_consensus.cc:2749] T 5e31e061d74146009d24d5cd57487a5e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:51:53.439355 1166 ts_tablet_manager.cc:1434] T 6c32c6f5e1cc40a0a12688201cac62a7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251029 02:51:53.440358 747 tablet_service.cc:1505] Processing CreateTablet for tablet b2b2d64dcf964da8be4b544dbbe31977 (DEFAULT_TABLE table=default.Table_e0e6e47b18d749e8ae80f822af1e23f0 [id=125ae5c7b5074378bcecb546e70f0c13]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.440354 898 tablet_service.cc:1505] Processing CreateTablet for tablet b2b2d64dcf964da8be4b544dbbe31977 (DEFAULT_TABLE table=default.Table_e0e6e47b18d749e8ae80f822af1e23f0 [id=125ae5c7b5074378bcecb546e70f0c13]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.440784 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2b2d64dcf964da8be4b544dbbe31977. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.440819 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2b2d64dcf964da8be4b544dbbe31977. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.441556 1166 tablet_bootstrap.cc:492] T 6dd4696ad4b44b6dab87a205f3f16ebd P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:51:53.445389 1166 tablet_bootstrap.cc:654] T 6dd4696ad4b44b6dab87a205f3f16ebd P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:53.448509 1167 tablet_bootstrap.cc:492] T d7ba1756532c4825a6c7024511dbae51 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:51:53.449590 1031 tablet_service.cc:1505] Processing CreateTablet for tablet c9e923724ca9402682f13b252dbc37aa (DEFAULT_TABLE table=default.Table_7680a98fa637454984a5d4da44855a48 [id=03a25420ce6a46fc93a9fb753246bc7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.449783 1037 tablet_service.cc:1505] Processing CreateTablet for tablet b2b2d64dcf964da8be4b544dbbe31977 (DEFAULT_TABLE table=default.Table_e0e6e47b18d749e8ae80f822af1e23f0 [id=125ae5c7b5074378bcecb546e70f0c13]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.450014 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9e923724ca9402682f13b252dbc37aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.450232 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2b2d64dcf964da8be4b544dbbe31977. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.451069 1167 ts_tablet_manager.cc:1403] T d7ba1756532c4825a6c7024511dbae51 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.075s user 0.001s sys 0.007s
I20251029 02:51:53.452900 1167 raft_consensus.cc:359] T d7ba1756532c4825a6c7024511dbae51 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:53.453159 1167 raft_consensus.cc:385] T d7ba1756532c4825a6c7024511dbae51 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:53.453325 1167 raft_consensus.cc:740] T d7ba1756532c4825a6c7024511dbae51 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:51:53.453619 1167 consensus_queue.cc:260] T d7ba1756532c4825a6c7024511dbae51 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:53.455214 748 tablet_service.cc:1505] Processing CreateTablet for tablet cedfb53078d94c81b5e294ce7438325e (DEFAULT_TABLE table=default.Table_ed3714840df249bfb7cd3f3d9abaa578 [id=86195563fb4a4d7abd746f4ac7cd241c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.455647 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cedfb53078d94c81b5e294ce7438325e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.456772 1167 ts_tablet_manager.cc:1434] T d7ba1756532c4825a6c7024511dbae51 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20251029 02:51:53.458909 1041 tablet_service.cc:1505] Processing CreateTablet for tablet f81aecb20d6e498bb9b498698886d65e (DEFAULT_TABLE table=default.Table_98034c2933664cefab94c26cb8349dc4 [id=efb8f381fea74cf48842af6b1a4d5a81]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.459391 755 tablet_service.cc:1505] Processing CreateTablet for tablet c9e923724ca9402682f13b252dbc37aa (DEFAULT_TABLE table=default.Table_7680a98fa637454984a5d4da44855a48 [id=03a25420ce6a46fc93a9fb753246bc7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.459969 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9e923724ca9402682f13b252dbc37aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.460650 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f81aecb20d6e498bb9b498698886d65e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.463996 1167 tablet_bootstrap.cc:492] T cb55b3d5bc63405f96cdc122f5a612cb P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:51:53.460247 744 tablet_service.cc:1505] Processing CreateTablet for tablet f81aecb20d6e498bb9b498698886d65e (DEFAULT_TABLE table=default.Table_98034c2933664cefab94c26cb8349dc4 [id=efb8f381fea74cf48842af6b1a4d5a81]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.466711 1167 tablet_bootstrap.cc:654] T cb55b3d5bc63405f96cdc122f5a612cb P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:53.469964 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f81aecb20d6e498bb9b498698886d65e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.470438 1166 tablet_bootstrap.cc:492] T 6dd4696ad4b44b6dab87a205f3f16ebd P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:51:53.470647 1166 ts_tablet_manager.cc:1403] T 6dd4696ad4b44b6dab87a205f3f16ebd P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.029s user 0.007s sys 0.003s
I20251029 02:51:53.474455 1167 tablet_bootstrap.cc:492] T cb55b3d5bc63405f96cdc122f5a612cb P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:51:53.475591 1166 raft_consensus.cc:359] T 6dd4696ad4b44b6dab87a205f3f16ebd P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:53.475817 1166 raft_consensus.cc:385] T 6dd4696ad4b44b6dab87a205f3f16ebd P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:53.475925 1166 raft_consensus.cc:740] T 6dd4696ad4b44b6dab87a205f3f16ebd P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:51:53.476771 1167 ts_tablet_manager.cc:1403] T cb55b3d5bc63405f96cdc122f5a612cb P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.005s
I20251029 02:51:53.476737 1166 consensus_queue.cc:260] T 6dd4696ad4b44b6dab87a205f3f16ebd P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:53.477984 1167 raft_consensus.cc:359] T cb55b3d5bc63405f96cdc122f5a612cb P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:53.479707 1166 ts_tablet_manager.cc:1434] T 6dd4696ad4b44b6dab87a205f3f16ebd P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20251029 02:51:53.480333 1167 raft_consensus.cc:385] T cb55b3d5bc63405f96cdc122f5a612cb P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:53.480618 1167 raft_consensus.cc:740] T cb55b3d5bc63405f96cdc122f5a612cb P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:51:53.481158 1167 consensus_queue.cc:260] T cb55b3d5bc63405f96cdc122f5a612cb P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:53.481521 1166 tablet_bootstrap.cc:492] T b867feae24e741aca6902868c35c1428 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:51:53.483250 1166 tablet_bootstrap.cc:654] T b867feae24e741aca6902868c35c1428 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:53.486056 1167 ts_tablet_manager.cc:1434] T cb55b3d5bc63405f96cdc122f5a612cb P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20251029 02:51:53.486764 525 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet aa776cde2f494fa5810539edf9f289a1
I20251029 02:51:53.487270 525 catalog_manager.cc:6719] ReplaceTablet: tablet aa776cde2f494fa5810539edf9f289a1 deleted and replaced by tablet c9bd3f2deb5d4af09b7918269a98509f
I20251029 02:51:53.487661 525 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_14156dbda95043dcb54dd316874be455" } new_table_name: "default.Table_b93cf815f69a4fe895e2ce461148fc71" modify_external_catalogs: true
I20251029 02:51:53.488039 1038 tablet_service.cc:1552] Processing DeleteTablet for tablet aa776cde2f494fa5810539edf9f289a1 with delete_type TABLET_DATA_DELETED (replaced by tablet c9bd3f2deb5d4af09b7918269a98509f) from {username='slave'} at 127.0.0.1:37626
W20251029 02:51:53.488765 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet aa776cde2f494fa5810539edf9f289a1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aa776cde2f494fa5810539edf9f289a1 already in progress: creating tablet
I20251029 02:51:53.489387 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_c73a6bc56eb44e08aa7d2ec65ad88949"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:53.489893 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c73a6bc56eb44e08aa7d2ec65ad88949 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:53.490737 518 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 3cded156380f45cc95893825c71ba0e9
I20251029 02:51:53.491693 518 catalog_manager.cc:6719] ReplaceTablet: tablet 3cded156380f45cc95893825c71ba0e9 deleted and replaced by tablet 02666f6113b24a8594a6baf840d68823
I20251029 02:51:53.491987 518 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_0fa5f3e932c846168255e7540c35a515" } modify_external_catalogs: true
I20251029 02:51:53.492141 1167 tablet_bootstrap.cc:492] T bf210435010d4dde8903fd116eb468e3 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:51:53.492163 518 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_0fa5f3e932c846168255e7540c35a515" } modify_external_catalogs: true
I20251029 02:51:53.492898 1038 tablet_service.cc:1552] Processing DeleteTablet for tablet 3cded156380f45cc95893825c71ba0e9 with delete_type TABLET_DATA_DELETED (replaced by tablet 02666f6113b24a8594a6baf840d68823) from {username='slave'} at 127.0.0.1:37626
W20251029 02:51:53.493610 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 3cded156380f45cc95893825c71ba0e9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3cded156380f45cc95893825c71ba0e9 already in progress: creating tablet
I20251029 02:51:53.494282 522 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet 8f7c746f8b6948b58676298c2c2ea221 (table default.Table_da4961f316764505bb9e0f191a62a327 [id=669ae9dd0a5749b1aa3557ab8183a057]). Expected version 4 got 3
I20251029 02:51:53.494325 1166 tablet_bootstrap.cc:492] T b867feae24e741aca6902868c35c1428 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:51:53.494483 1166 ts_tablet_manager.cc:1403] T b867feae24e741aca6902868c35c1428 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.001s
I20251029 02:51:53.494668 1167 tablet_bootstrap.cc:654] T bf210435010d4dde8903fd116eb468e3 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:53.495276 755 tablet_service.cc:1552] Processing DeleteTablet for tablet aa776cde2f494fa5810539edf9f289a1 with delete_type TABLET_DATA_DELETED (replaced by tablet c9bd3f2deb5d4af09b7918269a98509f) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:53.495560 744 tablet_service.cc:1552] Processing DeleteTablet for tablet 3cded156380f45cc95893825c71ba0e9 with delete_type TABLET_DATA_DELETED (replaced by tablet 02666f6113b24a8594a6baf840d68823) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:53.495513 1166 raft_consensus.cc:359] T b867feae24e741aca6902868c35c1428 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:53.495734 1166 raft_consensus.cc:385] T b867feae24e741aca6902868c35c1428 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:53.495703 523 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet 8f7c746f8b6948b58676298c2c2ea221 (table default.Table_da4961f316764505bb9e0f191a62a327 [id=669ae9dd0a5749b1aa3557ab8183a057]). Expected version 4 got 3
I20251029 02:51:53.495821 1166 raft_consensus.cc:740] T b867feae24e741aca6902868c35c1428 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:51:53.495903 523 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet 11f4252b5e694312aaa142468f8e68b4 (table default.Table_14156dbda95043dcb54dd316874be455 [id=d0cd3d1567c54463947443927e18169f]). Expected version 4 got 0
I20251029 02:51:53.496995 519 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_da0be17b8bdc4e579b6c3faff71d9c98" } modify_external_catalogs: true
I20251029 02:51:53.497170 519 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_da0be17b8bdc4e579b6c3faff71d9c98" } modify_external_catalogs: true
I20251029 02:51:53.498080 1166 consensus_queue.cc:260] T b867feae24e741aca6902868c35c1428 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:53.497576 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 3cded156380f45cc95893825c71ba0e9 with delete_type TABLET_DATA_DELETED (replaced by tablet 02666f6113b24a8594a6baf840d68823) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:53.499982 1166 ts_tablet_manager.cc:1434] T b867feae24e741aca6902868c35c1428 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251029 02:51:53.501536 1166 tablet_bootstrap.cc:492] T bc1a1cf274b64b52aba58811e9594330 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
W20251029 02:51:53.502694 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 3cded156380f45cc95893825c71ba0e9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3cded156380f45cc95893825c71ba0e9 already in progress: creating tablet
I20251029 02:51:53.503242 526 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 24cb2a71ba3841d4a5e3e5c6fe99ae2f
I20251029 02:51:53.503376 1166 tablet_bootstrap.cc:654] T bc1a1cf274b64b52aba58811e9594330 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:53.503755 526 catalog_manager.cc:6719] ReplaceTablet: tablet 24cb2a71ba3841d4a5e3e5c6fe99ae2f deleted and replaced by tablet f60e7c24982d40319801869ad060864d
I20251029 02:51:53.504093 526 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_e60149a6638a4b49843367e80eebda79" } new_table_name: "default.Table_db7380f4e5484484b1d7dc5c1039137a" modify_external_catalogs: true
I20251029 02:51:53.504225 1308 raft_consensus.cc:493] T cb55b3d5bc63405f96cdc122f5a612cb P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:53.504387 1308 raft_consensus.cc:515] T cb55b3d5bc63405f96cdc122f5a612cb P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:53.505261 1308 leader_election.cc:290] T cb55b3d5bc63405f96cdc122f5a612cb P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:51:53.505674 520 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 1f5b40c0403d4a49a4bfde222d8c947a
I20251029 02:51:53.506028 1062 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb55b3d5bc63405f96cdc122f5a612cb" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:51:53.506276 1062 raft_consensus.cc:2468] T cb55b3d5bc63405f96cdc122f5a612cb P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:51:53.506449 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_96a1c9ac99714785a6bb853741027180"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:53.506771 1038 tablet_service.cc:1552] Processing DeleteTablet for tablet 1f5b40c0403d4a49a4bfde222d8c947a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:53 UTC) from {username='slave'} at 127.0.0.1:37626
W20251029 02:51:53.507103 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_96a1c9ac99714785a6bb853741027180 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:51:53.507467 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 1f5b40c0403d4a49a4bfde222d8c947a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1f5b40c0403d4a49a4bfde222d8c947a already in progress: creating tablet
I20251029 02:51:53.507993 849 leader_election.cc:304] T cb55b3d5bc63405f96cdc122f5a612cb P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:51:53.508404 1041 tablet_service.cc:1552] Processing DeleteTablet for tablet 24cb2a71ba3841d4a5e3e5c6fe99ae2f with delete_type TABLET_DATA_DELETED (replaced by tablet f60e7c24982d40319801869ad060864d) from {username='slave'} at 127.0.0.1:37626
W20251029 02:51:53.508560 847 leader_election.cc:343] T cb55b3d5bc63405f96cdc122f5a612cb P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: cb55b3d5bc63405f96cdc122f5a612cb
I20251029 02:51:53.508867 1308 raft_consensus.cc:2804] T cb55b3d5bc63405f96cdc122f5a612cb P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:51:53.508957 1166 tablet_bootstrap.cc:492] T bc1a1cf274b64b52aba58811e9594330 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:51:53.508978 1308 raft_consensus.cc:493] T cb55b3d5bc63405f96cdc122f5a612cb P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:51:53.509073 1308 raft_consensus.cc:3060] T cb55b3d5bc63405f96cdc122f5a612cb P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:51:53.509122 1166 ts_tablet_manager.cc:1403] T bc1a1cf274b64b52aba58811e9594330 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
W20251029 02:51:53.509339 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 24cb2a71ba3841d4a5e3e5c6fe99ae2f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 24cb2a71ba3841d4a5e3e5c6fe99ae2f already in progress: creating tablet
I20251029 02:51:53.509618 744 tablet_service.cc:1552] Processing DeleteTablet for tablet 24cb2a71ba3841d4a5e3e5c6fe99ae2f with delete_type TABLET_DATA_DELETED (replaced by tablet f60e7c24982d40319801869ad060864d) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:53.509843 755 tablet_service.cc:1552] Processing DeleteTablet for tablet 1f5b40c0403d4a49a4bfde222d8c947a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:53 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:53.509905 1166 raft_consensus.cc:359] T bc1a1cf274b64b52aba58811e9594330 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:53.510080 1166 raft_consensus.cc:385] T bc1a1cf274b64b52aba58811e9594330 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:53.510143 1166 raft_consensus.cc:740] T bc1a1cf274b64b52aba58811e9594330 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:51:53.510360 1166 consensus_queue.cc:260] T bc1a1cf274b64b52aba58811e9594330 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:53.512526 1308 raft_consensus.cc:515] T cb55b3d5bc63405f96cdc122f5a612cb P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:53.513712 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 24cb2a71ba3841d4a5e3e5c6fe99ae2f with delete_type TABLET_DATA_DELETED (replaced by tablet f60e7c24982d40319801869ad060864d) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:53.513855 1062 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cb55b3d5bc63405f96cdc122f5a612cb" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:51:53.513978 1166 ts_tablet_manager.cc:1434] T bc1a1cf274b64b52aba58811e9594330 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251029 02:51:53.514041 1062 raft_consensus.cc:3060] T cb55b3d5bc63405f96cdc122f5a612cb P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
W20251029 02:51:53.514554 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 24cb2a71ba3841d4a5e3e5c6fe99ae2f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 24cb2a71ba3841d4a5e3e5c6fe99ae2f already in progress: creating tablet
I20251029 02:51:53.515720 1166 tablet_bootstrap.cc:492] T 41a6139fd9b84707a8c7eebc0381e383 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:51:53.517405 1062 raft_consensus.cc:2468] T cb55b3d5bc63405f96cdc122f5a612cb P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:51:53.517478 1166 tablet_bootstrap.cc:654] T 41a6139fd9b84707a8c7eebc0381e383 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:53.517946 849 leader_election.cc:304] T cb55b3d5bc63405f96cdc122f5a612cb P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters:
W20251029 02:51:53.518266 525 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 11f4252b5e694312aaa142468f8e68b4 consensus state has no leader
I20251029 02:51:53.518306 1307 raft_consensus.cc:2804] T cb55b3d5bc63405f96cdc122f5a612cb P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:51:53.518453 1307 raft_consensus.cc:697] T cb55b3d5bc63405f96cdc122f5a612cb P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:51:53.518765 1307 consensus_queue.cc:237] T cb55b3d5bc63405f96cdc122f5a612cb P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:53.523991 1166 tablet_bootstrap.cc:492] T 41a6139fd9b84707a8c7eebc0381e383 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:51:53.524143 1166 ts_tablet_manager.cc:1403] T 41a6139fd9b84707a8c7eebc0381e383 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
W20251029 02:51:53.524708 847 leader_election.cc:343] T cb55b3d5bc63405f96cdc122f5a612cb P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: cb55b3d5bc63405f96cdc122f5a612cb
I20251029 02:51:53.525094 1308 leader_election.cc:290] T cb55b3d5bc63405f96cdc122f5a612cb P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:51:53.524994 1166 raft_consensus.cc:359] T 41a6139fd9b84707a8c7eebc0381e383 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:53.525203 1166 raft_consensus.cc:385] T 41a6139fd9b84707a8c7eebc0381e383 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:53.525290 1166 raft_consensus.cc:740] T 41a6139fd9b84707a8c7eebc0381e383 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:51:53.525543 1166 consensus_queue.cc:260] T 41a6139fd9b84707a8c7eebc0381e383 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:53.529757 1166 ts_tablet_manager.cc:1434] T 41a6139fd9b84707a8c7eebc0381e383 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251029 02:51:53.533025 898 tablet_service.cc:1552] Processing DeleteTablet for tablet 1f5b40c0403d4a49a4bfde222d8c947a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:53 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:53.533370 775 raft_consensus.cc:1275] T 9be9add0274d41b688edd6292a253a06 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Refusing update from remote peer 0cfd4bee16f84b7c8b0ff3ce14db20d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20251029 02:51:53.533826 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 1f5b40c0403d4a49a4bfde222d8c947a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1f5b40c0403d4a49a4bfde222d8c947a already in progress: creating tablet
I20251029 02:51:53.533887 1351 consensus_queue.cc:1048] T 9be9add0274d41b688edd6292a253a06 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:51:53.535108 920 raft_consensus.cc:3060] T 9be9add0274d41b688edd6292a253a06 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:51:53.535730 1166 tablet_bootstrap.cc:492] T 70e8cfe5eaa9492c85874dc3214a1e0b P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:51:53.540261 1166 tablet_bootstrap.cc:654] T 70e8cfe5eaa9492c85874dc3214a1e0b P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:53.540727 920 raft_consensus.cc:1275] T 9be9add0274d41b688edd6292a253a06 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Refusing update from remote peer 0cfd4bee16f84b7c8b0ff3ce14db20d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251029 02:51:53.541682 1351 consensus_queue.cc:1048] T 9be9add0274d41b688edd6292a253a06 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:51:53.546090 1167 tablet_bootstrap.cc:492] T bf210435010d4dde8903fd116eb468e3 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:51:53.547437 1166 tablet_bootstrap.cc:492] T 70e8cfe5eaa9492c85874dc3214a1e0b P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:51:53.547627 1166 ts_tablet_manager.cc:1403] T 70e8cfe5eaa9492c85874dc3214a1e0b P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.000s
I20251029 02:51:53.547665 1167 ts_tablet_manager.cc:1403] T bf210435010d4dde8903fd116eb468e3 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.056s user 0.000s sys 0.004s
I20251029 02:51:53.548437 1166 raft_consensus.cc:359] T 70e8cfe5eaa9492c85874dc3214a1e0b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:53.548650 1166 raft_consensus.cc:385] T 70e8cfe5eaa9492c85874dc3214a1e0b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:53.548727 1166 raft_consensus.cc:740] T 70e8cfe5eaa9492c85874dc3214a1e0b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:51:53.549815 1342 tablet.cc:1722] T 9be9add0274d41b688edd6292a253a06 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 02:51:53.550084 1167 raft_consensus.cc:359] T bf210435010d4dde8903fd116eb468e3 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:53.550264 1167 raft_consensus.cc:385] T bf210435010d4dde8903fd116eb468e3 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:53.550374 1167 raft_consensus.cc:740] T bf210435010d4dde8903fd116eb468e3 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:51:53.550321 1339 tablet.cc:1722] T 9be9add0274d41b688edd6292a253a06 P c756528642a543eeacd93cef3e335a7d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 02:51:53.553530 1167 consensus_queue.cc:260] T bf210435010d4dde8903fd116eb468e3 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:53.548969 1166 consensus_queue.cc:260] T 70e8cfe5eaa9492c85874dc3214a1e0b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:53.558213 1340 tablet.cc:1722] T 9be9add0274d41b688edd6292a253a06 P b495a49349324f7d81908f2da700f0d9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 02:51:53.559123 1166 ts_tablet_manager.cc:1434] T 70e8cfe5eaa9492c85874dc3214a1e0b P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.011s user 0.002s sys 0.001s
I20251029 02:51:53.559429 1166 tablet_bootstrap.cc:492] T 5e31e061d74146009d24d5cd57487a5e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:51:53.562178 1167 ts_tablet_manager.cc:1434] T bf210435010d4dde8903fd116eb468e3 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.014s user 0.000s sys 0.002s
I20251029 02:51:53.563431 1167 tablet_bootstrap.cc:492] T b867feae24e741aca6902868c35c1428 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:51:53.564996 525 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_98034c2933664cefab94c26cb8349dc4" } modify_external_catalogs: true
I20251029 02:51:53.565219 525 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_98034c2933664cefab94c26cb8349dc4" } modify_external_catalogs: true
I20251029 02:51:53.567312 521 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 02666f6113b24a8594a6baf840d68823
I20251029 02:51:53.567831 1166 tablet_bootstrap.cc:654] T 5e31e061d74146009d24d5cd57487a5e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:53.567739 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_04a6cfdfbf8c4674b30a8c3024e3277c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:53.568338 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_04a6cfdfbf8c4674b30a8c3024e3277c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:53.569290 893 tablet_service.cc:1552] Processing DeleteTablet for tablet aa776cde2f494fa5810539edf9f289a1 with delete_type TABLET_DATA_DELETED (replaced by tablet c9bd3f2deb5d4af09b7918269a98509f) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:53.569506 524 catalog_manager.cc:6194] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: default.Table_da4961f316764505bb9e0f191a62a327 [id=669ae9dd0a5749b1aa3557ab8183a057] alter complete (version 4)
I20251029 02:51:53.571231 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_d6a9ff6b4cea442eb2394340301eac96"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:53.571756 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d6a9ff6b4cea442eb2394340301eac96 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:53.572489 1041 tablet_service.cc:1552] Processing DeleteTablet for tablet 5886b8d39f384426bcbed7289ae72e51 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:51 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:53.572774 1038 tablet_service.cc:1552] Processing DeleteTablet for tablet d32532973b3f46dd8158a2ce42d26365 with delete_type TABLET_DATA_DELETED (replaced by tablet 65fdec7cf5d143208d78d4c3806b1352) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:53.573011 518 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 6c22497760274380a832feff7c5a2fae
I20251029 02:51:53.573410 1176 tablet_replica.cc:333] T 5886b8d39f384426bcbed7289ae72e51 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:51:53.573660 1176 raft_consensus.cc:2243] T 5886b8d39f384426bcbed7289ae72e51 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:51:53.573961 1176 pending_rounds.cc:70] T 5886b8d39f384426bcbed7289ae72e51 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Trying to abort 1 pending ops.
I20251029 02:51:53.573966 518 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_7680a98fa637454984a5d4da44855a48" } new_table_name: "default.Table_b70ecfbeceed48a4bfdb94abe7c580f4" modify_external_catalogs: true
I20251029 02:51:53.574156 1176 pending_rounds.cc:77] T 5886b8d39f384426bcbed7289ae72e51 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215949059362476032 op_type: NO_OP noop_request { }
I20251029 02:51:53.574329 1176 raft_consensus.cc:2889] T 5886b8d39f384426bcbed7289ae72e51 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 02:51:53.574445 1176 raft_consensus.cc:2272] T 5886b8d39f384426bcbed7289ae72e51 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:53.575106 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_c90ddb84f3e044ba96f01a51fd463d6e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:53.575682 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c90ddb84f3e044ba96f01a51fd463d6e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:53.577484 522 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_4d5341f07b99483183d6aedc61c9ddcd" } new_table_name: "default.Table_ee95bb03ab1246e8826f28083411050d" modify_external_catalogs: true
I20251029 02:51:53.579466 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_528aa3f8d5e84754bc56f86ca7a480aa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:53.579752 1031 tablet_service.cc:1552] Processing DeleteTablet for tablet 6c22497760274380a832feff7c5a2fae with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:53 UTC) from {username='slave'} at 127.0.0.1:37626
W20251029 02:51:53.579984 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_528aa3f8d5e84754bc56f86ca7a480aa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:53.581264 519 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 9964ad774890422f8e14f6b99f28cb55
I20251029 02:51:53.582319 1176 ts_tablet_manager.cc:1916] T 5886b8d39f384426bcbed7289ae72e51 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:53.584789 894 tablet_service.cc:1552] Processing DeleteTablet for tablet d32532973b3f46dd8158a2ce42d26365 with delete_type TABLET_DATA_DELETED (replaced by tablet 65fdec7cf5d143208d78d4c3806b1352) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:53.585016 895 tablet_service.cc:1552] Processing DeleteTablet for tablet 5886b8d39f384426bcbed7289ae72e51 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:51 UTC) from {username='slave'} at 127.0.0.1:35702
W20251029 02:51:53.585247 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet aa776cde2f494fa5810539edf9f289a1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aa776cde2f494fa5810539edf9f289a1 already in progress: creating tablet
I20251029 02:51:53.585495 1174 tablet_replica.cc:333] T d32532973b3f46dd8158a2ce42d26365 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:51:53.585711 1174 raft_consensus.cc:2243] T d32532973b3f46dd8158a2ce42d26365 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:51:53.585970 1174 pending_rounds.cc:70] T d32532973b3f46dd8158a2ce42d26365 P c756528642a543eeacd93cef3e335a7d: Trying to abort 1 pending ops.
I20251029 02:51:53.586023 1174 pending_rounds.cc:77] T d32532973b3f46dd8158a2ce42d26365 P c756528642a543eeacd93cef3e335a7d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215949059383300096 op_type: NO_OP noop_request { }
I20251029 02:51:53.586155 1174 raft_consensus.cc:2889] T d32532973b3f46dd8158a2ce42d26365 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 02:51:53.586263 1174 raft_consensus.cc:2272] T d32532973b3f46dd8158a2ce42d26365 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:53.586530 1031 tablet_service.cc:1552] Processing DeleteTablet for tablet 9964ad774890422f8e14f6b99f28cb55 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:53 UTC) from {username='slave'} at 127.0.0.1:37626
W20251029 02:51:53.586800 526 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e1fa4980f2624321b617b84c68b90113 has no consensus state
I20251029 02:51:53.586764 1037 tablet_service.cc:1552] Processing DeleteTablet for tablet deca0c4a114441079fce22e74b403050 with delete_type TABLET_DATA_DELETED (replaced by tablet e19c8b10f32849f5885fc00176c8bc12) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:53.587024 893 tablet_service.cc:1552] Processing DeleteTablet for tablet 6c22497760274380a832feff7c5a2fae with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:53 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:53.552802 1175 ts_tablet_manager.cc:1929] T bc490b1327e44059a57e6cd7b63350e3 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:53.589520 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 9964ad774890422f8e14f6b99f28cb55 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:53 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:53.591373 1174 ts_tablet_manager.cc:1916] T d32532973b3f46dd8158a2ce42d26365 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 02:51:53.594053 1373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251029 02:51:53.594995 755 tablet_service.cc:1552] Processing DeleteTablet for tablet 6c22497760274380a832feff7c5a2fae with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:53 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:53.595531 1175 log.cc:1199] T bc490b1327e44059a57e6cd7b63350e3 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/bc490b1327e44059a57e6cd7b63350e3
I20251029 02:51:53.596000 1175 ts_tablet_manager.cc:1950] T bc490b1327e44059a57e6cd7b63350e3 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:51:53.597831 898 tablet_service.cc:1552] Processing DeleteTablet for tablet 0dd31dc5873d407d958a0f5199f78782 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:51 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:53.598619 755 tablet_service.cc:1552] Processing DeleteTablet for tablet 9964ad774890422f8e14f6b99f28cb55 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:53 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:53.601977 1176 ts_tablet_manager.cc:1929] T 5886b8d39f384426bcbed7289ae72e51 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:51:53.602283 1176 log.cc:1199] T 5886b8d39f384426bcbed7289ae72e51 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/5886b8d39f384426bcbed7289ae72e51
I20251029 02:51:53.602833 1176 ts_tablet_manager.cc:1950] T 5886b8d39f384426bcbed7289ae72e51 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:51:53.603957 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 5886b8d39f384426bcbed7289ae72e51 (table default.Table_839fd4e3c7874233afc8cea6d4c17026 [id=ace1554749d24504aad642b368b1781f]) successfully deleted
I20251029 02:51:53.604266 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet bc490b1327e44059a57e6cd7b63350e3 (table default.Table_6add80f3d25b4701b9a0fa5639dec66c [id=14bb2451b1c6422bb5fe048cd378d1b0]) successfully deleted
W20251029 02:51:53.604537 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet aa776cde2f494fa5810539edf9f289a1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aa776cde2f494fa5810539edf9f289a1 already in progress: creating tablet
W20251029 02:51:53.604848 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 24cb2a71ba3841d4a5e3e5c6fe99ae2f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 24cb2a71ba3841d4a5e3e5c6fe99ae2f already in progress: creating tablet
W20251029 02:51:53.605044 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 1f5b40c0403d4a49a4bfde222d8c947a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1f5b40c0403d4a49a4bfde222d8c947a already in progress: creating tablet
W20251029 02:51:53.605281 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 6c22497760274380a832feff7c5a2fae because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6c22497760274380a832feff7c5a2fae already in progress: creating tablet
I20251029 02:51:53.605530 1176 tablet_replica.cc:333] T d32532973b3f46dd8158a2ce42d26365 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
W20251029 02:51:53.605540 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 9964ad774890422f8e14f6b99f28cb55 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9964ad774890422f8e14f6b99f28cb55 already in progress: creating tablet
I20251029 02:51:53.606727 1176 raft_consensus.cc:2243] T d32532973b3f46dd8158a2ce42d26365 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:53.607246 1167 tablet_bootstrap.cc:654] T b867feae24e741aca6902868c35c1428 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:53.608952 1176 raft_consensus.cc:2272] T d32532973b3f46dd8158a2ce42d26365 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:53.608939 1174 ts_tablet_manager.cc:1929] T d32532973b3f46dd8158a2ce42d26365 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:51:53.610152 1174 log.cc:1199] T d32532973b3f46dd8158a2ce42d26365 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/d32532973b3f46dd8158a2ce42d26365
I20251029 02:51:53.610719 1174 ts_tablet_manager.cc:1950] T d32532973b3f46dd8158a2ce42d26365 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:51:53.612146 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet d32532973b3f46dd8158a2ce42d26365 (table default.Table_d1a375b674704cddb1c3bb13c566729e [id=909ecde47c7245b2b84dcd665e779b1e]) successfully deleted
I20251029 02:51:53.614341 1176 ts_tablet_manager.cc:1916] T d32532973b3f46dd8158a2ce42d26365 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:53.615381 521 master_service.cc:954] ReplaceTablet: received request to replace tablet aa776cde2f494fa5810539edf9f289a1 from {username='slave'} at 127.0.0.1:35314
I20251029 02:51:53.616144 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_0c1681c9cd0347f2ba2eab73b8cf11a9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:53.618297 1350 raft_consensus.cc:493] T 11f4252b5e694312aaa142468f8e68b4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:53.618542 1166 tablet_bootstrap.cc:492] T 5e31e061d74146009d24d5cd57487a5e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:51:53.618703 1166 ts_tablet_manager.cc:1403] T 5e31e061d74146009d24d5cd57487a5e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.059s user 0.002s sys 0.019s
I20251029 02:51:53.618464 1350 raft_consensus.cc:515] T 11f4252b5e694312aaa142468f8e68b4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
W20251029 02:51:53.619344 522 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 51effc9324034ae79e2b25cb57a996c1 consensus state has no leader
I20251029 02:51:53.619762 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_3126aa57075d44dc8b73f08fe21aabdf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:53.620342 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3126aa57075d44dc8b73f08fe21aabdf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:53.620899 1174 tablet_replica.cc:333] T 5886b8d39f384426bcbed7289ae72e51 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:51:53.621253 1174 raft_consensus.cc:2243] T 5886b8d39f384426bcbed7289ae72e51 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:53.621414 1166 raft_consensus.cc:359] T 5e31e061d74146009d24d5cd57487a5e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:53.621621 1166 raft_consensus.cc:385] T 5e31e061d74146009d24d5cd57487a5e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:53.621691 1350 leader_election.cc:290] T 11f4252b5e694312aaa142468f8e68b4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:51:53.621752 1166 raft_consensus.cc:740] T 5e31e061d74146009d24d5cd57487a5e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:51:53.621995 1166 consensus_queue.cc:260] T 5e31e061d74146009d24d5cd57487a5e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:53.622566 1166 ts_tablet_manager.cc:1434] T 5e31e061d74146009d24d5cd57487a5e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20251029 02:51:53.627146 1176 ts_tablet_manager.cc:1929] T d32532973b3f46dd8158a2ce42d26365 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:53.627379 1176 log.cc:1199] T d32532973b3f46dd8158a2ce42d26365 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/d32532973b3f46dd8158a2ce42d26365
I20251029 02:51:53.627815 1176 ts_tablet_manager.cc:1950] T d32532973b3f46dd8158a2ce42d26365 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:51:53.628183 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11f4252b5e694312aaa142468f8e68b4" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:51:53.628427 775 raft_consensus.cc:2468] T 11f4252b5e694312aaa142468f8e68b4 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 0.
I20251029 02:51:53.628950 1166 tablet_bootstrap.cc:492] T 61c1fec3ab4c402eb29f767b9bb7409f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:51:53.629099 989 leader_election.cc:304] T 11f4252b5e694312aaa142468f8e68b4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters:
I20251029 02:51:53.629662 1350 raft_consensus.cc:2804] T 11f4252b5e694312aaa142468f8e68b4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:51:53.629777 1350 raft_consensus.cc:493] T 11f4252b5e694312aaa142468f8e68b4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:51:53.629895 1350 raft_consensus.cc:3060] T 11f4252b5e694312aaa142468f8e68b4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:51:53.630141 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_77a5f27bd0ae4eb68a30c883268d2f9c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:53.630343 1176 tablet_replica.cc:333] T deca0c4a114441079fce22e74b403050 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:51:53.630559 1176 raft_consensus.cc:2243] T deca0c4a114441079fce22e74b403050 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:53.630751 1176 raft_consensus.cc:2272] T deca0c4a114441079fce22e74b403050 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251029 02:51:53.630824 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_77a5f27bd0ae4eb68a30c883268d2f9c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:51:53.631755 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0c1681c9cd0347f2ba2eab73b8cf11a9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:53.632707 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet d32532973b3f46dd8158a2ce42d26365 (table default.Table_d1a375b674704cddb1c3bb13c566729e [id=909ecde47c7245b2b84dcd665e779b1e]) successfully deleted
W20251029 02:51:53.633054 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 6c22497760274380a832feff7c5a2fae because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6c22497760274380a832feff7c5a2fae already in progress: creating tablet
W20251029 02:51:53.633340 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 9964ad774890422f8e14f6b99f28cb55 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9964ad774890422f8e14f6b99f28cb55 already in progress: creating tablet
I20251029 02:51:53.634931 517 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_8d5ec842f38f4e6f9c387ff7126ce94d" } new_table_name: "default.Table_bafe26b1fca14cc9b908984b7d8a2442" modify_external_catalogs: true
I20251029 02:51:53.635056 1172 tablet_bootstrap.cc:492] T dd90fb9eb167478cbc3ec9363af86295 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:51:53.635241 1172 ts_tablet_manager.cc:1403] T dd90fb9eb167478cbc3ec9363af86295 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.268s user 0.009s sys 0.013s
I20251029 02:51:53.641450 526 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet 11f4252b5e694312aaa142468f8e68b4 (table default.Table_b93cf815f69a4fe895e2ce461148fc71 [id=d0cd3d1567c54463947443927e18169f]). Expected version 5 got 0
I20251029 02:51:53.643445 520 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet 24016ceaa84147eca0b94b3ad288e772 (table default.Table_f06003da6e324504ade193bbbb11d36a [id=bd279b3ce623477ca20e8d35e3b7c0df]). Expected version 1 got 0
I20251029 02:51:53.643841 520 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet 4b76c4408a7340e59b8506171622175f (table default.Table_1281926bf1224c0cb4541bf561102021 [id=f3fa2dbbef7e47359492d5a5eef9b8fd]). Expected version 2 got 0
I20251029 02:51:53.645048 525 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet f81aecb20d6e498bb9b498698886d65e
I20251029 02:51:53.645337 755 tablet_service.cc:1505] Processing CreateTablet for tablet f246f6c202cf44d48c9f5e4f335202c5 (DEFAULT_TABLE table=default.Table_e4de02c7c1f04704bc39e1f0efd2c9d9 [id=40738627473f4a55adc5ed250af29d6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.645802 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f246f6c202cf44d48c9f5e4f335202c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.645951 1351 raft_consensus.cc:493] T 5e31e061d74146009d24d5cd57487a5e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:53.646533 1166 tablet_bootstrap.cc:654] T 61c1fec3ab4c402eb29f767b9bb7409f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:53.647884 1351 raft_consensus.cc:515] T 5e31e061d74146009d24d5cd57487a5e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:53.648995 1351 leader_election.cc:290] T 5e31e061d74146009d24d5cd57487a5e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:51:53.649508 1037 tablet_service.cc:1552] Processing DeleteTablet for tablet f81aecb20d6e498bb9b498698886d65e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:53 UTC) from {username='slave'} at 127.0.0.1:37626
W20251029 02:51:53.649893 518 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c9e923724ca9402682f13b252dbc37aa consensus state has no leader
I20251029 02:51:53.651866 519 catalog_manager.cc:6194] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: default.Table_c7daf9eda79e470186f5774de2b824d3 [id=f45ce142f37f4febbceb4e568e35e0f2] alter complete (version 1)
I20251029 02:51:53.652875 1350 raft_consensus.cc:515] T 11f4252b5e694312aaa142468f8e68b4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
W20251029 02:51:53.654620 517 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4bf06476c87a4549a73f5994314bb75f consensus state has no leader
I20251029 02:51:53.656369 1037 tablet_service.cc:1552] Processing DeleteTablet for tablet 0dd31dc5873d407d958a0f5199f78782 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:51 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:53.656726 1037 tablet_service.cc:1552] Processing DeleteTablet for tablet d7ba1756532c4825a6c7024511dbae51 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:51 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:53.658644 1038 tablet_service.cc:1552] Processing DeleteTablet for tablet 6dd4696ad4b44b6dab87a205f3f16ebd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:52 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:53.658915 1041 tablet_service.cc:1552] Processing DeleteTablet for tablet cb55b3d5bc63405f96cdc122f5a612cb with delete_type TABLET_DATA_DELETED (replaced by tablet aa776cde2f494fa5810539edf9f289a1) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:53.659287 1031 tablet_service.cc:1552] Processing DeleteTablet for tablet 5886b8d39f384426bcbed7289ae72e51 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:51 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:53.659546 1041 tablet_service.cc:1552] Processing DeleteTablet for tablet bc1a1cf274b64b52aba58811e9594330 with delete_type TABLET_DATA_DELETED (replaced by tablet f81aecb20d6e498bb9b498698886d65e) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:53.660660 1041 tablet_service.cc:1552] Processing DeleteTablet for tablet b867feae24e741aca6902868c35c1428 with delete_type TABLET_DATA_DELETED (replaced by tablet 1bab90fe904944af948dab3db06b7e34) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:53.660745 748 tablet_service.cc:1552] Processing DeleteTablet for tablet 64b0fb400e944ce2b2c7a49d12135e9c with delete_type TABLET_DATA_DELETED (replaced by tablet 9e07291d6b5645feb4d5890f58e1ef74) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:53.661175 749 tablet_service.cc:1505] Processing CreateTablet for tablet 6ca9bd37de9e4be388501b15af4bda31 (DEFAULT_TABLE table=default.Table_42bbf44edb134c16b76604d387907461 [id=a862192912a04942a8f682fc9825b014]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.661201 1041 tablet_service.cc:1552] Processing DeleteTablet for tablet 6c32c6f5e1cc40a0a12688201cac62a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:51 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:53.661432 1031 tablet_service.cc:1552] Processing DeleteTablet for tablet bf210435010d4dde8903fd116eb468e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:51 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:53.661669 744 tablet_service.cc:1552] Processing DeleteTablet for tablet f81aecb20d6e498bb9b498698886d65e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:53 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:53.661862 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e31e061d74146009d24d5cd57487a5e" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:51:53.662084 775 raft_consensus.cc:2468] T 5e31e061d74146009d24d5cd57487a5e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 0.
I20251029 02:51:53.662313 747 tablet_service.cc:1552] Processing DeleteTablet for tablet 7c041f4909274cd5a03b9fd78353faf9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:52 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:53.662498 754 tablet_service.cc:1552] Processing DeleteTablet for tablet ab421dfaa3f84593842bb732a2693160 with delete_type TABLET_DATA_DELETED (replaced by tablet c9e923724ca9402682f13b252dbc37aa) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:53.662633 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11f4252b5e694312aaa142468f8e68b4" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:51:53.662796 776 raft_consensus.cc:3060] T 11f4252b5e694312aaa142468f8e68b4 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:51:53.662904 744 tablet_service.cc:1505] Processing CreateTablet for tablet 6bd3fac2502e479aa9f4ceaabbb39504 (DEFAULT_TABLE table=default.Table_febb0d5740c847ce8aafa84d39ccbe91 [id=3513c69f6b3941849ac5b8ae0a5846d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.636951 1172 raft_consensus.cc:359] T dd90fb9eb167478cbc3ec9363af86295 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:53.663445 989 leader_election.cc:304] T 5e31e061d74146009d24d5cd57487a5e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters:
I20251029 02:51:53.663618 1172 raft_consensus.cc:385] T dd90fb9eb167478cbc3ec9363af86295 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:53.663751 1172 raft_consensus.cc:740] T dd90fb9eb167478cbc3ec9363af86295 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:51:53.663842 1351 raft_consensus.cc:2804] T 5e31e061d74146009d24d5cd57487a5e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:51:53.663945 1351 raft_consensus.cc:493] T 5e31e061d74146009d24d5cd57487a5e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:51:53.664060 1351 raft_consensus.cc:3060] T 5e31e061d74146009d24d5cd57487a5e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:51:53.664589 1172 consensus_queue.cc:260] T dd90fb9eb167478cbc3ec9363af86295 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:53.666493 745 tablet_service.cc:1505] Processing CreateTablet for tablet 74873a05ef634867a30aa63b31fcfcc7 (DEFAULT_TABLE table=default.Table_4e97f5e298434c16913bf7ca93ca30f5 [id=48dfcbe8b3654d72b51b77473dd918f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.667191 1031 tablet_service.cc:1505] Processing CreateTablet for tablet f246f6c202cf44d48c9f5e4f335202c5 (DEFAULT_TABLE table=default.Table_e4de02c7c1f04704bc39e1f0efd2c9d9 [id=40738627473f4a55adc5ed250af29d6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.667332 753 tablet_service.cc:1552] Processing DeleteTablet for tablet c4884e22cf2044a4a1070c1b4b759d9e with delete_type TABLET_DATA_DELETED (replaced by tablet 9964ad774890422f8e14f6b99f28cb55) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:53.667708 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f246f6c202cf44d48c9f5e4f335202c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.667695 754 tablet_service.cc:1505] Processing CreateTablet for tablet 0cc1957a845f4e9cb703a553b27d1347 (DEFAULT_TABLE table=default.Table_262db98ea44648baab95fc2b3081dc4e [id=843f681b71c64427b49becbabcad9451]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.668224 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ca9bd37de9e4be388501b15af4bda31. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.669870 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6bd3fac2502e479aa9f4ceaabbb39504. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.670189 1172 ts_tablet_manager.cc:1434] T dd90fb9eb167478cbc3ec9363af86295 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.033s user 0.002s sys 0.001s
I20251029 02:51:53.672588 1351 raft_consensus.cc:515] T 5e31e061d74146009d24d5cd57487a5e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:53.679199 1041 tablet_service.cc:1505] Processing CreateTablet for tablet 74873a05ef634867a30aa63b31fcfcc7 (DEFAULT_TABLE table=default.Table_4e97f5e298434c16913bf7ca93ca30f5 [id=48dfcbe8b3654d72b51b77473dd918f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.679809 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74873a05ef634867a30aa63b31fcfcc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.683293 1038 tablet_service.cc:1505] Processing CreateTablet for tablet 6ca9bd37de9e4be388501b15af4bda31 (DEFAULT_TABLE table=default.Table_42bbf44edb134c16b76604d387907461 [id=a862192912a04942a8f682fc9825b014]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.683835 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ca9bd37de9e4be388501b15af4bda31. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.686345 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74873a05ef634867a30aa63b31fcfcc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.687790 1037 tablet_service.cc:1505] Processing CreateTablet for tablet 6bd3fac2502e479aa9f4ceaabbb39504 (DEFAULT_TABLE table=default.Table_febb0d5740c847ce8aafa84d39ccbe91 [id=3513c69f6b3941849ac5b8ae0a5846d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.688061 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0cc1957a845f4e9cb703a553b27d1347. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.688215 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6bd3fac2502e479aa9f4ceaabbb39504. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.689216 1276 raft_consensus.cc:493] T 4b76c4408a7340e59b8506171622175f P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:53.689368 1276 raft_consensus.cc:515] T 4b76c4408a7340e59b8506171622175f P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:53.690591 1276 leader_election.cc:290] T 4b76c4408a7340e59b8506171622175f P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:51:53.690945 753 tablet_service.cc:1505] Processing CreateTablet for tablet 71068739b4384badbf305313d902f0f5 (DEFAULT_TABLE table=default.Table_4e899dcc04fe4cb1b96ced1fa1b03585 [id=992716bbc1694bd6b85e4c8a3ab51e40]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.691646 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71068739b4384badbf305313d902f0f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.692224 1033 tablet_service.cc:1505] Processing CreateTablet for tablet 0cc1957a845f4e9cb703a553b27d1347 (DEFAULT_TABLE table=default.Table_262db98ea44648baab95fc2b3081dc4e [id=843f681b71c64427b49becbabcad9451]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.692664 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0cc1957a845f4e9cb703a553b27d1347. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.694850 1039 tablet_service.cc:1505] Processing CreateTablet for tablet 71068739b4384badbf305313d902f0f5 (DEFAULT_TABLE table=default.Table_4e899dcc04fe4cb1b96ced1fa1b03585 [id=992716bbc1694bd6b85e4c8a3ab51e40]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.695232 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71068739b4384badbf305313d902f0f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.695286 776 raft_consensus.cc:2468] T 11f4252b5e694312aaa142468f8e68b4 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 1.
I20251029 02:51:53.696027 747 tablet_service.cc:1505] Processing CreateTablet for tablet da2a21c3236a4bd99872b89b12a36508 (DEFAULT_TABLE table=default.Table_b8bb03c0242d4262b78640fa8c88dc27 [id=fb42af08ec1f48498ee5e91cce62333e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.696467 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da2a21c3236a4bd99872b89b12a36508. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.696993 756 tablet_service.cc:1505] Processing CreateTablet for tablet e1fa4980f2624321b617b84c68b90113 (DEFAULT_TABLE table=default.Table_db7380f4e5484484b1d7dc5c1039137a [id=a0e161a5d9d94a3b936669e6dfe6fb4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.697328 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1fa4980f2624321b617b84c68b90113. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.697289 750 tablet_service.cc:1505] Processing CreateTablet for tablet 9e6eb099648549dbbf9969e59865f51f (DEFAULT_TABLE table=default.Table_218b1046ed7c4b0d8b9c55a9542a6a2e [id=0c5d3e4d34d045dc99ffc88e101cae1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.697599 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e6eb099648549dbbf9969e59865f51f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.698856 748 tablet_service.cc:1505] Processing CreateTablet for tablet bfec87f690a640c39c58c64e16cc67e7 (DEFAULT_TABLE table=default.Table_b313ce7369b24372b396d8e9a257a369 [id=6506de17cec5443b8a4628bd46a39b42]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.698917 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_f914dd26d0b846f8923187c2c8edf27e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:53.699615 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f914dd26d0b846f8923187c2c8edf27e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:53.700426 751 tablet_service.cc:1505] Processing CreateTablet for tablet daa05a34c0684c07a35966290fd7d44c (DEFAULT_TABLE table=default.Table_33eacbf00740465185d70b46a3782893 [id=22e43c4483ae488b81fb94a7d221bdce]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.701144 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bfec87f690a640c39c58c64e16cc67e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.701805 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet daa05a34c0684c07a35966290fd7d44c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.702193 1172 tablet_bootstrap.cc:492] T 6f93812578a940a3b7378015e2a46f95 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:51:53.703047 1175 tablet_replica.cc:333] T 64b0fb400e944ce2b2c7a49d12135e9c P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:51:53.703253 1175 raft_consensus.cc:2243] T 64b0fb400e944ce2b2c7a49d12135e9c P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:53.703464 1175 raft_consensus.cc:2272] T 64b0fb400e944ce2b2c7a49d12135e9c P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:53.700541 746 tablet_service.cc:1505] Processing CreateTablet for tablet c764062d3ba04d4ba8b2e367ad8ed433 (DEFAULT_TABLE table=default.Table_bce83bae13a94cd5bf6733b53e6a1e9e [id=761ebdd0095c414488aa221cbc99d3bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.705044 1034 tablet_service.cc:1505] Processing CreateTablet for tablet da2a21c3236a4bd99872b89b12a36508 (DEFAULT_TABLE table=default.Table_b8bb03c0242d4262b78640fa8c88dc27 [id=fb42af08ec1f48498ee5e91cce62333e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.705452 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da2a21c3236a4bd99872b89b12a36508. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.706184 526 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet 5e31e061d74146009d24d5cd57487a5e (table default.Table_f2bc74933428449197f8c28182c8698f [id=8e17d6b87b984d878787802bd6f627f2]). Expected version 2 got 0
I20251029 02:51:53.709118 752 tablet_service.cc:1505] Processing CreateTablet for tablet dd6ad641fbda472fab07aa35c5e4523a (DEFAULT_TABLE table=default.Table_e4bba3a0c06a4ff5bd8829e1081b9c68 [id=1d39d6caa9e24fe38b5cf9a20a6c66b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.711597 526 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_e0e6e47b18d749e8ae80f822af1e23f0" } modify_external_catalogs: true
I20251029 02:51:53.711805 526 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_e0e6e47b18d749e8ae80f822af1e23f0" } modify_external_catalogs: true
I20251029 02:51:53.713181 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd6ad641fbda472fab07aa35c5e4523a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.716768 1032 tablet_service.cc:1505] Processing CreateTablet for tablet dd6ad641fbda472fab07aa35c5e4523a (DEFAULT_TABLE table=default.Table_e4bba3a0c06a4ff5bd8829e1081b9c68 [id=1d39d6caa9e24fe38b5cf9a20a6c66b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.721933 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd6ad641fbda472fab07aa35c5e4523a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.721971 1351 leader_election.cc:290] T 5e31e061d74146009d24d5cd57487a5e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Requested vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:51:53.722395 526 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet b2b2d64dcf964da8be4b544dbbe31977
I20251029 02:51:53.723685 743 tablet_service.cc:1505] Processing CreateTablet for tablet 56d05e4a500043c995bb51a204cb295d (DEFAULT_TABLE table=default.Table_0c1681c9cd0347f2ba2eab73b8cf11a9 [id=fdb365438df14f12a382aa046aef41a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.724184 1175 ts_tablet_manager.cc:1916] T 64b0fb400e944ce2b2c7a49d12135e9c P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:53.724661 1176 ts_tablet_manager.cc:1916] T deca0c4a114441079fce22e74b403050 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:53.725064 1166 tablet_bootstrap.cc:492] T 61c1fec3ab4c402eb29f767b9bb7409f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:51:53.722280 1040 tablet_service.cc:1505] Processing CreateTablet for tablet e1fa4980f2624321b617b84c68b90113 (DEFAULT_TABLE table=default.Table_db7380f4e5484484b1d7dc5c1039137a [id=a0e161a5d9d94a3b936669e6dfe6fb4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.725598 1350 leader_election.cc:290] T 11f4252b5e694312aaa142468f8e68b4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Requested vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:51:53.623622 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11f4252b5e694312aaa142468f8e68b4" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:51:53.726768 920 raft_consensus.cc:2468] T 11f4252b5e694312aaa142468f8e68b4 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 0.
I20251029 02:51:53.635251 1308 raft_consensus.cc:493] T 6c32c6f5e1cc40a0a12688201cac62a7 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:53.727097 1308 raft_consensus.cc:515] T 6c32c6f5e1cc40a0a12688201cac62a7 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:53.727363 1039 tablet_service.cc:1505] Processing CreateTablet for tablet bfec87f690a640c39c58c64e16cc67e7 (DEFAULT_TABLE table=default.Table_b313ce7369b24372b396d8e9a257a369 [id=6506de17cec5443b8a4628bd46a39b42]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.727843 1308 leader_election.cc:290] T 6c32c6f5e1cc40a0a12688201cac62a7 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:51:53.728039 520 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_f44737ca5bae4763878b58e5ab45495a" } new_table_name: "default.Table_9dcb4bbb802b48b198c74e95f343200d" modify_external_catalogs: true
I20251029 02:51:53.651636 919 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e31e061d74146009d24d5cd57487a5e" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:51:53.651866 894 tablet_service.cc:1552] Processing DeleteTablet for tablet f81aecb20d6e498bb9b498698886d65e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:53 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:53.653970 896 tablet_service.cc:1552] Processing DeleteTablet for tablet bf210435010d4dde8903fd116eb468e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:51 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:53.654170 895 tablet_service.cc:1552] Processing DeleteTablet for tablet d7ba1756532c4825a6c7024511dbae51 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:51 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:53.729776 895 tablet_service.cc:1505] Processing CreateTablet for tablet f187dbd8a55b45b292ea61db43caf47d (DEFAULT_TABLE table=default.Table_d6a9ff6b4cea442eb2394340301eac96 [id=3f2d90da0f88434592a21c0687450ded]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.730177 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f187dbd8a55b45b292ea61db43caf47d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.730374 1276 raft_consensus.cc:493] T 7c041f4909274cd5a03b9fd78353faf9 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:53.654173 893 tablet_service.cc:1552] Processing DeleteTablet for tablet cb55b3d5bc63405f96cdc122f5a612cb with delete_type TABLET_DATA_DELETED (replaced by tablet aa776cde2f494fa5810539edf9f289a1) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:53.730921 893 tablet_service.cc:1505] Processing CreateTablet for tablet 4aa78f4ac38e4919864c924e50f1aaf4 (DEFAULT_TABLE table=default.Table_04a6cfdfbf8c4674b30a8c3024e3277c [id=a93c34a9cd344503954bfae8752a69a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.731276 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4aa78f4ac38e4919864c924e50f1aaf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.732151 1308 raft_consensus.cc:493] T deca0c4a114441079fce22e74b403050 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:53.732236 1276 raft_consensus.cc:515] T 7c041f4909274cd5a03b9fd78353faf9 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:53.732318 1308 raft_consensus.cc:515] T deca0c4a114441079fce22e74b403050 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:53.732954 1308 leader_election.cc:290] T deca0c4a114441079fce22e74b403050 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:51:53.655162 891 tablet_service.cc:1552] Processing DeleteTablet for tablet 6c32c6f5e1cc40a0a12688201cac62a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:51 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:53.734371 526 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet dd90fb9eb167478cbc3ec9363af86295 (table default.Table_eb28ff62f787444189cb7dfe9e735992 [id=052abf3dcc904bdcb869d5b885723dab]). Expected version 1 got 0
I20251029 02:51:53.734627 1174 raft_consensus.cc:2272] T 5886b8d39f384426bcbed7289ae72e51 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:53.734889 891 tablet_service.cc:1505] Processing CreateTablet for tablet 519754f5d8f7484480f973c53708c5f3 (DEFAULT_TABLE table=default.Table_96a1c9ac99714785a6bb853741027180 [id=d16cec879a3a45189e45088c4c0913ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.655344 899 tablet_service.cc:1552] Processing DeleteTablet for tablet d32532973b3f46dd8158a2ce42d26365 with delete_type TABLET_DATA_DELETED (replaced by tablet 65fdec7cf5d143208d78d4c3806b1352) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:53.735179 1174 ts_tablet_manager.cc:1916] T 5886b8d39f384426bcbed7289ae72e51 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:53.735406 899 tablet_service.cc:1505] Processing CreateTablet for tablet f60e7c24982d40319801869ad060864d (DEFAULT_TABLE table=default.Table_4350e92f14de4ba8a04afbe12ea7422a [id=226a2d412a7b40e59b6f5340b5a9b739]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.736084 896 tablet_service.cc:1505] Processing CreateTablet for tablet 86fc22bb84794149b598b97df41eb881 (DEFAULT_TABLE table=default.Table_77a5f27bd0ae4eb68a30c883268d2f9c [id=d62f017ad2514f29867c9cde35996faf]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 02:51:53.736384 847 leader_election.cc:343] T 6c32c6f5e1cc40a0a12688201cac62a7 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 6c32c6f5e1cc40a0a12688201cac62a7
I20251029 02:51:53.736584 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c764062d3ba04d4ba8b2e367ad8ed433. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.736694 894 tablet_service.cc:1505] Processing CreateTablet for tablet 56d05e4a500043c995bb51a204cb295d (DEFAULT_TABLE table=default.Table_0c1681c9cd0347f2ba2eab73b8cf11a9 [id=fdb365438df14f12a382aa046aef41a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.737370 1038 tablet_service.cc:1505] Processing CreateTablet for tablet 519754f5d8f7484480f973c53708c5f3 (DEFAULT_TABLE table=default.Table_96a1c9ac99714785a6bb853741027180 [id=d16cec879a3a45189e45088c4c0913ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.737788 1166 ts_tablet_manager.cc:1403] T 61c1fec3ab4c402eb29f767b9bb7409f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.096s user 0.002s sys 0.004s
I20251029 02:51:53.657164 898 tablet_service.cc:1505] Processing CreateTablet for tablet f246f6c202cf44d48c9f5e4f335202c5 (DEFAULT_TABLE table=default.Table_e4de02c7c1f04704bc39e1f0efd2c9d9 [id=40738627473f4a55adc5ed250af29d6c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.738389 742 tablet_service.cc:1505] Processing CreateTablet for tablet f187dbd8a55b45b292ea61db43caf47d (DEFAULT_TABLE table=default.Table_d6a9ff6b4cea442eb2394340301eac96 [id=3f2d90da0f88434592a21c0687450ded]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.657963 890 tablet_service.cc:1505] Processing CreateTablet for tablet 74873a05ef634867a30aa63b31fcfcc7 (DEFAULT_TABLE table=default.Table_4e97f5e298434c16913bf7ca93ca30f5 [id=48dfcbe8b3654d72b51b77473dd918f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.658671 892 tablet_service.cc:1505] Processing CreateTablet for tablet 6ca9bd37de9e4be388501b15af4bda31 (DEFAULT_TABLE table=default.Table_42bbf44edb134c16b76604d387907461 [id=a862192912a04942a8f682fc9825b014]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.660164 888 tablet_service.cc:1505] Processing CreateTablet for tablet 6bd3fac2502e479aa9f4ceaabbb39504 (DEFAULT_TABLE table=default.Table_febb0d5740c847ce8aafa84d39ccbe91 [id=3513c69f6b3941849ac5b8ae0a5846d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.668165 900 tablet_service.cc:1552] Processing DeleteTablet for tablet deca0c4a114441079fce22e74b403050 with delete_type TABLET_DATA_DELETED (replaced by tablet e19c8b10f32849f5885fc00176c8bc12) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:53.739380 900 tablet_service.cc:1505] Processing CreateTablet for tablet d5741dde463a43d6a776ab92dcfb04bf (DEFAULT_TABLE table=default.Table_0fc48a708ecc446990ad491afab2efdd [id=08e3b2ebcf37415d9a2886ad5bbda068]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.739574 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 519754f5d8f7484480f973c53708c5f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.673727 887 tablet_service.cc:1505] Processing CreateTablet for tablet 0cc1957a845f4e9cb703a553b27d1347 (DEFAULT_TABLE table=default.Table_262db98ea44648baab95fc2b3081dc4e [id=843f681b71c64427b49becbabcad9451]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.740054 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74873a05ef634867a30aa63b31fcfcc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.674201 886 tablet_service.cc:1505] Processing CreateTablet for tablet 71068739b4384badbf305313d902f0f5 (DEFAULT_TABLE table=default.Table_4e899dcc04fe4cb1b96ced1fa1b03585 [id=992716bbc1694bd6b85e4c8a3ab51e40]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.674643 885 tablet_service.cc:1505] Processing CreateTablet for tablet da2a21c3236a4bd99872b89b12a36508 (DEFAULT_TABLE table=default.Table_b8bb03c0242d4262b78640fa8c88dc27 [id=fb42af08ec1f48498ee5e91cce62333e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.740469 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6bd3fac2502e479aa9f4ceaabbb39504. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.675088 884 tablet_service.cc:1505] Processing CreateTablet for tablet bfec87f690a640c39c58c64e16cc67e7 (DEFAULT_TABLE table=default.Table_b313ce7369b24372b396d8e9a257a369 [id=6506de17cec5443b8a4628bd46a39b42]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.675689 883 tablet_service.cc:1505] Processing CreateTablet for tablet 9e6eb099648549dbbf9969e59865f51f (DEFAULT_TABLE table=default.Table_218b1046ed7c4b0d8b9c55a9542a6a2e [id=0c5d3e4d34d045dc99ffc88e101cae1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.676134 918 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e31e061d74146009d24d5cd57487a5e" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
I20251029 02:51:53.741408 1042 tablet_service.cc:1505] Processing CreateTablet for tablet f187dbd8a55b45b292ea61db43caf47d (DEFAULT_TABLE table=default.Table_d6a9ff6b4cea442eb2394340301eac96 [id=3f2d90da0f88434592a21c0687450ded]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 02:51:53.741637 520 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet faa835de84ff45b19525d1d603678987 consensus state has no leader
I20251029 02:51:53.679208 882 tablet_service.cc:1505] Processing CreateTablet for tablet e1fa4980f2624321b617b84c68b90113 (DEFAULT_TABLE table=default.Table_db7380f4e5484484b1d7dc5c1039137a [id=a0e161a5d9d94a3b936669e6dfe6fb4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.679994 881 tablet_service.cc:1505] Processing CreateTablet for tablet dd6ad641fbda472fab07aa35c5e4523a (DEFAULT_TABLE table=default.Table_e4bba3a0c06a4ff5bd8829e1081b9c68 [id=1d39d6caa9e24fe38b5cf9a20a6c66b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.680814 879 tablet_service.cc:1505] Processing CreateTablet for tablet daa05a34c0684c07a35966290fd7d44c (DEFAULT_TABLE table=default.Table_33eacbf00740465185d70b46a3782893 [id=22e43c4483ae488b81fb94a7d221bdce]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.681591 878 tablet_service.cc:1505] Processing CreateTablet for tablet c764062d3ba04d4ba8b2e367ad8ed433 (DEFAULT_TABLE table=default.Table_bce83bae13a94cd5bf6733b53e6a1e9e [id=761ebdd0095c414488aa221cbc99d3bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.701254 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b76c4408a7340e59b8506171622175f" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:51:53.715348 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "11f4252b5e694312aaa142468f8e68b4" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
I20251029 02:51:53.742204 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f60e7c24982d40319801869ad060864d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.742175 1030 tablet_service.cc:1505] Processing CreateTablet for tablet f60e7c24982d40319801869ad060864d (DEFAULT_TABLE table=default.Table_4350e92f14de4ba8a04afbe12ea7422a [id=226a2d412a7b40e59b6f5340b5a9b739]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.744522 1036 tablet_service.cc:1505] Processing CreateTablet for tablet 4aa78f4ac38e4919864c924e50f1aaf4 (DEFAULT_TABLE table=default.Table_04a6cfdfbf8c4674b30a8c3024e3277c [id=a93c34a9cd344503954bfae8752a69a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.745628 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5741dde463a43d6a776ab92dcfb04bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.745630 916 raft_consensus.cc:3060] T 11f4252b5e694312aaa142468f8e68b4 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
W20251029 02:51:53.746944 990 leader_election.cc:343] T 5e31e061d74146009d24d5cd57487a5e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
W20251029 02:51:53.747063 847 leader_election.cc:343] T deca0c4a114441079fce22e74b403050 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: deca0c4a114441079fce22e74b403050
W20251029 02:51:53.747344 990 leader_election.cc:343] T 5e31e061d74146009d24d5cd57487a5e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:53.747499 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ca9bd37de9e4be388501b15af4bda31. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.747924 1276 leader_election.cc:290] T 7c041f4909274cd5a03b9fd78353faf9 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:51:53.749049 1062 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6c32c6f5e1cc40a0a12688201cac62a7" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:51:53.744401 1035 tablet_service.cc:1505] Processing CreateTablet for tablet daa05a34c0684c07a35966290fd7d44c (DEFAULT_TABLE table=default.Table_33eacbf00740465185d70b46a3782893 [id=22e43c4483ae488b81fb94a7d221bdce]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.749737 1034 tablet_service.cc:1505] Processing CreateTablet for tablet 56d05e4a500043c995bb51a204cb295d (DEFAULT_TABLE table=default.Table_0c1681c9cd0347f2ba2eab73b8cf11a9 [id=fdb365438df14f12a382aa046aef41a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.750308 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1fa4980f2624321b617b84c68b90113. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.750535 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet daa05a34c0684c07a35966290fd7d44c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.749976 1061 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "deca0c4a114441079fce22e74b403050" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:51:53.754679 849 leader_election.cc:343] T deca0c4a114441079fce22e74b403050 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: Tablet not RUNNING: STOPPED
I20251029 02:51:53.754833 849 leader_election.cc:304] T deca0c4a114441079fce22e74b403050 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9
I20251029 02:51:53.755564 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bfec87f690a640c39c58c64e16cc67e7. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:51:53.757419 1390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251029 02:51:53.757966 1308 raft_consensus.cc:2749] T deca0c4a114441079fce22e74b403050 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:51:53.763324 1276 raft_consensus.cc:493] T 70e8cfe5eaa9492c85874dc3214a1e0b P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:53.763782 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f187dbd8a55b45b292ea61db43caf47d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.765909 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71068739b4384badbf305313d902f0f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.768918 524 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_0c1681c9cd0347f2ba2eab73b8cf11a9" } modify_external_catalogs: true
I20251029 02:51:53.769879 741 tablet_service.cc:1505] Processing CreateTablet for tablet 4aa78f4ac38e4919864c924e50f1aaf4 (DEFAULT_TABLE table=default.Table_04a6cfdfbf8c4674b30a8c3024e3277c [id=a93c34a9cd344503954bfae8752a69a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.769459 524 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_0c1681c9cd0347f2ba2eab73b8cf11a9" } modify_external_catalogs: true
I20251029 02:51:53.770725 1351 raft_consensus.cc:493] T b867feae24e741aca6902868c35c1428 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:53.771644 738 tablet_service.cc:1505] Processing CreateTablet for tablet 86fc22bb84794149b598b97df41eb881 (DEFAULT_TABLE table=default.Table_77a5f27bd0ae4eb68a30c883268d2f9c [id=d62f017ad2514f29867c9cde35996faf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.761173 1041 tablet_service.cc:1505] Processing CreateTablet for tablet c764062d3ba04d4ba8b2e367ad8ed433 (DEFAULT_TABLE table=default.Table_bce83bae13a94cd5bf6733b53e6a1e9e [id=761ebdd0095c414488aa221cbc99d3bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.774834 1062 raft_consensus.cc:2468] T 6c32c6f5e1cc40a0a12688201cac62a7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:51:53.775576 849 leader_election.cc:304] T 6c32c6f5e1cc40a0a12688201cac62a7 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters: b495a49349324f7d81908f2da700f0d9
I20251029 02:51:53.776067 1308 raft_consensus.cc:2804] T 6c32c6f5e1cc40a0a12688201cac62a7 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:51:53.776186 1308 raft_consensus.cc:493] T 6c32c6f5e1cc40a0a12688201cac62a7 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:51:53.776290 1308 raft_consensus.cc:3060] T 6c32c6f5e1cc40a0a12688201cac62a7 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:51:53.777228 520 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_262db98ea44648baab95fc2b3081dc4e" } modify_external_catalogs: true
I20251029 02:51:53.777448 520 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_262db98ea44648baab95fc2b3081dc4e" } modify_external_catalogs: true
I20251029 02:51:53.777750 739 tablet_service.cc:1505] Processing CreateTablet for tablet f60e7c24982d40319801869ad060864d (DEFAULT_TABLE table=default.Table_4350e92f14de4ba8a04afbe12ea7422a [id=226a2d412a7b40e59b6f5340b5a9b739]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.778064 1037 tablet_service.cc:1505] Processing CreateTablet for tablet 9e6eb099648549dbbf9969e59865f51f (DEFAULT_TABLE table=default.Table_218b1046ed7c4b0d8b9c55a9542a6a2e [id=0c5d3e4d34d045dc99ffc88e101cae1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.778097 1025 tablet_service.cc:1505] Processing CreateTablet for tablet c9bd3f2deb5d4af09b7918269a98509f (DEFAULT_TABLE table=default.Table_47a1a59db0444fa49f7274509385ff0c [id=283c09ba60114ba9b165e2ff144a6c35]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.778370 1027 tablet_service.cc:1552] Processing DeleteTablet for tablet b2b2d64dcf964da8be4b544dbbe31977 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:53 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:53.773828 1276 raft_consensus.cc:515] T 70e8cfe5eaa9492c85874dc3214a1e0b P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:53.781203 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56d05e4a500043c995bb51a204cb295d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.781654 1028 tablet_service.cc:1505] Processing CreateTablet for tablet 02e4588b03144ea2850e13a1f8f29f06 (DEFAULT_TABLE table=default.Table_c90ddb84f3e044ba96f01a51fd463d6e [id=fd6d31d379394857a35422211af00642]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.781893 1026 tablet_service.cc:1505] Processing CreateTablet for tablet cc74e83833754f0ca19046031a24ebb9 (DEFAULT_TABLE table=default.Table_528aa3f8d5e84754bc56f86ca7a480aa [id=57471769276f44299eae50b9d854b50c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.782121 1024 tablet_service.cc:1505] Processing CreateTablet for tablet 4add552cf4d940d2a0a36d6b58379a34 (DEFAULT_TABLE table=default.Table_c73a6bc56eb44e08aa7d2ec65ad88949 [id=903fed784c0243bbb60a39066f2dbc71]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.782344 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b76c4408a7340e59b8506171622175f" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:51:53.782825 1059 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7c041f4909274cd5a03b9fd78353faf9" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:51:53.782909 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_bdefc7775234406788f3f7e220769b43"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:53.783205 1029 tablet_service.cc:1505] Processing CreateTablet for tablet d5741dde463a43d6a776ab92dcfb04bf (DEFAULT_TABLE table=default.Table_0fc48a708ecc446990ad491afab2efdd [id=08e3b2ebcf37415d9a2886ad5bbda068]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.783485 1350 raft_consensus.cc:493] T 41a6139fd9b84707a8c7eebc0381e383 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251029 02:51:53.783561 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bdefc7775234406788f3f7e220769b43 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:53.783629 1350 raft_consensus.cc:515] T 41a6139fd9b84707a8c7eebc0381e383 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:53.778568 740 tablet_service.cc:1505] Processing CreateTablet for tablet 519754f5d8f7484480f973c53708c5f3 (DEFAULT_TABLE table=default.Table_96a1c9ac99714785a6bb853741027180 [id=d16cec879a3a45189e45088c4c0913ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.784324 1350 leader_election.cc:290] T 41a6139fd9b84707a8c7eebc0381e383 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:51:53.785633 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "41a6139fd9b84707a8c7eebc0381e383" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:51:53.781255 1166 raft_consensus.cc:359] T 61c1fec3ab4c402eb29f767b9bb7409f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:53.787055 522 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_c73a6bc56eb44e08aa7d2ec65ad88949" } modify_external_catalogs: true
I20251029 02:51:53.787257 522 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_c73a6bc56eb44e08aa7d2ec65ad88949" } modify_external_catalogs: true
I20251029 02:51:53.787607 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4aa78f4ac38e4919864c924e50f1aaf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.787853 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56d05e4a500043c995bb51a204cb295d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.789466 1167 tablet_bootstrap.cc:492] T b867feae24e741aca6902868c35c1428 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:51:53.791811 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4aa78f4ac38e4919864c924e50f1aaf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.791851 1351 raft_consensus.cc:515] T b867feae24e741aca6902868c35c1428 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:53.792091 523 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_b8bb03c0242d4262b78640fa8c88dc27" } new_table_name: "default.Table_baebbc03e01e4f809f9983e392d63ba6" modify_external_catalogs: true
I20251029 02:51:53.793368 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 519754f5d8f7484480f973c53708c5f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.794736 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f60e7c24982d40319801869ad060864d. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:51:53.796289 990 leader_election.cc:343] T 41a6139fd9b84707a8c7eebc0381e383 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:53.796363 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86fc22bb84794149b598b97df41eb881. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.796825 1166 raft_consensus.cc:385] T 61c1fec3ab4c402eb29f767b9bb7409f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:53.796931 1166 raft_consensus.cc:740] T 61c1fec3ab4c402eb29f767b9bb7409f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:51:53.797207 1166 consensus_queue.cc:260] T 61c1fec3ab4c402eb29f767b9bb7409f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:53.797827 1166 ts_tablet_manager.cc:1434] T 61c1fec3ab4c402eb29f767b9bb7409f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.060s user 0.002s sys 0.000s
I20251029 02:51:53.799430 1031 tablet_service.cc:1505] Processing CreateTablet for tablet 86fc22bb84794149b598b97df41eb881 (DEFAULT_TABLE table=default.Table_77a5f27bd0ae4eb68a30c883268d2f9c [id=d62f017ad2514f29867c9cde35996faf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.800556 989 leader_election.cc:304] T 11f4252b5e694312aaa142468f8e68b4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters:
I20251029 02:51:53.800839 1251 raft_consensus.cc:2804] T 11f4252b5e694312aaa142468f8e68b4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:51:53.802346 737 tablet_service.cc:1505] Processing CreateTablet for tablet d5741dde463a43d6a776ab92dcfb04bf (DEFAULT_TABLE table=default.Table_0fc48a708ecc446990ad491afab2efdd [id=08e3b2ebcf37415d9a2886ad5bbda068]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.805732 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b867feae24e741aca6902868c35c1428" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:51:53.805934 1167 ts_tablet_manager.cc:1403] T b867feae24e741aca6902868c35c1428 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.227s user 0.001s sys 0.004s
W20251029 02:51:53.807978 523 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet da2a21c3236a4bd99872b89b12a36508 consensus state has no leader
I20251029 02:51:53.808382 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_5c973aa3bd7f4a9ebf8435ed761552a5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:53.808950 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5c973aa3bd7f4a9ebf8435ed761552a5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:53.806377 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56d05e4a500043c995bb51a204cb295d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.810891 1307 raft_consensus.cc:493] T d7ba1756532c4825a6c7024511dbae51 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:53.811070 1307 raft_consensus.cc:515] T d7ba1756532c4825a6c7024511dbae51 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:53.811800 1307 leader_election.cc:290] T d7ba1756532c4825a6c7024511dbae51 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:51:53.804579 1276 leader_election.cc:290] T 70e8cfe5eaa9492c85874dc3214a1e0b P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:51:53.817505 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_d8875cc2c2f14a298da5a7c3c7678354"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:53.788643 1023 tablet_service.cc:1505] Processing CreateTablet for tablet 1bc45078a6a942c38ff3350c15885eff (DEFAULT_TABLE table=default.Table_3126aa57075d44dc8b73f08fe21aabdf [id=e8fa170a8a044f2fa1236949a91454de]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.818118 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9bd3f2deb5d4af09b7918269a98509f. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:51:53.818135 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d8875cc2c2f14a298da5a7c3c7678354 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:53.818338 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bfec87f690a640c39c58c64e16cc67e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.819181 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e6eb099648549dbbf9969e59865f51f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.819618 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86fc22bb84794149b598b97df41eb881. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.820063 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd6ad641fbda472fab07aa35c5e4523a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.820089 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_9efddcb184d74a75bd0d1b72cc522e3f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:53.820313 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da2a21c3236a4bd99872b89b12a36508. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.820592 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1fa4980f2624321b617b84c68b90113. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:51:53.820622 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9efddcb184d74a75bd0d1b72cc522e3f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:53.820996 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c764062d3ba04d4ba8b2e367ad8ed433. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.821520 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet daa05a34c0684c07a35966290fd7d44c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.821870 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f246f6c202cf44d48c9f5e4f335202c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.822256 519 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_baebbc03e01e4f809f9983e392d63ba6" } new_table_name: "default.Table_86c59704aa154b37b528fd68c2359e0f" modify_external_catalogs: true
I20251029 02:51:53.822860 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0cc1957a845f4e9cb703a553b27d1347. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:51:53.826440 847 leader_election.cc:343] T d7ba1756532c4825a6c7024511dbae51 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: d7ba1756532c4825a6c7024511dbae51
I20251029 02:51:53.826970 1350 raft_consensus.cc:493] T 70e8cfe5eaa9492c85874dc3214a1e0b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:53.827667 918 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "70e8cfe5eaa9492c85874dc3214a1e0b" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:51:53.828361 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7c041f4909274cd5a03b9fd78353faf9" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:51:53.832530 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5741dde463a43d6a776ab92dcfb04bf. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:51:53.834318 711 leader_election.cc:343] T 4b76c4408a7340e59b8506171622175f P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:53.835434 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_f65782ffe391498286f93d5e804bf177"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:53.835959 1172 tablet_bootstrap.cc:654] T 6f93812578a940a3b7378015e2a46f95 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
W20251029 02:51:53.836082 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f65782ffe391498286f93d5e804bf177 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:51:53.837189 711 leader_election.cc:343] T 70e8cfe5eaa9492c85874dc3214a1e0b P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
W20251029 02:51:53.837405 711 leader_election.cc:343] T 7c041f4909274cd5a03b9fd78353faf9 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:53.838474 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_b8ddec1d965444948171ef0cca0ff011"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:53.838984 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b8ddec1d965444948171ef0cca0ff011 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:53.843737 524 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 56d05e4a500043c995bb51a204cb295d
I20251029 02:51:53.844027 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 519754f5d8f7484480f973c53708c5f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.844887 1351 leader_election.cc:290] T b867feae24e741aca6902868c35c1428 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:51:53.849848 916 raft_consensus.cc:2468] T 11f4252b5e694312aaa142468f8e68b4 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 1.
I20251029 02:51:53.851627 900 tablet_service.cc:1552] Processing DeleteTablet for tablet b2b2d64dcf964da8be4b544dbbe31977 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:53 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:53.852938 756 tablet_service.cc:1505] Processing CreateTablet for tablet 02e4588b03144ea2850e13a1f8f29f06 (DEFAULT_TABLE table=default.Table_c90ddb84f3e044ba96f01a51fd463d6e [id=fd6d31d379394857a35422211af00642]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.853507 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02e4588b03144ea2850e13a1f8f29f06. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.854212 891 tablet_service.cc:1505] Processing CreateTablet for tablet cc74e83833754f0ca19046031a24ebb9 (DEFAULT_TABLE table=default.Table_528aa3f8d5e84754bc56f86ca7a480aa [id=57471769276f44299eae50b9d854b50c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.854662 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc74e83833754f0ca19046031a24ebb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.855293 524 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_96a1c9ac99714785a6bb853741027180" } modify_external_catalogs: true
I20251029 02:51:53.855489 524 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_96a1c9ac99714785a6bb853741027180" } modify_external_catalogs: true
I20251029 02:51:53.857060 745 tablet_service.cc:1552] Processing DeleteTablet for tablet b2b2d64dcf964da8be4b544dbbe31977 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:53 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:53.857633 745 tablet_service.cc:1505] Processing CreateTablet for tablet cc74e83833754f0ca19046031a24ebb9 (DEFAULT_TABLE table=default.Table_528aa3f8d5e84754bc56f86ca7a480aa [id=57471769276f44299eae50b9d854b50c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.857790 899 tablet_service.cc:1505] Processing CreateTablet for tablet 4add552cf4d940d2a0a36d6b58379a34 (DEFAULT_TABLE table=default.Table_c73a6bc56eb44e08aa7d2ec65ad88949 [id=903fed784c0243bbb60a39066f2dbc71]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.858026 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc74e83833754f0ca19046031a24ebb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.855325 1167 raft_consensus.cc:359] T b867feae24e741aca6902868c35c1428 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:53.858754 525 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_7df0e7c7f61d486891adb26e68972a1c" } new_table_name: "default.Table_3e28d37b4b6146bf95edf245d3f9d0cf" modify_external_catalogs: true
I20251029 02:51:53.861328 890 tablet_service.cc:1505] Processing CreateTablet for tablet 02e4588b03144ea2850e13a1f8f29f06 (DEFAULT_TABLE table=default.Table_c90ddb84f3e044ba96f01a51fd463d6e [id=fd6d31d379394857a35422211af00642]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.865002 748 tablet_service.cc:1505] Processing CreateTablet for tablet c9bd3f2deb5d4af09b7918269a98509f (DEFAULT_TABLE table=default.Table_47a1a59db0444fa49f7274509385ff0c [id=283c09ba60114ba9b165e2ff144a6c35]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.865092 1167 raft_consensus.cc:385] T b867feae24e741aca6902868c35c1428 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:53.865388 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9bd3f2deb5d4af09b7918269a98509f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.865419 1167 raft_consensus.cc:740] T b867feae24e741aca6902868c35c1428 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:51:53.868084 747 tablet_service.cc:1505] Processing CreateTablet for tablet 4add552cf4d940d2a0a36d6b58379a34 (DEFAULT_TABLE table=default.Table_c73a6bc56eb44e08aa7d2ec65ad88949 [id=903fed784c0243bbb60a39066f2dbc71]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.868528 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4add552cf4d940d2a0a36d6b58379a34. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:51:53.870273 990 leader_election.cc:343] T b867feae24e741aca6902868c35c1428 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:53.870739 892 tablet_service.cc:1505] Processing CreateTablet for tablet 1bc45078a6a942c38ff3350c15885eff (DEFAULT_TABLE table=default.Table_3126aa57075d44dc8b73f08fe21aabdf [id=e8fa170a8a044f2fa1236949a91454de]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.870915 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4add552cf4d940d2a0a36d6b58379a34. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.844796 1027 tablet_service.cc:1552] Processing DeleteTablet for tablet 61c1fec3ab4c402eb29f767b9bb7409f with delete_type TABLET_DATA_DELETED (replaced by tablet a5c3a15510cb4321bbb80d61fabd0c0a) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:53.874334 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f60e7c24982d40319801869ad060864d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.875718 517 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_f178e528491d416a99b0191e864f53d8" } new_table_name: "default.Table_e8685346a99d41078ecd38cd4ce4fec5" modify_external_catalogs: true
W20251029 02:51:53.876895 990 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:41701, user_credentials={real_user=slave}} blocked reactor thread for 54851.8us
I20251029 02:51:53.880575 1350 raft_consensus.cc:515] T 70e8cfe5eaa9492c85874dc3214a1e0b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:53.881170 1059 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d7ba1756532c4825a6c7024511dbae51" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:51:53.882695 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_d37ad2580f614a6e9c469258b145525b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:53.883282 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d37ad2580f614a6e9c469258b145525b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:51:53.884150 519 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet da2a21c3236a4bd99872b89b12a36508 consensus state has no leader
I20251029 02:51:53.884598 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_ba5ed91035ac405694232f7b7e997b59"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:53.885205 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ba5ed91035ac405694232f7b7e997b59 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:53.886668 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_5902a11255a049be9dd07acf8b50adda"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:53.886966 1394 raft_consensus.cc:493] T dd90fb9eb167478cbc3ec9363af86295 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251029 02:51:53.887687 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5902a11255a049be9dd07acf8b50adda in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:53.887140 1394 raft_consensus.cc:515] T dd90fb9eb167478cbc3ec9363af86295 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:53.890697 1394 leader_election.cc:290] T dd90fb9eb167478cbc3ec9363af86295 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
W20251029 02:51:53.894398 525 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 44784a3a760946e0be24743bc2fc8138 consensus state has no leader
W20251029 02:51:53.895548 712 leader_election.cc:343] T 4b76c4408a7340e59b8506171622175f P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:53.895701 712 leader_election.cc:304] T 4b76c4408a7340e59b8506171622175f P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:51:53.895983 1394 raft_consensus.cc:2749] T 4b76c4408a7340e59b8506171622175f P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 02:51:53.896441 712 leader_election.cc:343] T 7c041f4909274cd5a03b9fd78353faf9 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:53.896562 712 leader_election.cc:304] T 7c041f4909274cd5a03b9fd78353faf9 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:51:53.897505 1350 leader_election.cc:290] T 70e8cfe5eaa9492c85874dc3214a1e0b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:51:53.897925 744 tablet_service.cc:1505] Processing CreateTablet for tablet 1bc45078a6a942c38ff3350c15885eff (DEFAULT_TABLE table=default.Table_3126aa57075d44dc8b73f08fe21aabdf [id=e8fa170a8a044f2fa1236949a91454de]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.903043 742 tablet_service.cc:1552] Processing DeleteTablet for tablet 56d05e4a500043c995bb51a204cb295d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:53 UTC) from {username='slave'} at 127.0.0.1:44580
W20251029 02:51:53.903956 513 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 69597d48631d43b6a5b92abbf60d5ed3 (table default.Table_f2172770f9a94bb98d7813da6e1848fe [id=ffce392867b14b6ea9db44f4bdee57c3], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 69597d48631d43b6a5b92abbf60d5ed3 consensus state has no leader
W20251029 02:51:53.907810 517 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 576c4e6e70e24b0399ac9597a30f0569 consensus state has no leader
I20251029 02:51:53.908874 1027 tablet_service.cc:1552] Processing DeleteTablet for tablet 56d05e4a500043c995bb51a204cb295d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:53 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:53.909847 1394 raft_consensus.cc:2749] T 7c041f4909274cd5a03b9fd78353faf9 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:51:53.910792 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "70e8cfe5eaa9492c85874dc3214a1e0b" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:51:53.915437 1059 raft_consensus.cc:2468] T d7ba1756532c4825a6c7024511dbae51 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
W20251029 02:51:53.905665 1397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251029 02:51:53.916484 1251 raft_consensus.cc:697] T 11f4252b5e694312aaa142468f8e68b4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Becoming Leader. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Running, Role: LEADER
I20251029 02:51:53.916668 1027 tablet_service.cc:1505] Processing CreateTablet for tablet b3de6670357c4ae59eb77469b2fb434c (DEFAULT_TABLE table=default.Table_f914dd26d0b846f8923187c2c8edf27e [id=f9c7128476404646a9194e31f49c94e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.916782 1251 consensus_queue.cc:237] T 11f4252b5e694312aaa142468f8e68b4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:53.919173 1166 tablet_bootstrap.cc:492] T 4b76c4408a7340e59b8506171622175f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:51:53.919953 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f187dbd8a55b45b292ea61db43caf47d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.921581 1060 raft_consensus.cc:2468] T 70e8cfe5eaa9492c85874dc3214a1e0b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b495a49349324f7d81908f2da700f0d9 in term 0.
I20251029 02:51:53.921730 520 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 0cc1957a845f4e9cb703a553b27d1347
I20251029 02:51:53.927062 520 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_4350e92f14de4ba8a04afbe12ea7422a" } modify_external_catalogs: true
I20251029 02:51:53.927276 520 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_4350e92f14de4ba8a04afbe12ea7422a" } modify_external_catalogs: true
I20251029 02:51:53.927709 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02e4588b03144ea2850e13a1f8f29f06. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.929672 524 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 519754f5d8f7484480f973c53708c5f3
I20251029 02:51:53.929700 522 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 4add552cf4d940d2a0a36d6b58379a34
I20251029 02:51:53.935315 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5741dde463a43d6a776ab92dcfb04bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.939354 1040 tablet_service.cc:1552] Processing DeleteTablet for tablet 519754f5d8f7484480f973c53708c5f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:53 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:53.939675 1034 tablet_service.cc:1552] Processing DeleteTablet for tablet 4add552cf4d940d2a0a36d6b58379a34 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:53 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:53.942797 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e31e061d74146009d24d5cd57487a5e" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:51:53.943651 524 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_db7380f4e5484484b1d7dc5c1039137a" } new_table_name: "default.Table_8a5ae35851664e0f85324ec1ca672031" modify_external_catalogs: true
I20251029 02:51:53.945056 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bc45078a6a942c38ff3350c15885eff. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.948177 520 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet f60e7c24982d40319801869ad060864d
I20251029 02:51:53.954319 742 tablet_service.cc:1552] Processing DeleteTablet for tablet 4add552cf4d940d2a0a36d6b58379a34 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:53 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:53.957880 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc74e83833754f0ca19046031a24ebb9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.959267 750 tablet_service.cc:1552] Processing DeleteTablet for tablet 0cc1957a845f4e9cb703a553b27d1347 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:53 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:53.959347 849 leader_election.cc:304] T d7ba1756532c4825a6c7024511dbae51 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters: b495a49349324f7d81908f2da700f0d9
I20251029 02:51:53.959669 751 tablet_service.cc:1505] Processing CreateTablet for tablet b3de6670357c4ae59eb77469b2fb434c (DEFAULT_TABLE table=default.Table_f914dd26d0b846f8923187c2c8edf27e [id=f9c7128476404646a9194e31f49c94e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.959865 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_3b6a3dbb404a485dac0a7f73fc3685a8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:53.960592 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3b6a3dbb404a485dac0a7f73fc3685a8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:53.962635 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "41a6139fd9b84707a8c7eebc0381e383" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:51:53.962852 775 raft_consensus.cc:2468] T 41a6139fd9b84707a8c7eebc0381e383 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 0.
I20251029 02:51:53.965174 746 tablet_service.cc:1552] Processing DeleteTablet for tablet f60e7c24982d40319801869ad060864d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:53 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:53.965492 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3de6670357c4ae59eb77469b2fb434c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.968042 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd90fb9eb167478cbc3ec9363af86295" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:51:53.968503 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4add552cf4d940d2a0a36d6b58379a34. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:51:53.969203 989 leader_election.cc:343] T b867feae24e741aca6902868c35c1428 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: b867feae24e741aca6902868c35c1428
I20251029 02:51:53.969411 1034 tablet_service.cc:1552] Processing DeleteTablet for tablet f60e7c24982d40319801869ad060864d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:53 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:53.969496 776 raft_consensus.cc:3060] T 5e31e061d74146009d24d5cd57487a5e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:51:53.969679 1032 tablet_service.cc:1552] Processing DeleteTablet for tablet 0cc1957a845f4e9cb703a553b27d1347 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:53 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:53.971366 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e6eb099648549dbbf9969e59865f51f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.971798 1166 tablet_bootstrap.cc:654] T 4b76c4408a7340e59b8506171622175f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:53.964915 755 tablet_service.cc:1552] Processing DeleteTablet for tablet 519754f5d8f7484480f973c53708c5f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:53 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:53.973032 900 tablet_service.cc:1505] Processing CreateTablet for tablet c9bd3f2deb5d4af09b7918269a98509f (DEFAULT_TABLE table=default.Table_47a1a59db0444fa49f7274509385ff0c [id=283c09ba60114ba9b165e2ff144a6c35]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:53.972905 1167 consensus_queue.cc:260] T b867feae24e741aca6902868c35c1428 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:53.974700 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "70e8cfe5eaa9492c85874dc3214a1e0b" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:51:53.979635 989 leader_election.cc:304] T b867feae24e741aca6902868c35c1428 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5; no voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d
I20251029 02:51:53.980229 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3de6670357c4ae59eb77469b2fb434c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.980587 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bc45078a6a942c38ff3350c15885eff. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.981407 1350 raft_consensus.cc:2749] T b867feae24e741aca6902868c35c1428 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:51:53.976431 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "70e8cfe5eaa9492c85874dc3214a1e0b" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:51:53.981588 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c764062d3ba04d4ba8b2e367ad8ed433. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.981640 774 raft_consensus.cc:2468] T 70e8cfe5eaa9492c85874dc3214a1e0b P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 0.
I20251029 02:51:53.984412 776 raft_consensus.cc:2468] T 5e31e061d74146009d24d5cd57487a5e P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 1.
I20251029 02:51:53.985217 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86fc22bb84794149b598b97df41eb881. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:53.994030 1176 ts_tablet_manager.cc:1929] T deca0c4a114441079fce22e74b403050 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:53.994217 1176 log.cc:1199] T deca0c4a114441079fce22e74b403050 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/deca0c4a114441079fce22e74b403050
I20251029 02:51:53.998731 1176 ts_tablet_manager.cc:1950] T deca0c4a114441079fce22e74b403050 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:51:53.999468 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet deca0c4a114441079fce22e74b403050 (table default.Table_1bd19c9d52204fbf87ee3cdbc62984af [id=144a24e58314435d85f9c759ef79fa60]) successfully deleted
I20251029 02:51:54.000595 1307 raft_consensus.cc:2804] T d7ba1756532c4825a6c7024511dbae51 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:51:54.008633 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_fb891e36b29c4a3ab6f1eef96dde937d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:54.009344 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fb891e36b29c4a3ab6f1eef96dde937d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:54.011477 526 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_42a71464d71d487baa7245b5c3bdbbcd" } new_table_name: "default.Table_ea7ce466c0ea44a8ac497c7dd4ead10f" modify_external_catalogs: true
W20251029 02:51:54.012243 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet f81aecb20d6e498bb9b498698886d65e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f81aecb20d6e498bb9b498698886d65e already in progress: creating tablet
I20251029 02:51:54.013048 1174 ts_tablet_manager.cc:1929] T 5886b8d39f384426bcbed7289ae72e51 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:54.013159 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 02e4588b03144ea2850e13a1f8f29f06. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:51:54.015648 849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:45003, user_credentials={real_user=slave}} blocked reactor thread for 43837.9us
I20251029 02:51:54.015985 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd90fb9eb167478cbc3ec9363af86295" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:51:54.016316 1307 raft_consensus.cc:493] T d7ba1756532c4825a6c7024511dbae51 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:51:54.016445 1307 raft_consensus.cc:3060] T d7ba1756532c4825a6c7024511dbae51 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:51:54.016481 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bc45078a6a942c38ff3350c15885eff. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:54.017123 1175 ts_tablet_manager.cc:1929] T 64b0fb400e944ce2b2c7a49d12135e9c P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:54.017244 1175 log.cc:1199] T 64b0fb400e944ce2b2c7a49d12135e9c P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/64b0fb400e944ce2b2c7a49d12135e9c
I20251029 02:51:54.017616 1175 ts_tablet_manager.cc:1950] T 64b0fb400e944ce2b2c7a49d12135e9c P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
W20251029 02:51:54.017663 711 leader_election.cc:343] T dd90fb9eb167478cbc3ec9363af86295 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:54.017760 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9bd3f2deb5d4af09b7918269a98509f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:54.021513 1174 log.cc:1199] T 5886b8d39f384426bcbed7289ae72e51 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/5886b8d39f384426bcbed7289ae72e51
I20251029 02:51:54.022728 712 leader_election.cc:304] T 70e8cfe5eaa9492c85874dc3214a1e0b P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters: c756528642a543eeacd93cef3e335a7d
I20251029 02:51:54.023607 1172 tablet_bootstrap.cc:492] T 6f93812578a940a3b7378015e2a46f95 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:51:54.023824 1172 ts_tablet_manager.cc:1403] T 6f93812578a940a3b7378015e2a46f95 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.322s user 0.029s sys 0.031s
I20251029 02:51:54.025055 1394 raft_consensus.cc:2804] T 70e8cfe5eaa9492c85874dc3214a1e0b P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:51:54.025170 1394 raft_consensus.cc:493] T 70e8cfe5eaa9492c85874dc3214a1e0b P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:51:54.025265 1394 raft_consensus.cc:3060] T 70e8cfe5eaa9492c85874dc3214a1e0b P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
W20251029 02:51:54.025465 712 leader_election.cc:343] T dd90fb9eb167478cbc3ec9363af86295 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:54.025676 712 leader_election.cc:304] T dd90fb9eb167478cbc3ec9363af86295 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:51:54.025990 1276 raft_consensus.cc:2749] T dd90fb9eb167478cbc3ec9363af86295 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:51:54.026321 888 tablet_service.cc:1552] Processing DeleteTablet for tablet 56d05e4a500043c995bb51a204cb295d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:53 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:54.026844 888 tablet_service.cc:1505] Processing CreateTablet for tablet b3de6670357c4ae59eb77469b2fb434c (DEFAULT_TABLE table=default.Table_f914dd26d0b846f8923187c2c8edf27e [id=f9c7128476404646a9194e31f49c94e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.027266 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3de6670357c4ae59eb77469b2fb434c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:54.028391 1394 raft_consensus.cc:515] T 70e8cfe5eaa9492c85874dc3214a1e0b P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:54.029047 1394 leader_election.cc:290] T 70e8cfe5eaa9492c85874dc3214a1e0b P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 election: Requested vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:51:54.030040 1172 raft_consensus.cc:359] T 6f93812578a940a3b7378015e2a46f95 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:54.030277 1172 raft_consensus.cc:385] T 6f93812578a940a3b7378015e2a46f95 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:54.030260 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 64b0fb400e944ce2b2c7a49d12135e9c (table default.Table_9e6e0bf44516404ab570d4f843bd542f [id=e71aea48b9884bacb250412624ec4615]) successfully deleted
I20251029 02:51:54.030392 1172 raft_consensus.cc:740] T 6f93812578a940a3b7378015e2a46f95 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:51:54.030648 1172 consensus_queue.cc:260] T 6f93812578a940a3b7378015e2a46f95 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
W20251029 02:51:54.031267 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet f81aecb20d6e498bb9b498698886d65e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f81aecb20d6e498bb9b498698886d65e already in progress: creating tablet
I20251029 02:51:54.032936 1175 tablet_replica.cc:333] T 7c041f4909274cd5a03b9fd78353faf9 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:51:54.033205 1175 raft_consensus.cc:2243] T 7c041f4909274cd5a03b9fd78353faf9 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:54.033457 1175 raft_consensus.cc:2272] T 7c041f4909274cd5a03b9fd78353faf9 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:54.033900 1172 ts_tablet_manager.cc:1434] T 6f93812578a940a3b7378015e2a46f95 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20251029 02:51:54.034380 1172 tablet_bootstrap.cc:492] T fa5127bcfcda40a7b5ad4ef016af072a P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:51:54.034543 1174 ts_tablet_manager.cc:1950] T 5886b8d39f384426bcbed7289ae72e51 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:51:54.035820 1174 tablet_replica.cc:333] T 0dd31dc5873d407d958a0f5199f78782 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:51:54.036044 1174 raft_consensus.cc:2243] T 0dd31dc5873d407d958a0f5199f78782 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:54.036212 1174 raft_consensus.cc:2272] T 0dd31dc5873d407d958a0f5199f78782 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:54.036450 1172 tablet_bootstrap.cc:654] T fa5127bcfcda40a7b5ad4ef016af072a P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:54.036751 1174 ts_tablet_manager.cc:1916] T 0dd31dc5873d407d958a0f5199f78782 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:54.037778 1175 ts_tablet_manager.cc:1916] T 7c041f4909274cd5a03b9fd78353faf9 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:54.038574 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "70e8cfe5eaa9492c85874dc3214a1e0b" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
W20251029 02:51:54.039132 711 leader_election.cc:343] T 70e8cfe5eaa9492c85874dc3214a1e0b P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:54.041145 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 5886b8d39f384426bcbed7289ae72e51 (table default.Table_839fd4e3c7874233afc8cea6d4c17026 [id=ace1554749d24504aad642b368b1781f]) successfully deleted
W20251029 02:51:54.041421 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 6c22497760274380a832feff7c5a2fae because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6c22497760274380a832feff7c5a2fae already in progress: creating tablet
W20251029 02:51:54.041712 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 9964ad774890422f8e14f6b99f28cb55 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9964ad774890422f8e14f6b99f28cb55 already in progress: creating tablet
I20251029 02:51:54.048514 519 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_e523efa604564452a1f56dada6e21daf" } modify_external_catalogs: true
I20251029 02:51:54.048782 519 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_e523efa604564452a1f56dada6e21daf" } modify_external_catalogs: true
I20251029 02:51:54.049832 523 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet 6f93812578a940a3b7378015e2a46f95 (table default.Table_d84474a10a954e70afe50f3eb8e8c1ed [id=feb1f56ae44249c6adce65637017bb43]). Expected version 2 got 0
I20251029 02:51:54.060258 1175 ts_tablet_manager.cc:1929] T 7c041f4909274cd5a03b9fd78353faf9 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:54.060606 1307 raft_consensus.cc:515] T d7ba1756532c4825a6c7024511dbae51 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:54.061056 1175 log.cc:1199] T 7c041f4909274cd5a03b9fd78353faf9 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/7c041f4909274cd5a03b9fd78353faf9
I20251029 02:51:54.062005 1175 ts_tablet_manager.cc:1950] T 7c041f4909274cd5a03b9fd78353faf9 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:51:54.063083 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 7c041f4909274cd5a03b9fd78353faf9 (table default.Table_83d3fe1030e0461895fc2437732b4ff8 [id=e11618f2bd094526ad359c65bee6a9ba]) successfully deleted
I20251029 02:51:54.065474 1176 tablet_replica.cc:333] T 0dd31dc5873d407d958a0f5199f78782 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
W20251029 02:51:54.065693 990 leader_election.cc:343] T 70e8cfe5eaa9492c85874dc3214a1e0b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:54.065738 1176 raft_consensus.cc:2243] T 0dd31dc5873d407d958a0f5199f78782 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:54.070655 1175 tablet_replica.cc:333] T ab421dfaa3f84593842bb732a2693160 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:51:54.070919 1175 raft_consensus.cc:2243] T ab421dfaa3f84593842bb732a2693160 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:54.071345 1175 raft_consensus.cc:2272] T ab421dfaa3f84593842bb732a2693160 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:54.072218 1172 tablet_bootstrap.cc:492] T fa5127bcfcda40a7b5ad4ef016af072a P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:51:54.072383 1172 ts_tablet_manager.cc:1403] T fa5127bcfcda40a7b5ad4ef016af072a P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.038s user 0.011s sys 0.010s
W20251029 02:51:54.072625 524 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e1fa4980f2624321b617b84c68b90113 consensus state has no leader
I20251029 02:51:54.073127 1172 raft_consensus.cc:359] T fa5127bcfcda40a7b5ad4ef016af072a P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:54.073324 1172 raft_consensus.cc:385] T fa5127bcfcda40a7b5ad4ef016af072a P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:54.073390 1172 raft_consensus.cc:740] T fa5127bcfcda40a7b5ad4ef016af072a P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:51:54.073657 1172 consensus_queue.cc:260] T fa5127bcfcda40a7b5ad4ef016af072a P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:54.074661 1215 server_base.cc:1047] running on GCE node
I20251029 02:51:54.075706 1215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251029 02:51:54.078629 1172 ts_tablet_manager.cc:1434] T fa5127bcfcda40a7b5ad4ef016af072a P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251029 02:51:54.078696 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_f039c48d9cd4451e951c5113e8c24920"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:54.078961 1172 tablet_bootstrap.cc:492] T d6e1170dbb594a5696a3899cde8da7c9 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
W20251029 02:51:54.079291 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f039c48d9cd4451e951c5113e8c24920 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:54.080713 1172 tablet_bootstrap.cc:654] T d6e1170dbb594a5696a3899cde8da7c9 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
W20251029 02:51:54.081832 526 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a5c3a15510cb4321bbb80d61fabd0c0a consensus state has no leader
I20251029 02:51:54.082680 1394 raft_consensus.cc:493] T c4884e22cf2044a4a1070c1b4b759d9e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:54.086927 1394 raft_consensus.cc:515] T c4884e22cf2044a4a1070c1b4b759d9e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:54.087559 1394 leader_election.cc:290] T c4884e22cf2044a4a1070c1b4b759d9e P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:51:54.089838 1176 raft_consensus.cc:2272] T 0dd31dc5873d407d958a0f5199f78782 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:54.094079 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_78b81f8805aa49b8a07a940c8d886de5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:54.094969 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_78b81f8805aa49b8a07a940c8d886de5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:54.098146 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_75341a6f40d84ad3938442923cf3131e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:54.098837 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_75341a6f40d84ad3938442923cf3131e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:54.099995 517 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_d84474a10a954e70afe50f3eb8e8c1ed" } new_table_name: "default.Table_cd7b2ea5d1b240d087c135cfec7dd0be" modify_external_catalogs: true
I20251029 02:51:54.100482 1166 tablet_bootstrap.cc:492] T 4b76c4408a7340e59b8506171622175f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:51:54.100658 1166 ts_tablet_manager.cc:1403] T 4b76c4408a7340e59b8506171622175f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.219s user 0.007s sys 0.004s
I20251029 02:51:54.101395 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_9f5273b57e0b4d95af3e737377f031b6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:54.101995 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9f5273b57e0b4d95af3e737377f031b6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:54.102033 1059 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "70e8cfe5eaa9492c85874dc3214a1e0b" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:51:54.102669 522 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_ba5ed91035ac405694232f7b7e997b59" } new_table_name: "default.Table_d9fe74a49f4843b089aae7fca9e4a826" modify_external_catalogs: true
I20251029 02:51:54.103305 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_ca4e3b5646244cafa5a0429d5a8f2ac6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:54.103752 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ca4e3b5646244cafa5a0429d5a8f2ac6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:54.106197 1176 ts_tablet_manager.cc:1916] T 0dd31dc5873d407d958a0f5199f78782 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:54.107232 1175 ts_tablet_manager.cc:1916] T ab421dfaa3f84593842bb732a2693160 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 02:51:54.107637 990 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:41701, user_credentials={real_user=slave}} blocked reactor thread for 54519.1us
W20251029 02:51:54.117553 847 leader_election.cc:343] T d7ba1756532c4825a6c7024511dbae51 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: d7ba1756532c4825a6c7024511dbae51
I20251029 02:51:54.122274 1059 raft_consensus.cc:3060] T 70e8cfe5eaa9492c85874dc3214a1e0b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:51:54.123127 1394 raft_consensus.cc:493] T fa5127bcfcda40a7b5ad4ef016af072a P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:54.123315 1394 raft_consensus.cc:515] T fa5127bcfcda40a7b5ad4ef016af072a P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:54.124190 1394 leader_election.cc:290] T fa5127bcfcda40a7b5ad4ef016af072a P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:51:54.124361 893 tablet_service.cc:1552] Processing DeleteTablet for tablet 0cc1957a845f4e9cb703a553b27d1347 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:53 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:54.124629 893 tablet_service.cc:1552] Processing DeleteTablet for tablet 519754f5d8f7484480f973c53708c5f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:53 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:54.125164 1059 raft_consensus.cc:2468] T 70e8cfe5eaa9492c85874dc3214a1e0b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b495a49349324f7d81908f2da700f0d9 in term 1.
I20251029 02:51:54.125679 1215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251029 02:51:54.126236 893 tablet_service.cc:1552] Processing DeleteTablet for tablet 4add552cf4d940d2a0a36d6b58379a34 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:53 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:54.126490 893 tablet_service.cc:1552] Processing DeleteTablet for tablet f60e7c24982d40319801869ad060864d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:53 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:54.128078 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_9b6f49f9513d4a09a4782893f8f16e5c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:54.128718 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9b6f49f9513d4a09a4782893f8f16e5c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:54.130028 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_0d5730d385074a70b295b233e6ecd79e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:54.130582 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0d5730d385074a70b295b233e6ecd79e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:54.132586 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fa5127bcfcda40a7b5ad4ef016af072a" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:51:54.134196 893 tablet_service.cc:1552] Processing DeleteTablet for tablet b867feae24e741aca6902868c35c1428 with delete_type TABLET_DATA_DELETED (replaced by tablet 1bab90fe904944af948dab3db06b7e34) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:54.134836 523 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet 6f93812578a940a3b7378015e2a46f95 (table default.Table_d84474a10a954e70afe50f3eb8e8c1ed [id=feb1f56ae44249c6adce65637017bb43]). Expected version 2 got 0
W20251029 02:51:54.135174 711 leader_election.cc:343] T fa5127bcfcda40a7b5ad4ef016af072a P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:54.135836 521 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_86c59704aa154b37b528fd68c2359e0f" } new_table_name: "default.Table_db3418c43cde44f8a505e27220d0b5fd" modify_external_catalogs: true
I20251029 02:51:54.136183 747 tablet_service.cc:1505] Processing CreateTablet for tablet 5db519edc75d4bd0985b32322825ffae (DEFAULT_TABLE table=default.Table_b8ddec1d965444948171ef0cca0ff011 [id=60a528f028d24bbabd6f5a927fb7853b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.136452 744 tablet_service.cc:1505] Processing CreateTablet for tablet 777992acb1bd4765886b802940db74bb (DEFAULT_TABLE table=default.Table_ba5ed91035ac405694232f7b7e997b59 [id=152882c742b2459ba863fd41083284c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.136729 748 tablet_service.cc:1505] Processing CreateTablet for tablet 6f35412a66b74051a475dce5dafa7792 (DEFAULT_TABLE table=default.Table_f65782ffe391498286f93d5e804bf177 [id=d37290c8b4e8435a8b54bac83428a9ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.136943 751 tablet_service.cc:1505] Processing CreateTablet for tablet f529e17e97074dca91ec2f48954cfd32 (DEFAULT_TABLE table=default.Table_d37ad2580f614a6e9c469258b145525b [id=835aa84a7f8d426d9560b819e6864715]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.137244 738 tablet_service.cc:1505] Processing CreateTablet for tablet 1bdb67125493430398ff5c80b73de2ac (DEFAULT_TABLE table=default.Table_d8875cc2c2f14a298da5a7c3c7678354 [id=8c3d3d88395e4712af0f62c4ad14edea]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.137682 756 tablet_service.cc:1505] Processing CreateTablet for tablet dfa8b1b9a80e4e8689e0acb44e16346a (DEFAULT_TABLE table=default.Table_5c973aa3bd7f4a9ebf8435ed761552a5 [id=222ef1b283554565b4930328c2c8975b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.137833 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_af1ab75f523d41338d4471824997f935"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:54.137979 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d7ba1756532c4825a6c7024511dbae51" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:51:54.138180 1060 raft_consensus.cc:3060] T d7ba1756532c4825a6c7024511dbae51 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
W20251029 02:51:54.138460 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_af1ab75f523d41338d4471824997f935 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:54.126657 1166 raft_consensus.cc:359] T 4b76c4408a7340e59b8506171622175f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:54.126698 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4884e22cf2044a4a1070c1b4b759d9e" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:51:54.140504 739 tablet_service.cc:1505] Processing CreateTablet for tablet 785733da86ff425a90ccfab2933fc89e (DEFAULT_TABLE table=default.Table_bdefc7775234406788f3f7e220769b43 [id=b81476e2d6db40ef9f97211bdb79ce21]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 02:51:54.141148 711 leader_election.cc:343] T c4884e22cf2044a4a1070c1b4b759d9e P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:54.141721 753 tablet_service.cc:1505] Processing CreateTablet for tablet caef1c93e33a4390bcada59a41f43465 (DEFAULT_TABLE table=default.Table_9efddcb184d74a75bd0d1b72cc522e3f [id=8cccaacc256b40deabe999de1d75032e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.142695 743 tablet_service.cc:1505] Processing CreateTablet for tablet 0f80b840ed954230bfbfe7e40127cd2f (DEFAULT_TABLE table=default.Table_5902a11255a049be9dd07acf8b50adda [id=5bd71ce398e84db28f54262cd3c5528b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.144430 1166 raft_consensus.cc:385] T 4b76c4408a7340e59b8506171622175f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:54.146075 1166 raft_consensus.cc:740] T 4b76c4408a7340e59b8506171622175f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:51:54.146335 1166 consensus_queue.cc:260] T 4b76c4408a7340e59b8506171622175f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:54.150219 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5db519edc75d4bd0985b32322825ffae. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:54.150594 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f35412a66b74051a475dce5dafa7792. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:54.150794 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfa8b1b9a80e4e8689e0acb44e16346a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:54.151229 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f529e17e97074dca91ec2f48954cfd32. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:54.154671 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bdb67125493430398ff5c80b73de2ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:54.155005 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 777992acb1bd4765886b802940db74bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:54.155954 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet caef1c93e33a4390bcada59a41f43465. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:54.158638 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f80b840ed954230bfbfe7e40127cd2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:54.159359 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 785733da86ff425a90ccfab2933fc89e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:54.161334 1060 raft_consensus.cc:2468] T d7ba1756532c4825a6c7024511dbae51 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:51:54.162318 1166 ts_tablet_manager.cc:1434] T 4b76c4408a7340e59b8506171622175f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.062s user 0.002s sys 0.000s
I20251029 02:51:54.162684 1166 tablet_bootstrap.cc:492] T 24016ceaa84147eca0b94b3ad288e772 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:51:54.168159 1175 ts_tablet_manager.cc:1929] T ab421dfaa3f84593842bb732a2693160 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:54.168361 1175 log.cc:1199] T ab421dfaa3f84593842bb732a2693160 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/ab421dfaa3f84593842bb732a2693160
W20251029 02:51:54.168427 517 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6f93812578a940a3b7378015e2a46f95 consensus state has no leader
I20251029 02:51:54.169303 1175 ts_tablet_manager.cc:1950] T ab421dfaa3f84593842bb732a2693160 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:51:54.170377 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet ab421dfaa3f84593842bb732a2693160 (table default.Table_b70ecfbeceed48a4bfdb94abe7c580f4 [id=03a25420ce6a46fc93a9fb753246bc7b]) successfully deleted
I20251029 02:51:54.170769 1350 raft_consensus.cc:493] T bf210435010d4dde8903fd116eb468e3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:54.170946 1350 raft_consensus.cc:515] T bf210435010d4dde8903fd116eb468e3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:54.171134 1215 hybrid_clock.cc:648] HybridClock initialized: now 1761706314171021 us; error 112 us; skew 500 ppm
I20251029 02:51:54.171551 1350 leader_election.cc:290] T bf210435010d4dde8903fd116eb468e3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:51:54.171669 1215 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251029 02:51:54.172410 1166 tablet_bootstrap.cc:654] T 24016ceaa84147eca0b94b3ad288e772 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:54.173893 1394 raft_consensus.cc:493] T 6f93812578a940a3b7378015e2a46f95 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:54.174067 1394 raft_consensus.cc:515] T 6f93812578a940a3b7378015e2a46f95 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
W20251029 02:51:54.174616 522 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 777992acb1bd4765886b802940db74bb consensus state has no leader
I20251029 02:51:54.175488 1394 leader_election.cc:290] T 6f93812578a940a3b7378015e2a46f95 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:51:54.142963 893 tablet_service.cc:1505] Processing CreateTablet for tablet 777992acb1bd4765886b802940db74bb (DEFAULT_TABLE table=default.Table_ba5ed91035ac405694232f7b7e997b59 [id=152882c742b2459ba863fd41083284c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.185225 1171 raft_consensus.cc:493] T b867feae24e741aca6902868c35c1428 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:54.101838 989 leader_election.cc:304] T 41a6139fd9b84707a8c7eebc0381e383 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters: c756528642a543eeacd93cef3e335a7d
I20251029 02:51:54.187296 519 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 392a36a6b59a42f78e3ac7c8fb63214b
W20251029 02:51:54.187667 989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:38481, user_credentials={real_user=slave}} blocked reactor thread for 120404us
I20251029 02:51:54.187896 989 leader_election.cc:304] T 70e8cfe5eaa9492c85874dc3214a1e0b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters: c756528642a543eeacd93cef3e335a7d
I20251029 02:51:54.188167 1395 raft_consensus.cc:2804] T 41a6139fd9b84707a8c7eebc0381e383 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:51:54.188256 1395 raft_consensus.cc:493] T 41a6139fd9b84707a8c7eebc0381e383 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:51:54.188423 1395 raft_consensus.cc:3060] T 41a6139fd9b84707a8c7eebc0381e383 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:51:54.190732 1350 raft_consensus.cc:2764] T 70e8cfe5eaa9492c85874dc3214a1e0b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20251029 02:51:54.191253 989 leader_election.cc:304] T 5e31e061d74146009d24d5cd57487a5e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters: c756528642a543eeacd93cef3e335a7d
I20251029 02:51:54.191624 1350 raft_consensus.cc:2804] T 5e31e061d74146009d24d5cd57487a5e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:51:54.191766 1350 raft_consensus.cc:697] T 5e31e061d74146009d24d5cd57487a5e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Becoming Leader. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Running, Role: LEADER
I20251029 02:51:54.192003 1350 consensus_queue.cc:237] T 5e31e061d74146009d24d5cd57487a5e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:54.195350 712 leader_election.cc:304] T 70e8cfe5eaa9492c85874dc3214a1e0b P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters: c756528642a543eeacd93cef3e335a7d
I20251029 02:51:54.197038 1215 webserver.cc:492] Webserver started at http://127.0.104.60:42893/ using document root <none> and password file <none>
I20251029 02:51:54.196964 1395 raft_consensus.cc:515] T 41a6139fd9b84707a8c7eebc0381e383 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:54.197633 1395 leader_election.cc:290] T 41a6139fd9b84707a8c7eebc0381e383 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Requested vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
W20251029 02:51:54.197824 521 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet da2a21c3236a4bd99872b89b12a36508 consensus state has no leader
I20251029 02:51:54.198268 1215 fs_manager.cc:362] Metadata directory not provided
I20251029 02:51:54.198410 1215 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251029 02:51:54.199827 1394 raft_consensus.cc:2804] T 70e8cfe5eaa9492c85874dc3214a1e0b P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:51:54.200534 1394 raft_consensus.cc:697] T 70e8cfe5eaa9492c85874dc3214a1e0b P b495a49349324f7d81908f2da700f0d9 [term 1 LEADER]: Becoming Leader. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Running, Role: LEADER
I20251029 02:51:54.200819 1394 consensus_queue.cc:237] T 70e8cfe5eaa9492c85874dc3214a1e0b P b495a49349324f7d81908f2da700f0d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:54.185406 1171 raft_consensus.cc:515] T b867feae24e741aca6902868c35c1428 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:54.207746 1062 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f93812578a940a3b7378015e2a46f95" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:51:54.208045 746 tablet_service.cc:1505] Processing CreateTablet for tablet 44b518c3a0964a6fa863b2784d80c8a4 (DEFAULT_TABLE table=default.Table_3b6a3dbb404a485dac0a7f73fc3685a8 [id=e0c0cf3d456444c191a32bb180e83798]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.208206 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "41a6139fd9b84707a8c7eebc0381e383" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:51:54.208426 776 raft_consensus.cc:3060] T 41a6139fd9b84707a8c7eebc0381e383 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:51:54.208490 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44b518c3a0964a6fa863b2784d80c8a4. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:51:54.208475 989 leader_election.cc:343] T bf210435010d4dde8903fd116eb468e3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: bf210435010d4dde8903fd116eb468e3
W20251029 02:51:54.208810 712 leader_election.cc:343] T 6f93812578a940a3b7378015e2a46f95 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:54.209092 1059 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fa5127bcfcda40a7b5ad4ef016af072a" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:51:54.209393 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4884e22cf2044a4a1070c1b4b759d9e" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:51:54.209944 1176 ts_tablet_manager.cc:1929] T 0dd31dc5873d407d958a0f5199f78782 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:54.210050 1176 log.cc:1199] T 0dd31dc5873d407d958a0f5199f78782 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/0dd31dc5873d407d958a0f5199f78782
I20251029 02:51:54.210819 755 tablet_service.cc:1505] Processing CreateTablet for tablet ccdca31db2834de4b6ed0f2d67a10535 (DEFAULT_TABLE table=default.Table_75341a6f40d84ad3938442923cf3131e [id=72e9fc37abc14519b6b87ebfc83c292c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.211391 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ccdca31db2834de4b6ed0f2d67a10535. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:54.211860 1176 ts_tablet_manager.cc:1950] T 0dd31dc5873d407d958a0f5199f78782 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:51:54.212006 753 tablet_service.cc:1505] Processing CreateTablet for tablet 95f342d1cfb84c4eb31bd3d15ea3bdc2 (DEFAULT_TABLE table=default.Table_fb891e36b29c4a3ab6f1eef96dde937d [id=c63d52811db749d6badbed72fc133943]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.212241 1176 tablet_replica.cc:333] T d7ba1756532c4825a6c7024511dbae51 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:51:54.212426 1176 raft_consensus.cc:2243] T d7ba1756532c4825a6c7024511dbae51 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:54.212463 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95f342d1cfb84c4eb31bd3d15ea3bdc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:54.212589 1176 raft_consensus.cc:2272] T d7ba1756532c4825a6c7024511dbae51 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:54.213095 1176 ts_tablet_manager.cc:1916] T d7ba1756532c4825a6c7024511dbae51 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:54.213963 748 tablet_service.cc:1505] Processing CreateTablet for tablet 1c53fc1fff1a4bb988beb46ff91fd3be (DEFAULT_TABLE table=default.Table_f039c48d9cd4451e951c5113e8c24920 [id=9abdb27ccc0a48e991b7a4619808fdf8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.214267 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c53fc1fff1a4bb988beb46ff91fd3be. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:54.214823 1308 raft_consensus.cc:515] T 6c32c6f5e1cc40a0a12688201cac62a7 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:54.215612 1307 leader_election.cc:290] T d7ba1756532c4825a6c7024511dbae51 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:51:54.215837 756 tablet_service.cc:1505] Processing CreateTablet for tablet 7a1866afc7564217ad777e7ce53786c5 (DEFAULT_TABLE table=default.Table_78b81f8805aa49b8a07a940c8d886de5 [id=09ee821707fa4082889bfccb8667b0b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.216177 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a1866afc7564217ad777e7ce53786c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:54.219996 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f93812578a940a3b7378015e2a46f95" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:51:54.223424 711 leader_election.cc:343] T 6f93812578a940a3b7378015e2a46f95 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:54.223577 711 leader_election.cc:304] T 6f93812578a940a3b7378015e2a46f95 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:51:54.223975 1276 raft_consensus.cc:2749] T 6f93812578a940a3b7378015e2a46f95 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:51:54.224493 776 raft_consensus.cc:2468] T 41a6139fd9b84707a8c7eebc0381e383 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 1.
W20251029 02:51:54.226792 712 leader_election.cc:343] T fa5127bcfcda40a7b5ad4ef016af072a P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:54.226884 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 777992acb1bd4765886b802940db74bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:54.226950 1171 leader_election.cc:290] T b867feae24e741aca6902868c35c1428 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:51:54.226940 712 leader_election.cc:304] T fa5127bcfcda40a7b5ad4ef016af072a P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:51:54.227312 1276 raft_consensus.cc:2749] T fa5127bcfcda40a7b5ad4ef016af072a P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 02:51:54.227497 712 leader_election.cc:343] T c4884e22cf2044a4a1070c1b4b759d9e P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:54.227630 712 leader_election.cc:304] T c4884e22cf2044a4a1070c1b4b759d9e P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:51:54.227880 1276 raft_consensus.cc:2749] T c4884e22cf2044a4a1070c1b4b759d9e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:51:54.228328 1350 raft_consensus.cc:493] T 4b76c4408a7340e59b8506171622175f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:54.228477 1350 raft_consensus.cc:515] T 4b76c4408a7340e59b8506171622175f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:54.225241 849 leader_election.cc:304] T d7ba1756532c4825a6c7024511dbae51 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters: b495a49349324f7d81908f2da700f0d9
I20251029 02:51:54.229122 1350 leader_election.cc:290] T 4b76c4408a7340e59b8506171622175f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:51:54.229578 989 leader_election.cc:304] T 41a6139fd9b84707a8c7eebc0381e383 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters:
I20251029 02:51:54.229854 1350 raft_consensus.cc:2804] T 41a6139fd9b84707a8c7eebc0381e383 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:51:54.230026 1350 raft_consensus.cc:697] T 41a6139fd9b84707a8c7eebc0381e383 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Becoming Leader. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Running, Role: LEADER
I20251029 02:51:54.230295 1350 consensus_queue.cc:237] T 41a6139fd9b84707a8c7eebc0381e383 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:54.229883 1059 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b867feae24e741aca6902868c35c1428" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:51:54.230353 1215 fs_manager.cc:714] Time spent opening directory manager: real 0.029s user 0.005s sys 0.003s
I20251029 02:51:54.230585 1059 raft_consensus.cc:2468] T b867feae24e741aca6902868c35c1428 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:51:54.231346 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6c32c6f5e1cc40a0a12688201cac62a7" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:51:54.231542 1060 raft_consensus.cc:3060] T 6c32c6f5e1cc40a0a12688201cac62a7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:51:54.235720 1060 raft_consensus.cc:2468] T 6c32c6f5e1cc40a0a12688201cac62a7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
W20251029 02:51:54.237159 847 leader_election.cc:343] T b867feae24e741aca6902868c35c1428 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: b867feae24e741aca6902868c35c1428
I20251029 02:51:54.243376 1171 raft_consensus.cc:2804] T d7ba1756532c4825a6c7024511dbae51 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:51:54.245554 739 tablet_service.cc:1505] Processing CreateTablet for tablet dabb8d760aae4aa09eb83cdb586c9788 (DEFAULT_TABLE table=default.Table_9b6f49f9513d4a09a4782893f8f16e5c [id=b3e544a043444b77a597c6c65c15b795]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.245833 743 tablet_service.cc:1505] Processing CreateTablet for tablet 0d84dc8dc465496f9adec798b2e03e13 (DEFAULT_TABLE table=default.Table_0d5730d385074a70b295b233e6ecd79e [id=423993f75a9948aebe1dee01d3e12702]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.246057 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dabb8d760aae4aa09eb83cdb586c9788. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:54.195358 738 tablet_service.cc:1552] Processing DeleteTablet for tablet 392a36a6b59a42f78e3ac7c8fb63214b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:54 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:54.247591 918 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b76c4408a7340e59b8506171622175f" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:51:54.248111 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "41a6139fd9b84707a8c7eebc0381e383" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
I20251029 02:51:54.248447 1171 raft_consensus.cc:697] T d7ba1756532c4825a6c7024511dbae51 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:51:54.248634 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bf210435010d4dde8903fd116eb468e3" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:51:54.248807 917 raft_consensus.cc:2468] T bf210435010d4dde8903fd116eb468e3 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 0.
I20251029 02:51:54.249118 1411 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251029 02:51:54.249460 1167 ts_tablet_manager.cc:1434] T b867feae24e741aca6902868c35c1428 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.199s user 0.003s sys 0.003s
I20251029 02:51:54.249580 738 tablet_service.cc:1505] Processing CreateTablet for tablet 59d03a190b784ab18a3e63a572675891 (DEFAULT_TABLE table=default.Table_9f5273b57e0b4d95af3e737377f031b6 [id=3ede90ae95fb46b2ab910914a49a35a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.249967 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59d03a190b784ab18a3e63a572675891. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:51:54.250741 990 leader_election.cc:343] T 4b76c4408a7340e59b8506171622175f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:54.252198 1028 tablet_service.cc:1505] Processing CreateTablet for tablet 777992acb1bd4765886b802940db74bb (DEFAULT_TABLE table=default.Table_ba5ed91035ac405694232f7b7e997b59 [id=152882c742b2459ba863fd41083284c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.253154 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 777992acb1bd4765886b802940db74bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:54.254565 1037 tablet_service.cc:1505] Processing CreateTablet for tablet f529e17e97074dca91ec2f48954cfd32 (DEFAULT_TABLE table=default.Table_d37ad2580f614a6e9c469258b145525b [id=835aa84a7f8d426d9560b819e6864715]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.254966 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f529e17e97074dca91ec2f48954cfd32. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:54.255016 1042 tablet_service.cc:1505] Processing CreateTablet for tablet 6f35412a66b74051a475dce5dafa7792 (DEFAULT_TABLE table=default.Table_f65782ffe391498286f93d5e804bf177 [id=d37290c8b4e8435a8b54bac83428a9ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.255010 745 tablet_service.cc:1505] Processing CreateTablet for tablet 49d9ce32e4f04f11be2ecefc5d523a8b (DEFAULT_TABLE table=default.Table_af1ab75f523d41338d4471824997f935 [id=468c76c8c99a4aa59c8b1f6c8e4bfa30]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.255321 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f35412a66b74051a475dce5dafa7792. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:54.255396 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49d9ce32e4f04f11be2ecefc5d523a8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:54.256742 1029 tablet_service.cc:1505] Processing CreateTablet for tablet 1bdb67125493430398ff5c80b73de2ac (DEFAULT_TABLE table=default.Table_d8875cc2c2f14a298da5a7c3c7678354 [id=8c3d3d88395e4712af0f62c4ad14edea]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.256804 1025 tablet_service.cc:1505] Processing CreateTablet for tablet 785733da86ff425a90ccfab2933fc89e (DEFAULT_TABLE table=default.Table_bdefc7775234406788f3f7e220769b43 [id=b81476e2d6db40ef9f97211bdb79ce21]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.257089 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 785733da86ff425a90ccfab2933fc89e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:54.257308 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bdb67125493430398ff5c80b73de2ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:54.258250 1024 tablet_service.cc:1505] Processing CreateTablet for tablet dfa8b1b9a80e4e8689e0acb44e16346a (DEFAULT_TABLE table=default.Table_5c973aa3bd7f4a9ebf8435ed761552a5 [id=222ef1b283554565b4930328c2c8975b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.258661 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfa8b1b9a80e4e8689e0acb44e16346a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:54.259416 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 0dd31dc5873d407d958a0f5199f78782 (table default.Table_c60a78e1cea24e8e864803a5c6124df1 [id=99445ef970c644f8a39388b2015da927]) successfully deleted
I20251029 02:51:54.259593 1027 tablet_service.cc:1505] Processing CreateTablet for tablet 5db519edc75d4bd0985b32322825ffae (DEFAULT_TABLE table=default.Table_b8ddec1d965444948171ef0cca0ff011 [id=60a528f028d24bbabd6f5a927fb7853b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.259976 1215 fs_manager.cc:730] Time spent opening block manager: real 0.028s user 0.005s sys 0.000s
I20251029 02:51:54.260015 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5db519edc75d4bd0985b32322825ffae. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:51:54.251014 990 leader_election.cc:343] T 41a6139fd9b84707a8c7eebc0381e383 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:54.260169 1215 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/master-2/data,/tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/master-2/wal
uuid: "e5e5222b26fe4145948620e51edf53fb"
format_stamp: "Formatted at 2025-10-29 02:51:47 on dist-test-slave-qwrb"
I20251029 02:51:54.261157 1215 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251029 02:51:54.261662 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d84dc8dc465496f9adec798b2e03e13. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:54.262169 1038 tablet_service.cc:1505] Processing CreateTablet for tablet caef1c93e33a4390bcada59a41f43465 (DEFAULT_TABLE table=default.Table_9efddcb184d74a75bd0d1b72cc522e3f [id=8cccaacc256b40deabe999de1d75032e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.262765 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet caef1c93e33a4390bcada59a41f43465. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:54.262964 1026 tablet_service.cc:1505] Processing CreateTablet for tablet 1c53fc1fff1a4bb988beb46ff91fd3be (DEFAULT_TABLE table=default.Table_f039c48d9cd4451e951c5113e8c24920 [id=9abdb27ccc0a48e991b7a4619808fdf8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.263309 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c53fc1fff1a4bb988beb46ff91fd3be. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:54.266829 1030 tablet_service.cc:1505] Processing CreateTablet for tablet 7a1866afc7564217ad777e7ce53786c5 (DEFAULT_TABLE table=default.Table_78b81f8805aa49b8a07a940c8d886de5 [id=09ee821707fa4082889bfccb8667b0b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.267184 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a1866afc7564217ad777e7ce53786c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:54.269520 1041 tablet_service.cc:1552] Processing DeleteTablet for tablet 392a36a6b59a42f78e3ac7c8fb63214b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:54 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:54.270957 1023 tablet_service.cc:1505] Processing CreateTablet for tablet 0f80b840ed954230bfbfe7e40127cd2f (DEFAULT_TABLE table=default.Table_5902a11255a049be9dd07acf8b50adda [id=5bd71ce398e84db28f54262cd3c5528b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.271312 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f80b840ed954230bfbfe7e40127cd2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:54.277266 990 leader_election.cc:304] T bf210435010d4dde8903fd116eb468e3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters: b495a49349324f7d81908f2da700f0d9
I20251029 02:51:54.277710 1395 raft_consensus.cc:2804] T bf210435010d4dde8903fd116eb468e3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:51:54.277810 1395 raft_consensus.cc:493] T bf210435010d4dde8903fd116eb468e3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:51:54.277899 1395 raft_consensus.cc:3060] T bf210435010d4dde8903fd116eb468e3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:51:54.245842 747 tablet_service.cc:1505] Processing CreateTablet for tablet d61beda662d844cdaf67fd9a2e382ea2 (DEFAULT_TABLE table=default.Table_ca4e3b5646244cafa5a0429d5a8f2ac6 [id=481e8e175f364d31bb7dbbaa2df2dad1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.280685 1395 raft_consensus.cc:515] T bf210435010d4dde8903fd116eb468e3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:54.280910 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d61beda662d844cdaf67fd9a2e382ea2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:54.284415 1395 leader_election.cc:290] T bf210435010d4dde8903fd116eb468e3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Requested vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:51:54.289769 1041 tablet_service.cc:1505] Processing CreateTablet for tablet ccdca31db2834de4b6ed0f2d67a10535 (DEFAULT_TABLE table=default.Table_75341a6f40d84ad3938442923cf3131e [id=72e9fc37abc14519b6b87ebfc83c292c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.290150 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ccdca31db2834de4b6ed0f2d67a10535. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:54.291659 521 catalog_manager.cc:5649] T 11f4252b5e694312aaa142468f8e68b4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } health_report { overall_health: HEALTHY } } }
I20251029 02:51:54.292142 521 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet 11f4252b5e694312aaa142468f8e68b4 (table default.Table_b93cf815f69a4fe895e2ce461148fc71 [id=d0cd3d1567c54463947443927e18169f]). Expected version 5 got 0
I20251029 02:51:54.292626 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b76c4408a7340e59b8506171622175f" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:51:54.292870 776 raft_consensus.cc:2468] T 4b76c4408a7340e59b8506171622175f P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 0.
I20251029 02:51:54.293313 1030 tablet_service.cc:1505] Processing CreateTablet for tablet 95f342d1cfb84c4eb31bd3d15ea3bdc2 (DEFAULT_TABLE table=default.Table_fb891e36b29c4a3ab6f1eef96dde937d [id=c63d52811db749d6badbed72fc133943]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.294427 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95f342d1cfb84c4eb31bd3d15ea3bdc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:54.296301 895 tablet_service.cc:1505] Processing CreateTablet for tablet f529e17e97074dca91ec2f48954cfd32 (DEFAULT_TABLE table=default.Table_d37ad2580f614a6e9c469258b145525b [id=835aa84a7f8d426d9560b819e6864715]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.299391 1026 tablet_service.cc:1505] Processing CreateTablet for tablet 44b518c3a0964a6fa863b2784d80c8a4 (DEFAULT_TABLE table=default.Table_3b6a3dbb404a485dac0a7f73fc3685a8 [id=e0c0cf3d456444c191a32bb180e83798]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.302155 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44b518c3a0964a6fa863b2784d80c8a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:54.306182 1167 tablet_bootstrap.cc:492] T 6dd4696ad4b44b6dab87a205f3f16ebd P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:51:54.306244 849 leader_election.cc:304] T b867feae24e741aca6902868c35c1428 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters: b495a49349324f7d81908f2da700f0d9
I20251029 02:51:54.307523 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f529e17e97074dca91ec2f48954cfd32. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:54.313673 887 tablet_service.cc:1505] Processing CreateTablet for tablet 5db519edc75d4bd0985b32322825ffae (DEFAULT_TABLE table=default.Table_b8ddec1d965444948171ef0cca0ff011 [id=60a528f028d24bbabd6f5a927fb7853b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.314162 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5db519edc75d4bd0985b32322825ffae. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:54.314777 526 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_64e512f8ce034224aea9b887ed0d09dc" } modify_external_catalogs: true
I20251029 02:51:54.315002 526 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_64e512f8ce034224aea9b887ed0d09dc" } modify_external_catalogs: true
I20251029 02:51:54.316854 521 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_3e28d37b4b6146bf95edf245d3f9d0cf" } modify_external_catalogs: true
I20251029 02:51:54.317061 521 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_3e28d37b4b6146bf95edf245d3f9d0cf" } modify_external_catalogs: true
I20251029 02:51:54.322594 526 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet ba014ecfec7f4602ae1c09d931bac824
I20251029 02:51:54.324360 739 tablet_service.cc:1552] Processing DeleteTablet for tablet ba014ecfec7f4602ae1c09d931bac824 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:54 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:54.324712 1307 raft_consensus.cc:2804] T b867feae24e741aca6902868c35c1428 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:51:54.327356 1307 raft_consensus.cc:493] T b867feae24e741aca6902868c35c1428 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:51:54.328143 521 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 44784a3a760946e0be24743bc2fc8138
I20251029 02:51:54.328295 1307 raft_consensus.cc:3060] T b867feae24e741aca6902868c35c1428 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:51:54.329546 739 tablet_service.cc:1552] Processing DeleteTablet for tablet 44784a3a760946e0be24743bc2fc8138 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:54 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:54.329936 891 tablet_service.cc:1505] Processing CreateTablet for tablet 6f35412a66b74051a475dce5dafa7792 (DEFAULT_TABLE table=default.Table_f65782ffe391498286f93d5e804bf177 [id=d37290c8b4e8435a8b54bac83428a9ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.335103 1167 tablet_bootstrap.cc:654] T 6dd4696ad4b44b6dab87a205f3f16ebd P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:54.342207 521 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_8b39acd343e94119804478e17baaa785" } modify_external_catalogs: true
I20251029 02:51:54.342417 521 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_8b39acd343e94119804478e17baaa785" } modify_external_catalogs: true
I20251029 02:51:54.344098 526 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_198644e9ee1f45ecb6ad95a759b54607" } modify_external_catalogs: true
I20251029 02:51:54.344260 526 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_198644e9ee1f45ecb6ad95a759b54607" } modify_external_catalogs: true
I20251029 02:51:54.347324 1251 raft_consensus.cc:493] T bc1a1cf274b64b52aba58811e9594330 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:54.347494 1251 raft_consensus.cc:515] T bc1a1cf274b64b52aba58811e9594330 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:54.348162 1251 leader_election.cc:290] T bc1a1cf274b64b52aba58811e9594330 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:51:54.349226 879 tablet_service.cc:1505] Processing CreateTablet for tablet 1bdb67125493430398ff5c80b73de2ac (DEFAULT_TABLE table=default.Table_d8875cc2c2f14a298da5a7c3c7678354 [id=8c3d3d88395e4712af0f62c4ad14edea]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.349674 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bdb67125493430398ff5c80b73de2ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:54.352458 882 tablet_service.cc:1505] Processing CreateTablet for tablet dfa8b1b9a80e4e8689e0acb44e16346a (DEFAULT_TABLE table=default.Table_5c973aa3bd7f4a9ebf8435ed761552a5 [id=222ef1b283554565b4930328c2c8975b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.352829 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfa8b1b9a80e4e8689e0acb44e16346a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:54.354449 526 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 6cd7009e06a646acad0802918c2b425e
I20251029 02:51:54.355683 747 tablet_service.cc:1552] Processing DeleteTablet for tablet 6cd7009e06a646acad0802918c2b425e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:54 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:54.356534 885 tablet_service.cc:1505] Processing CreateTablet for tablet 785733da86ff425a90ccfab2933fc89e (DEFAULT_TABLE table=default.Table_bdefc7775234406788f3f7e220769b43 [id=b81476e2d6db40ef9f97211bdb79ce21]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.356927 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 785733da86ff425a90ccfab2933fc89e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:54.357162 521 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet b5d5b737c86844d58c51b2388c0b1705
I20251029 02:51:54.358390 747 tablet_service.cc:1552] Processing DeleteTablet for tablet b5d5b737c86844d58c51b2388c0b1705 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:54 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:54.358806 1171 consensus_queue.cc:237] T d7ba1756532c4825a6c7024511dbae51 P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:54.360773 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bf210435010d4dde8903fd116eb468e3" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
I20251029 02:51:54.360962 917 raft_consensus.cc:3060] T bf210435010d4dde8903fd116eb468e3 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:51:54.363621 1251 raft_consensus.cc:493] T 6dd4696ad4b44b6dab87a205f3f16ebd P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:54.363942 917 raft_consensus.cc:2468] T bf210435010d4dde8903fd116eb468e3 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 1.
I20251029 02:51:54.363837 1251 raft_consensus.cc:515] T 6dd4696ad4b44b6dab87a205f3f16ebd P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:54.365031 1251 leader_election.cc:290] T 6dd4696ad4b44b6dab87a205f3f16ebd P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:51:54.365669 519 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_42bbf44edb134c16b76604d387907461" } new_table_name: "default.Table_3ca4a6512b174cb897b142dfe1259549" modify_external_catalogs: true
I20251029 02:51:54.366629 990 leader_election.cc:304] T bf210435010d4dde8903fd116eb468e3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:51:54.367864 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc1a1cf274b64b52aba58811e9594330" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:51:54.368901 526 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_f2172770f9a94bb98d7813da6e1848fe" } new_table_name: "default.Table_cec9500b890447609a5feb7346bcf7a0" modify_external_catalogs: true
I20251029 02:51:54.371241 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_73eec04729fd4870aca2779a5738d772"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:54.371984 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_73eec04729fd4870aca2779a5738d772 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:54.376235 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_cdc08bccd00a493e850b242c3da4b0f3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:54.376849 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cdc08bccd00a493e850b242c3da4b0f3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:54.376845 886 tablet_service.cc:1505] Processing CreateTablet for tablet caef1c93e33a4390bcada59a41f43465 (DEFAULT_TABLE table=default.Table_9efddcb184d74a75bd0d1b72cc522e3f [id=8cccaacc256b40deabe999de1d75032e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.377301 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet caef1c93e33a4390bcada59a41f43465. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:54.378504 518 master_service.cc:954] ReplaceTablet: received request to replace tablet fd0f89180aba43eabb20f926abccbaa6 from {username='slave'} at 127.0.0.1:35314
I20251029 02:51:54.380527 524 master_service.cc:954] ReplaceTablet: received request to replace tablet 6f93812578a940a3b7378015e2a46f95 from {username='slave'} at 127.0.0.1:35314
I20251029 02:51:54.382720 1350 raft_consensus.cc:2804] T bf210435010d4dde8903fd116eb468e3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:51:54.383280 1350 raft_consensus.cc:697] T bf210435010d4dde8903fd116eb468e3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Becoming Leader. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Running, Role: LEADER
I20251029 02:51:54.383916 1350 consensus_queue.cc:237] T bf210435010d4dde8903fd116eb468e3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
W20251029 02:51:54.388994 519 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6ca9bd37de9e4be388501b15af4bda31 consensus state has no leader
I20251029 02:51:54.396502 518 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet fd0f89180aba43eabb20f926abccbaa6
I20251029 02:51:54.396983 518 catalog_manager.cc:6719] ReplaceTablet: tablet fd0f89180aba43eabb20f926abccbaa6 deleted and replaced by tablet 9bb2166a590d4210b02dd779ef6654ad
I20251029 02:51:54.398398 524 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 6f93812578a940a3b7378015e2a46f95
I20251029 02:51:54.398955 524 catalog_manager.cc:6719] ReplaceTablet: tablet 6f93812578a940a3b7378015e2a46f95 deleted and replaced by tablet ba699862e6f646c6beaa39f080d0faba
I20251029 02:51:54.400280 739 tablet_service.cc:1552] Processing DeleteTablet for tablet 6f93812578a940a3b7378015e2a46f95 with delete_type TABLET_DATA_DELETED (replaced by tablet ba699862e6f646c6beaa39f080d0faba) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:54.400274 747 tablet_service.cc:1552] Processing DeleteTablet for tablet fd0f89180aba43eabb20f926abccbaa6 with delete_type TABLET_DATA_DELETED (replaced by tablet 9bb2166a590d4210b02dd779ef6654ad) from {username='slave'} at 127.0.0.1:44580
W20251029 02:51:54.400604 526 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 69597d48631d43b6a5b92abbf60d5ed3 consensus state has no leader
W20251029 02:51:54.400872 990 leader_election.cc:343] T bc1a1cf274b64b52aba58811e9594330 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:54.401259 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dd4696ad4b44b6dab87a205f3f16ebd" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:51:54.405210 521 master_service.cc:954] ReplaceTablet: received request to replace tablet 1bab90fe904944af948dab3db06b7e34 from {username='slave'} at 127.0.0.1:35314
I20251029 02:51:54.411441 989 leader_election.cc:304] T 4b76c4408a7340e59b8506171622175f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters: c756528642a543eeacd93cef3e335a7d
I20251029 02:51:54.419464 747 tablet_service.cc:1505] Processing CreateTablet for tablet 1275e192871440408a8345fef9b56260 (DEFAULT_TABLE table=default.Table_cdc08bccd00a493e850b242c3da4b0f3 [id=5c4f8f38dfc748dda3abf03f7882cf35]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.419958 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1275e192871440408a8345fef9b56260. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:54.425393 1307 raft_consensus.cc:515] T b867feae24e741aca6902868c35c1428 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:54.427784 739 tablet_service.cc:1505] Processing CreateTablet for tablet 9bb2166a590d4210b02dd779ef6654ad (DEFAULT_TABLE table=default.Table_50252b3b3cc4403da5e259f9f91be915 [id=ff64a323de354792997beb99bae4a52c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.427804 756 tablet_service.cc:1505] Processing CreateTablet for tablet ba699862e6f646c6beaa39f080d0faba (DEFAULT_TABLE table=default.Table_cd7b2ea5d1b240d087c135cfec7dd0be [id=feb1f56ae44249c6adce65637017bb43]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.428440 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bb2166a590d4210b02dd779ef6654ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:54.428766 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba699862e6f646c6beaa39f080d0faba. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:51:54.434706 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.3:45003, user_credentials={real_user=slave}} blocked reactor thread for 63140.3us
I20251029 02:51:54.444711 521 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 1bab90fe904944af948dab3db06b7e34
I20251029 02:51:54.445626 743 tablet_service.cc:1552] Processing DeleteTablet for tablet 1bab90fe904944af948dab3db06b7e34 with delete_type TABLET_DATA_DELETED (replaced by tablet b6d4f7bf702f4834bc5f4ef2b50b92a7) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:54.445866 521 catalog_manager.cc:6719] ReplaceTablet: tablet 1bab90fe904944af948dab3db06b7e34 deleted and replaced by tablet b6d4f7bf702f4834bc5f4ef2b50b92a7
W20251029 02:51:54.415969 990 leader_election.cc:343] T 6dd4696ad4b44b6dab87a205f3f16ebd P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
W20251029 02:51:54.450393 989 leader_election.cc:343] T bf210435010d4dde8903fd116eb468e3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: bf210435010d4dde8903fd116eb468e3
I20251029 02:51:54.451944 1251 raft_consensus.cc:2804] T 4b76c4408a7340e59b8506171622175f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:51:54.452093 1251 raft_consensus.cc:493] T 4b76c4408a7340e59b8506171622175f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:51:54.452265 1251 raft_consensus.cc:3060] T 4b76c4408a7340e59b8506171622175f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:51:54.457460 756 tablet_service.cc:1505] Processing CreateTablet for tablet 13ee62fc2339410b9db70dfd3d3c6fa6 (DEFAULT_TABLE table=default.Table_73eec04729fd4870aca2779a5738d772 [id=b54d7930e083434680f3934fc4023653]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.458287 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13ee62fc2339410b9db70dfd3d3c6fa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:54.473116 747 tablet_service.cc:1505] Processing CreateTablet for tablet b6d4f7bf702f4834bc5f4ef2b50b92a7 (DEFAULT_TABLE table=default.Table_c70eb9f284cd49109fd09e430a1cea6c [id=be817eb6ec33406eb316d36575c134fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.473836 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6d4f7bf702f4834bc5f4ef2b50b92a7. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:51:54.482578 989 leader_election.cc:343] T bc1a1cf274b64b52aba58811e9594330 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: bc1a1cf274b64b52aba58811e9594330
I20251029 02:51:54.482807 989 leader_election.cc:304] T bc1a1cf274b64b52aba58811e9594330 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5; no voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d
I20251029 02:51:54.483124 1350 raft_consensus.cc:2749] T bc1a1cf274b64b52aba58811e9594330 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:51:54.485268 1031 tablet_service.cc:1505] Processing CreateTablet for tablet d61beda662d844cdaf67fd9a2e382ea2 (DEFAULT_TABLE table=default.Table_ca4e3b5646244cafa5a0429d5a8f2ac6 [id=481e8e175f364d31bb7dbbaa2df2dad1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.440325 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f35412a66b74051a475dce5dafa7792. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:54.441352 888 tablet_service.cc:1552] Processing DeleteTablet for tablet 392a36a6b59a42f78e3ac7c8fb63214b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:54 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:54.442498 896 tablet_service.cc:1505] Processing CreateTablet for tablet 7a1866afc7564217ad777e7ce53786c5 (DEFAULT_TABLE table=default.Table_78b81f8805aa49b8a07a940c8d886de5 [id=09ee821707fa4082889bfccb8667b0b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.485745 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d61beda662d844cdaf67fd9a2e382ea2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:54.444250 894 tablet_service.cc:1505] Processing CreateTablet for tablet 1c53fc1fff1a4bb988beb46ff91fd3be (DEFAULT_TABLE table=default.Table_f039c48d9cd4451e951c5113e8c24920 [id=9abdb27ccc0a48e991b7a4619808fdf8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.445303 881 tablet_service.cc:1505] Processing CreateTablet for tablet 95f342d1cfb84c4eb31bd3d15ea3bdc2 (DEFAULT_TABLE table=default.Table_fb891e36b29c4a3ab6f1eef96dde937d [id=c63d52811db749d6badbed72fc133943]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.446687 884 tablet_service.cc:1505] Processing CreateTablet for tablet 44b518c3a0964a6fa863b2784d80c8a4 (DEFAULT_TABLE table=default.Table_3b6a3dbb404a485dac0a7f73fc3685a8 [id=e0c0cf3d456444c191a32bb180e83798]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.449116 899 tablet_service.cc:1505] Processing CreateTablet for tablet 0f80b840ed954230bfbfe7e40127cd2f (DEFAULT_TABLE table=default.Table_5902a11255a049be9dd07acf8b50adda [id=5bd71ce398e84db28f54262cd3c5528b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.462690 1308 leader_election.cc:290] T 6c32c6f5e1cc40a0a12688201cac62a7 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
W20251029 02:51:54.462458 847 leader_election.cc:343] T 6c32c6f5e1cc40a0a12688201cac62a7 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 6c32c6f5e1cc40a0a12688201cac62a7
I20251029 02:51:54.467594 898 tablet_service.cc:1505] Processing CreateTablet for tablet ccdca31db2834de4b6ed0f2d67a10535 (DEFAULT_TABLE table=default.Table_75341a6f40d84ad3938442923cf3131e [id=72e9fc37abc14519b6b87ebfc83c292c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.486824 521 catalog_manager.cc:5649] T 5e31e061d74146009d24d5cd57487a5e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } health_report { overall_health: UNKNOWN } } }
I20251029 02:51:54.487952 521 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet 5e31e061d74146009d24d5cd57487a5e (table default.Table_f2bc74933428449197f8c28182c8698f [id=8e17d6b87b984d878787802bd6f627f2]). Expected version 2 got 0
W20251029 02:51:54.487950 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.3:45003, user_credentials={real_user=slave}} blocked reactor thread for 50673.1us
I20251029 02:51:54.488255 521 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet 4b76c4408a7340e59b8506171622175f (table default.Table_1281926bf1224c0cb4541bf561102021 [id=f3fa2dbbef7e47359492d5a5eef9b8fd]). Expected version 2 got 0
I20251029 02:51:54.488514 521 catalog_manager.cc:5649] T 41a6139fd9b84707a8c7eebc0381e383 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } health_report { overall_health: UNKNOWN } } }
W20251029 02:51:54.492884 847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:38481, user_credentials={real_user=slave}} blocked reactor thread for 73723.8us
I20251029 02:51:54.503197 892 tablet_service.cc:1505] Processing CreateTablet for tablet dabb8d760aae4aa09eb83cdb586c9788 (DEFAULT_TABLE table=default.Table_9b6f49f9513d4a09a4782893f8f16e5c [id=b3e544a043444b77a597c6c65c15b795]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 02:51:54.510102 847 leader_election.cc:343] T b867feae24e741aca6902868c35c1428 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: b867feae24e741aca6902868c35c1428
I20251029 02:51:54.511260 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a1866afc7564217ad777e7ce53786c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:54.521564 888 tablet_service.cc:1505] Processing CreateTablet for tablet d61beda662d844cdaf67fd9a2e382ea2 (DEFAULT_TABLE table=default.Table_ca4e3b5646244cafa5a0429d5a8f2ac6 [id=481e8e175f364d31bb7dbbaa2df2dad1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.527734 1174 ts_tablet_manager.cc:1929] T 0dd31dc5873d407d958a0f5199f78782 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:54.530313 1307 leader_election.cc:290] T b867feae24e741aca6902868c35c1428 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:51:54.195358 1175 tablet_replica.cc:333] T c4884e22cf2044a4a1070c1b4b759d9e P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:51:54.533617 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dabb8d760aae4aa09eb83cdb586c9788. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:54.535156 1175 raft_consensus.cc:2243] T c4884e22cf2044a4a1070c1b4b759d9e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:54.535617 1175 raft_consensus.cc:2272] T c4884e22cf2044a4a1070c1b4b759d9e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:54.539810 1038 tablet_service.cc:1505] Processing CreateTablet for tablet dabb8d760aae4aa09eb83cdb586c9788 (DEFAULT_TABLE table=default.Table_9b6f49f9513d4a09a4782893f8f16e5c [id=b3e544a043444b77a597c6c65c15b795]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.540436 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dabb8d760aae4aa09eb83cdb586c9788. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:54.561914 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_24be2b894899463a810d2fe7511d4d6b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:54.562532 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_435f2dfe42134760abc9e8880fa19f32"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:54.562701 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_24be2b894899463a810d2fe7511d4d6b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:54.562698 1251 raft_consensus.cc:515] T 4b76c4408a7340e59b8506171622175f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
W20251029 02:51:54.563050 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_435f2dfe42134760abc9e8880fa19f32 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:54.563715 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_4e701fdc7b034edc80ccc5ae0c9c4439"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:54.564282 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4e701fdc7b034edc80ccc5ae0c9c4439 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:54.571836 1251 leader_election.cc:290] T 4b76c4408a7340e59b8506171622175f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Requested vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:51:54.572422 1175 ts_tablet_manager.cc:1916] T c4884e22cf2044a4a1070c1b4b759d9e P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:54.584628 1174 log.cc:1199] T 0dd31dc5873d407d958a0f5199f78782 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/0dd31dc5873d407d958a0f5199f78782
I20251029 02:51:54.586542 747 tablet_service.cc:1505] Processing CreateTablet for tablet 2e97d1bc06a24e8d8ba11acb6fd280d8 (DEFAULT_TABLE table=default.Table_24be2b894899463a810d2fe7511d4d6b [id=43e922b1efe44ad9882b5084b22d1560]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.586694 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44b518c3a0964a6fa863b2784d80c8a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:54.586978 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e97d1bc06a24e8d8ba11acb6fd280d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:54.594676 1174 ts_tablet_manager.cc:1950] T 0dd31dc5873d407d958a0f5199f78782 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:51:54.591655 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b76c4408a7340e59b8506171622175f" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
I20251029 02:51:54.598562 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c53fc1fff1a4bb988beb46ff91fd3be. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:54.603669 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95f342d1cfb84c4eb31bd3d15ea3bdc2. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:51:54.604830 512 consensus_peers.cc:597] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0 -> Peer e5e5222b26fe4145948620e51edf53fb (127.0.104.60:41233): Couldn't send request to peer e5e5222b26fe4145948620e51edf53fb. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.60:41233: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20251029 02:51:54.617761 526 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_febb0d5740c847ce8aafa84d39ccbe91" } new_table_name: "default.Table_a0b78d19117a4bd69361d316b318decf" modify_external_catalogs: true
W20251029 02:51:54.619372 990 leader_election.cc:343] T 4b76c4408a7340e59b8506171622175f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:54.620999 890 tablet_service.cc:1552] Processing DeleteTablet for tablet 44784a3a760946e0be24743bc2fc8138 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:54 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:54.621058 1028 tablet_service.cc:1505] Processing CreateTablet for tablet 0d84dc8dc465496f9adec798b2e03e13 (DEFAULT_TABLE table=default.Table_0d5730d385074a70b295b233e6ecd79e [id=423993f75a9948aebe1dee01d3e12702]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 02:51:54.625797 526 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6bd3fac2502e479aa9f4ceaabbb39504 consensus state has no leader
I20251029 02:51:54.625985 1025 tablet_service.cc:1505] Processing CreateTablet for tablet 49d9ce32e4f04f11be2ecefc5d523a8b (DEFAULT_TABLE table=default.Table_af1ab75f523d41338d4471824997f935 [id=468c76c8c99a4aa59c8b1f6c8e4bfa30]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.629662 747 tablet_service.cc:1505] Processing CreateTablet for tablet 30d03e1e42d34242bd354f9e9ec4f127 (DEFAULT_TABLE table=default.Table_435f2dfe42134760abc9e8880fa19f32 [id=c6e1c357fbfc49be972748da564cc9a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.630285 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30d03e1e42d34242bd354f9e9ec4f127. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:54.636391 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d84dc8dc465496f9adec798b2e03e13. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:54.637269 1026 tablet_service.cc:1552] Processing DeleteTablet for tablet 44784a3a760946e0be24743bc2fc8138 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:54 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:54.639415 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_049332132e3142ca86d49635656135f5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:54.640012 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_049332132e3142ca86d49635656135f5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:54.641938 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49d9ce32e4f04f11be2ecefc5d523a8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:54.643117 521 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_f7c46eb111f04bcc8824f60ec45a3a6b" } new_table_name: "default.Table_9306136142dd41cca83965ab5a92c5a0" modify_external_catalogs: true
I20251029 02:51:54.643347 526 master_service.cc:954] ReplaceTablet: received request to replace tablet bc1a1cf274b64b52aba58811e9594330 from {username='slave'} at 127.0.0.1:35314
I20251029 02:51:54.647292 1031 tablet_service.cc:1505] Processing CreateTablet for tablet 59d03a190b784ab18a3e63a572675891 (DEFAULT_TABLE table=default.Table_9f5273b57e0b4d95af3e737377f031b6 [id=3ede90ae95fb46b2ab910914a49a35a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.647753 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59d03a190b784ab18a3e63a572675891. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:54.647979 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_0f40dc35be6d46b3a991e93384fa1e1f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:54.648646 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0f40dc35be6d46b3a991e93384fa1e1f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:54.651597 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet ba014ecfec7f4602ae1c09d931bac824 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:54 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:54.652015 756 tablet_service.cc:1505] Processing CreateTablet for tablet 9269622ca92645d5aa8e73341201ccbb (DEFAULT_TABLE table=default.Table_4e701fdc7b034edc80ccc5ae0c9c4439 [id=d0ca7c93c8cf4dc19962a1957321c5b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.652593 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9269622ca92645d5aa8e73341201ccbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:54.652976 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_a8568b9dabb942a9a88179114c8e1240"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:54.653472 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a8568b9dabb942a9a88179114c8e1240 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:54.654665 1024 tablet_service.cc:1552] Processing DeleteTablet for tablet b5d5b737c86844d58c51b2388c0b1705 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:54 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:54.654944 1037 tablet_service.cc:1552] Processing DeleteTablet for tablet 6cd7009e06a646acad0802918c2b425e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:54 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:54.657629 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_19165ff9f7f24a97bf835e72701455d3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:54.658218 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_19165ff9f7f24a97bf835e72701455d3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:54.661553 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b76c4408a7340e59b8506171622175f" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:51:54.661804 776 raft_consensus.cc:3060] T 4b76c4408a7340e59b8506171622175f P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
W20251029 02:51:54.663156 521 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1fc87e2d7b4240e08f2b4ac7277293ff consensus state has no leader
I20251029 02:51:54.664834 776 raft_consensus.cc:2468] T 4b76c4408a7340e59b8506171622175f P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 1.
I20251029 02:51:54.666018 518 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_bdefc7775234406788f3f7e220769b43" } modify_external_catalogs: true
I20251029 02:51:54.666227 518 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_bdefc7775234406788f3f7e220769b43" } modify_external_catalogs: true
I20251029 02:51:54.670559 1026 tablet_service.cc:1505] Processing CreateTablet for tablet ba699862e6f646c6beaa39f080d0faba (DEFAULT_TABLE table=default.Table_cd7b2ea5d1b240d087c135cfec7dd0be [id=feb1f56ae44249c6adce65637017bb43]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.671517 520 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_41af15055e73444a9fbe64c4c4f63dba" } new_table_name: "default.Table_3b26d01c7bce4d678c1b65f1753e3662" modify_external_catalogs: true
I20251029 02:51:54.675676 518 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 785733da86ff425a90ccfab2933fc89e
I20251029 02:51:54.676874 739 tablet_service.cc:1552] Processing DeleteTablet for tablet 785733da86ff425a90ccfab2933fc89e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:54 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:54.678114 1042 tablet_service.cc:1505] Processing CreateTablet for tablet 9bb2166a590d4210b02dd779ef6654ad (DEFAULT_TABLE table=default.Table_50252b3b3cc4403da5e259f9f91be915 [id=ff64a323de354792997beb99bae4a52c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.678480 1029 tablet_service.cc:1505] Processing CreateTablet for tablet 1275e192871440408a8345fef9b56260 (DEFAULT_TABLE table=default.Table_cdc08bccd00a493e850b242c3da4b0f3 [id=5c4f8f38dfc748dda3abf03f7882cf35]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.678809 1037 tablet_service.cc:1552] Processing DeleteTablet for tablet fd0f89180aba43eabb20f926abccbaa6 with delete_type TABLET_DATA_DELETED (replaced by tablet 9bb2166a590d4210b02dd779ef6654ad) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:54.678740 1024 tablet_service.cc:1552] Processing DeleteTablet for tablet 6f93812578a940a3b7378015e2a46f95 with delete_type TABLET_DATA_DELETED (replaced by tablet ba699862e6f646c6beaa39f080d0faba) from {username='slave'} at 127.0.0.1:37626
W20251029 02:51:54.684092 989 leader_election.cc:343] T 6dd4696ad4b44b6dab87a205f3f16ebd P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 6dd4696ad4b44b6dab87a205f3f16ebd
W20251029 02:51:54.688602 1172 thread.cc:641] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.599s user 0.045s sys 0.077s
I20251029 02:51:54.689706 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_0badd99ae4d346928005ec415468a5a1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:54.690523 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0badd99ae4d346928005ec415468a5a1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:54.694164 518 master_service.cc:954] ReplaceTablet: received request to replace tablet dbc2092b63e0417997565c8fadb53ae0 from {username='slave'} at 127.0.0.1:35314
W20251029 02:51:54.696434 1172 thread.cc:608] log-alloc [worker] (thread pool) Time spent starting thread: real 0.607s user 0.045s sys 0.078s
I20251029 02:51:54.697216 524 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_4e97f5e298434c16913bf7ca93ca30f5" } modify_external_catalogs: true
I20251029 02:51:54.697434 524 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_4e97f5e298434c16913bf7ca93ca30f5" } modify_external_catalogs: true
I20251029 02:51:54.698379 1172 tablet_bootstrap.cc:492] T d6e1170dbb594a5696a3899cde8da7c9 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:51:54.698611 1172 ts_tablet_manager.cc:1403] T d6e1170dbb594a5696a3899cde8da7c9 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.620s user 0.049s sys 0.078s
I20251029 02:51:54.699620 1172 raft_consensus.cc:359] T d6e1170dbb594a5696a3899cde8da7c9 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:54.699901 1172 raft_consensus.cc:385] T d6e1170dbb594a5696a3899cde8da7c9 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:54.700053 1172 raft_consensus.cc:740] T d6e1170dbb594a5696a3899cde8da7c9 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:51:54.700416 1172 consensus_queue.cc:260] T d6e1170dbb594a5696a3899cde8da7c9 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
W20251029 02:51:54.701107 520 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 42cfdf88472b44979d86836561005ed1 consensus state has no leader
I20251029 02:51:54.701128 1041 tablet_service.cc:1505] Processing CreateTablet for tablet 13ee62fc2339410b9db70dfd3d3c6fa6 (DEFAULT_TABLE table=default.Table_73eec04729fd4870aca2779a5738d772 [id=b54d7930e083434680f3934fc4023653]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.701874 1172 ts_tablet_manager.cc:1434] T d6e1170dbb594a5696a3899cde8da7c9 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251029 02:51:54.702157 878 tablet_service.cc:1505] Processing CreateTablet for tablet 59d03a190b784ab18a3e63a572675891 (DEFAULT_TABLE table=default.Table_9f5273b57e0b4d95af3e737377f031b6 [id=3ede90ae95fb46b2ab910914a49a35a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.703065 1172 tablet_bootstrap.cc:492] T f71388036478404d878a52fc20ece216 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:51:54.703415 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba699862e6f646c6beaa39f080d0faba. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:54.704849 1030 tablet_service.cc:1505] Processing CreateTablet for tablet b6d4f7bf702f4834bc5f4ef2b50b92a7 (DEFAULT_TABLE table=default.Table_c70eb9f284cd49109fd09e430a1cea6c [id=be817eb6ec33406eb316d36575c134fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.705688 989 leader_election.cc:304] T 6dd4696ad4b44b6dab87a205f3f16ebd P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5; no voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d
I20251029 02:51:54.705996 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13ee62fc2339410b9db70dfd3d3c6fa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:54.706836 1172 tablet_bootstrap.cc:654] T f71388036478404d878a52fc20ece216 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:54.706876 1251 raft_consensus.cc:2749] T 6dd4696ad4b44b6dab87a205f3f16ebd P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:51:54.707320 989 leader_election.cc:304] T 4b76c4408a7340e59b8506171622175f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters: c756528642a543eeacd93cef3e335a7d
I20251029 02:51:54.707988 1167 tablet_bootstrap.cc:492] T 6dd4696ad4b44b6dab87a205f3f16ebd P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:51:54.708196 1167 ts_tablet_manager.cc:1403] T 6dd4696ad4b44b6dab87a205f3f16ebd P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.458s user 0.008s sys 0.000s
I20251029 02:51:54.709198 1027 tablet_service.cc:1552] Processing DeleteTablet for tablet 1bab90fe904944af948dab3db06b7e34 with delete_type TABLET_DATA_DELETED (replaced by tablet b6d4f7bf702f4834bc5f4ef2b50b92a7) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:54.709661 1251 raft_consensus.cc:2804] T 4b76c4408a7340e59b8506171622175f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:51:54.709807 1251 raft_consensus.cc:697] T 4b76c4408a7340e59b8506171622175f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Becoming Leader. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Running, Role: LEADER
I20251029 02:51:54.710884 885 tablet_service.cc:1505] Processing CreateTablet for tablet 0d84dc8dc465496f9adec798b2e03e13 (DEFAULT_TABLE table=default.Table_0d5730d385074a70b295b233e6ecd79e [id=423993f75a9948aebe1dee01d3e12702]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.710996 520 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_75341a6f40d84ad3938442923cf3131e" } modify_external_catalogs: true
I20251029 02:51:54.711047 1251 consensus_queue.cc:237] T 4b76c4408a7340e59b8506171622175f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
W20251029 02:51:54.711143 849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:45003, user_credentials={real_user=slave}} blocked reactor thread for 407081us
I20251029 02:51:54.711264 520 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_75341a6f40d84ad3938442923cf3131e" } modify_external_catalogs: true
I20251029 02:51:54.712002 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b867feae24e741aca6902868c35c1428" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:51:54.712549 1060 raft_consensus.cc:3060] T b867feae24e741aca6902868c35c1428 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:51:54.714742 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bb2166a590d4210b02dd779ef6654ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:54.717993 1060 raft_consensus.cc:2468] T b867feae24e741aca6902868c35c1428 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:51:54.727622 524 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 74873a05ef634867a30aa63b31fcfcc7
I20251029 02:51:54.729691 747 tablet_service.cc:1552] Processing DeleteTablet for tablet 74873a05ef634867a30aa63b31fcfcc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:54 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:54.731467 520 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet ccdca31db2834de4b6ed0f2d67a10535
I20251029 02:51:54.732640 747 tablet_service.cc:1552] Processing DeleteTablet for tablet ccdca31db2834de4b6ed0f2d67a10535 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:54 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:54.733716 518 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet dbc2092b63e0417997565c8fadb53ae0
I20251029 02:51:54.734290 518 catalog_manager.cc:6719] ReplaceTablet: tablet dbc2092b63e0417997565c8fadb53ae0 deleted and replaced by tablet 290d9d46914b4f328c7f2d9afe901889
I20251029 02:51:54.734903 756 tablet_service.cc:1552] Processing DeleteTablet for tablet dbc2092b63e0417997565c8fadb53ae0 with delete_type TABLET_DATA_DELETED (replaced by tablet 290d9d46914b4f328c7f2d9afe901889) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:54.735757 747 tablet_service.cc:1505] Processing CreateTablet for tablet 738046ab57334f2dab9f4aab979c5b42 (DEFAULT_TABLE table=default.Table_a8568b9dabb942a9a88179114c8e1240 [id=33b26e2658ff4270a5f1306b6cde34db]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.736233 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 738046ab57334f2dab9f4aab979c5b42. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:54.740958 518 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_a873546f69334d12b0cbecce2cd489fb" } modify_external_catalogs: true
I20251029 02:51:54.741154 518 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_a873546f69334d12b0cbecce2cd489fb" } modify_external_catalogs: true
I20251029 02:51:54.743260 1350 raft_consensus.cc:493] T 61c1fec3ab4c402eb29f767b9bb7409f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:54.743448 1350 raft_consensus.cc:515] T 61c1fec3ab4c402eb29f767b9bb7409f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:54.743762 520 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_3b6a3dbb404a485dac0a7f73fc3685a8" } new_table_name: "default.Table_9120016827e946cc8a145252d2569d2b" modify_external_catalogs: true
W20251029 02:51:54.745098 1166 thread.cc:641] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.552s user 0.036s sys 0.039s
I20251029 02:51:54.745361 1350 leader_election.cc:290] T 61c1fec3ab4c402eb29f767b9bb7409f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:51:54.745504 524 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_e8685346a99d41078ecd38cd4ce4fec5" } modify_external_catalogs: true
I20251029 02:51:54.746460 524 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_e8685346a99d41078ecd38cd4ce4fec5" } modify_external_catalogs: true
I20251029 02:51:54.747133 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1275e192871440408a8345fef9b56260. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:51:54.747819 1166 thread.cc:608] log-alloc [worker] (thread pool) Time spent starting thread: real 0.572s user 0.036s sys 0.039s
I20251029 02:51:54.750417 1166 tablet_bootstrap.cc:492] T 24016ceaa84147eca0b94b3ad288e772 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:51:54.750803 1166 ts_tablet_manager.cc:1403] T 24016ceaa84147eca0b94b3ad288e772 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.588s user 0.039s sys 0.040s
I20251029 02:51:54.747900 526 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_e4bba3a0c06a4ff5bd8829e1081b9c68" } new_table_name: "default.Table_29766c136dbc44e486d4256e7dbba851" modify_external_catalogs: true
I20251029 02:51:54.749835 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_e461f80fc20349ec9ab63da15b9fb436"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:54.753057 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e461f80fc20349ec9ab63da15b9fb436 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:54.758841 883 tablet_service.cc:1552] Processing DeleteTablet for tablet ba014ecfec7f4602ae1c09d931bac824 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:54 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:54.759088 883 tablet_service.cc:1552] Processing DeleteTablet for tablet 6f93812578a940a3b7378015e2a46f95 with delete_type TABLET_DATA_DELETED (replaced by tablet ba699862e6f646c6beaa39f080d0faba) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:54.760215 1167 raft_consensus.cc:359] T 6dd4696ad4b44b6dab87a205f3f16ebd P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:54.760430 1167 raft_consensus.cc:385] T 6dd4696ad4b44b6dab87a205f3f16ebd P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:54.760768 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 190) took 1010 ms. Trace:
I20251029 02:51:54.760843 756 tablet_service.cc:1505] Processing CreateTablet for tablet 061f3d38c4604fa4b57e74ff6f6fc7da (DEFAULT_TABLE table=default.Table_0f40dc35be6d46b3a991e93384fa1e1f [id=7f5f847a87ab464b932a46f3c04e43d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.763388 518 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 290d9d46914b4f328c7f2d9afe901889
I20251029 02:51:54.764166 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 061f3d38c4604fa4b57e74ff6f6fc7da. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:54.768926 739 tablet_service.cc:1505] Processing CreateTablet for tablet 6d830d30b26b4ba799785fec8a6f0769 (DEFAULT_TABLE table=default.Table_19165ff9f7f24a97bf835e72701455d3 [id=ae6032816e6c4b80aa294478cbd49460]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.769503 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d830d30b26b4ba799785fec8a6f0769. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:54.770539 893 tablet_service.cc:1505] Processing CreateTablet for tablet 9bb2166a590d4210b02dd779ef6654ad (DEFAULT_TABLE table=default.Table_50252b3b3cc4403da5e259f9f91be915 [id=ff64a323de354792997beb99bae4a52c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.770566 890 tablet_service.cc:1552] Processing DeleteTablet for tablet 6cd7009e06a646acad0802918c2b425e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:54 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:54.771075 890 tablet_service.cc:1505] Processing CreateTablet for tablet ba699862e6f646c6beaa39f080d0faba (DEFAULT_TABLE table=default.Table_cd7b2ea5d1b240d087c135cfec7dd0be [id=feb1f56ae44249c6adce65637017bb43]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.770592 883 tablet_service.cc:1505] Processing CreateTablet for tablet 1275e192871440408a8345fef9b56260 (DEFAULT_TABLE table=default.Table_cdc08bccd00a493e850b242c3da4b0f3 [id=5c4f8f38dfc748dda3abf03f7882cf35]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.772923 895 tablet_service.cc:1552] Processing DeleteTablet for tablet b5d5b737c86844d58c51b2388c0b1705 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:54 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:54.773154 895 tablet_service.cc:1552] Processing DeleteTablet for tablet 1bab90fe904944af948dab3db06b7e34 with delete_type TABLET_DATA_DELETED (replaced by tablet b6d4f7bf702f4834bc5f4ef2b50b92a7) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:54.773524 895 tablet_service.cc:1505] Processing CreateTablet for tablet 13ee62fc2339410b9db70dfd3d3c6fa6 (DEFAULT_TABLE table=default.Table_73eec04729fd4870aca2779a5738d772 [id=b54d7930e083434680f3934fc4023653]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.773857 900 tablet_service.cc:1552] Processing DeleteTablet for tablet fd0f89180aba43eabb20f926abccbaa6 with delete_type TABLET_DATA_DELETED (replaced by tablet 9bb2166a590d4210b02dd779ef6654ad) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:54.774194 900 tablet_service.cc:1505] Processing CreateTablet for tablet b6d4f7bf702f4834bc5f4ef2b50b92a7 (DEFAULT_TABLE table=default.Table_c70eb9f284cd49109fd09e430a1cea6c [id=be817eb6ec33406eb316d36575c134fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.774688 849 leader_election.cc:304] T 6c32c6f5e1cc40a0a12688201cac62a7 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters: b495a49349324f7d81908f2da700f0d9
W20251029 02:51:54.775106 849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:45003, user_credentials={real_user=slave}} blocked reactor thread for 62672.4us
I20251029 02:51:54.775759 849 leader_election.cc:304] T b867feae24e741aca6902868c35c1428 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters: b495a49349324f7d81908f2da700f0d9
I20251029 02:51:54.776809 1167 raft_consensus.cc:740] T 6dd4696ad4b44b6dab87a205f3f16ebd P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:51:54.776773 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "61c1fec3ab4c402eb29f767b9bb7409f" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:51:54.777076 1167 consensus_queue.cc:260] T 6dd4696ad4b44b6dab87a205f3f16ebd P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:54.777232 882 tablet_service.cc:1505] Processing CreateTablet for tablet 49d9ce32e4f04f11be2ecefc5d523a8b (DEFAULT_TABLE table=default.Table_af1ab75f523d41338d4471824997f935 [id=468c76c8c99a4aa59c8b1f6c8e4bfa30]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.777770 1167 ts_tablet_manager.cc:1434] T 6dd4696ad4b44b6dab87a205f3f16ebd P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.069s user 0.001s sys 0.001s
I20251029 02:51:54.777951 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ccdca31db2834de4b6ed0f2d67a10535. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:54.778235 1167 tablet_bootstrap.cc:492] T 61c1fec3ab4c402eb29f767b9bb7409f P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:51:54.778699 1308 raft_consensus.cc:2804] T 6c32c6f5e1cc40a0a12688201cac62a7 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:51:54.779066 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9bb2166a590d4210b02dd779ef6654ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:54.780113 1167 tablet_bootstrap.cc:654] T 61c1fec3ab4c402eb29f767b9bb7409f P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:54.781487 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6d4f7bf702f4834bc5f4ef2b50b92a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:54.785002 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_f5db3e3913ca4ea98b02981cd04a5965"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:54.785581 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f5db3e3913ca4ea98b02981cd04a5965 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:51:54.789389 520 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 44b518c3a0964a6fa863b2784d80c8a4 consensus state has no leader
I20251029 02:51:54.789587 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba699862e6f646c6beaa39f080d0faba. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:54.793920 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1275e192871440408a8345fef9b56260. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:54.794895 747 tablet_service.cc:1505] Processing CreateTablet for tablet e7f846b294cc4eb687a26c8b5931ac19 (DEFAULT_TABLE table=default.Table_049332132e3142ca86d49635656135f5 [id=74c72bc1206b4247b1b1c994ccc402f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.795533 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7f846b294cc4eb687a26c8b5931ac19. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:51:54.800025 526 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dd6ad641fbda472fab07aa35c5e4523a consensus state has no leader
I20251029 02:51:54.798501 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d61beda662d844cdaf67fd9a2e382ea2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:54.800391 1171 raft_consensus.cc:2804] T b867feae24e741aca6902868c35c1428 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:51:54.805596 524 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 576c4e6e70e24b0399ac9597a30f0569
I20251029 02:51:54.807377 756 tablet_service.cc:1552] Processing DeleteTablet for tablet 576c4e6e70e24b0399ac9597a30f0569 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:54 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:54.808272 898 tablet_service.cc:1505] Processing CreateTablet for tablet 2e97d1bc06a24e8d8ba11acb6fd280d8 (DEFAULT_TABLE table=default.Table_24be2b894899463a810d2fe7511d4d6b [id=43e922b1efe44ad9882b5084b22d1560]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.808706 1308 raft_consensus.cc:697] T 6c32c6f5e1cc40a0a12688201cac62a7 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:51:54.769074 1174 rpcz_store.cc:276] 1029 02:51:53.584537 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:53.597203 (+ 12666us) service_pool.cc:225] Handling call
1029 02:51:54.035762 (+438559us) ts_tablet_manager.cc:1151] Deleting tablet 0dd31dc5873d407d958a0f5199f78782
1029 02:51:54.035769 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:54.409927 (+374158us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:54.527553 (+117626us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:54.595109 (+ 67556us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":437691}
I20251029 02:51:54.814236 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13ee62fc2339410b9db70dfd3d3c6fa6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:54.815024 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 0dd31dc5873d407d958a0f5199f78782 (table default.Table_c60a78e1cea24e8e864803a5c6124df1 [id=99445ef970c644f8a39388b2015da927]) successfully deleted
I20251029 02:51:54.815922 1172 tablet_bootstrap.cc:492] T f71388036478404d878a52fc20ece216 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:51:54.816102 1172 ts_tablet_manager.cc:1403] T f71388036478404d878a52fc20ece216 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.113s user 0.012s sys 0.015s
I20251029 02:51:54.817019 1172 raft_consensus.cc:359] T f71388036478404d878a52fc20ece216 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:54.817286 1172 raft_consensus.cc:385] T f71388036478404d878a52fc20ece216 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:54.817363 1172 raft_consensus.cc:740] T f71388036478404d878a52fc20ece216 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:51:54.817659 1172 consensus_queue.cc:260] T f71388036478404d878a52fc20ece216 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:54.818233 1172 ts_tablet_manager.cc:1434] T f71388036478404d878a52fc20ece216 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 02:51:54.819233 1172 tablet_bootstrap.cc:492] T 59af906061d049c99ae2153945597007 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:51:54.821440 1172 tablet_bootstrap.cc:654] T 59af906061d049c99ae2153945597007 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:54.827600 1308 consensus_queue.cc:237] T 6c32c6f5e1cc40a0a12688201cac62a7 P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:54.834815 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 196) took 1176 ms. Trace:
I20251029 02:51:54.834945 1174 rpcz_store.cc:276] 1029 02:51:53.651403 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:53.651789 (+ 386us) service_pool.cc:225] Handling call
1029 02:51:54.827705 (+1175916us) ts_tablet_manager.cc:1151] Deleting tablet f81aecb20d6e498bb9b498698886d65e
1029 02:51:54.827718 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:54.827832 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1098302}
W20251029 02:51:54.835705 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet f81aecb20d6e498bb9b498698886d65e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f81aecb20d6e498bb9b498698886d65e already in progress: creating tablet
I20251029 02:51:54.836710 1174 tablet_replica.cc:333] T bf210435010d4dde8903fd116eb468e3 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:51:54.837201 1172 tablet_bootstrap.cc:492] T 59af906061d049c99ae2153945597007 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:51:54.837265 893 tablet_service.cc:1505] Processing CreateTablet for tablet 30d03e1e42d34242bd354f9e9ec4f127 (DEFAULT_TABLE table=default.Table_435f2dfe42134760abc9e8880fa19f32 [id=c6e1c357fbfc49be972748da564cc9a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.837648 1172 ts_tablet_manager.cc:1403] T 59af906061d049c99ae2153945597007 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.007s
I20251029 02:51:54.837806 1174 raft_consensus.cc:2243] T bf210435010d4dde8903fd116eb468e3 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus shutting down.
W20251029 02:51:54.837857 847 consensus_peers.cc:597] T cb55b3d5bc63405f96cdc122f5a612cb P c756528642a543eeacd93cef3e335a7d -> Peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Couldn't send request to peer b495a49349324f7d81908f2da700f0d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cb55b3d5bc63405f96cdc122f5a612cb. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:51:54.838891 1172 raft_consensus.cc:359] T 59af906061d049c99ae2153945597007 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:54.839269 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49d9ce32e4f04f11be2ecefc5d523a8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:54.839262 1172 raft_consensus.cc:385] T 59af906061d049c99ae2153945597007 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:54.839491 1172 raft_consensus.cc:740] T 59af906061d049c99ae2153945597007 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:51:54.839942 1172 consensus_queue.cc:260] T 59af906061d049c99ae2153945597007 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:54.843926 1174 raft_consensus.cc:2272] T bf210435010d4dde8903fd116eb468e3 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:54.846037 1172 ts_tablet_manager.cc:1434] T 59af906061d049c99ae2153945597007 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20251029 02:51:54.850371 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59d03a190b784ab18a3e63a572675891. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:54.850622 1174 ts_tablet_manager.cc:1916] T bf210435010d4dde8903fd116eb468e3 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:54.851346 1172 tablet_bootstrap.cc:492] T 6c22497760274380a832feff7c5a2fae P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:51:54.853739 1172 tablet_bootstrap.cc:654] T 6c22497760274380a832feff7c5a2fae P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:54.860337 1172 tablet_bootstrap.cc:492] T 6c22497760274380a832feff7c5a2fae P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:51:54.861140 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f80b840ed954230bfbfe7e40127cd2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:54.861173 1172 ts_tablet_manager.cc:1403] T 6c22497760274380a832feff7c5a2fae P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.004s
I20251029 02:51:54.862385 900 tablet_service.cc:1505] Processing CreateTablet for tablet 9269622ca92645d5aa8e73341201ccbb (DEFAULT_TABLE table=default.Table_4e701fdc7b034edc80ccc5ae0c9c4439 [id=d0ca7c93c8cf4dc19962a1957321c5b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.863056 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9269622ca92645d5aa8e73341201ccbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:54.863667 1172 raft_consensus.cc:359] T 6c22497760274380a832feff7c5a2fae P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:54.863962 1172 raft_consensus.cc:385] T 6c22497760274380a832feff7c5a2fae P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:54.864061 1172 raft_consensus.cc:740] T 6c22497760274380a832feff7c5a2fae P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:51:54.865185 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e97d1bc06a24e8d8ba11acb6fd280d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:54.865115 1172 consensus_queue.cc:260] T 6c22497760274380a832feff7c5a2fae P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:54.867393 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30d03e1e42d34242bd354f9e9ec4f127. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:54.868160 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d84dc8dc465496f9adec798b2e03e13. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:54.871923 1172 ts_tablet_manager.cc:1434] T 6c22497760274380a832feff7c5a2fae P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20251029 02:51:54.876895 1175 ts_tablet_manager.cc:1929] T c4884e22cf2044a4a1070c1b4b759d9e P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:54.877077 1175 log.cc:1199] T c4884e22cf2044a4a1070c1b4b759d9e P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/c4884e22cf2044a4a1070c1b4b759d9e
I20251029 02:51:54.877228 879 tablet_service.cc:1552] Processing DeleteTablet for tablet 785733da86ff425a90ccfab2933fc89e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:54 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:54.877759 879 tablet_service.cc:1505] Processing CreateTablet for tablet 738046ab57334f2dab9f4aab979c5b42 (DEFAULT_TABLE table=default.Table_a8568b9dabb942a9a88179114c8e1240 [id=33b26e2658ff4270a5f1306b6cde34db]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.878050 1172 tablet_bootstrap.cc:492] T 8d9722c764ac4b27a540595e9c6aafcd P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:51:54.880957 887 tablet_service.cc:1552] Processing DeleteTablet for tablet 74873a05ef634867a30aa63b31fcfcc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:54 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:54.881078 1172 tablet_bootstrap.cc:654] T 8d9722c764ac4b27a540595e9c6aafcd P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:54.881852 887 tablet_service.cc:1552] Processing DeleteTablet for tablet ccdca31db2834de4b6ed0f2d67a10535 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:54 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:54.882114 887 tablet_service.cc:1552] Processing DeleteTablet for tablet dbc2092b63e0417997565c8fadb53ae0 with delete_type TABLET_DATA_DELETED (replaced by tablet 290d9d46914b4f328c7f2d9afe901889) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:54.883342 1171 raft_consensus.cc:697] T b867feae24e741aca6902868c35c1428 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:51:54.884649 1175 ts_tablet_manager.cc:1950] T c4884e22cf2044a4a1070c1b4b759d9e P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:51:54.885622 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 191) took 1225 ms. Trace:
I20251029 02:51:54.885767 1175 rpcz_store.cc:276] 1029 02:51:53.660034 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:53.660925 (+ 891us) service_pool.cc:225] Handling call
1029 02:51:54.183237 (+522312us) ts_tablet_manager.cc:1151] Deleting tablet c4884e22cf2044a4a1070c1b4b759d9e
1029 02:51:54.183247 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:54.766002 (+582755us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:54.876740 (+110738us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:54.885556 (+ 8816us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1839,"tablet-delete.queue_time_us":515783}
I20251029 02:51:54.887367 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 214) took 1123 ms. Trace:
I20251029 02:51:54.887358 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet c4884e22cf2044a4a1070c1b4b759d9e (table default.Table_da0be17b8bdc4e579b6c3faff71d9c98 [id=6e06ac70c6814e7d89a008bbaddccc7e]) successfully deleted
I20251029 02:51:54.887491 1175 rpcz_store.cc:276] 1029 02:51:53.764204 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:53.856944 (+ 92740us) service_pool.cc:225] Handling call
1029 02:51:54.887163 (+1030219us) ts_tablet_manager.cc:1151] Deleting tablet b2b2d64dcf964da8be4b544dbbe31977
1029 02:51:54.887176 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:54.887302 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1029982}
W20251029 02:51:54.888054 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet b2b2d64dcf964da8be4b544dbbe31977 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b2b2d64dcf964da8be4b544dbbe31977 already in progress: creating tablet
I20251029 02:51:54.888113 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 219) took 1001 ms. Trace:
I20251029 02:51:54.888219 1175 rpcz_store.cc:276] 1029 02:51:53.886741 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:53.892963 (+ 6222us) service_pool.cc:225] Handling call
1029 02:51:54.887926 (+994963us) ts_tablet_manager.cc:1151] Deleting tablet 56d05e4a500043c995bb51a204cb295d
1029 02:51:54.887937 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:54.888050 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":984730}
W20251029 02:51:54.888765 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 56d05e4a500043c995bb51a204cb295d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 56d05e4a500043c995bb51a204cb295d already in progress: creating tablet
W20251029 02:51:54.889039 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 4add552cf4d940d2a0a36d6b58379a34 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4add552cf4d940d2a0a36d6b58379a34 already in progress: creating tablet
I20251029 02:51:54.890554 1172 tablet_bootstrap.cc:492] T 8d9722c764ac4b27a540595e9c6aafcd P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:51:54.890806 1172 ts_tablet_manager.cc:1403] T 8d9722c764ac4b27a540595e9c6aafcd P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.001s
I20251029 02:51:54.891673 1172 raft_consensus.cc:359] T 8d9722c764ac4b27a540595e9c6aafcd P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:54.891865 1172 raft_consensus.cc:385] T 8d9722c764ac4b27a540595e9c6aafcd P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:54.891957 1172 raft_consensus.cc:740] T 8d9722c764ac4b27a540595e9c6aafcd P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
W20251029 02:51:54.893891 989 leader_election.cc:343] T 61c1fec3ab4c402eb29f767b9bb7409f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 61c1fec3ab4c402eb29f767b9bb7409f
I20251029 02:51:54.894277 1172 consensus_queue.cc:260] T 8d9722c764ac4b27a540595e9c6aafcd P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
W20251029 02:51:54.895431 990 leader_election.cc:343] T 61c1fec3ab4c402eb29f767b9bb7409f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:54.895597 990 leader_election.cc:304] T 61c1fec3ab4c402eb29f767b9bb7409f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5; no voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d
I20251029 02:51:54.895963 1350 raft_consensus.cc:2749] T 61c1fec3ab4c402eb29f767b9bb7409f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:51:54.897156 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6d4f7bf702f4834bc5f4ef2b50b92a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:54.902498 519 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet fa5127bcfcda40a7b5ad4ef016af072a (table default.Table_762d41caedac41b09d445c6bfea97e75 [id=ae5d56d40fa941a68f2b10d5e6e9ac83]). Expected version 1 got 0
I20251029 02:51:54.902860 1172 ts_tablet_manager.cc:1434] T 8d9722c764ac4b27a540595e9c6aafcd P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20251029 02:51:54.903074 519 catalog_manager.cc:5649] T 70e8cfe5eaa9492c85874dc3214a1e0b P b495a49349324f7d81908f2da700f0d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to b495a49349324f7d81908f2da700f0d9 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "b495a49349324f7d81908f2da700f0d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } health_report { overall_health: HEALTHY } } }
I20251029 02:51:54.904498 747 tablet_service.cc:1505] Processing CreateTablet for tablet 28f7ceb91a34450cad4b5b45f5285b41 (DEFAULT_TABLE table=default.Table_e461f80fc20349ec9ab63da15b9fb436 [id=89406c2781fd4674b634e3b1dbd06dc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.905004 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28f7ceb91a34450cad4b5b45f5285b41. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:51:54.905674 512 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 42cfdf88472b44979d86836561005ed1 (table default.Table_3b26d01c7bce4d678c1b65f1753e3662 [id=5e0c72957b9a491d960b33d3fffd8f70], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 42cfdf88472b44979d86836561005ed1 consensus state has no leader
I20251029 02:51:54.908013 1308 consensus_queue.cc:1048] T cb55b3d5bc63405f96cdc122f5a612cb P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251029 02:51:54.914633 989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:38481, user_credentials={real_user=slave}} blocked reactor thread for 140292us
I20251029 02:51:54.917071 519 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet 59af906061d049c99ae2153945597007 (table default.Table_3bc6a1815ce646fb9a89e91fdda46ae0 [id=25f16d95843d41c981d6a07ec0d4b48a]). Expected version 1 got 0
I20251029 02:51:54.918646 1172 tablet_bootstrap.cc:492] T c519832030884a049839d309cc738c28 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:51:54.918726 756 tablet_service.cc:1552] Processing DeleteTablet for tablet f71388036478404d878a52fc20ece216 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:52 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:54.919275 756 tablet_service.cc:1552] Processing DeleteTablet for tablet d6e1170dbb594a5696a3899cde8da7c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:52 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:54.919603 756 tablet_service.cc:1552] Processing DeleteTablet for tablet 6c22497760274380a832feff7c5a2fae with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:53 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:54.920256 747 tablet_service.cc:1552] Processing DeleteTablet for tablet 8d9722c764ac4b27a540595e9c6aafcd with delete_type TABLET_DATA_DELETED (replaced by tablet 51effc9324034ae79e2b25cb57a996c1) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:54.922209 1172 tablet_bootstrap.cc:654] T c519832030884a049839d309cc738c28 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:54.924150 890 tablet_service.cc:1505] Processing CreateTablet for tablet e7f846b294cc4eb687a26c8b5931ac19 (DEFAULT_TABLE table=default.Table_049332132e3142ca86d49635656135f5 [id=74c72bc1206b4247b1b1c994ccc402f8]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 02:51:54.924536 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 0cc1957a845f4e9cb703a553b27d1347 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0cc1957a845f4e9cb703a553b27d1347 already in progress: creating tablet
W20251029 02:51:54.924800 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 519754f5d8f7484480f973c53708c5f3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 519754f5d8f7484480f973c53708c5f3 already in progress: creating tablet
I20251029 02:51:54.924876 1172 tablet_bootstrap.cc:492] T c519832030884a049839d309cc738c28 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:51:54.925021 1172 ts_tablet_manager.cc:1403] T c519832030884a049839d309cc738c28 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.001s
W20251029 02:51:54.925015 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet f60e7c24982d40319801869ad060864d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f60e7c24982d40319801869ad060864d already in progress: creating tablet
I20251029 02:51:54.925750 1172 raft_consensus.cc:359] T c519832030884a049839d309cc738c28 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:54.925966 1172 raft_consensus.cc:385] T c519832030884a049839d309cc738c28 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:54.926043 1172 raft_consensus.cc:740] T c519832030884a049839d309cc738c28 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:51:54.926276 1172 consensus_queue.cc:260] T c519832030884a049839d309cc738c28 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
W20251029 02:51:54.926644 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 392a36a6b59a42f78e3ac7c8fb63214b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 392a36a6b59a42f78e3ac7c8fb63214b already in progress: creating tablet
I20251029 02:51:54.927265 1175 tablet_replica.cc:333] T ba014ecfec7f4602ae1c09d931bac824 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:51:54.927522 1175 raft_consensus.cc:2243] T ba014ecfec7f4602ae1c09d931bac824 P b495a49349324f7d81908f2da700f0d9 [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:51:54.927789 887 tablet_service.cc:1505] Processing CreateTablet for tablet 061f3d38c4604fa4b57e74ff6f6fc7da (DEFAULT_TABLE table=default.Table_0f40dc35be6d46b3a991e93384fa1e1f [id=7f5f847a87ab464b932a46f3c04e43d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.930033 1172 ts_tablet_manager.cc:1434] T c519832030884a049839d309cc738c28 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20251029 02:51:54.931123 1172 tablet_bootstrap.cc:492] T 75236b2fa82f4f28b82efae9a2c5160d P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:51:54.931793 1350 consensus_queue.cc:1048] T 11f4252b5e694312aaa142468f8e68b4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
I20251029 02:51:54.932843 1172 tablet_bootstrap.cc:654] T 75236b2fa82f4f28b82efae9a2c5160d P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:54.937162 1175 raft_consensus.cc:2272] T ba014ecfec7f4602ae1c09d931bac824 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:54.937860 1175 ts_tablet_manager.cc:1916] T ba014ecfec7f4602ae1c09d931bac824 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:54.944780 747 tablet_service.cc:1552] Processing DeleteTablet for tablet c519832030884a049839d309cc738c28 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:52 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:54.944988 756 tablet_service.cc:1505] Processing CreateTablet for tablet 8048506ced4540a5aee9f79bfc4599bf (DEFAULT_TABLE table=default.Table_0badd99ae4d346928005ec415468a5a1 [id=a047685192f34bf1a1f3d1361cdaf272]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.945446 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8048506ced4540a5aee9f79bfc4599bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:54.945778 1041 tablet_service.cc:1552] Processing DeleteTablet for tablet 785733da86ff425a90ccfab2933fc89e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:54 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:54.950363 1027 tablet_service.cc:1505] Processing CreateTablet for tablet 2e97d1bc06a24e8d8ba11acb6fd280d8 (DEFAULT_TABLE table=default.Table_24be2b894899463a810d2fe7511d4d6b [id=43e922b1efe44ad9882b5084b22d1560]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.951102 1024 tablet_service.cc:1505] Processing CreateTablet for tablet 738046ab57334f2dab9f4aab979c5b42 (DEFAULT_TABLE table=default.Table_a8568b9dabb942a9a88179114c8e1240 [id=33b26e2658ff4270a5f1306b6cde34db]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.950732 1171 consensus_queue.cc:237] T b867feae24e741aca6902868c35c1428 P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:54.953373 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet 74873a05ef634867a30aa63b31fcfcc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:54 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:54.951821 1041 tablet_service.cc:1505] Processing CreateTablet for tablet 9269622ca92645d5aa8e73341201ccbb (DEFAULT_TABLE table=default.Table_4e701fdc7b034edc80ccc5ae0c9c4439 [id=d0ca7c93c8cf4dc19962a1957321c5b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.954006 883 tablet_service.cc:1505] Processing CreateTablet for tablet 6d830d30b26b4ba799785fec8a6f0769 (DEFAULT_TABLE table=default.Table_19165ff9f7f24a97bf835e72701455d3 [id=ae6032816e6c4b80aa294478cbd49460]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.955103 1037 tablet_service.cc:1505] Processing CreateTablet for tablet 061f3d38c4604fa4b57e74ff6f6fc7da (DEFAULT_TABLE table=default.Table_0f40dc35be6d46b3a991e93384fa1e1f [id=7f5f847a87ab464b932a46f3c04e43d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.955546 1025 tablet_service.cc:1552] Processing DeleteTablet for tablet dbc2092b63e0417997565c8fadb53ae0 with delete_type TABLET_DATA_DELETED (replaced by tablet 290d9d46914b4f328c7f2d9afe901889) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:54.955865 1308 raft_consensus.cc:493] T 6dd4696ad4b44b6dab87a205f3f16ebd P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:54.956061 1308 raft_consensus.cc:515] T 6dd4696ad4b44b6dab87a205f3f16ebd P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:54.957192 1172 tablet_bootstrap.cc:492] T 75236b2fa82f4f28b82efae9a2c5160d P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:51:54.957358 1172 ts_tablet_manager.cc:1403] T 75236b2fa82f4f28b82efae9a2c5160d P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.026s user 0.001s sys 0.006s
I20251029 02:51:54.957955 1059 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dd4696ad4b44b6dab87a205f3f16ebd" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:51:54.958305 1026 tablet_service.cc:1552] Processing DeleteTablet for tablet ccdca31db2834de4b6ed0f2d67a10535 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:54 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:54.958320 1172 raft_consensus.cc:359] T 75236b2fa82f4f28b82efae9a2c5160d P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:54.958536 1172 raft_consensus.cc:385] T 75236b2fa82f4f28b82efae9a2c5160d P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:54.958678 1172 raft_consensus.cc:740] T 75236b2fa82f4f28b82efae9a2c5160d P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:51:54.958940 1172 consensus_queue.cc:260] T 75236b2fa82f4f28b82efae9a2c5160d P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:54.959352 1059 raft_consensus.cc:2468] T 6dd4696ad4b44b6dab87a205f3f16ebd P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:51:54.960157 1025 tablet_service.cc:1505] Processing CreateTablet for tablet 6d830d30b26b4ba799785fec8a6f0769 (DEFAULT_TABLE table=default.Table_19165ff9f7f24a97bf835e72701455d3 [id=ae6032816e6c4b80aa294478cbd49460]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.960552 1172 ts_tablet_manager.cc:1434] T 75236b2fa82f4f28b82efae9a2c5160d P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20251029 02:51:54.961524 1039 tablet_service.cc:1505] Processing CreateTablet for tablet 28f7ceb91a34450cad4b5b45f5285b41 (DEFAULT_TABLE table=default.Table_e461f80fc20349ec9ab63da15b9fb436 [id=89406c2781fd4674b634e3b1dbd06dc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.962395 1036 tablet_service.cc:1505] Processing CreateTablet for tablet 8048506ced4540a5aee9f79bfc4599bf (DEFAULT_TABLE table=default.Table_0badd99ae4d346928005ec415468a5a1 [id=a047685192f34bf1a1f3d1361cdaf272]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:54.962903 1172 tablet_bootstrap.cc:492] T 14698af6348f41e384f10569392d9b23 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:51:54.964637 1172 tablet_bootstrap.cc:654] T 14698af6348f41e384f10569392d9b23 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:54.964462 1031 tablet_service.cc:1552] Processing DeleteTablet for tablet 576c4e6e70e24b0399ac9597a30f0569 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:54 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:54.966307 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_bc32d7beda4f4957817b20124bf48e9c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:54.966929 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bc32d7beda4f4957817b20124bf48e9c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:54.967759 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_ed30c800e4eb4970a4b03299c936e7ed"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:54.968367 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ed30c800e4eb4970a4b03299c936e7ed in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:54.955464 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e97d1bc06a24e8d8ba11acb6fd280d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:54.971163 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 738046ab57334f2dab9f4aab979c5b42. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:54.972649 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28f7ceb91a34450cad4b5b45f5285b41. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:54.973310 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_b6110a49346640fb8926a871940e1aad"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:54.974849 847 leader_election.cc:343] T 6dd4696ad4b44b6dab87a205f3f16ebd P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 6dd4696ad4b44b6dab87a205f3f16ebd
I20251029 02:51:54.975004 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8048506ced4540a5aee9f79bfc4599bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:54.978794 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 061f3d38c4604fa4b57e74ff6f6fc7da. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:54.981035 1175 ts_tablet_manager.cc:1929] T ba014ecfec7f4602ae1c09d931bac824 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251029 02:51:54.981194 1175 log.cc:1199] T ba014ecfec7f4602ae1c09d931bac824 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/ba014ecfec7f4602ae1c09d931bac824
I20251029 02:51:54.981427 1308 leader_election.cc:290] T 6dd4696ad4b44b6dab87a205f3f16ebd P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:51:54.982544 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 738046ab57334f2dab9f4aab979c5b42. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:51:54.982522 711 consensus_peers.cc:597] T 70e8cfe5eaa9492c85874dc3214a1e0b P b495a49349324f7d81908f2da700f0d9 -> Peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Couldn't send request to peer c756528642a543eeacd93cef3e335a7d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:51:54.983388 1175 ts_tablet_manager.cc:1950] T ba014ecfec7f4602ae1c09d931bac824 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:51:54.993527 1175 tablet_replica.cc:333] T 6f93812578a940a3b7378015e2a46f95 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:51:54.994108 1175 raft_consensus.cc:2243] T 6f93812578a940a3b7378015e2a46f95 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:54.994318 1175 raft_consensus.cc:2272] T 6f93812578a940a3b7378015e2a46f95 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:54.995024 1175 ts_tablet_manager.cc:1916] T 6f93812578a940a3b7378015e2a46f95 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:54.995041 849 leader_election.cc:304] T 6dd4696ad4b44b6dab87a205f3f16ebd P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters: b495a49349324f7d81908f2da700f0d9
I20251029 02:51:54.996011 882 tablet_service.cc:1505] Processing CreateTablet for tablet 28f7ceb91a34450cad4b5b45f5285b41 (DEFAULT_TABLE table=default.Table_e461f80fc20349ec9ab63da15b9fb436 [id=89406c2781fd4674b634e3b1dbd06dc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:55.000204 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9269622ca92645d5aa8e73341201ccbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:55.001088 1308 raft_consensus.cc:2804] T 6dd4696ad4b44b6dab87a205f3f16ebd P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:51:55.001389 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d830d30b26b4ba799785fec8a6f0769. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:55.010746 1028 tablet_service.cc:1505] Processing CreateTablet for tablet e7f846b294cc4eb687a26c8b5931ac19 (DEFAULT_TABLE table=default.Table_049332132e3142ca86d49635656135f5 [id=74c72bc1206b4247b1b1c994ccc402f8]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 02:51:55.011214 1426 log.cc:927] Time spent T 6c32c6f5e1cc40a0a12688201cac62a7 P c756528642a543eeacd93cef3e335a7d: Append to log took a long time: real 0.052s user 0.000s sys 0.000s
I20251029 02:51:55.019398 520 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_cdc08bccd00a493e850b242c3da4b0f3" } new_table_name: "default.Table_5265805d6c3746209f9466df88b164b9" modify_external_catalogs: true
I20251029 02:51:55.019640 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_0ecb5a1d51bb40d79afa038e4f2aa4d2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:55.019820 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_d4caf12f7662463c824210f3c9cd7e0f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:55.020179 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_6896073ee0134bfa8c8075ef5ae1a01c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:55.020752 517 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_dea0766540d24b5e9bd6cc8b6338b47c" } modify_external_catalogs: true
I20251029 02:51:55.019886 525 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_762d41caedac41b09d445c6bfea97e75" } new_table_name: "default.Table_5113d1a33e8e4bd2af5bc51e689f304f" modify_external_catalogs: true
I20251029 02:51:55.010428 1175 ts_tablet_manager.cc:1929] T 6f93812578a940a3b7378015e2a46f95 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:55.022495 1175 log.cc:1199] T 6f93812578a940a3b7378015e2a46f95 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/6f93812578a940a3b7378015e2a46f95
I20251029 02:51:55.022949 1175 ts_tablet_manager.cc:1950] T 6f93812578a940a3b7378015e2a46f95 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:51:55.028599 1350 consensus_queue.cc:1048] T 11f4252b5e694312aaa142468f8e68b4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:51:55.030390 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7f846b294cc4eb687a26c8b5931ac19. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:55.033756 1175 tablet_replica.cc:333] T f71388036478404d878a52fc20ece216 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:51:55.034013 1175 raft_consensus.cc:2243] T f71388036478404d878a52fc20ece216 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:55.034248 1175 raft_consensus.cc:2272] T f71388036478404d878a52fc20ece216 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:55.035995 1175 ts_tablet_manager.cc:1916] T f71388036478404d878a52fc20ece216 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:55.036073 881 tablet_service.cc:1505] Processing CreateTablet for tablet 8048506ced4540a5aee9f79bfc4599bf (DEFAULT_TABLE table=default.Table_0badd99ae4d346928005ec415468a5a1 [id=a047685192f34bf1a1f3d1361cdaf272]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:55.036504 1308 raft_consensus.cc:493] T 6dd4696ad4b44b6dab87a205f3f16ebd P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:51:55.036631 1308 raft_consensus.cc:3060] T 6dd4696ad4b44b6dab87a205f3f16ebd P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:51:55.037195 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7f846b294cc4eb687a26c8b5931ac19. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:55.038122 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 061f3d38c4604fa4b57e74ff6f6fc7da. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:55.038578 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8048506ced4540a5aee9f79bfc4599bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:55.040035 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28f7ceb91a34450cad4b5b45f5285b41. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:55.047132 1308 raft_consensus.cc:515] T 6dd4696ad4b44b6dab87a205f3f16ebd P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
W20251029 02:51:55.049057 847 leader_election.cc:343] T 6dd4696ad4b44b6dab87a205f3f16ebd P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 6dd4696ad4b44b6dab87a205f3f16ebd
I20251029 02:51:55.049261 1308 leader_election.cc:290] T 6dd4696ad4b44b6dab87a205f3f16ebd P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:51:55.050139 1059 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dd4696ad4b44b6dab87a205f3f16ebd" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:51:55.050369 1059 raft_consensus.cc:3060] T 6dd4696ad4b44b6dab87a205f3f16ebd P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:51:55.051829 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d830d30b26b4ba799785fec8a6f0769. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:55.051208 886 tablet_service.cc:1552] Processing DeleteTablet for tablet 576c4e6e70e24b0399ac9597a30f0569 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:54 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:55.053972 1059 raft_consensus.cc:2468] T 6dd4696ad4b44b6dab87a205f3f16ebd P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:51:55.054872 849 leader_election.cc:304] T 6dd4696ad4b44b6dab87a205f3f16ebd P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters: b495a49349324f7d81908f2da700f0d9
I20251029 02:51:55.055595 1308 raft_consensus.cc:2804] T 6dd4696ad4b44b6dab87a205f3f16ebd P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:51:55.055879 1308 raft_consensus.cc:697] T 6dd4696ad4b44b6dab87a205f3f16ebd P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:51:55.056416 1308 consensus_queue.cc:237] T 6dd4696ad4b44b6dab87a205f3f16ebd P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:55.057971 1394 raft_consensus.cc:493] T 75236b2fa82f4f28b82efae9a2c5160d P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:55.058221 1394 raft_consensus.cc:515] T 75236b2fa82f4f28b82efae9a2c5160d P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:55.059022 1394 leader_election.cc:290] T 75236b2fa82f4f28b82efae9a2c5160d P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:51:55.059788 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75236b2fa82f4f28b82efae9a2c5160d" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:51:55.060415 711 leader_election.cc:343] T 75236b2fa82f4f28b82efae9a2c5160d P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:55.067391 517 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_dea0766540d24b5e9bd6cc8b6338b47c" } modify_external_catalogs: true
W20251029 02:51:55.071046 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0ecb5a1d51bb40d79afa038e4f2aa4d2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:51:55.072235 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b6110a49346640fb8926a871940e1aad in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:55.072316 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_341f16ec415a43b7a29d194cef0ae86b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:55.079792 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d4caf12f7662463c824210f3c9cd7e0f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:54.984633 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet ba014ecfec7f4602ae1c09d931bac824 (table default.Table_64e512f8ce034224aea9b887ed0d09dc [id=7bb8b246fbf1431bb71bbb7c5520e301]) successfully deleted
W20251029 02:51:55.083334 511 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.1:38481, user_credentials={real_user=slave}} blocked reactor thread for 98712.5us
I20251029 02:51:55.087642 1175 ts_tablet_manager.cc:1929] T f71388036478404d878a52fc20ece216 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:55.088981 1175 log.cc:1199] T f71388036478404d878a52fc20ece216 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/f71388036478404d878a52fc20ece216
I20251029 02:51:55.091043 1175 ts_tablet_manager.cc:1950] T f71388036478404d878a52fc20ece216 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:51:55.098176 1175 tablet_replica.cc:333] T d6e1170dbb594a5696a3899cde8da7c9 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:51:55.098887 1175 raft_consensus.cc:2243] T d6e1170dbb594a5696a3899cde8da7c9 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:55.099476 1175 raft_consensus.cc:2272] T d6e1170dbb594a5696a3899cde8da7c9 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:55.100823 1175 ts_tablet_manager.cc:1916] T d6e1170dbb594a5696a3899cde8da7c9 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:55.103971 1167 tablet_bootstrap.cc:492] T 61c1fec3ab4c402eb29f767b9bb7409f P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:51:55.111079 1394 raft_consensus.cc:493] T 8d9722c764ac4b27a540595e9c6aafcd P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:55.111303 1394 raft_consensus.cc:515] T 8d9722c764ac4b27a540595e9c6aafcd P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:55.112113 1394 leader_election.cc:290] T 8d9722c764ac4b27a540595e9c6aafcd P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
W20251029 02:51:55.117020 525 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fa5127bcfcda40a7b5ad4ef016af072a consensus state has no leader
I20251029 02:51:55.119793 1440 tablet.cc:1722] T 11f4252b5e694312aaa142468f8e68b4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20251029 02:51:55.120357 1059 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75236b2fa82f4f28b82efae9a2c5160d" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:51:55.043897 1042 tablet_service.cc:1505] Processing CreateTablet for tablet 30d03e1e42d34242bd354f9e9ec4f127 (DEFAULT_TABLE table=default.Table_435f2dfe42134760abc9e8880fa19f32 [id=c6e1c357fbfc49be972748da564cc9a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:55.123178 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30d03e1e42d34242bd354f9e9ec4f127. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:55.127529 522 catalog_manager.cc:5649] T 4b76c4408a7340e59b8506171622175f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } health_report { overall_health: UNKNOWN } } }
I20251029 02:51:55.130581 1166 raft_consensus.cc:359] T 24016ceaa84147eca0b94b3ad288e772 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:55.130978 1166 raft_consensus.cc:385] T 24016ceaa84147eca0b94b3ad288e772 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:55.131114 1166 raft_consensus.cc:740] T 24016ceaa84147eca0b94b3ad288e772 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:51:55.131517 1166 consensus_queue.cc:260] T 24016ceaa84147eca0b94b3ad288e772 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:55.132175 1166 ts_tablet_manager.cc:1434] T 24016ceaa84147eca0b94b3ad288e772 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.381s user 0.001s sys 0.001s
I20251029 02:51:55.132532 1166 tablet_bootstrap.cc:492] T fa5127bcfcda40a7b5ad4ef016af072a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:51:55.146684 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d9722c764ac4b27a540595e9c6aafcd" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:51:55.146852 1176 ts_tablet_manager.cc:1929] T d7ba1756532c4825a6c7024511dbae51 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:55.147010 1176 log.cc:1199] T d7ba1756532c4825a6c7024511dbae51 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/d7ba1756532c4825a6c7024511dbae51
I20251029 02:51:55.147384 1176 ts_tablet_manager.cc:1950] T d7ba1756532c4825a6c7024511dbae51 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
W20251029 02:51:55.147603 711 leader_election.cc:343] T 8d9722c764ac4b27a540595e9c6aafcd P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:55.147878 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 200) took 1491 ms. Trace:
I20251029 02:51:55.148015 1176 rpcz_store.cc:276] 1029 02:51:53.656603 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:53.656654 (+ 51us) service_pool.cc:225] Handling call
1029 02:51:54.212190 (+555536us) ts_tablet_manager.cc:1151] Deleting tablet d7ba1756532c4825a6c7024511dbae51
1029 02:51:54.212196 (+ 6us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:54.476225 (+264029us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:55.146719 (+670494us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:55.147823 (+ 1104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":555349}
I20251029 02:51:55.148483 1176 tablet_replica.cc:333] T 6dd4696ad4b44b6dab87a205f3f16ebd P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:51:55.148810 1176 raft_consensus.cc:2243] T 6dd4696ad4b44b6dab87a205f3f16ebd P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:55.149134 1176 raft_consensus.cc:2272] T 6dd4696ad4b44b6dab87a205f3f16ebd P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:55.149969 1176 ts_tablet_manager.cc:1916] T 6dd4696ad4b44b6dab87a205f3f16ebd P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:55.160477 1175 ts_tablet_manager.cc:1929] T d6e1170dbb594a5696a3899cde8da7c9 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:55.163991 1175 log.cc:1199] T d6e1170dbb594a5696a3899cde8da7c9 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/d6e1170dbb594a5696a3899cde8da7c9
I20251029 02:51:55.169865 1175 ts_tablet_manager.cc:1950] T d6e1170dbb594a5696a3899cde8da7c9 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:51:55.171350 1175 tablet_replica.cc:333] T 6c22497760274380a832feff7c5a2fae P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:51:55.179760 1175 raft_consensus.cc:2243] T 6c22497760274380a832feff7c5a2fae P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:55.180461 1175 raft_consensus.cc:2272] T 6c22497760274380a832feff7c5a2fae P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:55.182897 1175 ts_tablet_manager.cc:1916] T 6c22497760274380a832feff7c5a2fae P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:55.197053 1059 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d9722c764ac4b27a540595e9c6aafcd" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:51:55.204231 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6896073ee0134bfa8c8075ef5ae1a01c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:51:55.204313 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 44784a3a760946e0be24743bc2fc8138 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 44784a3a760946e0be24743bc2fc8138 already in progress: creating tablet
W20251029 02:51:55.204527 511 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.1:38481, user_credentials={real_user=slave}} blocked reactor thread for 120767us
W20251029 02:51:55.204845 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 6cd7009e06a646acad0802918c2b425e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6cd7009e06a646acad0802918c2b425e already in progress: creating tablet
W20251029 02:51:55.205076 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet b5d5b737c86844d58c51b2388c0b1705 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b5d5b737c86844d58c51b2388c0b1705 already in progress: creating tablet
I20251029 02:51:55.205310 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 6f93812578a940a3b7378015e2a46f95 (table default.Table_cd7b2ea5d1b240d087c135cfec7dd0be [id=feb1f56ae44249c6adce65637017bb43]) successfully deleted
W20251029 02:51:55.205529 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet fd0f89180aba43eabb20f926abccbaa6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fd0f89180aba43eabb20f926abccbaa6 already in progress: creating tablet
W20251029 02:51:55.205753 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 1bab90fe904944af948dab3db06b7e34 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1bab90fe904944af948dab3db06b7e34 already in progress: creating tablet
I20251029 02:51:55.205840 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_d1050ce4c18f47c89dab970ce5edc755"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:55.206032 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 785733da86ff425a90ccfab2933fc89e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 785733da86ff425a90ccfab2933fc89e already in progress: creating tablet
W20251029 02:51:55.206246 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 74873a05ef634867a30aa63b31fcfcc7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 74873a05ef634867a30aa63b31fcfcc7 already in progress: creating tablet
I20251029 02:51:55.206424 524 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_9e6e0bf44516404ab570d4f843bd542f" } new_table_name: "default.Table_91cd5889c90148e5bde6051f2847832a" modify_external_catalogs: true
W20251029 02:51:55.206578 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d1050ce4c18f47c89dab970ce5edc755 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:51:55.206449 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet ccdca31db2834de4b6ed0f2d67a10535 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ccdca31db2834de4b6ed0f2d67a10535 already in progress: creating tablet
W20251029 02:51:55.207075 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 576c4e6e70e24b0399ac9597a30f0569 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 576c4e6e70e24b0399ac9597a30f0569 already in progress: creating tablet
I20251029 02:51:55.207284 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet f71388036478404d878a52fc20ece216 (table default.Table_03f7d6f02b064939ba7464c5173e02e3 [id=9898cffc2b964059ba3163ee4d1395c2]) successfully deleted
I20251029 02:51:55.207509 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet d6e1170dbb594a5696a3899cde8da7c9 (table default.Table_4d2c99159e924a06858977465efa38a6 [id=ad583c111f524bb0a40dcf1eb4d5c527]) successfully deleted
W20251029 02:51:55.204191 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_341f16ec415a43b7a29d194cef0ae86b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:55.207882 525 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet 11f4252b5e694312aaa142468f8e68b4 (table default.Table_b93cf815f69a4fe895e2ce461148fc71 [id=d0cd3d1567c54463947443927e18169f]). Expected version 5 got 0
I20251029 02:51:55.208015 517 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 7e6c4f30be624769961c9ec17c5d5dcb
I20251029 02:51:55.208981 517 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_cb594ce4077d4d02aa70ef94d5f801cc" } modify_external_catalogs: true
I20251029 02:51:55.209028 525 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_22184a76d5034feea912425485a578a4" } new_table_name: "default.Table_2a10c9dfc91d4974845dbcd500d1d8be" modify_external_catalogs: true
I20251029 02:51:55.209208 517 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_cb594ce4077d4d02aa70ef94d5f801cc" } modify_external_catalogs: true
I20251029 02:51:55.210085 522 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet 4b76c4408a7340e59b8506171622175f (table default.Table_1281926bf1224c0cb4541bf561102021 [id=f3fa2dbbef7e47359492d5a5eef9b8fd]). Expected version 2 got 0
I20251029 02:51:55.210238 756 tablet_service.cc:1552] Processing DeleteTablet for tablet 7e6c4f30be624769961c9ec17c5d5dcb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:55 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:55.210400 747 tablet_service.cc:1552] Processing DeleteTablet for tablet 75236b2fa82f4f28b82efae9a2c5160d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:52 UTC) from {username='slave'} at 127.0.0.1:44580
W20251029 02:51:55.210950 712 leader_election.cc:343] T 75236b2fa82f4f28b82efae9a2c5160d P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:55.211056 526 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_50252b3b3cc4403da5e259f9f91be915" } modify_external_catalogs: true
I20251029 02:51:55.211133 712 leader_election.cc:304] T 75236b2fa82f4f28b82efae9a2c5160d P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:51:55.211225 526 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_50252b3b3cc4403da5e259f9f91be915" } modify_external_catalogs: true
I20251029 02:51:55.211647 1436 mvcc.cc:204] Tried to move back new op lower bound from 7215949066013978624 to 7215949061837979648. Current Snapshot: MvccSnapshot[applied={T|T < 7215949066013978624}]
I20251029 02:51:55.212049 1394 raft_consensus.cc:2749] T 75236b2fa82f4f28b82efae9a2c5160d P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:51:55.213402 518 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_b8ddec1d965444948171ef0cca0ff011" } modify_external_catalogs: true
I20251029 02:51:55.213617 518 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_b8ddec1d965444948171ef0cca0ff011" } modify_external_catalogs: true
W20251029 02:51:55.214754 520 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1275e192871440408a8345fef9b56260 consensus state has no leader
I20251029 02:51:55.215359 520 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_04a6cfdfbf8c4674b30a8c3024e3277c" } new_table_name: "default.Table_e168a30d3a1e48ed8053847b304036e9" modify_external_catalogs: true
I20251029 02:51:55.221289 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet d7ba1756532c4825a6c7024511dbae51 (table default.Table_4c5fd91d917e4f1a99baef291b2d77e0 [id=5587d5c964034bd7ad0efa338b74df65]) successfully deleted
I20251029 02:51:55.227774 1215 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251029 02:51:55.230012 1175 ts_tablet_manager.cc:1929] T 6c22497760274380a832feff7c5a2fae P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:55.239889 1215 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251029 02:51:55.240200 1175 log.cc:1199] T 6c22497760274380a832feff7c5a2fae P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/6c22497760274380a832feff7c5a2fae
I20251029 02:51:55.240842 1175 ts_tablet_manager.cc:1950] T 6c22497760274380a832feff7c5a2fae P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:51:55.241262 1215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251029 02:51:55.247412 523 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet 11f4252b5e694312aaa142468f8e68b4 (table default.Table_b93cf815f69a4fe895e2ce461148fc71 [id=d0cd3d1567c54463947443927e18169f]). Expected version 5 got 0
I20251029 02:51:55.233353 1444 tablet.cc:1722] T 11f4252b5e694312aaa142468f8e68b4 P b495a49349324f7d81908f2da700f0d9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20251029 02:51:55.248907 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 6c22497760274380a832feff7c5a2fae (table default.Table_0fa5f3e932c846168255e7540c35a515 [id=c2475b97464f45da8990e004660c42b3]) successfully deleted
I20251029 02:51:55.249382 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_7dd1cd855bb74080b6e242847e5faac6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:55.249830 712 leader_election.cc:343] T 8d9722c764ac4b27a540595e9c6aafcd P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:55.250061 712 leader_election.cc:304] T 8d9722c764ac4b27a540595e9c6aafcd P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:51:55.250401 1394 consensus_queue.cc:1048] T 70e8cfe5eaa9492c85874dc3214a1e0b P b495a49349324f7d81908f2da700f0d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251029 02:51:55.250875 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7dd1cd855bb74080b6e242847e5faac6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:55.250973 1394 raft_consensus.cc:2749] T 8d9722c764ac4b27a540595e9c6aafcd P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:51:55.257030 1175 tablet_replica.cc:333] T 8d9722c764ac4b27a540595e9c6aafcd P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:51:55.257618 1175 raft_consensus.cc:2243] T 8d9722c764ac4b27a540595e9c6aafcd P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:55.257867 1175 raft_consensus.cc:2272] T 8d9722c764ac4b27a540595e9c6aafcd P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:55.257292 1445 raft_consensus.cc:493] T c519832030884a049839d309cc738c28 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:55.258234 1445 raft_consensus.cc:515] T c519832030884a049839d309cc738c28 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:55.259384 1445 leader_election.cc:290] T c519832030884a049839d309cc738c28 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:51:55.259681 1175 ts_tablet_manager.cc:1916] T 8d9722c764ac4b27a540595e9c6aafcd P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:55.257865 1172 tablet_bootstrap.cc:492] T 14698af6348f41e384f10569392d9b23 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:51:55.267580 1172 ts_tablet_manager.cc:1403] T 14698af6348f41e384f10569392d9b23 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.305s user 0.016s sys 0.104s
W20251029 02:51:55.268005 525 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8afc01d52d784f2fa559cab7ce07c8be consensus state has no leader
I20251029 02:51:55.268342 525 master_service.cc:954] ReplaceTablet: received request to replace tablet dfa8b1b9a80e4e8689e0acb44e16346a from {username='slave'} at 127.0.0.1:35314
I20251029 02:51:55.268384 1172 raft_consensus.cc:359] T 14698af6348f41e384f10569392d9b23 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:55.268584 1172 raft_consensus.cc:385] T 14698af6348f41e384f10569392d9b23 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:55.268664 1172 raft_consensus.cc:740] T 14698af6348f41e384f10569392d9b23 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:51:55.271490 1172 consensus_queue.cc:260] T 14698af6348f41e384f10569392d9b23 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:55.272025 1172 ts_tablet_manager.cc:1434] T 14698af6348f41e384f10569392d9b23 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251029 02:51:55.270880 1251 consensus_queue.cc:1048] T 5e31e061d74146009d24d5cd57487a5e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.022s
I20251029 02:51:55.273995 1172 tablet_bootstrap.cc:492] T fd900eec3f0c43bf965c148ea90870a3 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:51:55.275705 1172 tablet_bootstrap.cc:654] T fd900eec3f0c43bf965c148ea90870a3 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:55.285678 1175 ts_tablet_manager.cc:1929] T 8d9722c764ac4b27a540595e9c6aafcd P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:55.290048 1175 log.cc:1199] T 8d9722c764ac4b27a540595e9c6aafcd P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/8d9722c764ac4b27a540595e9c6aafcd
I20251029 02:51:55.290794 1175 ts_tablet_manager.cc:1950] T 8d9722c764ac4b27a540595e9c6aafcd P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:51:55.293548 1175 tablet_replica.cc:333] T c519832030884a049839d309cc738c28 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:51:55.293731 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 8d9722c764ac4b27a540595e9c6aafcd (table default.Table_ee95bb03ab1246e8826f28083411050d [id=09fbf31759f047be924a5ef1f0030765]) successfully deleted
I20251029 02:51:55.295626 1167 ts_tablet_manager.cc:1403] T 61c1fec3ab4c402eb29f767b9bb7409f P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.326s user 0.009s sys 0.001s
I20251029 02:51:55.297521 1175 raft_consensus.cc:2243] T c519832030884a049839d309cc738c28 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:55.298863 1175 raft_consensus.cc:2272] T c519832030884a049839d309cc738c28 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:55.300019 1175 ts_tablet_manager.cc:1916] T c519832030884a049839d309cc738c28 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:55.301429 1167 raft_consensus.cc:359] T 61c1fec3ab4c402eb29f767b9bb7409f P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:55.301724 1167 raft_consensus.cc:385] T 61c1fec3ab4c402eb29f767b9bb7409f P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:55.301810 1167 raft_consensus.cc:740] T 61c1fec3ab4c402eb29f767b9bb7409f P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:51:55.301898 1172 tablet_bootstrap.cc:492] T fd900eec3f0c43bf965c148ea90870a3 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:51:55.302029 1172 ts_tablet_manager.cc:1403] T fd900eec3f0c43bf965c148ea90870a3 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.028s user 0.000s sys 0.005s
I20251029 02:51:55.302047 1167 consensus_queue.cc:260] T 61c1fec3ab4c402eb29f767b9bb7409f P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:55.302683 1167 ts_tablet_manager.cc:1434] T 61c1fec3ab4c402eb29f767b9bb7409f P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20251029 02:51:55.302861 1172 raft_consensus.cc:359] T fd900eec3f0c43bf965c148ea90870a3 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:55.303023 1167 tablet_bootstrap.cc:492] T 41a6139fd9b84707a8c7eebc0381e383 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:51:55.303042 1172 raft_consensus.cc:385] T fd900eec3f0c43bf965c148ea90870a3 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:55.303128 1172 raft_consensus.cc:740] T fd900eec3f0c43bf965c148ea90870a3 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:51:55.303462 1172 consensus_queue.cc:260] T fd900eec3f0c43bf965c148ea90870a3 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:55.304172 1172 ts_tablet_manager.cc:1434] T fd900eec3f0c43bf965c148ea90870a3 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251029 02:51:55.304620 1172 tablet_bootstrap.cc:492] T aa776cde2f494fa5810539edf9f289a1 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:51:55.310606 1175 ts_tablet_manager.cc:1929] T c519832030884a049839d309cc738c28 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:55.310817 1175 log.cc:1199] T c519832030884a049839d309cc738c28 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/c519832030884a049839d309cc738c28
I20251029 02:51:55.311329 1175 ts_tablet_manager.cc:1950] T c519832030884a049839d309cc738c28 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:51:55.312142 1175 tablet_replica.cc:333] T 75236b2fa82f4f28b82efae9a2c5160d P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:51:55.312391 1175 raft_consensus.cc:2243] T 75236b2fa82f4f28b82efae9a2c5160d P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:55.312358 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet c519832030884a049839d309cc738c28 (table default.Table_e95a0439ea534f538683392bbc8e5219 [id=8ad554168c7e4b86a373a3c4579e6211]) successfully deleted
I20251029 02:51:55.312620 1175 raft_consensus.cc:2272] T 75236b2fa82f4f28b82efae9a2c5160d P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251029 02:51:55.312649 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 7e6c4f30be624769961c9ec17c5d5dcb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7e6c4f30be624769961c9ec17c5d5dcb already in progress: creating tablet
I20251029 02:51:55.312897 1444 tablet.cc:1717] T 11f4252b5e694312aaa142468f8e68b4 P b495a49349324f7d81908f2da700f0d9: Skipping requested alter to schema version 5, tablet already version 5
I20251029 02:51:55.313237 1440 tablet.cc:1717] T 11f4252b5e694312aaa142468f8e68b4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Skipping requested alter to schema version 5, tablet already version 5
I20251029 02:51:55.313596 1175 ts_tablet_manager.cc:1916] T 75236b2fa82f4f28b82efae9a2c5160d P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:55.314417 1172 tablet_bootstrap.cc:654] T aa776cde2f494fa5810539edf9f289a1 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:55.314673 1025 tablet_service.cc:1552] Processing DeleteTablet for tablet 7e6c4f30be624769961c9ec17c5d5dcb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:55 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:55.315277 882 tablet_service.cc:1552] Processing DeleteTablet for tablet 7e6c4f30be624769961c9ec17c5d5dcb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:55 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:55.315546 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c519832030884a049839d309cc738c28" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:51:55.319919 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c519832030884a049839d309cc738c28" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:51:55.320695 711 leader_election.cc:343] T c519832030884a049839d309cc738c28 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:55.321488 1175 ts_tablet_manager.cc:1929] T 75236b2fa82f4f28b82efae9a2c5160d P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:55.321734 1175 log.cc:1199] T 75236b2fa82f4f28b82efae9a2c5160d P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/75236b2fa82f4f28b82efae9a2c5160d
I20251029 02:51:55.322168 1175 ts_tablet_manager.cc:1950] T 75236b2fa82f4f28b82efae9a2c5160d P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:51:55.323300 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 75236b2fa82f4f28b82efae9a2c5160d (table default.Table_0b84a2f666074fd18085ff4ce249a2fe [id=715a1df4e483412da9a63eca19b8f02d]) successfully deleted
I20251029 02:51:55.324337 1166 tablet_bootstrap.cc:654] T fa5127bcfcda40a7b5ad4ef016af072a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
W20251029 02:51:55.325464 712 leader_election.cc:343] T c519832030884a049839d309cc738c28 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:55.325702 712 leader_election.cc:304] T c519832030884a049839d309cc738c28 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
W20251029 02:51:55.326094 712 raft_consensus.cc:2686] T c519832030884a049839d309cc738c28 P b495a49349324f7d81908f2da700f0d9: unable to run election callback: Service unavailable: Thread pool token was shut down
I20251029 02:51:55.326333 1174 ts_tablet_manager.cc:1929] T bf210435010d4dde8903fd116eb468e3 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:55.326539 1174 log.cc:1199] T bf210435010d4dde8903fd116eb468e3 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/bf210435010d4dde8903fd116eb468e3
I20251029 02:51:55.341640 878 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:35702 (request call id 252) took 1094 ms. Trace:
I20251029 02:51:55.341852 878 rpcz_store.cc:276] 1029 02:51:54.246963 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:54.578773 (+331810us) service_pool.cc:225] Handling call
1029 02:51:54.702420 (+123647us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1029 02:51:54.702448 (+ 28us) ts_tablet_manager.cc:682] Creating new metadata...
1029 02:51:54.983960 (+281512us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:55.340859 (+356899us) tablet_replica.cc:206] Creating consensus instance
1029 02:51:55.341034 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251029 02:51:55.342643 1174 ts_tablet_manager.cc:1950] T bf210435010d4dde8903fd116eb468e3 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:51:55.348263 1167 tablet_bootstrap.cc:654] T 41a6139fd9b84707a8c7eebc0381e383 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:55.349192 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 198) took 1689 ms. Trace:
I20251029 02:51:55.349766 1174 rpcz_store.cc:276] 1029 02:51:53.653800 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:53.653871 (+ 71us) service_pool.cc:225] Handling call
1029 02:51:54.835233 (+1181362us) ts_tablet_manager.cc:1151] Deleting tablet bf210435010d4dde8903fd116eb468e3
1029 02:51:54.835244 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:54.970499 (+135255us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:55.325368 (+354869us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:55.343102 (+ 17734us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1105824}
I20251029 02:51:55.350195 1174 tablet_replica.cc:333] T d7ba1756532c4825a6c7024511dbae51 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:51:55.350435 1174 raft_consensus.cc:2243] T d7ba1756532c4825a6c7024511dbae51 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:51:55.350867 1174 pending_rounds.cc:70] T d7ba1756532c4825a6c7024511dbae51 P c756528642a543eeacd93cef3e335a7d: Trying to abort 1 pending ops.
I20251029 02:51:55.350991 1174 pending_rounds.cc:77] T d7ba1756532c4825a6c7024511dbae51 P c756528642a543eeacd93cef3e335a7d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215949063616143360 op_type: NO_OP noop_request { }
I20251029 02:51:55.351164 1174 raft_consensus.cc:2889] T d7ba1756532c4825a6c7024511dbae51 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 02:51:55.351346 1174 raft_consensus.cc:2272] T d7ba1756532c4825a6c7024511dbae51 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:55.352073 1174 ts_tablet_manager.cc:1916] T d7ba1756532c4825a6c7024511dbae51 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 02:51:55.355054 990 consensus_peers.cc:597] T 41a6139fd9b84707a8c7eebc0381e383 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 -> Peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Couldn't send request to peer c756528642a543eeacd93cef3e335a7d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:51:55.359952 1176 ts_tablet_manager.cc:1929] T 6dd4696ad4b44b6dab87a205f3f16ebd P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:55.379382 896 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:35702 (request call id 244) took 1136 ms. Trace:
I20251029 02:51:55.379503 896 rpcz_store.cc:276] 1029 02:51:54.243042 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:54.442181 (+199139us) service_pool.cc:225] Handling call
1029 02:51:54.485803 (+ 43622us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1029 02:51:54.485828 (+ 25us) ts_tablet_manager.cc:682] Creating new metadata...
1029 02:51:54.934996 (+449168us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:55.379147 (+444151us) tablet_replica.cc:206] Creating consensus instance
1029 02:51:55.379312 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251029 02:51:55.366331 990 consensus_peers.cc:597] T bf210435010d4dde8903fd116eb468e3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 -> Peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Couldn't send request to peer c756528642a543eeacd93cef3e335a7d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bf210435010d4dde8903fd116eb468e3. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:51:55.391111 1176 log.cc:1199] T 6dd4696ad4b44b6dab87a205f3f16ebd P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/6dd4696ad4b44b6dab87a205f3f16ebd
I20251029 02:51:55.391597 1176 ts_tablet_manager.cc:1950] T 6dd4696ad4b44b6dab87a205f3f16ebd P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:51:55.392455 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 199) took 1735 ms. Trace:
I20251029 02:51:55.392607 1176 rpcz_store.cc:276] 1029 02:51:53.656494 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:53.658567 (+ 2073us) service_pool.cc:225] Handling call
1029 02:51:55.148401 (+1489834us) ts_tablet_manager.cc:1151] Deleting tablet 6dd4696ad4b44b6dab87a205f3f16ebd
1029 02:51:55.148411 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:55.288734 (+140323us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:55.359806 (+ 71072us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:55.392407 (+ 32601us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1489602}
I20251029 02:51:55.393277 1424 raft_consensus.cc:493] T 61c1fec3ab4c402eb29f767b9bb7409f P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:55.393453 1424 raft_consensus.cc:515] T 61c1fec3ab4c402eb29f767b9bb7409f P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:55.394182 1424 leader_election.cc:290] T 61c1fec3ab4c402eb29f767b9bb7409f P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:51:55.394260 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 6dd4696ad4b44b6dab87a205f3f16ebd (table default.Table_69af0d3b63c44157abd3263a9deb02da [id=90dfeabf0e8945c9bd86472d68884221]) successfully deleted
W20251029 02:51:55.398474 847 leader_election.cc:343] T 61c1fec3ab4c402eb29f767b9bb7409f P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 61c1fec3ab4c402eb29f767b9bb7409f
I20251029 02:51:55.385550 1166 tablet_bootstrap.cc:492] T fa5127bcfcda40a7b5ad4ef016af072a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:51:55.386000 1350 consensus_queue.cc:1048] T 41a6139fd9b84707a8c7eebc0381e383 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:51:55.401715 1176 tablet_replica.cc:333] T cb55b3d5bc63405f96cdc122f5a612cb P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:51:55.401959 1176 raft_consensus.cc:2243] T cb55b3d5bc63405f96cdc122f5a612cb P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:55.402138 1176 raft_consensus.cc:2272] T cb55b3d5bc63405f96cdc122f5a612cb P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:55.404592 1176 ts_tablet_manager.cc:1916] T cb55b3d5bc63405f96cdc122f5a612cb P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:55.411610 1059 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "61c1fec3ab4c402eb29f767b9bb7409f" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:51:55.412513 1059 raft_consensus.cc:2468] T 61c1fec3ab4c402eb29f767b9bb7409f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:51:55.413619 849 leader_election.cc:304] T 61c1fec3ab4c402eb29f767b9bb7409f P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters: b495a49349324f7d81908f2da700f0d9
I20251029 02:51:55.414014 1171 raft_consensus.cc:2804] T 61c1fec3ab4c402eb29f767b9bb7409f P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:51:55.414183 1171 raft_consensus.cc:493] T 61c1fec3ab4c402eb29f767b9bb7409f P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:51:55.414309 1171 raft_consensus.cc:3060] T 61c1fec3ab4c402eb29f767b9bb7409f P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
W20251029 02:51:55.416229 990 consensus_peers.cc:597] T 5e31e061d74146009d24d5cd57487a5e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 -> Peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Couldn't send request to peer c756528642a543eeacd93cef3e335a7d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:51:55.416798 1166 ts_tablet_manager.cc:1403] T fa5127bcfcda40a7b5ad4ef016af072a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.284s user 0.001s sys 0.008s
I20251029 02:51:55.417784 1171 raft_consensus.cc:515] T 61c1fec3ab4c402eb29f767b9bb7409f P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:55.418924 1059 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "61c1fec3ab4c402eb29f767b9bb7409f" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:51:55.419132 1059 raft_consensus.cc:3060] T 61c1fec3ab4c402eb29f767b9bb7409f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
W20251029 02:51:55.419626 847 leader_election.cc:343] T 61c1fec3ab4c402eb29f767b9bb7409f P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 61c1fec3ab4c402eb29f767b9bb7409f
I20251029 02:51:55.422926 1171 leader_election.cc:290] T 61c1fec3ab4c402eb29f767b9bb7409f P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:51:55.424099 1166 raft_consensus.cc:359] T fa5127bcfcda40a7b5ad4ef016af072a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:55.427155 1166 raft_consensus.cc:385] T fa5127bcfcda40a7b5ad4ef016af072a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:55.427512 1166 raft_consensus.cc:740] T fa5127bcfcda40a7b5ad4ef016af072a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:51:55.428833 1166 consensus_queue.cc:260] T fa5127bcfcda40a7b5ad4ef016af072a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:55.431658 1174 ts_tablet_manager.cc:1929] T d7ba1756532c4825a6c7024511dbae51 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:51:55.431998 1174 log.cc:1199] T d7ba1756532c4825a6c7024511dbae51 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/d7ba1756532c4825a6c7024511dbae51
I20251029 02:51:55.437494 526 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 9bb2166a590d4210b02dd779ef6654ad
W20251029 02:51:55.438902 520 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4aa78f4ac38e4919864c924e50f1aaf4 consensus state has no leader
I20251029 02:51:55.439198 520 master_service.cc:954] ReplaceTablet: received request to replace tablet 777992acb1bd4765886b802940db74bb from {username='slave'} at 127.0.0.1:35314
I20251029 02:51:55.439921 747 tablet_service.cc:1552] Processing DeleteTablet for tablet 9bb2166a590d4210b02dd779ef6654ad with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:55 UTC) from {username='slave'} at 127.0.0.1:44580
W20251029 02:51:55.440856 524 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9e07291d6b5645feb4d5890f58e1ef74 consensus state has no leader
W20251029 02:51:55.441560 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 9bb2166a590d4210b02dd779ef6654ad because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9bb2166a590d4210b02dd779ef6654ad already in progress: creating tablet
I20251029 02:51:55.443893 521 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_24be2b894899463a810d2fe7511d4d6b" } modify_external_catalogs: true
I20251029 02:51:55.444069 521 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_24be2b894899463a810d2fe7511d4d6b" } modify_external_catalogs: true
I20251029 02:51:55.444758 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_a63991c194724381a84c0ce2adab25da"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:55.445363 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a63991c194724381a84c0ce2adab25da in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:55.426954 1059 raft_consensus.cc:2468] T 61c1fec3ab4c402eb29f767b9bb7409f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:51:55.446580 849 leader_election.cc:304] T 61c1fec3ab4c402eb29f767b9bb7409f P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters: b495a49349324f7d81908f2da700f0d9
I20251029 02:51:55.446950 1171 raft_consensus.cc:2804] T 61c1fec3ab4c402eb29f767b9bb7409f P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:51:55.447103 1171 raft_consensus.cc:697] T 61c1fec3ab4c402eb29f767b9bb7409f P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:51:55.447398 1171 consensus_queue.cc:237] T 61c1fec3ab4c402eb29f767b9bb7409f P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:55.448171 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_bd9202984dc14b3b983326f60258c754"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:55.448798 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bd9202984dc14b3b983326f60258c754 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:55.453240 1042 tablet_service.cc:1552] Processing DeleteTablet for tablet 9bb2166a590d4210b02dd779ef6654ad with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:55 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:55.454069 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_094b4e33c9654fc7b1b4bf460be40fcd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:55.455403 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_094b4e33c9654fc7b1b4bf460be40fcd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:55.457489 1176 ts_tablet_manager.cc:1929] T cb55b3d5bc63405f96cdc122f5a612cb P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:51:55.466711 1176 log.cc:1199] T cb55b3d5bc63405f96cdc122f5a612cb P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/cb55b3d5bc63405f96cdc122f5a612cb
I20251029 02:51:55.467203 1176 ts_tablet_manager.cc:1950] T cb55b3d5bc63405f96cdc122f5a612cb P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:51:55.468221 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 201) took 1811 ms. Trace:
I20251029 02:51:55.468355 1176 rpcz_store.cc:276] 1029 02:51:53.656718 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:53.658854 (+ 2136us) service_pool.cc:225] Handling call
1029 02:51:55.392950 (+1734096us) ts_tablet_manager.cc:1151] Deleting tablet cb55b3d5bc63405f96cdc122f5a612cb
1029 02:51:55.401165 (+ 8215us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:55.413173 (+ 12008us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:55.457324 (+ 44151us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:55.468153 (+ 10829us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1733917}
I20251029 02:51:55.468851 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet cb55b3d5bc63405f96cdc122f5a612cb (table default.Table_47a1a59db0444fa49f7274509385ff0c [id=283c09ba60114ba9b165e2ff144a6c35]) successfully deleted
I20251029 02:51:55.469048 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 198) took 1812 ms. Trace:
I20251029 02:51:55.469149 1176 rpcz_store.cc:276] 1029 02:51:53.656344 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:53.658423 (+ 2079us) service_pool.cc:225] Handling call
1029 02:51:55.468889 (+1810466us) ts_tablet_manager.cc:1151] Deleting tablet 5886b8d39f384426bcbed7289ae72e51
1029 02:51:55.468902 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:55.468994 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1809469}
I20251029 02:51:55.461826 1166 ts_tablet_manager.cc:1434] T fa5127bcfcda40a7b5ad4ef016af072a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.021s user 0.000s sys 0.002s
W20251029 02:51:55.469765 513 catalog_manager.cc:4952] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 5886b8d39f384426bcbed7289ae72e51 because the tablet was not found. No further retry: Not found: Tablet not found: 5886b8d39f384426bcbed7289ae72e51
I20251029 02:51:55.469815 1166 tablet_bootstrap.cc:492] T c4884e22cf2044a4a1070c1b4b759d9e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:51:55.470739 1176 tablet_replica.cc:333] T bc1a1cf274b64b52aba58811e9594330 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:51:55.470928 1176 raft_consensus.cc:2243] T bc1a1cf274b64b52aba58811e9594330 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:55.471578 1176 raft_consensus.cc:2272] T bc1a1cf274b64b52aba58811e9594330 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:55.472093 1176 ts_tablet_manager.cc:1916] T bc1a1cf274b64b52aba58811e9594330 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:55.483736 1166 tablet_bootstrap.cc:654] T c4884e22cf2044a4a1070c1b4b759d9e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:55.490413 1176 ts_tablet_manager.cc:1929] T bc1a1cf274b64b52aba58811e9594330 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:55.502924 1167 tablet_bootstrap.cc:492] T 41a6139fd9b84707a8c7eebc0381e383 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:51:55.503114 1167 ts_tablet_manager.cc:1403] T 41a6139fd9b84707a8c7eebc0381e383 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.200s user 0.004s sys 0.005s
I20251029 02:51:55.503990 1167 raft_consensus.cc:359] T 41a6139fd9b84707a8c7eebc0381e383 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:55.504177 1167 raft_consensus.cc:385] T 41a6139fd9b84707a8c7eebc0381e383 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:55.504266 1167 raft_consensus.cc:740] T 41a6139fd9b84707a8c7eebc0381e383 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:51:55.497166 1166 tablet_bootstrap.cc:492] T c4884e22cf2044a4a1070c1b4b759d9e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:51:55.504565 1167 consensus_queue.cc:260] T 41a6139fd9b84707a8c7eebc0381e383 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:55.505218 1167 ts_tablet_manager.cc:1434] T 41a6139fd9b84707a8c7eebc0381e383 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 02:51:55.505549 1167 tablet_bootstrap.cc:492] T 5e31e061d74146009d24d5cd57487a5e P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:51:55.504717 1176 log.cc:1199] T bc1a1cf274b64b52aba58811e9594330 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/bc1a1cf274b64b52aba58811e9594330
I20251029 02:51:55.507509 1167 tablet_bootstrap.cc:654] T 5e31e061d74146009d24d5cd57487a5e P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:55.507030 1166 ts_tablet_manager.cc:1403] T c4884e22cf2044a4a1070c1b4b759d9e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.037s user 0.003s sys 0.005s
I20251029 02:51:55.507830 1176 ts_tablet_manager.cc:1950] T bc1a1cf274b64b52aba58811e9594330 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
W20251029 02:51:55.509222 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.3:45003, user_credentials={real_user=slave}} blocked reactor thread for 85744.4us
I20251029 02:51:55.509866 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 202) took 1850 ms. Trace:
I20251029 02:51:55.510048 1176 rpcz_store.cc:276] 1029 02:51:53.659222 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:53.659453 (+ 231us) service_pool.cc:225] Handling call
1029 02:51:55.470660 (+1811207us) ts_tablet_manager.cc:1151] Deleting tablet bc1a1cf274b64b52aba58811e9594330
1029 02:51:55.470668 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:55.486470 (+ 15802us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:55.490256 (+ 3786us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:55.509803 (+ 19547us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1811000}
I20251029 02:51:55.510475 1176 tablet_replica.cc:333] T b867feae24e741aca6902868c35c1428 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:51:55.510636 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet bc1a1cf274b64b52aba58811e9594330 (table default.Table_98034c2933664cefab94c26cb8349dc4 [id=efb8f381fea74cf48842af6b1a4d5a81]) successfully deleted
I20251029 02:51:55.510721 1176 raft_consensus.cc:2243] T b867feae24e741aca6902868c35c1428 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:55.510877 1176 raft_consensus.cc:2272] T b867feae24e741aca6902868c35c1428 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:55.511425 1176 ts_tablet_manager.cc:1916] T b867feae24e741aca6902868c35c1428 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:55.508816 1166 raft_consensus.cc:359] T c4884e22cf2044a4a1070c1b4b759d9e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:55.513587 1166 raft_consensus.cc:385] T c4884e22cf2044a4a1070c1b4b759d9e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:55.513706 1166 raft_consensus.cc:740] T c4884e22cf2044a4a1070c1b4b759d9e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:51:55.514025 1166 consensus_queue.cc:260] T c4884e22cf2044a4a1070c1b4b759d9e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:55.515661 1166 ts_tablet_manager.cc:1434] T c4884e22cf2044a4a1070c1b4b759d9e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20251029 02:51:55.516772 1166 tablet_bootstrap.cc:492] T 64b0fb400e944ce2b2c7a49d12135e9c P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:51:55.518836 1166 tablet_bootstrap.cc:654] T 64b0fb400e944ce2b2c7a49d12135e9c P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:55.529565 518 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 5db519edc75d4bd0985b32322825ffae
I20251029 02:51:55.531219 747 tablet_service.cc:1552] Processing DeleteTablet for tablet 5db519edc75d4bd0985b32322825ffae with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:55 UTC) from {username='slave'} at 127.0.0.1:44580
W20251029 02:51:55.532770 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 5db519edc75d4bd0985b32322825ffae because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5db519edc75d4bd0985b32322825ffae already in progress: creating tablet
I20251029 02:51:55.534636 1172 tablet_bootstrap.cc:492] T aa776cde2f494fa5810539edf9f289a1 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:51:55.534935 1172 ts_tablet_manager.cc:1403] T aa776cde2f494fa5810539edf9f289a1 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.230s user 0.020s sys 0.041s
I20251029 02:51:55.534999 518 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet 11f4252b5e694312aaa142468f8e68b4 (table default.Table_b93cf815f69a4fe895e2ce461148fc71 [id=d0cd3d1567c54463947443927e18169f]). Expected version 5 got 0
I20251029 02:51:55.535789 523 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_a8568b9dabb942a9a88179114c8e1240" } new_table_name: "default.Table_31d42031714e415fab63cc5b9933fe0c" modify_external_catalogs: true
I20251029 02:51:55.536186 1172 raft_consensus.cc:359] T aa776cde2f494fa5810539edf9f289a1 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:55.536474 1172 raft_consensus.cc:385] T aa776cde2f494fa5810539edf9f289a1 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:55.536604 1172 raft_consensus.cc:740] T aa776cde2f494fa5810539edf9f289a1 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:51:55.536918 1172 consensus_queue.cc:260] T aa776cde2f494fa5810539edf9f289a1 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:55.537477 1172 ts_tablet_manager.cc:1434] T aa776cde2f494fa5810539edf9f289a1 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 02:51:55.538231 1172 tablet_bootstrap.cc:492] T d2990e9f626c40618c118266bc595684 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:51:55.540650 1172 tablet_bootstrap.cc:654] T d2990e9f626c40618c118266bc595684 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:55.547914 1172 tablet_bootstrap.cc:492] T d2990e9f626c40618c118266bc595684 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:51:55.547928 525 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet dfa8b1b9a80e4e8689e0acb44e16346a
I20251029 02:51:55.548166 1172 ts_tablet_manager.cc:1403] T d2990e9f626c40618c118266bc595684 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20251029 02:51:55.548468 525 catalog_manager.cc:6719] ReplaceTablet: tablet dfa8b1b9a80e4e8689e0acb44e16346a deleted and replaced by tablet 97b2ecae572a4b7ea1b9595074556708
I20251029 02:51:55.549000 747 tablet_service.cc:1552] Processing DeleteTablet for tablet dfa8b1b9a80e4e8689e0acb44e16346a with delete_type TABLET_DATA_DELETED (replaced by tablet 97b2ecae572a4b7ea1b9595074556708) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:55.549126 1465 raft_consensus.cc:493] T c4884e22cf2044a4a1070c1b4b759d9e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:55.549335 1465 raft_consensus.cc:515] T c4884e22cf2044a4a1070c1b4b759d9e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
W20251029 02:51:55.550242 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet dfa8b1b9a80e4e8689e0acb44e16346a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dfa8b1b9a80e4e8689e0acb44e16346a already in progress: creating tablet
I20251029 02:51:55.550180 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_03b43c22b7714ab7a0f80d77b69c9f36"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:55.550935 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_03b43c22b7714ab7a0f80d77b69c9f36 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:55.551483 1172 raft_consensus.cc:359] T d2990e9f626c40618c118266bc595684 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:55.551734 1172 raft_consensus.cc:385] T d2990e9f626c40618c118266bc595684 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:55.551872 1172 raft_consensus.cc:740] T d2990e9f626c40618c118266bc595684 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:51:55.552196 1172 consensus_queue.cc:260] T d2990e9f626c40618c118266bc595684 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:55.552771 1172 ts_tablet_manager.cc:1434] T d2990e9f626c40618c118266bc595684 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251029 02:51:55.553290 1172 tablet_bootstrap.cc:492] T 012ce243fd07466ab5301d1c9ce4b413 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:51:55.557941 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_c2719a3251c84739a6681f80f1d8f94d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:55.558738 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c2719a3251c84739a6681f80f1d8f94d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:55.559710 1042 tablet_service.cc:1552] Processing DeleteTablet for tablet 5db519edc75d4bd0985b32322825ffae with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:55 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:55.559947 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet dfa8b1b9a80e4e8689e0acb44e16346a with delete_type TABLET_DATA_DELETED (replaced by tablet 97b2ecae572a4b7ea1b9595074556708) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:55.560111 1251 raft_consensus.cc:493] T fa5127bcfcda40a7b5ad4ef016af072a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:55.560287 1251 raft_consensus.cc:515] T fa5127bcfcda40a7b5ad4ef016af072a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:55.560424 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_3d94f8b10e884dc0b338dcfbb9c82309"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:55.560726 1172 tablet_bootstrap.cc:654] T 012ce243fd07466ab5301d1c9ce4b413 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
W20251029 02:51:55.561028 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3d94f8b10e884dc0b338dcfbb9c82309 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:55.561103 1251 leader_election.cc:290] T fa5127bcfcda40a7b5ad4ef016af072a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:51:55.561542 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fa5127bcfcda40a7b5ad4ef016af072a" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:51:55.561743 774 raft_consensus.cc:2468] T fa5127bcfcda40a7b5ad4ef016af072a P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 0.
I20251029 02:51:55.561818 1465 leader_election.cc:290] T c4884e22cf2044a4a1070c1b4b759d9e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:51:55.562463 989 leader_election.cc:304] T fa5127bcfcda40a7b5ad4ef016af072a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters:
I20251029 02:51:55.563004 1251 raft_consensus.cc:2804] T fa5127bcfcda40a7b5ad4ef016af072a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:51:55.563128 1251 raft_consensus.cc:493] T fa5127bcfcda40a7b5ad4ef016af072a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:51:55.563221 1251 raft_consensus.cc:3060] T fa5127bcfcda40a7b5ad4ef016af072a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
W20251029 02:51:55.563561 989 leader_election.cc:343] T c4884e22cf2044a4a1070c1b4b759d9e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: c4884e22cf2044a4a1070c1b4b759d9e
W20251029 02:51:55.564925 989 consensus_peers.cc:597] T bf210435010d4dde8903fd116eb468e3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 -> Peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Couldn't send request to peer b495a49349324f7d81908f2da700f0d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bf210435010d4dde8903fd116eb468e3. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:51:55.566629 1251 raft_consensus.cc:515] T fa5127bcfcda40a7b5ad4ef016af072a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:55.567349 1251 leader_election.cc:290] T fa5127bcfcda40a7b5ad4ef016af072a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Requested vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:51:55.567983 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fa5127bcfcda40a7b5ad4ef016af072a" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:51:55.568161 774 raft_consensus.cc:3060] T fa5127bcfcda40a7b5ad4ef016af072a P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:51:55.569880 520 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 777992acb1bd4765886b802940db74bb
I20251029 02:51:55.570473 520 catalog_manager.cc:6719] ReplaceTablet: tablet 777992acb1bd4765886b802940db74bb deleted and replaced by tablet e65eaaae43ac435ebd665546de9d68dd
I20251029 02:51:55.571203 774 raft_consensus.cc:2468] T fa5127bcfcda40a7b5ad4ef016af072a P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 1.
I20251029 02:51:55.571872 747 tablet_service.cc:1552] Processing DeleteTablet for tablet 777992acb1bd4765886b802940db74bb with delete_type TABLET_DATA_DELETED (replaced by tablet e65eaaae43ac435ebd665546de9d68dd) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:55.572453 989 leader_election.cc:304] T fa5127bcfcda40a7b5ad4ef016af072a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters:
W20251029 02:51:55.572552 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 777992acb1bd4765886b802940db74bb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 777992acb1bd4765886b802940db74bb already in progress: creating tablet
I20251029 02:51:55.573130 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet 777992acb1bd4765886b802940db74bb with delete_type TABLET_DATA_DELETED (replaced by tablet e65eaaae43ac435ebd665546de9d68dd) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:55.573424 1251 raft_consensus.cc:2804] T fa5127bcfcda40a7b5ad4ef016af072a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:51:55.573644 1251 raft_consensus.cc:697] T fa5127bcfcda40a7b5ad4ef016af072a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Becoming Leader. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Running, Role: LEADER
I20251029 02:51:55.574272 1251 consensus_queue.cc:237] T fa5127bcfcda40a7b5ad4ef016af072a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
W20251029 02:51:55.575709 512 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.60:41233, user_credentials={real_user=slave}} blocked reactor thread for 76832.2us
I20251029 02:51:55.577682 1176 ts_tablet_manager.cc:1929] T b867feae24e741aca6902868c35c1428 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:55.577855 1176 log.cc:1199] T b867feae24e741aca6902868c35c1428 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/b867feae24e741aca6902868c35c1428
I20251029 02:51:55.578248 1176 ts_tablet_manager.cc:1950] T b867feae24e741aca6902868c35c1428 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:51:55.581547 1172 tablet_bootstrap.cc:492] T 012ce243fd07466ab5301d1c9ce4b413 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
W20251029 02:51:55.581820 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.3:45003, user_credentials={real_user=slave}} blocked reactor thread for 67340.7us
I20251029 02:51:55.582237 1172 ts_tablet_manager.cc:1403] T 012ce243fd07466ab5301d1c9ce4b413 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.004s
I20251029 02:51:55.582841 520 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_ca4e3b5646244cafa5a0429d5a8f2ac6" } modify_external_catalogs: true
I20251029 02:51:55.583091 520 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_ca4e3b5646244cafa5a0429d5a8f2ac6" } modify_external_catalogs: true
I20251029 02:51:55.583145 1172 raft_consensus.cc:359] T 012ce243fd07466ab5301d1c9ce4b413 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:55.583384 1172 raft_consensus.cc:385] T 012ce243fd07466ab5301d1c9ce4b413 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:55.583452 1172 raft_consensus.cc:740] T 012ce243fd07466ab5301d1c9ce4b413 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:51:55.585136 1172 consensus_queue.cc:260] T 012ce243fd07466ab5301d1c9ce4b413 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:55.585404 519 catalog_manager.cc:6194] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: default.Table_b93cf815f69a4fe895e2ce461148fc71 [id=d0cd3d1567c54463947443927e18169f] alter complete (version 5)
I20251029 02:51:55.585678 1172 ts_tablet_manager.cc:1434] T 012ce243fd07466ab5301d1c9ce4b413 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20251029 02:51:55.585917 526 master_service.cc:954] ReplaceTablet: received request to replace tablet 2242f5b2c20346d9b753d7cd29242441 from {username='slave'} at 127.0.0.1:35314
I20251029 02:51:55.586021 1172 tablet_bootstrap.cc:492] T 078ace5eada044c88d2462efb2b57aa8 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:51:55.588531 519 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_6896073ee0134bfa8c8075ef5ae1a01c" } new_table_name: "default.Table_746054018e16496ea35decf9614e9f5c" modify_external_catalogs: true
I20251029 02:51:55.590422 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_a6a9f0fa8983438db6cece4c2111a3c9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:55.591032 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a6a9f0fa8983438db6cece4c2111a3c9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:55.591467 1172 tablet_bootstrap.cc:654] T 078ace5eada044c88d2462efb2b57aa8 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:55.592841 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet bf210435010d4dde8903fd116eb468e3 (table default.Table_4bdbcbb87e5b40aeb9f98eb1aa0e6fe3 [id=78615d8807a74b4496a3223ef6fe709d]) successfully deleted
W20251029 02:51:55.593333 523 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 738046ab57334f2dab9f4aab979c5b42 consensus state has no leader
I20251029 02:51:55.598270 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_06562b86d9c44611a457aef29c434785"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:55.598896 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_06562b86d9c44611a457aef29c434785 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:55.598948 1172 tablet_bootstrap.cc:492] T 078ace5eada044c88d2462efb2b57aa8 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:51:55.599094 1172 ts_tablet_manager.cc:1403] T 078ace5eada044c88d2462efb2b57aa8 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.000s
I20251029 02:51:55.599910 1172 raft_consensus.cc:359] T 078ace5eada044c88d2462efb2b57aa8 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:55.600116 1172 raft_consensus.cc:385] T 078ace5eada044c88d2462efb2b57aa8 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:55.600207 1172 raft_consensus.cc:740] T 078ace5eada044c88d2462efb2b57aa8 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:51:55.597663 525 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_b313ce7369b24372b396d8e9a257a369" } new_table_name: "default.Table_11de4e6789214efca8b43cbcdae81a93" modify_external_catalogs: true
I20251029 02:51:55.602180 1172 consensus_queue.cc:260] T 078ace5eada044c88d2462efb2b57aa8 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:55.602849 1172 ts_tablet_manager.cc:1434] T 078ace5eada044c88d2462efb2b57aa8 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20251029 02:51:55.603191 1172 tablet_bootstrap.cc:492] T b5d5b737c86844d58c51b2388c0b1705 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:51:55.603406 1251 raft_consensus.cc:493] T 24016ceaa84147eca0b94b3ad288e772 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:55.603659 1251 raft_consensus.cc:515] T 24016ceaa84147eca0b94b3ad288e772 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:55.605095 1251 leader_election.cc:290] T 24016ceaa84147eca0b94b3ad288e772 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:51:55.605377 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24016ceaa84147eca0b94b3ad288e772" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:51:55.605635 774 raft_consensus.cc:2468] T 24016ceaa84147eca0b94b3ad288e772 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 0.
I20251029 02:51:55.606118 989 leader_election.cc:304] T 24016ceaa84147eca0b94b3ad288e772 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters:
I20251029 02:51:55.606492 1251 raft_consensus.cc:2804] T 24016ceaa84147eca0b94b3ad288e772 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:51:55.606667 1251 raft_consensus.cc:493] T 24016ceaa84147eca0b94b3ad288e772 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:51:55.606803 1251 raft_consensus.cc:3060] T 24016ceaa84147eca0b94b3ad288e772 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:51:55.610766 1251 raft_consensus.cc:515] T 24016ceaa84147eca0b94b3ad288e772 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:55.612556 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24016ceaa84147eca0b94b3ad288e772" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:51:55.612807 774 raft_consensus.cc:3060] T 24016ceaa84147eca0b94b3ad288e772 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:51:55.615998 774 raft_consensus.cc:2468] T 24016ceaa84147eca0b94b3ad288e772 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 1.
I20251029 02:51:55.616465 989 leader_election.cc:304] T 24016ceaa84147eca0b94b3ad288e772 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters:
I20251029 02:51:55.616829 1465 raft_consensus.cc:2804] T 24016ceaa84147eca0b94b3ad288e772 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:51:55.617019 1465 raft_consensus.cc:697] T 24016ceaa84147eca0b94b3ad288e772 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Becoming Leader. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Running, Role: LEADER
I20251029 02:51:55.617300 1465 consensus_queue.cc:237] T 24016ceaa84147eca0b94b3ad288e772 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:55.619472 1251 leader_election.cc:290] T 24016ceaa84147eca0b94b3ad288e772 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Requested vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:51:55.621755 1172 tablet_bootstrap.cc:654] T b5d5b737c86844d58c51b2388c0b1705 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:55.634089 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_a66eb4754b854a998e458627d738338b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:55.634856 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a66eb4754b854a998e458627d738338b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:55.641590 1172 tablet_bootstrap.cc:492] T b5d5b737c86844d58c51b2388c0b1705 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:51:55.641808 1172 ts_tablet_manager.cc:1403] T b5d5b737c86844d58c51b2388c0b1705 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.039s user 0.000s sys 0.003s
I20251029 02:51:55.642666 1172 raft_consensus.cc:359] T b5d5b737c86844d58c51b2388c0b1705 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:55.642886 1172 raft_consensus.cc:385] T b5d5b737c86844d58c51b2388c0b1705 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:55.642969 1172 raft_consensus.cc:740] T b5d5b737c86844d58c51b2388c0b1705 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:51:55.645427 522 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet 4b76c4408a7340e59b8506171622175f (table default.Table_1281926bf1224c0cb4541bf561102021 [id=f3fa2dbbef7e47359492d5a5eef9b8fd]). Expected version 2 got 0
I20251029 02:51:55.646159 522 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet 5e31e061d74146009d24d5cd57487a5e (table default.Table_f2bc74933428449197f8c28182c8698f [id=8e17d6b87b984d878787802bd6f627f2]). Expected version 2 got 0
I20251029 02:51:55.646124 747 tablet_service.cc:1505] Processing CreateTablet for tablet af4860c6b72f4d988b206b5ef2f22906 (DEFAULT_TABLE table=default.Table_f5db3e3913ca4ea98b02981cd04a5965 [id=7eb08c679e0e441bb1ad4be13b2494ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:55.647143 1042 tablet_service.cc:1505] Processing CreateTablet for tablet af4860c6b72f4d988b206b5ef2f22906 (DEFAULT_TABLE table=default.Table_f5db3e3913ca4ea98b02981cd04a5965 [id=7eb08c679e0e441bb1ad4be13b2494ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:55.647413 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af4860c6b72f4d988b206b5ef2f22906. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:55.647694 1166 tablet_bootstrap.cc:492] T 64b0fb400e944ce2b2c7a49d12135e9c P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:51:55.647835 1166 ts_tablet_manager.cc:1403] T 64b0fb400e944ce2b2c7a49d12135e9c P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.131s user 0.026s sys 0.055s
I20251029 02:51:55.648486 890 tablet_service.cc:1552] Processing DeleteTablet for tablet 6dd4696ad4b44b6dab87a205f3f16ebd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:52 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:55.649063 898 tablet_service.cc:1552] Processing DeleteTablet for tablet 777992acb1bd4765886b802940db74bb with delete_type TABLET_DATA_DELETED (replaced by tablet e65eaaae43ac435ebd665546de9d68dd) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:55.649091 919 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24016ceaa84147eca0b94b3ad288e772" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
I20251029 02:51:55.649308 887 tablet_service.cc:1552] Processing DeleteTablet for tablet dfa8b1b9a80e4e8689e0acb44e16346a with delete_type TABLET_DATA_DELETED (replaced by tablet 97b2ecae572a4b7ea1b9595074556708) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:55.646463 522 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet 24016ceaa84147eca0b94b3ad288e772 (table default.Table_f06003da6e324504ade193bbbb11d36a [id=bd279b3ce623477ca20e8d35e3b7c0df]). Expected version 1 got 0
W20251029 02:51:55.649663 990 leader_election.cc:343] T 24016ceaa84147eca0b94b3ad288e772 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:55.649807 1166 raft_consensus.cc:359] T 64b0fb400e944ce2b2c7a49d12135e9c P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:55.649983 1166 raft_consensus.cc:385] T 64b0fb400e944ce2b2c7a49d12135e9c P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:55.650086 1166 raft_consensus.cc:740] T 64b0fb400e944ce2b2c7a49d12135e9c P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:51:55.650386 1166 consensus_queue.cc:260] T 64b0fb400e944ce2b2c7a49d12135e9c P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:55.650678 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c4884e22cf2044a4a1070c1b4b759d9e" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:51:55.650678 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fa5127bcfcda40a7b5ad4ef016af072a" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:51:55.651352 1174 ts_tablet_manager.cc:1950] T d7ba1756532c4825a6c7024511dbae51 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
W20251029 02:51:55.651369 990 leader_election.cc:343] T c4884e22cf2044a4a1070c1b4b759d9e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:55.651497 990 leader_election.cc:304] T c4884e22cf2044a4a1070c1b4b759d9e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5; no voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d
I20251029 02:51:55.651939 883 tablet_service.cc:1552] Processing DeleteTablet for tablet 9bb2166a590d4210b02dd779ef6654ad with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:55 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:55.652200 896 tablet_service.cc:1552] Processing DeleteTablet for tablet 61c1fec3ab4c402eb29f767b9bb7409f with delete_type TABLET_DATA_DELETED (replaced by tablet a5c3a15510cb4321bbb80d61fabd0c0a) from {username='slave'} at 127.0.0.1:35702
W20251029 02:51:55.652463 990 leader_election.cc:343] T fa5127bcfcda40a7b5ad4ef016af072a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:55.652482 881 tablet_service.cc:1552] Processing DeleteTablet for tablet 5db519edc75d4bd0985b32322825ffae with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:55 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:55.652544 918 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fa5127bcfcda40a7b5ad4ef016af072a" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
I20251029 02:51:55.652488 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24016ceaa84147eca0b94b3ad288e772" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:51:55.653221 1251 raft_consensus.cc:2749] T c4884e22cf2044a4a1070c1b4b759d9e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:51:55.652769 890 tablet_service.cc:1505] Processing CreateTablet for tablet af4860c6b72f4d988b206b5ef2f22906 (DEFAULT_TABLE table=default.Table_f5db3e3913ca4ea98b02981cd04a5965 [id=7eb08c679e0e441bb1ad4be13b2494ef]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 02:51:55.653540 990 leader_election.cc:343] T fa5127bcfcda40a7b5ad4ef016af072a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
W20251029 02:51:55.653852 990 leader_election.cc:343] T 24016ceaa84147eca0b94b3ad288e772 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:55.654641 522 master_service.cc:954] ReplaceTablet: received request to replace tablet 44784a3a760946e0be24743bc2fc8138 from {username='slave'} at 127.0.0.1:35314
I20251029 02:51:55.655023 1166 ts_tablet_manager.cc:1434] T 64b0fb400e944ce2b2c7a49d12135e9c P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.007s user 0.002s sys 0.001s
I20251029 02:51:55.655249 522 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_c70eb9f284cd49109fd09e430a1cea6c" } modify_external_catalogs: true
I20251029 02:51:55.655472 522 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_c70eb9f284cd49109fd09e430a1cea6c" } modify_external_catalogs: true
I20251029 02:51:55.656033 1041 tablet_service.cc:1552] Processing DeleteTablet for tablet cb55b3d5bc63405f96cdc122f5a612cb with delete_type TABLET_DATA_DELETED (replaced by tablet aa776cde2f494fa5810539edf9f289a1) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:55.659924 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af4860c6b72f4d988b206b5ef2f22906. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:55.659941 1172 consensus_queue.cc:260] T b5d5b737c86844d58c51b2388c0b1705 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:55.660485 1172 ts_tablet_manager.cc:1434] T b5d5b737c86844d58c51b2388c0b1705 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.019s user 0.000s sys 0.002s
I20251029 02:51:55.660843 1172 tablet_bootstrap.cc:492] T 9e07291d6b5645feb4d5890f58e1ef74 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:51:55.662297 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af4860c6b72f4d988b206b5ef2f22906. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:55.662500 1172 tablet_bootstrap.cc:654] T 9e07291d6b5645feb4d5890f58e1ef74 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:55.664374 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 199) took 2003 ms. Trace:
I20251029 02:51:55.670145 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 203) took 2010 ms. Trace:
I20251029 02:51:55.670262 1176 rpcz_store.cc:276] 1029 02:51:53.660521 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:53.660580 (+ 59us) service_pool.cc:225] Handling call
1029 02:51:55.510356 (+1849776us) ts_tablet_manager.cc:1151] Deleting tablet b867feae24e741aca6902868c35c1428
1029 02:51:55.510366 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:55.525573 (+ 15207us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:55.577523 (+ 51950us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:55.670093 (+ 92570us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1849545}
I20251029 02:51:55.670694 1174 rpcz_store.cc:276] 1029 02:51:53.653931 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:53.653981 (+ 50us) service_pool.cc:225] Handling call
1029 02:51:55.350132 (+1696151us) ts_tablet_manager.cc:1151] Deleting tablet d7ba1756532c4825a6c7024511dbae51
1029 02:51:55.350141 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:55.361406 (+ 11265us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:55.431453 (+ 70047us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:55.657323 (+225870us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1620584}
I20251029 02:51:55.670857 1166 tablet_bootstrap.cc:492] T ab421dfaa3f84593842bb732a2693160 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:51:55.671061 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet b867feae24e741aca6902868c35c1428 (table default.Table_c70eb9f284cd49109fd09e430a1cea6c [id=be817eb6ec33406eb316d36575c134fb]) successfully deleted
I20251029 02:51:55.671188 1174 tablet_replica.cc:333] T cb55b3d5bc63405f96cdc122f5a612cb P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:51:55.671528 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet d7ba1756532c4825a6c7024511dbae51 (table default.Table_4c5fd91d917e4f1a99baef291b2d77e0 [id=5587d5c964034bd7ad0efa338b74df65]) successfully deleted
I20251029 02:51:55.671721 1470 tablet.cc:1722] T 11f4252b5e694312aaa142468f8e68b4 P c756528642a543eeacd93cef3e335a7d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20251029 02:51:55.672964 1166 tablet_bootstrap.cc:654] T ab421dfaa3f84593842bb732a2693160 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:55.680718 1172 tablet_bootstrap.cc:492] T 9e07291d6b5645feb4d5890f58e1ef74 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:51:55.680898 1172 ts_tablet_manager.cc:1403] T 9e07291d6b5645feb4d5890f58e1ef74 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.002s
I20251029 02:51:55.681679 1172 raft_consensus.cc:359] T 9e07291d6b5645feb4d5890f58e1ef74 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:55.681896 1172 raft_consensus.cc:385] T 9e07291d6b5645feb4d5890f58e1ef74 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:55.681994 1172 raft_consensus.cc:740] T 9e07291d6b5645feb4d5890f58e1ef74 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:51:55.684216 1172 consensus_queue.cc:260] T 9e07291d6b5645feb4d5890f58e1ef74 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:55.685911 1172 ts_tablet_manager.cc:1434] T 9e07291d6b5645feb4d5890f58e1ef74 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251029 02:51:55.686244 1172 tablet_bootstrap.cc:492] T 4bf06476c87a4549a73f5994314bb75f P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:51:55.688056 1172 tablet_bootstrap.cc:654] T 4bf06476c87a4549a73f5994314bb75f P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:55.688622 1174 raft_consensus.cc:2243] T cb55b3d5bc63405f96cdc122f5a612cb P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:51:55.689836 1174 raft_consensus.cc:2272] T cb55b3d5bc63405f96cdc122f5a612cb P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:55.690819 1174 ts_tablet_manager.cc:1916] T cb55b3d5bc63405f96cdc122f5a612cb P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:55.690832 1176 tablet_replica.cc:333] T 6c32c6f5e1cc40a0a12688201cac62a7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:51:55.691046 1176 raft_consensus.cc:2243] T 6c32c6f5e1cc40a0a12688201cac62a7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus shutting down.
W20251029 02:51:55.694192 519 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet adf6e736cb9c4920807ee58682e134ef has no consensus state
I20251029 02:51:55.691259 1176 raft_consensus.cc:2272] T 6c32c6f5e1cc40a0a12688201cac62a7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:55.695840 1176 ts_tablet_manager.cc:1916] T 6c32c6f5e1cc40a0a12688201cac62a7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:55.701407 519 catalog_manager.cc:5649] T fa5127bcfcda40a7b5ad4ef016af072a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } health_report { overall_health: UNKNOWN } } }
I20251029 02:51:55.702121 519 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet fa5127bcfcda40a7b5ad4ef016af072a (table default.Table_5113d1a33e8e4bd2af5bc51e689f304f [id=ae5d56d40fa941a68f2b10d5e6e9ac83]). Expected version 2 got 0
I20251029 02:51:55.703006 519 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet 5e31e061d74146009d24d5cd57487a5e (table default.Table_f2bc74933428449197f8c28182c8698f [id=8e17d6b87b984d878787802bd6f627f2]). Expected version 2 got 0
I20251029 02:51:55.703321 519 catalog_manager.cc:5649] T 24016ceaa84147eca0b94b3ad288e772 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } health_report { overall_health: UNKNOWN } } }
I20251029 02:51:55.705574 519 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet 24016ceaa84147eca0b94b3ad288e772 (table default.Table_f06003da6e324504ade193bbbb11d36a [id=bd279b3ce623477ca20e8d35e3b7c0df]). Expected version 1 got 0
I20251029 02:51:55.706362 518 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_03b43c22b7714ab7a0f80d77b69c9f36" } new_table_name: "default.Table_457a3835f7d84fd7bf722dfc1d18d301" modify_external_catalogs: true
I20251029 02:51:55.709949 1167 tablet_bootstrap.cc:492] T 5e31e061d74146009d24d5cd57487a5e P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:51:55.710242 1167 ts_tablet_manager.cc:1403] T 5e31e061d74146009d24d5cd57487a5e P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.205s user 0.004s sys 0.000s
I20251029 02:51:55.711391 1167 raft_consensus.cc:359] T 5e31e061d74146009d24d5cd57487a5e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:55.711755 1167 raft_consensus.cc:385] T 5e31e061d74146009d24d5cd57487a5e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:55.711896 1167 raft_consensus.cc:740] T 5e31e061d74146009d24d5cd57487a5e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:51:55.712244 1167 consensus_queue.cc:260] T 5e31e061d74146009d24d5cd57487a5e P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:55.712414 1445 raft_consensus.cc:493] T 59af906061d049c99ae2153945597007 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:55.712596 1445 raft_consensus.cc:515] T 59af906061d049c99ae2153945597007 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:55.712940 1167 ts_tablet_manager.cc:1434] T 5e31e061d74146009d24d5cd57487a5e P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20251029 02:51:55.713304 1445 leader_election.cc:290] T 59af906061d049c99ae2153945597007 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:51:55.713478 1167 tablet_bootstrap.cc:492] T bc1a1cf274b64b52aba58811e9594330 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:51:55.714268 1059 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59af906061d049c99ae2153945597007" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:51:55.715169 712 leader_election.cc:343] T 59af906061d049c99ae2153945597007 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:55.715771 1470 tablet.cc:1717] T 11f4252b5e694312aaa142468f8e68b4 P c756528642a543eeacd93cef3e335a7d: Skipping requested alter to schema version 5, tablet already version 5
I20251029 02:51:55.719734 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59af906061d049c99ae2153945597007" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:51:55.720243 711 leader_election.cc:343] T 59af906061d049c99ae2153945597007 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:55.720376 711 leader_election.cc:304] T 59af906061d049c99ae2153945597007 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:51:55.720790 1167 tablet_bootstrap.cc:654] T bc1a1cf274b64b52aba58811e9594330 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:55.721284 1445 raft_consensus.cc:2749] T 59af906061d049c99ae2153945597007 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:51:55.722123 1172 tablet_bootstrap.cc:492] T 4bf06476c87a4549a73f5994314bb75f P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:51:55.722344 1172 ts_tablet_manager.cc:1403] T 4bf06476c87a4549a73f5994314bb75f P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.036s user 0.005s sys 0.005s
I20251029 02:51:55.723434 1172 raft_consensus.cc:359] T 4bf06476c87a4549a73f5994314bb75f P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:55.724081 1174 ts_tablet_manager.cc:1929] T cb55b3d5bc63405f96cdc122f5a612cb P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:51:55.724284 1174 log.cc:1199] T cb55b3d5bc63405f96cdc122f5a612cb P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/cb55b3d5bc63405f96cdc122f5a612cb
I20251029 02:51:55.724383 1172 raft_consensus.cc:385] T 4bf06476c87a4549a73f5994314bb75f P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:55.724474 1172 raft_consensus.cc:740] T 4bf06476c87a4549a73f5994314bb75f P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:51:55.725104 1174 ts_tablet_manager.cc:1950] T cb55b3d5bc63405f96cdc122f5a612cb P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:51:55.725060 1172 consensus_queue.cc:260] T 4bf06476c87a4549a73f5994314bb75f P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:55.726006 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 197) took 2072 ms. Trace:
I20251029 02:51:55.726562 1166 tablet_bootstrap.cc:492] T ab421dfaa3f84593842bb732a2693160 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:51:55.726145 1174 rpcz_store.cc:276] 1029 02:51:53.653668 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:53.654085 (+ 417us) service_pool.cc:225] Handling call
1029 02:51:55.671122 (+2017037us) ts_tablet_manager.cc:1151] Deleting tablet cb55b3d5bc63405f96cdc122f5a612cb
1029 02:51:55.671133 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:55.689042 (+ 17909us) spinlock_profiling.cc:243] Waited 17.2 ms on lock 0x616000160f54. stack: 00007f1e09cad9f6 00007f1e09cad7b0 00007f1e08c893ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f1e15b78100 00007f1e16ec5ae3 00007f1e19a86287 00007f1e19ab92d4 00007f1e19a99d70 00007f1e19a99afd 0000000000a8867c 00007f1e09cf3d7f 00007f1e09cf704b 00007f1e09cf6ec8
1029 02:51:55.711721 (+ 22679us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:55.723956 (+ 12235us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:55.725952 (+ 1996us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":39585024,"tablet-delete.queue_time_us":1940482}
I20251029 02:51:55.726887 1166 ts_tablet_manager.cc:1403] T ab421dfaa3f84593842bb732a2693160 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.056s user 0.002s sys 0.002s
I20251029 02:51:55.726976 1172 ts_tablet_manager.cc:1434] T 4bf06476c87a4549a73f5994314bb75f P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20251029 02:51:55.727169 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet cb55b3d5bc63405f96cdc122f5a612cb (table default.Table_47a1a59db0444fa49f7274509385ff0c [id=283c09ba60114ba9b165e2ff144a6c35]) successfully deleted
I20251029 02:51:55.727869 1174 tablet_replica.cc:333] T 6c32c6f5e1cc40a0a12688201cac62a7 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:51:55.728116 1174 raft_consensus.cc:2243] T 6c32c6f5e1cc40a0a12688201cac62a7 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:51:55.728216 1172 tablet_bootstrap.cc:492] T 9964ad774890422f8e14f6b99f28cb55 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:51:55.728502 1174 pending_rounds.cc:70] T 6c32c6f5e1cc40a0a12688201cac62a7 P c756528642a543eeacd93cef3e335a7d: Trying to abort 1 pending ops.
I20251029 02:51:55.728622 1174 pending_rounds.cc:77] T 6c32c6f5e1cc40a0a12688201cac62a7 P c756528642a543eeacd93cef3e335a7d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215949065604403200 op_type: NO_OP noop_request { }
I20251029 02:51:55.728814 1174 raft_consensus.cc:2889] T 6c32c6f5e1cc40a0a12688201cac62a7 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 02:51:55.728971 1174 raft_consensus.cc:2272] T 6c32c6f5e1cc40a0a12688201cac62a7 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:55.730446 1174 ts_tablet_manager.cc:1916] T 6c32c6f5e1cc40a0a12688201cac62a7 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 02:51:55.731029 525 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bfec87f690a640c39c58c64e16cc67e7 consensus state has no leader
I20251029 02:51:55.731948 1445 raft_consensus.cc:493] T aa776cde2f494fa5810539edf9f289a1 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:55.732129 1445 raft_consensus.cc:515] T aa776cde2f494fa5810539edf9f289a1 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:55.732743 1445 leader_election.cc:290] T aa776cde2f494fa5810539edf9f289a1 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:51:55.733489 1059 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aa776cde2f494fa5810539edf9f289a1" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:51:55.734102 712 leader_election.cc:343] T aa776cde2f494fa5810539edf9f289a1 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:55.734264 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aa776cde2f494fa5810539edf9f289a1" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:51:55.734844 711 leader_election.cc:343] T aa776cde2f494fa5810539edf9f289a1 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:55.734964 711 leader_election.cc:304] T aa776cde2f494fa5810539edf9f289a1 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:51:55.735440 1445 raft_consensus.cc:2749] T aa776cde2f494fa5810539edf9f289a1 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:51:55.735531 525 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_f039c48d9cd4451e951c5113e8c24920" } new_table_name: "default.Table_dd4934aabcd442039db111954d819f6d" modify_external_catalogs: true
I20251029 02:51:55.736503 1172 tablet_bootstrap.cc:654] T 9964ad774890422f8e14f6b99f28cb55 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:55.737957 524 master_service.cc:954] ReplaceTablet: received request to replace tablet c764062d3ba04d4ba8b2e367ad8ed433 from {username='slave'} at 127.0.0.1:35314
I20251029 02:51:55.740114 1174 ts_tablet_manager.cc:1929] T 6c32c6f5e1cc40a0a12688201cac62a7 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:51:55.740694 1174 log.cc:1199] T 6c32c6f5e1cc40a0a12688201cac62a7 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/6c32c6f5e1cc40a0a12688201cac62a7
I20251029 02:51:55.740690 1166 raft_consensus.cc:359] T ab421dfaa3f84593842bb732a2693160 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:55.740972 1166 raft_consensus.cc:385] T ab421dfaa3f84593842bb732a2693160 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:55.741063 1166 raft_consensus.cc:740] T ab421dfaa3f84593842bb732a2693160 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:51:55.741398 1174 ts_tablet_manager.cc:1950] T 6c32c6f5e1cc40a0a12688201cac62a7 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:51:55.741360 1166 consensus_queue.cc:260] T ab421dfaa3f84593842bb732a2693160 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:55.741983 1166 ts_tablet_manager.cc:1434] T ab421dfaa3f84593842bb732a2693160 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.015s user 0.001s sys 0.000s
I20251029 02:51:55.742386 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 200) took 2087 ms. Trace:
I20251029 02:51:55.742547 1174 rpcz_store.cc:276] 1029 02:51:53.655023 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:53.655083 (+ 60us) service_pool.cc:225] Handling call
1029 02:51:55.727783 (+2072700us) ts_tablet_manager.cc:1151] Deleting tablet 6c32c6f5e1cc40a0a12688201cac62a7
1029 02:51:55.727795 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:55.735503 (+ 7708us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:55.739998 (+ 4495us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:55.742330 (+ 2332us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1993155}
I20251029 02:51:55.743095 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 6c32c6f5e1cc40a0a12688201cac62a7 (table default.Table_e6ca0a151fc14c5085c6ea454f460242 [id=2a711e1172654147a58b09ee43cac845]) successfully deleted
I20251029 02:51:55.743561 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 201) took 2088 ms. Trace:
I20251029 02:51:55.743709 1174 rpcz_store.cc:276] 1029 02:51:53.655173 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:53.655272 (+ 99us) service_pool.cc:225] Handling call
1029 02:51:55.743416 (+2088144us) ts_tablet_manager.cc:1151] Deleting tablet d32532973b3f46dd8158a2ce42d26365
1029 02:51:55.743427 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:55.743508 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2008222}
W20251029 02:51:55.744278 512 catalog_manager.cc:4952] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet d32532973b3f46dd8158a2ce42d26365 because the tablet was not found. No further retry: Not found: Tablet not found: d32532973b3f46dd8158a2ce42d26365
I20251029 02:51:55.745000 1174 tablet_replica.cc:333] T deca0c4a114441079fce22e74b403050 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:51:55.745258 1174 raft_consensus.cc:2243] T deca0c4a114441079fce22e74b403050 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:55.745556 1174 raft_consensus.cc:2272] T deca0c4a114441079fce22e74b403050 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:55.745780 1172 tablet_bootstrap.cc:492] T 9964ad774890422f8e14f6b99f28cb55 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:51:55.745957 1172 ts_tablet_manager.cc:1403] T 9964ad774890422f8e14f6b99f28cb55 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.002s
I20251029 02:51:55.746248 1174 ts_tablet_manager.cc:1916] T deca0c4a114441079fce22e74b403050 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:55.748250 1172 raft_consensus.cc:359] T 9964ad774890422f8e14f6b99f28cb55 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:55.748484 1172 raft_consensus.cc:385] T 9964ad774890422f8e14f6b99f28cb55 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:55.748610 1172 raft_consensus.cc:740] T 9964ad774890422f8e14f6b99f28cb55 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:51:55.749478 1172 consensus_queue.cc:260] T 9964ad774890422f8e14f6b99f28cb55 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:55.751358 1172 ts_tablet_manager.cc:1434] T 9964ad774890422f8e14f6b99f28cb55 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251029 02:51:55.751690 1172 tablet_bootstrap.cc:492] T 0a47781d036149569e1164dd938a45e9 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:51:55.753116 521 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 2e97d1bc06a24e8d8ba11acb6fd280d8
I20251029 02:51:55.753849 521 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_341f16ec415a43b7a29d194cef0ae86b" } modify_external_catalogs: true
I20251029 02:51:55.754056 521 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_341f16ec415a43b7a29d194cef0ae86b" } modify_external_catalogs: true
I20251029 02:51:55.754115 1042 tablet_service.cc:1552] Processing DeleteTablet for tablet 2e97d1bc06a24e8d8ba11acb6fd280d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:55 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:55.756268 1172 tablet_bootstrap.cc:654] T 0a47781d036149569e1164dd938a45e9 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:55.756888 890 tablet_service.cc:1552] Processing DeleteTablet for tablet 2e97d1bc06a24e8d8ba11acb6fd280d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:55 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:55.757154 747 tablet_service.cc:1552] Processing DeleteTablet for tablet 2e97d1bc06a24e8d8ba11acb6fd280d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:55 UTC) from {username='slave'} at 127.0.0.1:44580
W20251029 02:51:55.757779 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 2e97d1bc06a24e8d8ba11acb6fd280d8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2e97d1bc06a24e8d8ba11acb6fd280d8 already in progress: creating tablet
I20251029 02:51:55.762640 1172 tablet_bootstrap.cc:492] T 0a47781d036149569e1164dd938a45e9 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:51:55.762903 1172 ts_tablet_manager.cc:1403] T 0a47781d036149569e1164dd938a45e9 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.001s
I20251029 02:51:55.763772 1172 raft_consensus.cc:359] T 0a47781d036149569e1164dd938a45e9 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:55.764034 1172 raft_consensus.cc:385] T 0a47781d036149569e1164dd938a45e9 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:55.764161 1172 raft_consensus.cc:740] T 0a47781d036149569e1164dd938a45e9 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:51:55.764490 1172 consensus_queue.cc:260] T 0a47781d036149569e1164dd938a45e9 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:55.765794 1172 ts_tablet_manager.cc:1434] T 0a47781d036149569e1164dd938a45e9 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20251029 02:51:55.766541 1172 tablet_bootstrap.cc:492] T 65fdec7cf5d143208d78d4c3806b1352 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:51:55.770639 1172 tablet_bootstrap.cc:654] T 65fdec7cf5d143208d78d4c3806b1352 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:55.772145 1440 tablet.cc:1722] T 5e31e061d74146009d24d5cd57487a5e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 02:51:55.773036 1444 tablet.cc:1722] T 5e31e061d74146009d24d5cd57487a5e P b495a49349324f7d81908f2da700f0d9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 02:51:55.775043 1166 tablet_bootstrap.cc:492] T dd90fb9eb167478cbc3ec9363af86295 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:51:55.776883 1166 tablet_bootstrap.cc:654] T dd90fb9eb167478cbc3ec9363af86295 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:55.778831 1445 raft_consensus.cc:493] T 0a47781d036149569e1164dd938a45e9 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:55.779003 1445 raft_consensus.cc:515] T 0a47781d036149569e1164dd938a45e9 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:55.779610 1445 leader_election.cc:290] T 0a47781d036149569e1164dd938a45e9 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:51:55.780326 1059 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0a47781d036149569e1164dd938a45e9" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:51:55.781957 712 leader_election.cc:343] T 0a47781d036149569e1164dd938a45e9 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:55.787035 1176 ts_tablet_manager.cc:1929] T 6c32c6f5e1cc40a0a12688201cac62a7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:55.787184 1176 log.cc:1199] T 6c32c6f5e1cc40a0a12688201cac62a7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/6c32c6f5e1cc40a0a12688201cac62a7
I20251029 02:51:55.787633 1176 ts_tablet_manager.cc:1950] T 6c32c6f5e1cc40a0a12688201cac62a7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:51:55.787982 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 204) took 2127 ms. Trace:
I20251029 02:51:55.788101 1176 rpcz_store.cc:276] 1029 02:51:53.660964 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:53.661123 (+ 159us) service_pool.cc:225] Handling call
1029 02:51:55.690706 (+2029583us) ts_tablet_manager.cc:1151] Deleting tablet 6c32c6f5e1cc40a0a12688201cac62a7
1029 02:51:55.690724 (+ 18us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:55.717474 (+ 26750us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:55.786884 (+ 69410us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:55.787942 (+ 1058us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2029338}
I20251029 02:51:55.789173 1176 tablet_replica.cc:333] T bf210435010d4dde8903fd116eb468e3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:51:55.789422 1176 raft_consensus.cc:2243] T bf210435010d4dde8903fd116eb468e3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:51:55.789921 1176 pending_rounds.cc:70] T bf210435010d4dde8903fd116eb468e3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Trying to abort 1 pending ops.
I20251029 02:51:55.790063 1176 pending_rounds.cc:77] T bf210435010d4dde8903fd116eb468e3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215949063719301120 op_type: NO_OP noop_request { }
I20251029 02:51:55.790108 520 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet d61beda662d844cdaf67fd9a2e382ea2
I20251029 02:51:55.790194 1176 raft_consensus.cc:2889] T bf210435010d4dde8903fd116eb468e3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 02:51:55.790549 1176 raft_consensus.cc:2272] T bf210435010d4dde8903fd116eb468e3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:55.790959 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_b9f2a2e3d9f8446fbd46ca6c48155ccf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:55.792171 747 tablet_service.cc:1552] Processing DeleteTablet for tablet d61beda662d844cdaf67fd9a2e382ea2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:55 UTC) from {username='slave'} at 127.0.0.1:44580
W20251029 02:51:55.792809 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b9f2a2e3d9f8446fbd46ca6c48155ccf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:51:55.792919 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet d61beda662d844cdaf67fd9a2e382ea2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d61beda662d844cdaf67fd9a2e382ea2 already in progress: creating tablet
I20251029 02:51:55.792970 1176 ts_tablet_manager.cc:1916] T bf210435010d4dde8903fd116eb468e3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:55.794863 1042 tablet_service.cc:1552] Processing DeleteTablet for tablet d61beda662d844cdaf67fd9a2e382ea2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:55 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:55.795574 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 6c32c6f5e1cc40a0a12688201cac62a7 (table default.Table_e6ca0a151fc14c5085c6ea454f460242 [id=2a711e1172654147a58b09ee43cac845]) successfully deleted
W20251029 02:51:55.798637 518 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 71b4b91071a54d6aa66160e7955bc915 has no consensus state
I20251029 02:51:55.802445 523 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet 5e31e061d74146009d24d5cd57487a5e (table default.Table_f2bc74933428449197f8c28182c8698f [id=8e17d6b87b984d878787802bd6f627f2]). Expected version 2 got 0
I20251029 02:51:55.805680 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_0100f32eabf24d88a739a3cdac3597e1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:55.806394 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0100f32eabf24d88a739a3cdac3597e1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:55.807215 517 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 012ce243fd07466ab5301d1c9ce4b413
I20251029 02:51:55.808068 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_a6472497e2e940c3b6a79f15665aa636"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:55.808633 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a6472497e2e940c3b6a79f15665aa636 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:55.808597 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_644024f7ee3a44ce988f60dedec9419b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:55.809185 1167 tablet_bootstrap.cc:492] T bc1a1cf274b64b52aba58811e9594330 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
W20251029 02:51:55.809209 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_644024f7ee3a44ce988f60dedec9419b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:55.809350 1167 ts_tablet_manager.cc:1403] T bc1a1cf274b64b52aba58811e9594330 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.096s user 0.001s sys 0.003s
I20251029 02:51:55.810248 1167 raft_consensus.cc:359] T bc1a1cf274b64b52aba58811e9594330 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:55.810457 1167 raft_consensus.cc:385] T bc1a1cf274b64b52aba58811e9594330 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:55.810590 1167 raft_consensus.cc:740] T bc1a1cf274b64b52aba58811e9594330 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:51:55.810612 747 tablet_service.cc:1552] Processing DeleteTablet for tablet 012ce243fd07466ab5301d1c9ce4b413 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:55 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:55.810819 1167 consensus_queue.cc:260] T bc1a1cf274b64b52aba58811e9594330 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:55.810904 1175 tablet_replica.cc:333] T 012ce243fd07466ab5301d1c9ce4b413 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:51:55.811095 1175 raft_consensus.cc:2243] T 012ce243fd07466ab5301d1c9ce4b413 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:55.811272 1175 raft_consensus.cc:2272] T 012ce243fd07466ab5301d1c9ce4b413 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:55.811985 1042 tablet_service.cc:1552] Processing DeleteTablet for tablet 64b0fb400e944ce2b2c7a49d12135e9c with delete_type TABLET_DATA_DELETED (replaced by tablet 9e07291d6b5645feb4d5890f58e1ef74) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:55.812067 522 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet b6d4f7bf702f4834bc5f4ef2b50b92a7
I20251029 02:51:55.812414 1175 ts_tablet_manager.cc:1916] T 012ce243fd07466ab5301d1c9ce4b413 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:55.812923 1042 tablet_service.cc:1552] Processing DeleteTablet for tablet c4884e22cf2044a4a1070c1b4b759d9e with delete_type TABLET_DATA_DELETED (replaced by tablet 9964ad774890422f8e14f6b99f28cb55) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:55.812915 522 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_06562b86d9c44611a457aef29c434785" } new_table_name: "default.Table_2ccb14f87f774da3b62823c798b31a7c" modify_external_catalogs: true
I20251029 02:51:55.813145 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet 012ce243fd07466ab5301d1c9ce4b413 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:55 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:55.813915 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet b6d4f7bf702f4834bc5f4ef2b50b92a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:55 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:55.815626 747 tablet_service.cc:1552] Processing DeleteTablet for tablet b6d4f7bf702f4834bc5f4ef2b50b92a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:55 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:55.817514 524 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet c764062d3ba04d4ba8b2e367ad8ed433
I20251029 02:51:55.817953 524 catalog_manager.cc:6719] ReplaceTablet: tablet c764062d3ba04d4ba8b2e367ad8ed433 deleted and replaced by tablet 372d93914abc41b38a702742b7db0bc2
I20251029 02:51:55.818253 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_0c02e8daa2e74175bb2fdbdba0ec0eab"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:55.818825 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0c02e8daa2e74175bb2fdbdba0ec0eab in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:55.819676 1175 ts_tablet_manager.cc:1929] T 012ce243fd07466ab5301d1c9ce4b413 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:55.819802 1175 log.cc:1199] T 012ce243fd07466ab5301d1c9ce4b413 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/012ce243fd07466ab5301d1c9ce4b413
W20251029 02:51:55.820000 525 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1c53fc1fff1a4bb988beb46ff91fd3be consensus state has no leader
I20251029 02:51:55.820158 1175 ts_tablet_manager.cc:1950] T 012ce243fd07466ab5301d1c9ce4b413 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:51:55.820277 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0a47781d036149569e1164dd938a45e9" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:51:55.823443 1421 raft_consensus.cc:493] T 41a6139fd9b84707a8c7eebc0381e383 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:55.823611 1421 raft_consensus.cc:515] T 41a6139fd9b84707a8c7eebc0381e383 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:55.824227 1421 leader_election.cc:290] T 41a6139fd9b84707a8c7eebc0381e383 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:51:55.824151 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_8cbb25deb1a043959698717a9af07e52"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:55.824832 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8cbb25deb1a043959698717a9af07e52 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:55.824867 1059 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "41a6139fd9b84707a8c7eebc0381e383" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:51:55.826794 1166 tablet_bootstrap.cc:492] T dd90fb9eb167478cbc3ec9363af86295 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:51:55.826957 1166 ts_tablet_manager.cc:1403] T dd90fb9eb167478cbc3ec9363af86295 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.052s user 0.002s sys 0.002s
I20251029 02:51:55.827742 1166 raft_consensus.cc:359] T dd90fb9eb167478cbc3ec9363af86295 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:55.827934 1166 raft_consensus.cc:385] T dd90fb9eb167478cbc3ec9363af86295 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:55.828028 1166 raft_consensus.cc:740] T dd90fb9eb167478cbc3ec9363af86295 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:51:55.828083 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 012ce243fd07466ab5301d1c9ce4b413 (table default.Table_cb594ce4077d4d02aa70ef94d5f801cc [id=b792f15b5ea34e44b32b3087b983567c]) successfully deleted
W20251029 02:51:55.828366 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet b6d4f7bf702f4834bc5f4ef2b50b92a7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b6d4f7bf702f4834bc5f4ef2b50b92a7 already in progress: creating tablet
I20251029 02:51:55.828457 1166 consensus_queue.cc:260] T dd90fb9eb167478cbc3ec9363af86295 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:55.828850 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet c764062d3ba04d4ba8b2e367ad8ed433 with delete_type TABLET_DATA_DELETED (replaced by tablet 372d93914abc41b38a702742b7db0bc2) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:55.829138 1166 ts_tablet_manager.cc:1434] T dd90fb9eb167478cbc3ec9363af86295 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251029 02:51:55.829567 1166 tablet_bootstrap.cc:492] T 59af906061d049c99ae2153945597007 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:51:55.830233 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_8f1e631e578a4010927f77c77cc29cb0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:55.830816 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8f1e631e578a4010927f77c77cc29cb0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:55.833236 1166 tablet_bootstrap.cc:654] T 59af906061d049c99ae2153945597007 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:55.835173 523 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet dd90fb9eb167478cbc3ec9363af86295 (table default.Table_eb28ff62f787444189cb7dfe9e735992 [id=052abf3dcc904bdcb869d5b885723dab]). Expected version 1 got 0
W20251029 02:51:55.836508 711 leader_election.cc:343] T 0a47781d036149569e1164dd938a45e9 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:55.836686 711 leader_election.cc:304] T 0a47781d036149569e1164dd938a45e9 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:51:55.837354 521 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 43ec1295692d40c3871d12478dd318f8
I20251029 02:51:55.838088 1174 ts_tablet_manager.cc:1929] T deca0c4a114441079fce22e74b403050 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:55.838231 1174 log.cc:1199] T deca0c4a114441079fce22e74b403050 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/deca0c4a114441079fce22e74b403050
I20251029 02:51:55.838351 890 tablet_service.cc:1552] Processing DeleteTablet for tablet d61beda662d844cdaf67fd9a2e382ea2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:55 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:55.841687 1445 raft_consensus.cc:2749] T 0a47781d036149569e1164dd938a45e9 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:51:55.842082 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "41a6139fd9b84707a8c7eebc0381e383" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:51:55.843428 756 tablet_service.cc:1552] Processing DeleteTablet for tablet 14698af6348f41e384f10569392d9b23 with delete_type TABLET_DATA_DELETED (replaced by tablet 7309d3190cfa415281c5253a634fa7f3) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:55.843835 739 tablet_service.cc:1552] Processing DeleteTablet for tablet aa776cde2f494fa5810539edf9f289a1 with delete_type TABLET_DATA_DELETED (replaced by tablet c9bd3f2deb5d4af09b7918269a98509f) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:55.843832 1175 tablet_replica.cc:333] T 14698af6348f41e384f10569392d9b23 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:51:55.844071 738 tablet_service.cc:1552] Processing DeleteTablet for tablet 012ce243fd07466ab5301d1c9ce4b413 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:55 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:55.844166 521 master_service.cc:954] ReplaceTablet: received request to replace tablet 6ca9bd37de9e4be388501b15af4bda31 from {username='slave'} at 127.0.0.1:35314
I20251029 02:51:55.844404 743 tablet_service.cc:1552] Processing DeleteTablet for tablet fd900eec3f0c43bf965c148ea90870a3 with delete_type TABLET_DATA_DELETED (replaced by tablet b2b2d64dcf964da8be4b544dbbe31977) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:55.844424 1175 raft_consensus.cc:2243] T 14698af6348f41e384f10569392d9b23 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:55.844595 747 tablet_service.cc:1552] Processing DeleteTablet for tablet c764062d3ba04d4ba8b2e367ad8ed433 with delete_type TABLET_DATA_DELETED (replaced by tablet 372d93914abc41b38a702742b7db0bc2) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:55.844671 1175 raft_consensus.cc:2272] T 14698af6348f41e384f10569392d9b23 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:55.844851 774 raft_consensus.cc:1275] T fa5127bcfcda40a7b5ad4ef016af072a P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Refusing update from remote peer 0cfd4bee16f84b7c8b0ff3ce14db20d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251029 02:51:55.845266 1175 ts_tablet_manager.cc:1916] T 14698af6348f41e384f10569392d9b23 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:55.845872 1251 consensus_queue.cc:1048] T fa5127bcfcda40a7b5ad4ef016af072a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:51:55.852276 1175 ts_tablet_manager.cc:1929] T 14698af6348f41e384f10569392d9b23 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:55.852416 1176 ts_tablet_manager.cc:1929] T bf210435010d4dde8903fd116eb468e3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:51:55.852453 1175 log.cc:1199] T 14698af6348f41e384f10569392d9b23 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/14698af6348f41e384f10569392d9b23
I20251029 02:51:55.852845 1175 ts_tablet_manager.cc:1950] T 14698af6348f41e384f10569392d9b23 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:51:55.854485 1174 ts_tablet_manager.cc:1950] T deca0c4a114441079fce22e74b403050 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:51:55.855013 1176 log.cc:1199] T bf210435010d4dde8903fd116eb468e3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/bf210435010d4dde8903fd116eb468e3
I20251029 02:51:55.855419 1176 ts_tablet_manager.cc:1950] T bf210435010d4dde8903fd116eb468e3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
W20251029 02:51:55.856055 849 consensus_peers.cc:597] T 6dd4696ad4b44b6dab87a205f3f16ebd P c756528642a543eeacd93cef3e335a7d -> Peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Couldn't send request to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6dd4696ad4b44b6dab87a205f3f16ebd. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:51:55.856284 1167 ts_tablet_manager.cc:1434] T bc1a1cf274b64b52aba58811e9594330 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.045s user 0.002s sys 0.000s
I20251029 02:51:55.856488 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 189) took 2272 ms. Trace:
I20251029 02:51:55.857015 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 205) took 2195 ms. Trace:
I20251029 02:51:55.858783 847 leader_election.cc:304] T 41a6139fd9b84707a8c7eebc0381e383 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9
I20251029 02:51:55.858817 1174 rpcz_store.cc:276] 1029 02:51:53.584421 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:53.667606 (+ 83185us) service_pool.cc:225] Handling call
1029 02:51:55.744592 (+2076986us) ts_tablet_manager.cc:1151] Deleting tablet deca0c4a114441079fce22e74b403050
1029 02:51:55.744601 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:55.758668 (+ 14067us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:55.826557 (+ 67889us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:55.856444 (+ 29887us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2005511}
I20251029 02:51:55.859306 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 222) took 2127 ms. Trace:
I20251029 02:51:55.859666 1176 rpcz_store.cc:276] 1029 02:51:53.661080 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:53.661374 (+ 294us) service_pool.cc:225] Handling call
1029 02:51:55.789111 (+2127737us) ts_tablet_manager.cc:1151] Deleting tablet bf210435010d4dde8903fd116eb468e3
1029 02:51:55.789121 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:55.797173 (+ 8052us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:55.852271 (+ 55098us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:55.856972 (+ 4701us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2127583}
I20251029 02:51:55.860087 1167 tablet_bootstrap.cc:492] T 70e8cfe5eaa9492c85874dc3214a1e0b P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:51:55.860750 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 227) took 2117 ms. Trace:
I20251029 02:51:55.860932 1176 rpcz_store.cc:276] 1029 02:51:53.743050 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:53.743525 (+ 475us) service_pool.cc:225] Handling call
1029 02:51:55.860589 (+2117064us) ts_tablet_manager.cc:1151] Deleting tablet b2b2d64dcf964da8be4b544dbbe31977
1029 02:51:55.860600 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:55.860712 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2079524}
I20251029 02:51:55.862411 1176 tablet_replica.cc:333] T 61c1fec3ab4c402eb29f767b9bb7409f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:51:55.862413 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet deca0c4a114441079fce22e74b403050 (table default.Table_1bd19c9d52204fbf87ee3cdbc62984af [id=144a24e58314435d85f9c759ef79fa60]) successfully deleted
I20251029 02:51:55.863049 890 tablet_service.cc:1552] Processing DeleteTablet for tablet 012ce243fd07466ab5301d1c9ce4b413 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:55 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:55.863225 883 tablet_service.cc:1552] Processing DeleteTablet for tablet cb55b3d5bc63405f96cdc122f5a612cb with delete_type TABLET_DATA_DELETED (replaced by tablet aa776cde2f494fa5810539edf9f289a1) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:55.862928 1176 raft_consensus.cc:2243] T 61c1fec3ab4c402eb29f767b9bb7409f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:55.863492 1176 raft_consensus.cc:2272] T 61c1fec3ab4c402eb29f767b9bb7409f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:55.863626 881 tablet_service.cc:1552] Processing DeleteTablet for tablet b6d4f7bf702f4834bc5f4ef2b50b92a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:55 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:55.864146 896 tablet_service.cc:1552] Processing DeleteTablet for tablet c764062d3ba04d4ba8b2e367ad8ed433 with delete_type TABLET_DATA_DELETED (replaced by tablet 372d93914abc41b38a702742b7db0bc2) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:55.864051 1174 rpcz_store.cc:276] 1029 02:51:53.731638 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:53.849857 (+118219us) service_pool.cc:225] Handling call
1029 02:51:55.859138 (+2009281us) ts_tablet_manager.cc:1151] Deleting tablet b2b2d64dcf964da8be4b544dbbe31977
1029 02:51:55.859150 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:55.859265 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2004992}
I20251029 02:51:55.864360 1421 raft_consensus.cc:2749] T 41a6139fd9b84707a8c7eebc0381e383 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 02:51:55.864029 990 consensus_peers.cc:597] T fa5127bcfcda40a7b5ad4ef016af072a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 -> Peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Couldn't send request to peer c756528642a543eeacd93cef3e335a7d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:51:55.865372 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 14698af6348f41e384f10569392d9b23 (table default.Table_ef563eb489f947d4a4772678806a0777 [id=7de93d6d48ab4d58b5aed9319d7299a4]) successfully deleted
I20251029 02:51:55.866058 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 227) took 2017 ms. Trace:
I20251029 02:51:55.866189 1174 rpcz_store.cc:276] 1029 02:51:53.847444 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:54.026207 (+178763us) service_pool.cc:225] Handling call
1029 02:51:55.864299 (+1838092us) ts_tablet_manager.cc:1151] Deleting tablet 56d05e4a500043c995bb51a204cb295d
1029 02:51:55.864797 (+ 498us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:55.865088 (+ 291us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1837830}
W20251029 02:51:55.866328 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet b2b2d64dcf964da8be4b544dbbe31977 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b2b2d64dcf964da8be4b544dbbe31977 already in progress: creating tablet
I20251029 02:51:55.866583 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 229) took 1909 ms. Trace:
I20251029 02:51:55.866698 1174 rpcz_store.cc:276] 1029 02:51:54.001994 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:54.107720 (+105726us) service_pool.cc:225] Handling call
1029 02:51:55.866424 (+1758704us) ts_tablet_manager.cc:1151] Deleting tablet 0cc1957a845f4e9cb703a553b27d1347
1029 02:51:55.866432 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:55.866555 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1741921}
W20251029 02:51:55.866909 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 56d05e4a500043c995bb51a204cb295d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 56d05e4a500043c995bb51a204cb295d already in progress: creating tablet
I20251029 02:51:55.867043 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 230) took 1853 ms. Trace:
I20251029 02:51:55.867151 1174 rpcz_store.cc:276] 1029 02:51:54.013939 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:54.124536 (+110597us) service_pool.cc:225] Handling call
1029 02:51:55.866932 (+1742396us) ts_tablet_manager.cc:1151] Deleting tablet 519754f5d8f7484480f973c53708c5f3
1029 02:51:55.866941 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:55.867014 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1742185}
W20251029 02:51:55.867497 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 0cc1957a845f4e9cb703a553b27d1347 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0cc1957a845f4e9cb703a553b27d1347 already in progress: creating tablet
I20251029 02:51:55.867635 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 231) took 1843 ms. Trace:
I20251029 02:51:55.867749 1174 rpcz_store.cc:276] 1029 02:51:54.028575 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:54.124768 (+ 96193us) service_pool.cc:225] Handling call
1029 02:51:55.867375 (+1742607us) ts_tablet_manager.cc:1151] Deleting tablet 4add552cf4d940d2a0a36d6b58379a34
1029 02:51:55.867383 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:55.867478 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1741008}
W20251029 02:51:55.867935 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 519754f5d8f7484480f973c53708c5f3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 519754f5d8f7484480f973c53708c5f3 already in progress: creating tablet
I20251029 02:51:55.868090 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 232) took 1836 ms. Trace:
I20251029 02:51:55.868129 1175 tablet_replica.cc:333] T aa776cde2f494fa5810539edf9f289a1 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:51:55.868198 1174 rpcz_store.cc:276] 1029 02:51:54.031481 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:54.126397 (+ 94916us) service_pool.cc:225] Handling call
1029 02:51:55.867971 (+1741574us) ts_tablet_manager.cc:1151] Deleting tablet f60e7c24982d40319801869ad060864d
1029 02:51:55.867980 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:55.868049 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1741352}
I20251029 02:51:55.868353 1175 raft_consensus.cc:2243] T aa776cde2f494fa5810539edf9f289a1 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
W20251029 02:51:55.868463 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 4add552cf4d940d2a0a36d6b58379a34 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4add552cf4d940d2a0a36d6b58379a34 already in progress: creating tablet
I20251029 02:51:55.868520 1174 tablet_replica.cc:333] T b867feae24e741aca6902868c35c1428 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:51:55.868543 1175 raft_consensus.cc:2272] T aa776cde2f494fa5810539edf9f289a1 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:55.868721 1174 raft_consensus.cc:2243] T b867feae24e741aca6902868c35c1428 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:51:55.868959 1174 pending_rounds.cc:70] T b867feae24e741aca6902868c35c1428 P c756528642a543eeacd93cef3e335a7d: Trying to abort 1 pending ops.
W20251029 02:51:55.868975 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet f60e7c24982d40319801869ad060864d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f60e7c24982d40319801869ad060864d already in progress: creating tablet
I20251029 02:51:55.869048 1174 pending_rounds.cc:77] T b867feae24e741aca6902868c35c1428 P c756528642a543eeacd93cef3e335a7d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215949066047234048 op_type: NO_OP noop_request { }
I20251029 02:51:55.869168 1174 raft_consensus.cc:2889] T b867feae24e741aca6902868c35c1428 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 02:51:55.869293 1174 raft_consensus.cc:2272] T b867feae24e741aca6902868c35c1428 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:55.869493 1167 tablet_bootstrap.cc:654] T 70e8cfe5eaa9492c85874dc3214a1e0b P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:55.870272 1174 ts_tablet_manager.cc:1916] T b867feae24e741aca6902868c35c1428 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:55.870801 1176 ts_tablet_manager.cc:1916] T 61c1fec3ab4c402eb29f767b9bb7409f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:55.871403 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet bf210435010d4dde8903fd116eb468e3 (table default.Table_4bdbcbb87e5b40aeb9f98eb1aa0e6fe3 [id=78615d8807a74b4496a3223ef6fe709d]) successfully deleted
W20251029 02:51:55.871758 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet b2b2d64dcf964da8be4b544dbbe31977 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b2b2d64dcf964da8be4b544dbbe31977 already in progress: creating tablet
I20251029 02:51:55.872641 1175 ts_tablet_manager.cc:1916] T aa776cde2f494fa5810539edf9f289a1 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:55.873728 518 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet 5e31e061d74146009d24d5cd57487a5e (table default.Table_f2bc74933428449197f8c28182c8698f [id=8e17d6b87b984d878787802bd6f627f2]). Expected version 2 got 0
I20251029 02:51:55.881659 517 master_service.cc:954] ReplaceTablet: received request to replace tablet fd900eec3f0c43bf965c148ea90870a3 from {username='slave'} at 127.0.0.1:35314
I20251029 02:51:55.883502 517 master_service.cc:954] ReplaceTablet: received request to replace tablet 74873a05ef634867a30aa63b31fcfcc7 from {username='slave'} at 127.0.0.1:35314
I20251029 02:51:55.892213 1167 tablet_bootstrap.cc:492] T 70e8cfe5eaa9492c85874dc3214a1e0b P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:51:55.892416 1167 ts_tablet_manager.cc:1403] T 70e8cfe5eaa9492c85874dc3214a1e0b P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.036s user 0.007s sys 0.001s
I20251029 02:51:55.893406 1167 raft_consensus.cc:359] T 70e8cfe5eaa9492c85874dc3214a1e0b P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:55.893682 1167 raft_consensus.cc:385] T 70e8cfe5eaa9492c85874dc3214a1e0b P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:55.893775 1167 raft_consensus.cc:740] T 70e8cfe5eaa9492c85874dc3214a1e0b P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:51:55.894170 1167 consensus_queue.cc:260] T 70e8cfe5eaa9492c85874dc3214a1e0b P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:55.894729 1167 ts_tablet_manager.cc:1434] T 70e8cfe5eaa9492c85874dc3214a1e0b P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20251029 02:51:55.895231 1167 tablet_bootstrap.cc:492] T 4b76c4408a7340e59b8506171622175f P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:51:55.896270 1175 ts_tablet_manager.cc:1929] T aa776cde2f494fa5810539edf9f289a1 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:55.896422 1175 log.cc:1199] T aa776cde2f494fa5810539edf9f289a1 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/aa776cde2f494fa5810539edf9f289a1
I20251029 02:51:55.896826 1175 ts_tablet_manager.cc:1950] T aa776cde2f494fa5810539edf9f289a1 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:51:55.896739 519 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_eb28ff62f787444189cb7dfe9e735992" } new_table_name: "default.Table_b3ad8b7021294ca484b65c1773165849" modify_external_catalogs: true
I20251029 02:51:55.897877 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet aa776cde2f494fa5810539edf9f289a1 (table default.Table_47a1a59db0444fa49f7274509385ff0c [id=283c09ba60114ba9b165e2ff144a6c35]) successfully deleted
I20251029 02:51:55.898152 1167 tablet_bootstrap.cc:654] T 4b76c4408a7340e59b8506171622175f P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:55.901651 1176 ts_tablet_manager.cc:1929] T 61c1fec3ab4c402eb29f767b9bb7409f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:55.901813 1176 log.cc:1199] T 61c1fec3ab4c402eb29f767b9bb7409f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/61c1fec3ab4c402eb29f767b9bb7409f
I20251029 02:51:55.902200 1176 ts_tablet_manager.cc:1950] T 61c1fec3ab4c402eb29f767b9bb7409f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
W20251029 02:51:55.903349 522 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 46cd75ff18b14212b556b9bcb3012dd9 has no consensus state
W20251029 02:51:55.903834 511 catalog_manager.cc:4952] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 012ce243fd07466ab5301d1c9ce4b413 because the tablet was not found. No further retry: Not found: Tablet not found: 012ce243fd07466ab5301d1c9ce4b413
I20251029 02:51:55.904536 1175 tablet_replica.cc:333] T fd900eec3f0c43bf965c148ea90870a3 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:51:55.903205 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 232) took 2090 ms. Trace:
W20251029 02:51:55.903762 990 consensus_peers.cc:597] T 4b76c4408a7340e59b8506171622175f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 -> Peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Couldn't send request to peer c756528642a543eeacd93cef3e335a7d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:51:55.904707 1176 rpcz_store.cc:276] 1029 02:51:53.812482 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:53.812549 (+ 67us) service_pool.cc:225] Handling call
1029 02:51:55.862326 (+2049777us) ts_tablet_manager.cc:1151] Deleting tablet 61c1fec3ab4c402eb29f767b9bb7409f
1029 02:51:55.862338 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:55.881221 (+ 18883us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:55.901514 (+ 20293us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:55.902559 (+ 1045us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1990824}
I20251029 02:51:55.905099 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 61c1fec3ab4c402eb29f767b9bb7409f (table default.Table_ea7ce466c0ea44a8ac497c7dd4ead10f [id=c124cfa700724f6db0f1a12fe495c2af]) successfully deleted
I20251029 02:51:55.905704 1166 tablet_bootstrap.cc:492] T 59af906061d049c99ae2153945597007 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:51:55.905866 1166 ts_tablet_manager.cc:1403] T 59af906061d049c99ae2153945597007 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.076s user 0.001s sys 0.010s
I20251029 02:51:55.905766 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_19ec7789551742df938b5c458a04326d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:55.906315 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_19ec7789551742df938b5c458a04326d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:55.907208 1172 tablet_bootstrap.cc:492] T 65fdec7cf5d143208d78d4c3806b1352 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:51:55.907379 1172 ts_tablet_manager.cc:1403] T 65fdec7cf5d143208d78d4c3806b1352 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.141s user 0.008s sys 0.023s
I20251029 02:51:55.907666 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_4903da2131c140bca97524eda175ce3c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:55.907740 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 233) took 2035 ms. Trace:
I20251029 02:51:55.907835 1176 rpcz_store.cc:276] 1029 02:51:53.872457 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:53.872820 (+ 363us) service_pool.cc:225] Handling call
1029 02:51:55.905015 (+2032195us) ts_tablet_manager.cc:1151] Deleting tablet 56d05e4a500043c995bb51a204cb295d
1029 02:51:55.905024 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:55.907688 (+ 2664us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1995975}
W20251029 02:51:55.908154 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4903da2131c140bca97524eda175ce3c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:55.908226 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 237) took 1971 ms. Trace:
I20251029 02:51:55.908166 1172 raft_consensus.cc:359] T 65fdec7cf5d143208d78d4c3806b1352 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
W20251029 02:51:55.908226 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 56d05e4a500043c995bb51a204cb295d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 56d05e4a500043c995bb51a204cb295d already in progress: creating tablet
I20251029 02:51:55.908380 1172 raft_consensus.cc:385] T 65fdec7cf5d143208d78d4c3806b1352 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:55.908356 1176 rpcz_store.cc:276] 1029 02:51:53.936763 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:53.936873 (+ 110us) service_pool.cc:225] Handling call
1029 02:51:55.908075 (+1971202us) ts_tablet_manager.cc:1151] Deleting tablet 519754f5d8f7484480f973c53708c5f3
1029 02:51:55.908085 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:55.908174 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1968560}
I20251029 02:51:55.908494 1172 raft_consensus.cc:740] T 65fdec7cf5d143208d78d4c3806b1352 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:51:55.908701 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 236) took 1973 ms. Trace:
W20251029 02:51:55.908797 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 519754f5d8f7484480f973c53708c5f3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 519754f5d8f7484480f973c53708c5f3 already in progress: creating tablet
I20251029 02:51:55.908735 1172 consensus_queue.cc:260] T 65fdec7cf5d143208d78d4c3806b1352 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:55.908779 1176 rpcz_store.cc:276] 1029 02:51:53.935633 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:53.936410 (+ 777us) service_pool.cc:225] Handling call
1029 02:51:55.908568 (+1972158us) ts_tablet_manager.cc:1151] Deleting tablet 4add552cf4d940d2a0a36d6b58379a34
1029 02:51:55.908575 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:55.908651 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1968786}
I20251029 02:51:55.909108 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 238) took 1951 ms. Trace:
I20251029 02:51:55.909170 1172 ts_tablet_manager.cc:1434] T 65fdec7cf5d143208d78d4c3806b1352 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251029 02:51:55.909191 1176 rpcz_store.cc:276] 1029 02:51:53.957931 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:53.958012 (+ 81us) service_pool.cc:225] Handling call
1029 02:51:55.908969 (+1950957us) ts_tablet_manager.cc:1151] Deleting tablet f60e7c24982d40319801869ad060864d
1029 02:51:55.908977 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:55.909054 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1939405}
W20251029 02:51:55.909308 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 4add552cf4d940d2a0a36d6b58379a34 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4add552cf4d940d2a0a36d6b58379a34 already in progress: creating tablet
I20251029 02:51:55.909554 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 235) took 1981 ms. Trace:
I20251029 02:51:55.909643 1176 rpcz_store.cc:276] 1029 02:51:53.928119 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:53.937520 (+ 9401us) service_pool.cc:225] Handling call
1029 02:51:55.909403 (+1971883us) ts_tablet_manager.cc:1151] Deleting tablet 0cc1957a845f4e9cb703a553b27d1347
1029 02:51:55.909410 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:55.909500 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1938935}
W20251029 02:51:55.909858 512 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 4aa78f4ac38e4919864c924e50f1aaf4 (table default.Table_e168a30d3a1e48ed8053847b304036e9 [id=a93c34a9cd344503954bfae8752a69a6], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 4aa78f4ac38e4919864c924e50f1aaf4 consensus state has no leader
I20251029 02:51:55.909957 1166 raft_consensus.cc:359] T 59af906061d049c99ae2153945597007 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
W20251029 02:51:55.910140 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet f60e7c24982d40319801869ad060864d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f60e7c24982d40319801869ad060864d already in progress: creating tablet
W20251029 02:51:55.910588 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 0cc1957a845f4e9cb703a553b27d1347 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0cc1957a845f4e9cb703a553b27d1347 already in progress: creating tablet
I20251029 02:51:55.910723 1166 raft_consensus.cc:385] T 59af906061d049c99ae2153945597007 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:55.910830 1166 raft_consensus.cc:740] T 59af906061d049c99ae2153945597007 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:51:55.911185 1166 consensus_queue.cc:260] T 59af906061d049c99ae2153945597007 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:55.911913 1166 ts_tablet_manager.cc:1434] T 59af906061d049c99ae2153945597007 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20251029 02:51:55.912045 1172 tablet_bootstrap.cc:492] T 44784a3a760946e0be24743bc2fc8138 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:51:55.913121 1167 tablet_bootstrap.cc:492] T 4b76c4408a7340e59b8506171622175f P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:51:55.913293 1167 ts_tablet_manager.cc:1403] T 4b76c4408a7340e59b8506171622175f P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.000s
I20251029 02:51:55.914386 1167 raft_consensus.cc:359] T 4b76c4408a7340e59b8506171622175f P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:55.914657 1167 raft_consensus.cc:385] T 4b76c4408a7340e59b8506171622175f P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:55.914711 1172 tablet_bootstrap.cc:654] T 44784a3a760946e0be24743bc2fc8138 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:55.914774 1167 raft_consensus.cc:740] T 4b76c4408a7340e59b8506171622175f P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:51:55.915019 1167 consensus_queue.cc:260] T 4b76c4408a7340e59b8506171622175f P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:55.915903 1175 raft_consensus.cc:2243] T fd900eec3f0c43bf965c148ea90870a3 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:55.916070 1175 raft_consensus.cc:2272] T fd900eec3f0c43bf965c148ea90870a3 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:55.916601 1175 ts_tablet_manager.cc:1916] T fd900eec3f0c43bf965c148ea90870a3 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:55.917963 1166 tablet_bootstrap.cc:492] T c519832030884a049839d309cc738c28 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:51:55.920320 1166 tablet_bootstrap.cc:654] T c519832030884a049839d309cc738c28 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:55.921129 1251 consensus_queue.cc:1048] T 4b76c4408a7340e59b8506171622175f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:51:55.926596 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 248) took 1647 ms. Trace:
I20251029 02:51:55.926774 1176 rpcz_store.cc:276] 1029 02:51:54.262166 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:54.269451 (+ 7285us) service_pool.cc:225] Handling call
1029 02:51:55.909853 (+1640402us) ts_tablet_manager.cc:1151] Deleting tablet 392a36a6b59a42f78e3ac7c8fb63214b
1029 02:51:55.909861 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:55.909931 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1640187}
W20251029 02:51:55.927357 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 392a36a6b59a42f78e3ac7c8fb63214b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 392a36a6b59a42f78e3ac7c8fb63214b already in progress: creating tablet
I20251029 02:51:55.927372 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 260) took 1292 ms. Trace:
I20251029 02:51:55.927479 1176 rpcz_store.cc:276] 1029 02:51:54.634405 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:54.634496 (+ 91us) service_pool.cc:225] Handling call
1029 02:51:55.927143 (+1292647us) ts_tablet_manager.cc:1151] Deleting tablet 44784a3a760946e0be24743bc2fc8138
1029 02:51:55.927157 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:55.927302 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1289735}
W20251029 02:51:55.928047 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 44784a3a760946e0be24743bc2fc8138 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 44784a3a760946e0be24743bc2fc8138 already in progress: creating tablet
I20251029 02:51:55.930656 1445 raft_consensus.cc:493] T 9964ad774890422f8e14f6b99f28cb55 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:55.930830 1445 raft_consensus.cc:515] T 9964ad774890422f8e14f6b99f28cb55 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:55.931425 1445 leader_election.cc:290] T 9964ad774890422f8e14f6b99f28cb55 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:51:55.932327 1059 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9964ad774890422f8e14f6b99f28cb55" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:51:55.932682 1166 tablet_bootstrap.cc:492] T c519832030884a049839d309cc738c28 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:51:55.932847 1166 ts_tablet_manager.cc:1403] T c519832030884a049839d309cc738c28 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.000s
W20251029 02:51:55.933279 712 leader_election.cc:343] T 9964ad774890422f8e14f6b99f28cb55 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:55.933526 1176 tablet_replica.cc:333] T ba014ecfec7f4602ae1c09d931bac824 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:51:55.933753 1176 raft_consensus.cc:2243] T ba014ecfec7f4602ae1c09d931bac824 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:55.935890 1167 ts_tablet_manager.cc:1434] T 4b76c4408a7340e59b8506171622175f P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20251029 02:51:55.936259 1167 tablet_bootstrap.cc:492] T ab421dfaa3f84593842bb732a2693160 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:51:55.936431 1444 tablet.cc:1722] T fa5127bcfcda40a7b5ad4ef016af072a P b495a49349324f7d81908f2da700f0d9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 02:51:55.938905 526 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_cd7b2ea5d1b240d087c135cfec7dd0be" } new_table_name: "default.Table_ae16b33c42244fc393dd1f163867961a" modify_external_catalogs: true
I20251029 02:51:55.938942 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9964ad774890422f8e14f6b99f28cb55" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:51:55.940342 711 leader_election.cc:343] T 9964ad774890422f8e14f6b99f28cb55 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:55.940490 711 leader_election.cc:304] T 9964ad774890422f8e14f6b99f28cb55 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:51:55.940654 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_eb12e18b799a47d98ae1f3e8ebfa3588"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:55.941090 1167 tablet_bootstrap.cc:654] T ab421dfaa3f84593842bb732a2693160 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:55.941071 1445 raft_consensus.cc:2749] T 9964ad774890422f8e14f6b99f28cb55 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 02:51:55.941195 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eb12e18b799a47d98ae1f3e8ebfa3588 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:55.941509 1176 raft_consensus.cc:2272] T ba014ecfec7f4602ae1c09d931bac824 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:55.941797 1174 ts_tablet_manager.cc:1929] T b867feae24e741aca6902868c35c1428 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:51:55.942094 1174 log.cc:1199] T b867feae24e741aca6902868c35c1428 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/b867feae24e741aca6902868c35c1428
I20251029 02:51:55.942107 1176 ts_tablet_manager.cc:1916] T ba014ecfec7f4602ae1c09d931bac824 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:55.942229 1440 tablet.cc:1722] T fa5127bcfcda40a7b5ad4ef016af072a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 02:51:55.943408 1172 tablet_bootstrap.cc:492] T 44784a3a760946e0be24743bc2fc8138 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:51:55.943584 1172 ts_tablet_manager.cc:1403] T 44784a3a760946e0be24743bc2fc8138 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.033s user 0.001s sys 0.004s
I20251029 02:51:55.944381 1172 raft_consensus.cc:359] T 44784a3a760946e0be24743bc2fc8138 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:55.944587 1172 raft_consensus.cc:385] T 44784a3a760946e0be24743bc2fc8138 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:55.946012 1174 ts_tablet_manager.cc:1950] T b867feae24e741aca6902868c35c1428 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:51:55.947072 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 233) took 1914 ms. Trace:
I20251029 02:51:55.964732 520 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet 4bf06476c87a4549a73f5994314bb75f (table default.Table_bafe26b1fca14cc9b908984b7d8a2442 [id=046010a03d774e91b0ad10df97848c56]). Expected version 1 got 0
I20251029 02:51:55.965111 520 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet 9e07291d6b5645feb4d5890f58e1ef74 (table default.Table_91cd5889c90148e5bde6051f2847832a [id=e71aea48b9884bacb250412624ec4615]). Expected version 4 got 0
I20251029 02:51:55.966166 523 catalog_manager.cc:6194] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: default.Table_f2bc74933428449197f8c28182c8698f [id=8e17d6b87b984d878787802bd6f627f2] alter complete (version 2)
I20251029 02:51:55.971736 743 tablet_service.cc:1505] Processing CreateTablet for tablet 946f65014d97425aab3851009b742807 (DEFAULT_TABLE table=default.Table_7dd1cd855bb74080b6e242847e5faac6 [id=8a3311a64d244193985716c908a42f27]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:55.972278 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 946f65014d97425aab3851009b742807. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:55.975075 738 tablet_service.cc:1552] Processing DeleteTablet for tablet b5d5b737c86844d58c51b2388c0b1705 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:54 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:55.971771 747 tablet_service.cc:1505] Processing CreateTablet for tablet 44ae2b07edca44afbe1b1fb3a5b1115d (DEFAULT_TABLE table=default.Table_c2719a3251c84739a6681f80f1d8f94d [id=a746921b2aba4cd6af22a293991968e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:55.980016 756 tablet_service.cc:1505] Processing CreateTablet for tablet c32c28b0e74a4af3a0e69bbf24ab1a1f (DEFAULT_TABLE table=default.Table_3d94f8b10e884dc0b338dcfbb9c82309 [id=3db1589f811041eaa1cd7412a26391bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:55.980053 524 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_e4de02c7c1f04704bc39e1f0efd2c9d9" } modify_external_catalogs: true
I20251029 02:51:55.980276 524 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_e4de02c7c1f04704bc39e1f0efd2c9d9" } modify_external_catalogs: true
I20251029 02:51:55.980432 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44ae2b07edca44afbe1b1fb3a5b1115d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:55.981829 520 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet 4b76c4408a7340e59b8506171622175f (table default.Table_1281926bf1224c0cb4541bf561102021 [id=f3fa2dbbef7e47359492d5a5eef9b8fd]). Expected version 2 got 0
I20251029 02:51:55.982637 739 tablet_service.cc:1552] Processing DeleteTablet for tablet 9964ad774890422f8e14f6b99f28cb55 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:53 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:55.984380 518 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet 4b76c4408a7340e59b8506171622175f (table default.Table_1281926bf1224c0cb4541bf561102021 [id=f3fa2dbbef7e47359492d5a5eef9b8fd]). Expected version 2 got 0
I20251029 02:51:55.986343 1350 raft_consensus.cc:493] T dd90fb9eb167478cbc3ec9363af86295 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:55.986496 748 tablet_service.cc:1505] Processing CreateTablet for tablet 97b2ecae572a4b7ea1b9595074556708 (DEFAULT_TABLE table=default.Table_5c973aa3bd7f4a9ebf8435ed761552a5 [id=222ef1b283554565b4930328c2c8975b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:55.987098 738 tablet_service.cc:1505] Processing CreateTablet for tablet 71b4b91071a54d6aa66160e7955bc915 (DEFAULT_TABLE table=default.Table_457a3835f7d84fd7bf722dfc1d18d301 [id=68645701bb494d9fa65e7c1ff7c9e14c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:55.987547 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c32c28b0e74a4af3a0e69bbf24ab1a1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:55.988423 1042 tablet_service.cc:1505] Processing CreateTablet for tablet 946f65014d97425aab3851009b742807 (DEFAULT_TABLE table=default.Table_7dd1cd855bb74080b6e242847e5faac6 [id=8a3311a64d244193985716c908a42f27]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 02:51:55.990351 519 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dd90fb9eb167478cbc3ec9363af86295 consensus state has no leader
I20251029 02:51:55.990474 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71b4b91071a54d6aa66160e7955bc915. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:55.964737 896 tablet_service.cc:1505] Processing CreateTablet for tablet 44ae2b07edca44afbe1b1fb3a5b1115d (DEFAULT_TABLE table=default.Table_c2719a3251c84739a6681f80f1d8f94d [id=a746921b2aba4cd6af22a293991968e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:55.991493 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97b2ecae572a4b7ea1b9595074556708. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:55.991753 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44ae2b07edca44afbe1b1fb3a5b1115d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:55.975080 881 tablet_service.cc:1552] Processing DeleteTablet for tablet bc1a1cf274b64b52aba58811e9594330 with delete_type TABLET_DATA_DELETED (replaced by tablet f81aecb20d6e498bb9b498698886d65e) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:55.975639 883 tablet_service.cc:1505] Processing CreateTablet for tablet 946f65014d97425aab3851009b742807 (DEFAULT_TABLE table=default.Table_7dd1cd855bb74080b6e242847e5faac6 [id=8a3311a64d244193985716c908a42f27]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:55.995224 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 946f65014d97425aab3851009b742807. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:55.976195 890 tablet_service.cc:1505] Processing CreateTablet for tablet c32c28b0e74a4af3a0e69bbf24ab1a1f (DEFAULT_TABLE table=default.Table_3d94f8b10e884dc0b338dcfbb9c82309 [id=3db1589f811041eaa1cd7412a26391bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:55.976835 887 tablet_service.cc:1505] Processing CreateTablet for tablet 71b4b91071a54d6aa66160e7955bc915 (DEFAULT_TABLE table=default.Table_457a3835f7d84fd7bf722dfc1d18d301 [id=68645701bb494d9fa65e7c1ff7c9e14c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:55.979511 898 tablet_service.cc:1505] Processing CreateTablet for tablet 97b2ecae572a4b7ea1b9595074556708 (DEFAULT_TABLE table=default.Table_5c973aa3bd7f4a9ebf8435ed761552a5 [id=222ef1b283554565b4930328c2c8975b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:56.001375 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c32c28b0e74a4af3a0e69bbf24ab1a1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:56.001924 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71b4b91071a54d6aa66160e7955bc915. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:56.002367 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97b2ecae572a4b7ea1b9595074556708. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:56.011885 523 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_d8875cc2c2f14a298da5a7c3c7678354" } new_table_name: "default.Table_7f609ea711d84843b92af7fc7c328804" modify_external_catalogs: true
I20251029 02:51:56.012301 519 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_b93cf815f69a4fe895e2ce461148fc71" } modify_external_catalogs: true
I20251029 02:51:56.013406 519 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_b93cf815f69a4fe895e2ce461148fc71" } modify_external_catalogs: true
I20251029 02:51:56.017098 518 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_f2bc74933428449197f8c28182c8698f" } modify_external_catalogs: true
I20251029 02:51:56.017310 518 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_f2bc74933428449197f8c28182c8698f" } modify_external_catalogs: true
I20251029 02:51:56.022038 920 raft_consensus.cc:3060] T 41a6139fd9b84707a8c7eebc0381e383 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:51:55.944660 1172 raft_consensus.cc:740] T 44784a3a760946e0be24743bc2fc8138 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:51:56.024078 1172 consensus_queue.cc:260] T 44784a3a760946e0be24743bc2fc8138 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:56.024840 1172 ts_tablet_manager.cc:1434] T 44784a3a760946e0be24743bc2fc8138 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.081s user 0.000s sys 0.002s
I20251029 02:51:56.027367 738 tablet_service.cc:1505] Processing CreateTablet for tablet 82289e0f823c4d43bfd8c4e878572b2b (DEFAULT_TABLE table=default.Table_b6110a49346640fb8926a871940e1aad [id=2cca3b7c50b243b5883b8ac1f353f253]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:56.027868 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82289e0f823c4d43bfd8c4e878572b2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:56.029147 881 tablet_service.cc:1505] Processing CreateTablet for tablet 82289e0f823c4d43bfd8c4e878572b2b (DEFAULT_TABLE table=default.Table_b6110a49346640fb8926a871940e1aad [id=2cca3b7c50b243b5883b8ac1f353f253]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:56.035547 522 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_0d5730d385074a70b295b233e6ecd79e" } modify_external_catalogs: true
I20251029 02:51:56.035799 522 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_0d5730d385074a70b295b233e6ecd79e" } modify_external_catalogs: true
I20251029 02:51:56.037842 1029 tablet_service.cc:1505] Processing CreateTablet for tablet 82289e0f823c4d43bfd8c4e878572b2b (DEFAULT_TABLE table=default.Table_b6110a49346640fb8926a871940e1aad [id=2cca3b7c50b243b5883b8ac1f353f253]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:56.038614 1176 ts_tablet_manager.cc:1929] T ba014ecfec7f4602ae1c09d931bac824 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251029 02:51:56.039371 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82289e0f823c4d43bfd8c4e878572b2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:56.040474 1172 tablet_bootstrap.cc:492] T fd0f89180aba43eabb20f926abccbaa6 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:51:56.043984 738 tablet_service.cc:1505] Processing CreateTablet for tablet 217144957930495aa3e9dfcfde268e71 (DEFAULT_TABLE table=default.Table_ed30c800e4eb4970a4b03299c936e7ed [id=3241aaa6ea9148b7a101833462ab3eeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:56.044646 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 217144957930495aa3e9dfcfde268e71. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:56.046119 878 tablet_service.cc:1505] Processing CreateTablet for tablet 217144957930495aa3e9dfcfde268e71 (DEFAULT_TABLE table=default.Table_ed30c800e4eb4970a4b03299c936e7ed [id=3241aaa6ea9148b7a101833462ab3eeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:56.046697 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 217144957930495aa3e9dfcfde268e71. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:56.053462 517 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_ea7ce466c0ea44a8ac497c7dd4ead10f" } new_table_name: "default.Table_8cf076425649436289d327edc92d4642" modify_external_catalogs: true
I20251029 02:51:56.063573 887 tablet_service.cc:1505] Processing CreateTablet for tablet c1222e30bc4141b3ae686d80053c4ef5 (DEFAULT_TABLE table=default.Table_a63991c194724381a84c0ce2adab25da [id=31801defa6db4ca7a08fbf7f029b65c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:56.064258 1175 ts_tablet_manager.cc:1929] T fd900eec3f0c43bf965c148ea90870a3 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:56.064474 1175 log.cc:1199] T fd900eec3f0c43bf965c148ea90870a3 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/fd900eec3f0c43bf965c148ea90870a3
I20251029 02:51:56.064913 1175 ts_tablet_manager.cc:1950] T fd900eec3f0c43bf965c148ea90870a3 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:51:56.065469 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1222e30bc4141b3ae686d80053c4ef5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:56.066259 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet fd900eec3f0c43bf965c148ea90870a3 (table default.Table_e0e6e47b18d749e8ae80f822af1e23f0 [id=125ae5c7b5074378bcecb546e70f0c13]) successfully deleted
I20251029 02:51:56.066846 1350 raft_consensus.cc:515] T dd90fb9eb167478cbc3ec9363af86295 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:56.067651 1350 leader_election.cc:290] T dd90fb9eb167478cbc3ec9363af86295 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:51:56.068470 918 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd90fb9eb167478cbc3ec9363af86295" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:51:56.068542 1028 tablet_service.cc:1505] Processing CreateTablet for tablet 44ae2b07edca44afbe1b1fb3a5b1115d (DEFAULT_TABLE table=default.Table_c2719a3251c84739a6681f80f1d8f94d [id=a746921b2aba4cd6af22a293991968e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:56.069777 1038 tablet_service.cc:1505] Processing CreateTablet for tablet 217144957930495aa3e9dfcfde268e71 (DEFAULT_TABLE table=default.Table_ed30c800e4eb4970a4b03299c936e7ed [id=3241aaa6ea9148b7a101833462ab3eeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:56.069846 1041 tablet_service.cc:1505] Processing CreateTablet for tablet c32c28b0e74a4af3a0e69bbf24ab1a1f (DEFAULT_TABLE table=default.Table_3d94f8b10e884dc0b338dcfbb9c82309 [id=3db1589f811041eaa1cd7412a26391bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:56.070128 1025 tablet_service.cc:1505] Processing CreateTablet for tablet 71b4b91071a54d6aa66160e7955bc915 (DEFAULT_TABLE table=default.Table_457a3835f7d84fd7bf722dfc1d18d301 [id=68645701bb494d9fa65e7c1ff7c9e14c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:56.070477 1027 tablet_service.cc:1505] Processing CreateTablet for tablet 97b2ecae572a4b7ea1b9595074556708 (DEFAULT_TABLE table=default.Table_5c973aa3bd7f4a9ebf8435ed761552a5 [id=222ef1b283554565b4930328c2c8975b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:56.070492 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd90fb9eb167478cbc3ec9363af86295" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:51:56.070648 1176 log.cc:1199] T ba014ecfec7f4602ae1c09d931bac824 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/ba014ecfec7f4602ae1c09d931bac824
I20251029 02:51:56.070740 776 raft_consensus.cc:2468] T dd90fb9eb167478cbc3ec9363af86295 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 0.
I20251029 02:51:56.071040 1030 tablet_service.cc:1552] Processing DeleteTablet for tablet ab421dfaa3f84593842bb732a2693160 with delete_type TABLET_DATA_DELETED (replaced by tablet c9e923724ca9402682f13b252dbc37aa) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:56.071158 1176 ts_tablet_manager.cc:1950] T ba014ecfec7f4602ae1c09d931bac824 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:51:56.071261 989 leader_election.cc:304] T dd90fb9eb167478cbc3ec9363af86295 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters:
W20251029 02:51:56.071379 526 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ba699862e6f646c6beaa39f080d0faba consensus state has no leader
I20251029 02:51:56.071524 1465 raft_consensus.cc:493] T ab421dfaa3f84593842bb732a2693160 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:56.071554 1395 raft_consensus.cc:493] T 64b0fb400e944ce2b2c7a49d12135e9c P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:56.071702 1465 raft_consensus.cc:515] T ab421dfaa3f84593842bb732a2693160 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:56.072319 1465 leader_election.cc:290] T ab421dfaa3f84593842bb732a2693160 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:51:56.071704 1395 raft_consensus.cc:515] T 64b0fb400e944ce2b2c7a49d12135e9c P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:56.072474 1350 raft_consensus.cc:2804] T dd90fb9eb167478cbc3ec9363af86295 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:51:56.072578 1350 raft_consensus.cc:493] T dd90fb9eb167478cbc3ec9363af86295 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:51:56.072685 1350 raft_consensus.cc:3060] T dd90fb9eb167478cbc3ec9363af86295 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:51:56.072983 1395 leader_election.cc:290] T 64b0fb400e944ce2b2c7a49d12135e9c P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
W20251029 02:51:56.073704 989 leader_election.cc:343] T 64b0fb400e944ce2b2c7a49d12135e9c P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 64b0fb400e944ce2b2c7a49d12135e9c
I20251029 02:51:56.074306 526 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_f5db3e3913ca4ea98b02981cd04a5965" } modify_external_catalogs: true
I20251029 02:51:56.074481 526 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_f5db3e3913ca4ea98b02981cd04a5965" } modify_external_catalogs: true
I20251029 02:51:56.075505 1350 raft_consensus.cc:515] T dd90fb9eb167478cbc3ec9363af86295 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
W20251029 02:51:56.076671 990 leader_election.cc:343] T dd90fb9eb167478cbc3ec9363af86295 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:56.078150 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "64b0fb400e944ce2b2c7a49d12135e9c" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:51:56.077898 918 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ab421dfaa3f84593842bb732a2693160" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:51:56.078702 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd90fb9eb167478cbc3ec9363af86295" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
I20251029 02:51:56.079221 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 946f65014d97425aab3851009b742807. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:56.082574 1174 rpcz_store.cc:276] 1029 02:51:54.032479 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:54.126635 (+ 94156us) service_pool.cc:225] Handling call
1029 02:51:55.868466 (+1741831us) ts_tablet_manager.cc:1151] Deleting tablet b867feae24e741aca6902868c35c1428
1029 02:51:55.868475 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:55.873749 (+ 5274us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:55.916674 (+ 42925us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:55.947032 (+ 30358us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1734123}
W20251029 02:51:56.082855 990 leader_election.cc:343] T 64b0fb400e944ce2b2c7a49d12135e9c P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:56.082978 990 leader_election.cc:304] T 64b0fb400e944ce2b2c7a49d12135e9c P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5; no voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d
I20251029 02:51:56.083004 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 243) took 1840 ms. Trace:
I20251029 02:51:56.083110 1174 rpcz_store.cc:276] 1029 02:51:54.242955 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:54.441223 (+198268us) service_pool.cc:225] Handling call
1029 02:51:56.082866 (+1641643us) ts_tablet_manager.cc:1151] Deleting tablet 392a36a6b59a42f78e3ac7c8fb63214b
1029 02:51:56.082876 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:56.082973 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1597224}
I20251029 02:51:56.083382 1395 raft_consensus.cc:2749] T 64b0fb400e944ce2b2c7a49d12135e9c P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:51:56.083729 1030 tablet_service.cc:1505] Processing CreateTablet for tablet c1222e30bc4141b3ae686d80053c4ef5 (DEFAULT_TABLE table=default.Table_a63991c194724381a84c0ce2adab25da [id=31801defa6db4ca7a08fbf7f029b65c1]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 02:51:56.084087 990 leader_election.cc:343] T ab421dfaa3f84593842bb732a2693160 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
W20251029 02:51:56.084283 990 leader_election.cc:343] T dd90fb9eb167478cbc3ec9363af86295 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:56.084422 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82289e0f823c4d43bfd8c4e878572b2b. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:51:56.084731 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 392a36a6b59a42f78e3ac7c8fb63214b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 392a36a6b59a42f78e3ac7c8fb63214b already in progress: creating tablet
I20251029 02:51:56.085105 1350 leader_election.cc:290] T dd90fb9eb167478cbc3ec9363af86295 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Requested vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:51:56.085461 1166 raft_consensus.cc:359] T c519832030884a049839d309cc738c28 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:56.085654 1166 raft_consensus.cc:385] T c519832030884a049839d309cc738c28 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:56.085748 1166 raft_consensus.cc:740] T c519832030884a049839d309cc738c28 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:51:56.085987 1166 consensus_queue.cc:260] T c519832030884a049839d309cc738c28 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:56.086540 1166 ts_tablet_manager.cc:1434] T c519832030884a049839d309cc738c28 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.154s user 0.003s sys 0.000s
I20251029 02:51:56.087913 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd90fb9eb167478cbc3ec9363af86295" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:51:56.088126 774 raft_consensus.cc:3060] T dd90fb9eb167478cbc3ec9363af86295 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:51:56.089517 525 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet 59af906061d049c99ae2153945597007 (table default.Table_3bc6a1815ce646fb9a89e91fdda46ae0 [id=25f16d95843d41c981d6a07ec0d4b48a]). Expected version 1 got 0
I20251029 02:51:56.089792 525 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet 4b76c4408a7340e59b8506171622175f (table default.Table_1281926bf1224c0cb4541bf561102021 [id=f3fa2dbbef7e47359492d5a5eef9b8fd]). Expected version 2 got 0
W20251029 02:51:56.091049 989 leader_election.cc:343] T ab421dfaa3f84593842bb732a2693160 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: ab421dfaa3f84593842bb732a2693160
I20251029 02:51:56.091213 989 leader_election.cc:304] T ab421dfaa3f84593842bb732a2693160 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5; no voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d
I20251029 02:51:56.091341 774 raft_consensus.cc:2468] T dd90fb9eb167478cbc3ec9363af86295 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 1.
I20251029 02:51:56.091809 989 leader_election.cc:304] T dd90fb9eb167478cbc3ec9363af86295 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters: c756528642a543eeacd93cef3e335a7d
I20251029 02:51:56.087730 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 255) took 1722 ms. Trace:
I20251029 02:51:56.094323 1395 raft_consensus.cc:2749] T ab421dfaa3f84593842bb732a2693160 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:51:56.094581 1395 raft_consensus.cc:2804] T dd90fb9eb167478cbc3ec9363af86295 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:51:56.094722 1395 raft_consensus.cc:697] T dd90fb9eb167478cbc3ec9363af86295 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Becoming Leader. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Running, Role: LEADER
I20251029 02:51:56.094830 1174 rpcz_store.cc:276] 1029 02:51:54.365205 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:54.620920 (+255715us) service_pool.cc:225] Handling call
1029 02:51:56.087587 (+1466667us) ts_tablet_manager.cc:1151] Deleting tablet 44784a3a760946e0be24743bc2fc8138
1029 02:51:56.087598 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:56.087698 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1462206}
I20251029 02:51:56.094969 1395 consensus_queue.cc:237] T dd90fb9eb167478cbc3ec9363af86295 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
W20251029 02:51:56.095690 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 44784a3a760946e0be24743bc2fc8138 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 44784a3a760946e0be24743bc2fc8138 already in progress: creating tablet
I20251029 02:51:56.096189 1172 tablet_bootstrap.cc:654] T fd0f89180aba43eabb20f926abccbaa6 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:56.096890 878 tablet_service.cc:1505] Processing CreateTablet for tablet 2f198c3259864f7ab3625cb7bfae20b0 (DEFAULT_TABLE table=default.Table_bd9202984dc14b3b983326f60258c754 [id=991a151cc5d84421bf50858b1fa864e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:56.097406 1024 tablet_service.cc:1505] Processing CreateTablet for tablet 2f198c3259864f7ab3625cb7bfae20b0 (DEFAULT_TABLE table=default.Table_bd9202984dc14b3b983326f60258c754 [id=991a151cc5d84421bf50858b1fa864e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:56.097587 1166 tablet_bootstrap.cc:492] T 7c041f4909274cd5a03b9fd78353faf9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:51:56.099943 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 259) took 1466 ms. Trace:
I20251029 02:51:56.100298 1176 rpcz_store.cc:276] 1029 02:51:54.633122 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:54.651496 (+ 18374us) service_pool.cc:225] Handling call
1029 02:51:55.932854 (+1281358us) ts_tablet_manager.cc:1151] Deleting tablet ba014ecfec7f4602ae1c09d931bac824
1029 02:51:55.932863 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:55.956398 (+ 23535us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:55.978884 (+ 22486us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:56.099812 (+120928us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1281117}
I20251029 02:51:56.101352 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f198c3259864f7ab3625cb7bfae20b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:56.101348 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 262) took 1454 ms. Trace:
I20251029 02:51:56.101783 521 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 6ca9bd37de9e4be388501b15af4bda31
I20251029 02:51:56.101658 1176 rpcz_store.cc:276] 1029 02:51:54.646896 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:54.654560 (+ 7664us) service_pool.cc:225] Handling call
1029 02:51:56.100952 (+1446392us) ts_tablet_manager.cc:1151] Deleting tablet b5d5b737c86844d58c51b2388c0b1705
1029 02:51:56.101001 (+ 49us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:56.101219 (+ 218us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1446157}
I20251029 02:51:56.102285 521 catalog_manager.cc:6719] ReplaceTablet: tablet 6ca9bd37de9e4be388501b15af4bda31 deleted and replaced by tablet 20a762344c5f46f88481c4ac7ad6e43b
I20251029 02:51:56.102720 1174 tablet_replica.cc:333] T ba014ecfec7f4602ae1c09d931bac824 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:51:56.102967 1174 raft_consensus.cc:2243] T ba014ecfec7f4602ae1c09d931bac824 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:56.102689 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 261) took 1465 ms. Trace:
I20251029 02:51:56.103169 1174 raft_consensus.cc:2272] T ba014ecfec7f4602ae1c09d931bac824 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:56.103300 748 tablet_service.cc:1552] Processing DeleteTablet for tablet 6ca9bd37de9e4be388501b15af4bda31 with delete_type TABLET_DATA_DELETED (replaced by tablet 20a762344c5f46f88481c4ac7ad6e43b) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:56.103173 1176 rpcz_store.cc:276] 1029 02:51:54.637165 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:54.654863 (+ 17698us) service_pool.cc:225] Handling call
1029 02:51:56.102274 (+1447411us) ts_tablet_manager.cc:1151] Deleting tablet 6cd7009e06a646acad0802918c2b425e
1029 02:51:56.102285 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:56.102572 (+ 287us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1447123}
I20251029 02:51:56.104672 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97b2ecae572a4b7ea1b9595074556708. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:56.104914 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44ae2b07edca44afbe1b1fb3a5b1115d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:56.105906 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet ba014ecfec7f4602ae1c09d931bac824 (table default.Table_64e512f8ce034224aea9b887ed0d09dc [id=7bb8b246fbf1431bb71bbb7c5520e301]) successfully deleted
W20251029 02:51:56.106205 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet b5d5b737c86844d58c51b2388c0b1705 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b5d5b737c86844d58c51b2388c0b1705 already in progress: creating tablet
W20251029 02:51:56.106418 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 6cd7009e06a646acad0802918c2b425e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6cd7009e06a646acad0802918c2b425e already in progress: creating tablet
I20251029 02:51:56.107332 1037 tablet_service.cc:1552] Processing DeleteTablet for tablet 6ca9bd37de9e4be388501b15af4bda31 with delete_type TABLET_DATA_DELETED (replaced by tablet 20a762344c5f46f88481c4ac7ad6e43b) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:56.107509 898 tablet_service.cc:1552] Processing DeleteTablet for tablet 6ca9bd37de9e4be388501b15af4bda31 with delete_type TABLET_DATA_DELETED (replaced by tablet 20a762344c5f46f88481c4ac7ad6e43b) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:56.107745 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_c4333d8c61d346b1ac1a809d3ccb1915"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:56.108340 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c4333d8c61d346b1ac1a809d3ccb1915 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:56.110638 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c32c28b0e74a4af3a0e69bbf24ab1a1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:56.110387 748 tablet_service.cc:1505] Processing CreateTablet for tablet 2f198c3259864f7ab3625cb7bfae20b0 (DEFAULT_TABLE table=default.Table_bd9202984dc14b3b983326f60258c754 [id=991a151cc5d84421bf50858b1fa864e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:56.111011 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f198c3259864f7ab3625cb7bfae20b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:56.113942 898 tablet_service.cc:1505] Processing CreateTablet for tablet e65eaaae43ac435ebd665546de9d68dd (DEFAULT_TABLE table=default.Table_d9fe74a49f4843b089aae7fca9e4a826 [id=152882c742b2459ba863fd41083284c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:56.114996 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 71b4b91071a54d6aa66160e7955bc915. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:56.115046 524 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet f246f6c202cf44d48c9f5e4f335202c5
I20251029 02:51:56.115825 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 217144957930495aa3e9dfcfde268e71. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:56.116303 743 tablet_service.cc:1505] Processing CreateTablet for tablet e65eaaae43ac435ebd665546de9d68dd (DEFAULT_TABLE table=default.Table_d9fe74a49f4843b089aae7fca9e4a826 [id=152882c742b2459ba863fd41083284c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:56.116710 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e65eaaae43ac435ebd665546de9d68dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:56.117012 883 tablet_service.cc:1552] Processing DeleteTablet for tablet f246f6c202cf44d48c9f5e4f335202c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:55 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:56.118111 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_93b405a4f257481289518c5ff28832a0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:56.118752 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_93b405a4f257481289518c5ff28832a0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:56.119083 1421 raft_consensus.cc:493] T 5e31e061d74146009d24d5cd57487a5e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:56.119217 1037 tablet_service.cc:1552] Processing DeleteTablet for tablet f246f6c202cf44d48c9f5e4f335202c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:55 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:56.119226 1421 raft_consensus.cc:515] T 5e31e061d74146009d24d5cd57487a5e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:56.119876 1421 leader_election.cc:290] T 5e31e061d74146009d24d5cd57487a5e P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:51:56.120488 1059 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e31e061d74146009d24d5cd57487a5e" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:51:56.120632 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5e31e061d74146009d24d5cd57487a5e" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:51:56.121347 847 leader_election.cc:304] T 5e31e061d74146009d24d5cd57487a5e P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9
I20251029 02:51:56.121445 883 tablet_service.cc:1505] Processing CreateTablet for tablet 44c0488eba0543078ff4f0b3d69366f2 (DEFAULT_TABLE table=default.Table_bc32d7beda4f4957817b20124bf48e9c [id=df809b5c66d64eb1846fa5eb9da11781]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:56.121770 1421 raft_consensus.cc:2749] T 5e31e061d74146009d24d5cd57487a5e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:51:56.121888 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44c0488eba0543078ff4f0b3d69366f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:56.125484 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e65eaaae43ac435ebd665546de9d68dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:56.127045 747 tablet_service.cc:1552] Processing DeleteTablet for tablet f246f6c202cf44d48c9f5e4f335202c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:55 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:56.127468 756 tablet_service.cc:1505] Processing CreateTablet for tablet 44c0488eba0543078ff4f0b3d69366f2 (DEFAULT_TABLE table=default.Table_bc32d7beda4f4957817b20124bf48e9c [id=df809b5c66d64eb1846fa5eb9da11781]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:56.127861 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44c0488eba0543078ff4f0b3d69366f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:56.129634 520 catalog_manager.cc:6194] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: default.Table_5113d1a33e8e4bd2af5bc51e689f304f [id=ae5d56d40fa941a68f2b10d5e6e9ac83] alter complete (version 2)
I20251029 02:51:56.131393 1037 tablet_service.cc:1505] Processing CreateTablet for tablet 44c0488eba0543078ff4f0b3d69366f2 (DEFAULT_TABLE table=default.Table_bc32d7beda4f4957817b20124bf48e9c [id=df809b5c66d64eb1846fa5eb9da11781]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:56.131834 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44c0488eba0543078ff4f0b3d69366f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:56.134984 1174 ts_tablet_manager.cc:1916] T ba014ecfec7f4602ae1c09d931bac824 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:56.134997 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_eae5a69444004b54bf8bf18ea29fe73f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:56.135226 1166 tablet_bootstrap.cc:654] T 7c041f4909274cd5a03b9fd78353faf9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
W20251029 02:51:56.135635 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eae5a69444004b54bf8bf18ea29fe73f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:56.104259 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 263) took 1444 ms. Trace:
I20251029 02:51:56.136394 1176 rpcz_store.cc:276] 1029 02:51:54.659699 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:54.661696 (+ 1997us) service_pool.cc:225] Handling call
1029 02:51:56.103855 (+1442159us) ts_tablet_manager.cc:1151] Deleting tablet fd0f89180aba43eabb20f926abccbaa6
1029 02:51:56.103900 (+ 45us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:56.104125 (+ 225us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1424933}
I20251029 02:51:56.138172 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1222e30bc4141b3ae686d80053c4ef5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:56.138691 520 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet 4b76c4408a7340e59b8506171622175f (table default.Table_1281926bf1224c0cb4541bf561102021 [id=f3fa2dbbef7e47359492d5a5eef9b8fd]). Expected version 2 got 0
I20251029 02:51:56.139299 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 264) took 1479 ms. Trace:
I20251029 02:51:56.139432 1176 rpcz_store.cc:276] 1029 02:51:54.659840 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:54.661672 (+ 1832us) service_pool.cc:225] Handling call
1029 02:51:56.139077 (+1477405us) ts_tablet_manager.cc:1151] Deleting tablet 6f93812578a940a3b7378015e2a46f95
1029 02:51:56.139089 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:56.139260 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1456951}
W20251029 02:51:56.140419 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 6f93812578a940a3b7378015e2a46f95 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6f93812578a940a3b7378015e2a46f95 already in progress: creating tablet
I20251029 02:51:56.140439 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_d9cfb3211dcd4bb295955b8750600b5d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:56.141038 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d9cfb3211dcd4bb295955b8750600b5d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:56.142726 1445 raft_consensus.cc:493] T 44784a3a760946e0be24743bc2fc8138 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:56.142983 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f198c3259864f7ab3625cb7bfae20b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:56.142935 1445 raft_consensus.cc:515] T 44784a3a760946e0be24743bc2fc8138 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:56.143194 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 268) took 1482 ms. Trace:
I20251029 02:51:56.143316 1176 rpcz_store.cc:276] 1029 02:51:54.660248 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:54.696672 (+ 36424us) service_pool.cc:225] Handling call
1029 02:51:56.143033 (+1446361us) ts_tablet_manager.cc:1151] Deleting tablet 1bab90fe904944af948dab3db06b7e34
1029 02:51:56.143043 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:56.143152 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1433557}
I20251029 02:51:56.143592 1445 leader_election.cc:290] T 44784a3a760946e0be24743bc2fc8138 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:51:56.144416 1175 tablet_replica.cc:333] T b5d5b737c86844d58c51b2388c0b1705 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:51:56.144925 1059 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "44784a3a760946e0be24743bc2fc8138" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:51:56.145656 896 tablet_service.cc:1505] Processing CreateTablet for tablet d7abaf84ba5d445aa17433b3be891548 (DEFAULT_TABLE table=default.Table_0ecb5a1d51bb40d79afa038e4f2aa4d2 [id=9e33c530a35544a4ae22e2a1238592db]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:56.146054 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7abaf84ba5d445aa17433b3be891548. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:56.145957 1038 tablet_service.cc:1505] Processing CreateTablet for tablet d7abaf84ba5d445aa17433b3be891548 (DEFAULT_TABLE table=default.Table_0ecb5a1d51bb40d79afa038e4f2aa4d2 [id=9e33c530a35544a4ae22e2a1238592db]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:56.146936 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7abaf84ba5d445aa17433b3be891548. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:56.147970 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "44784a3a760946e0be24743bc2fc8138" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:51:56.148249 519 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 11f4252b5e694312aaa142468f8e68b4
I20251029 02:51:56.148640 1025 tablet_service.cc:1505] Processing CreateTablet for tablet e65eaaae43ac435ebd665546de9d68dd (DEFAULT_TABLE table=default.Table_d9fe74a49f4843b089aae7fca9e4a826 [id=152882c742b2459ba863fd41083284c1]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 02:51:56.149567 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet c764062d3ba04d4ba8b2e367ad8ed433 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c764062d3ba04d4ba8b2e367ad8ed433 already in progress: creating tablet
I20251029 02:51:56.149051 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 275) took 1204 ms. Trace:
I20251029 02:51:56.149809 1176 rpcz_store.cc:276] 1029 02:51:54.944228 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:54.944290 (+ 62us) service_pool.cc:225] Handling call
1029 02:51:56.148908 (+1204618us) ts_tablet_manager.cc:1151] Deleting tablet 785733da86ff425a90ccfab2933fc89e
1029 02:51:56.148916 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:56.149005 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1202934}
I20251029 02:51:56.150718 881 tablet_service.cc:1552] Processing DeleteTablet for tablet 11f4252b5e694312aaa142468f8e68b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:56 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:56.151212 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e65eaaae43ac435ebd665546de9d68dd. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:51:56.153680 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 785733da86ff425a90ccfab2933fc89e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 785733da86ff425a90ccfab2933fc89e already in progress: creating tablet
I20251029 02:51:56.155619 519 master_service.cc:954] ReplaceTablet: received request to replace tablet 6bd3fac2502e479aa9f4ceaabbb39504 from {username='slave'} at 127.0.0.1:35314
I20251029 02:51:56.151556 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 278) took 1201 ms. Trace:
W20251029 02:51:56.145432 712 leader_election.cc:343] T 44784a3a760946e0be24743bc2fc8138 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:56.153985 1027 tablet_service.cc:1552] Processing DeleteTablet for tablet 11f4252b5e694312aaa142468f8e68b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:56 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:56.148116 739 tablet_service.cc:1552] Processing DeleteTablet for tablet 44784a3a760946e0be24743bc2fc8138 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:54 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:56.148267 747 tablet_service.cc:1552] Processing DeleteTablet for tablet 65fdec7cf5d143208d78d4c3806b1352 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:52 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:56.158730 1175 raft_consensus.cc:2243] T b5d5b737c86844d58c51b2388c0b1705 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:56.158414 1176 rpcz_store.cc:276] 1029 02:51:54.949693 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:54.950131 (+ 438us) service_pool.cc:225] Handling call
1029 02:51:56.151413 (+1201282us) ts_tablet_manager.cc:1151] Deleting tablet 74873a05ef634867a30aa63b31fcfcc7
1029 02:51:56.151422 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:56.151521 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1197797}
I20251029 02:51:56.158946 1175 raft_consensus.cc:2272] T b5d5b737c86844d58c51b2388c0b1705 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251029 02:51:56.159119 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 74873a05ef634867a30aa63b31fcfcc7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 74873a05ef634867a30aa63b31fcfcc7 already in progress: creating tablet
I20251029 02:51:56.159551 1175 ts_tablet_manager.cc:1916] T b5d5b737c86844d58c51b2388c0b1705 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 02:51:56.161338 711 leader_election.cc:343] T 44784a3a760946e0be24743bc2fc8138 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:56.161540 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 279) took 1211 ms. Trace:
I20251029 02:51:56.161590 711 leader_election.cc:304] T 44784a3a760946e0be24743bc2fc8138 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:51:56.161653 1176 rpcz_store.cc:276] 1029 02:51:54.949788 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:54.950053 (+ 265us) service_pool.cc:225] Handling call
1029 02:51:56.161394 (+1211341us) ts_tablet_manager.cc:1151] Deleting tablet dbc2092b63e0417997565c8fadb53ae0
1029 02:51:56.161406 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:56.161508 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1205571}
I20251029 02:51:56.162377 1445 raft_consensus.cc:2749] T 44784a3a760946e0be24743bc2fc8138 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:51:56.162878 739 tablet_service.cc:1505] Processing CreateTablet for tablet d7abaf84ba5d445aa17433b3be891548 (DEFAULT_TABLE table=default.Table_0ecb5a1d51bb40d79afa038e4f2aa4d2 [id=9e33c530a35544a4ae22e2a1238592db]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:56.163275 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7abaf84ba5d445aa17433b3be891548. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:56.163364 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 277) took 1213 ms. Trace:
I20251029 02:51:56.165109 881 tablet_service.cc:1505] Processing CreateTablet for tablet afb19ba39399410bb90b6225d7b5b5b1 (DEFAULT_TABLE table=default.Table_d4caf12f7662463c824210f3c9cd7e0f [id=7dfb670d18824f5dbdd321940fea965e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:56.165531 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet afb19ba39399410bb90b6225d7b5b5b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:56.166162 747 tablet_service.cc:1552] Processing DeleteTablet for tablet 11f4252b5e694312aaa142468f8e68b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:56 UTC) from {username='slave'} at 127.0.0.1:44580
W20251029 02:51:56.166249 523 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1bdb67125493430398ff5c80b73de2ac consensus state has no leader
I20251029 02:51:56.170061 523 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_bd9202984dc14b3b983326f60258c754" } new_table_name: "default.Table_a3377614e4a54266a4457944d48b0060" modify_external_catalogs: true
I20251029 02:51:56.163473 1176 rpcz_store.cc:276] 1029 02:51:54.949388 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:54.949521 (+ 133us) service_pool.cc:225] Handling call
1029 02:51:56.163210 (+1213689us) ts_tablet_manager.cc:1151] Deleting tablet ccdca31db2834de4b6ed0f2d67a10535
1029 02:51:56.163220 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:56.163320 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1199694}
I20251029 02:51:56.170830 920 raft_consensus.cc:3060] T 70e8cfe5eaa9492c85874dc3214a1e0b P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:51:56.171214 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 284) took 1214 ms. Trace:
I20251029 02:51:56.171231 1421 raft_consensus.cc:493] T 4b76c4408a7340e59b8506171622175f P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:56.171355 1176 rpcz_store.cc:276] 1029 02:51:54.956680 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:54.956949 (+ 269us) service_pool.cc:225] Handling call
1029 02:51:56.171055 (+1214106us) ts_tablet_manager.cc:1151] Deleting tablet 576c4e6e70e24b0399ac9597a30f0569
1029 02:51:56.171067 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:56.171167 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1206094}
I20251029 02:51:56.171375 1421 raft_consensus.cc:515] T 4b76c4408a7340e59b8506171622175f P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:56.172538 1059 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b76c4408a7340e59b8506171622175f" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:51:56.172674 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4b76c4408a7340e59b8506171622175f" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
W20251029 02:51:56.172988 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet ccdca31db2834de4b6ed0f2d67a10535 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ccdca31db2834de4b6ed0f2d67a10535 already in progress: creating tablet
W20251029 02:51:56.173274 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 576c4e6e70e24b0399ac9597a30f0569 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 576c4e6e70e24b0399ac9597a30f0569 already in progress: creating tablet
W20251029 02:51:56.173509 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 7e6c4f30be624769961c9ec17c5d5dcb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7e6c4f30be624769961c9ec17c5d5dcb already in progress: creating tablet
I20251029 02:51:56.173633 1421 leader_election.cc:290] T 4b76c4408a7340e59b8506171622175f P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:51:56.173805 747 tablet_service.cc:1505] Processing CreateTablet for tablet afb19ba39399410bb90b6225d7b5b5b1 (DEFAULT_TABLE table=default.Table_d4caf12f7662463c824210f3c9cd7e0f [id=7dfb670d18824f5dbdd321940fea965e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:56.174103 1172 tablet_bootstrap.cc:492] T fd0f89180aba43eabb20f926abccbaa6 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:51:56.174073 1027 tablet_service.cc:1505] Processing CreateTablet for tablet afb19ba39399410bb90b6225d7b5b5b1 (DEFAULT_TABLE table=default.Table_d4caf12f7662463c824210f3c9cd7e0f [id=7dfb670d18824f5dbdd321940fea965e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:56.174232 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet afb19ba39399410bb90b6225d7b5b5b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:56.174199 849 leader_election.cc:304] T 4b76c4408a7340e59b8506171622175f P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9
I20251029 02:51:56.174253 1172 ts_tablet_manager.cc:1403] T fd0f89180aba43eabb20f926abccbaa6 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.135s user 0.001s sys 0.003s
I20251029 02:51:56.174472 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet afb19ba39399410bb90b6225d7b5b5b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:56.174615 1421 raft_consensus.cc:2749] T 4b76c4408a7340e59b8506171622175f P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:51:56.175014 1445 raft_consensus.cc:493] T d2990e9f626c40618c118266bc595684 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:56.175177 1445 raft_consensus.cc:515] T d2990e9f626c40618c118266bc595684 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:56.175244 1172 raft_consensus.cc:359] T fd0f89180aba43eabb20f926abccbaa6 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:56.175436 1172 raft_consensus.cc:385] T fd0f89180aba43eabb20f926abccbaa6 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:56.176201 1445 leader_election.cc:290] T d2990e9f626c40618c118266bc595684 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:51:56.176455 1172 raft_consensus.cc:740] T fd0f89180aba43eabb20f926abccbaa6 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:51:56.176595 1059 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2990e9f626c40618c118266bc595684" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:51:56.176750 1172 consensus_queue.cc:260] T fd0f89180aba43eabb20f926abccbaa6 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:56.177069 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2990e9f626c40618c118266bc595684" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:51:56.177299 1172 ts_tablet_manager.cc:1434] T fd0f89180aba43eabb20f926abccbaa6 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20251029 02:51:56.177670 712 leader_election.cc:343] T d2990e9f626c40618c118266bc595684 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:56.178292 1172 tablet_bootstrap.cc:492] T 69597d48631d43b6a5b92abbf60d5ed3 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:51:56.182260 890 tablet_service.cc:1505] Processing CreateTablet for tablet adf6e736cb9c4920807ee58682e134ef (DEFAULT_TABLE table=default.Table_746054018e16496ea35decf9614e9f5c [id=64650da66d464074828f2dc955045f68]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:56.182909 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet adf6e736cb9c4920807ee58682e134ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:56.184829 518 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 5e31e061d74146009d24d5cd57487a5e
I20251029 02:51:56.186622 879 tablet_service.cc:1552] Processing DeleteTablet for tablet 5e31e061d74146009d24d5cd57487a5e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:56 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:56.187134 1041 tablet_service.cc:1552] Processing DeleteTablet for tablet 5e31e061d74146009d24d5cd57487a5e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:56 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:56.191723 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_e8e977ce21ca40d4a22d0434d73ef70b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:56.192363 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e8e977ce21ca40d4a22d0434d73ef70b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:51:56.192468 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 9bb2166a590d4210b02dd779ef6654ad because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9bb2166a590d4210b02dd779ef6654ad already in progress: creating tablet
W20251029 02:51:56.192723 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 5db519edc75d4bd0985b32322825ffae because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5db519edc75d4bd0985b32322825ffae already in progress: creating tablet
W20251029 02:51:56.192936 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet dfa8b1b9a80e4e8689e0acb44e16346a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dfa8b1b9a80e4e8689e0acb44e16346a already in progress: creating tablet
W20251029 02:51:56.193146 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 777992acb1bd4765886b802940db74bb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 777992acb1bd4765886b802940db74bb already in progress: creating tablet
W20251029 02:51:56.195291 847 consensus_peers.cc:597] T 6dd4696ad4b44b6dab87a205f3f16ebd P c756528642a543eeacd93cef3e335a7d -> Peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Couldn't send request to peer b495a49349324f7d81908f2da700f0d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6dd4696ad4b44b6dab87a205f3f16ebd. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:51:56.196137 1176 tablet_replica.cc:333] T 64b0fb400e944ce2b2c7a49d12135e9c P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:51:56.196380 1176 raft_consensus.cc:2243] T 64b0fb400e944ce2b2c7a49d12135e9c P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:56.196553 1176 raft_consensus.cc:2272] T 64b0fb400e944ce2b2c7a49d12135e9c P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:56.197120 1176 ts_tablet_manager.cc:1916] T 64b0fb400e944ce2b2c7a49d12135e9c P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 02:51:56.198853 513 catalog_manager.cc:4952] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet cb55b3d5bc63405f96cdc122f5a612cb because the tablet was not found. No further retry: Not found: Tablet not found: cb55b3d5bc63405f96cdc122f5a612cb
W20251029 02:51:56.199112 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 2e97d1bc06a24e8d8ba11acb6fd280d8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2e97d1bc06a24e8d8ba11acb6fd280d8 already in progress: creating tablet
W20251029 02:51:56.199375 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet d61beda662d844cdaf67fd9a2e382ea2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d61beda662d844cdaf67fd9a2e382ea2 already in progress: creating tablet
I20251029 02:51:56.200577 1041 tablet_service.cc:1505] Processing CreateTablet for tablet adf6e736cb9c4920807ee58682e134ef (DEFAULT_TABLE table=default.Table_746054018e16496ea35decf9614e9f5c [id=64650da66d464074828f2dc955045f68]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:56.201012 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet adf6e736cb9c4920807ee58682e134ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:56.201452 879 tablet_service.cc:1505] Processing CreateTablet for tablet 643e20d9974447daa78885b3401f3b77 (DEFAULT_TABLE table=default.Table_d1050ce4c18f47c89dab970ce5edc755 [id=b4fc453bf54d45c4b8dddc3c49bf48f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:56.202782 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 643e20d9974447daa78885b3401f3b77. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:56.204633 522 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 0d84dc8dc465496f9adec798b2e03e13
I20251029 02:51:56.211340 522 catalog_manager.cc:5649] T dd90fb9eb167478cbc3ec9363af86295 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } health_report { overall_health: UNKNOWN } } }
I20251029 02:51:56.211794 522 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet dd90fb9eb167478cbc3ec9363af86295 (table default.Table_b3ad8b7021294ca484b65c1773165849 [id=052abf3dcc904bdcb869d5b885723dab]). Expected version 2 got 0
I20251029 02:51:56.214169 1172 tablet_bootstrap.cc:654] T 69597d48631d43b6a5b92abbf60d5ed3 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
W20251029 02:51:56.215224 711 leader_election.cc:343] T d2990e9f626c40618c118266bc595684 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:56.215373 711 leader_election.cc:304] T d2990e9f626c40618c118266bc595684 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:51:56.218741 890 tablet_service.cc:1552] Processing DeleteTablet for tablet 0d84dc8dc465496f9adec798b2e03e13 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:56 UTC) from {username='slave'} at 127.0.0.1:35702
W20251029 02:51:56.219811 517 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a5c3a15510cb4321bbb80d61fabd0c0a consensus state has no leader
I20251029 02:51:56.222633 1042 tablet_service.cc:1505] Processing CreateTablet for tablet 643e20d9974447daa78885b3401f3b77 (DEFAULT_TABLE table=default.Table_d1050ce4c18f47c89dab970ce5edc755 [id=b4fc453bf54d45c4b8dddc3c49bf48f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:56.223165 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 643e20d9974447daa78885b3401f3b77. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:56.223942 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_dbf2e434d7964eac80709c205f61191e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:56.224645 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dbf2e434d7964eac80709c205f61191e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:56.226923 881 tablet_service.cc:1505] Processing CreateTablet for tablet 731c0acfbe204e8199ad246e38ae2ccd (DEFAULT_TABLE table=default.Table_094b4e33c9654fc7b1b4bf460be40fcd [id=93e27f360fbb436aa652b936c9046510]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:56.227412 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 731c0acfbe204e8199ad246e38ae2ccd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:56.231971 1028 tablet_service.cc:1505] Processing CreateTablet for tablet 731c0acfbe204e8199ad246e38ae2ccd (DEFAULT_TABLE table=default.Table_094b4e33c9654fc7b1b4bf460be40fcd [id=93e27f360fbb436aa652b936c9046510]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:56.232483 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 731c0acfbe204e8199ad246e38ae2ccd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:56.237875 1027 tablet_service.cc:1552] Processing DeleteTablet for tablet 0d84dc8dc465496f9adec798b2e03e13 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:56 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:56.239349 526 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet af4860c6b72f4d988b206b5ef2f22906
I20251029 02:51:56.240455 1027 tablet_service.cc:1552] Processing DeleteTablet for tablet af4860c6b72f4d988b206b5ef2f22906 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:56 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:56.241087 883 tablet_service.cc:1552] Processing DeleteTablet for tablet af4860c6b72f4d988b206b5ef2f22906 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:56 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:56.244146 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_31aa9d49c83b4ab29823acea7480bc96"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:56.244832 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_31aa9d49c83b4ab29823acea7480bc96 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:56.247670 1174 ts_tablet_manager.cc:1929] T ba014ecfec7f4602ae1c09d931bac824 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251029 02:51:56.249383 1421 raft_consensus.cc:493] T bc1a1cf274b64b52aba58811e9594330 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:56.250065 1445 raft_consensus.cc:2749] T d2990e9f626c40618c118266bc595684 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:51:56.250203 1421 raft_consensus.cc:515] T bc1a1cf274b64b52aba58811e9594330 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:56.250613 738 tablet_service.cc:1505] Processing CreateTablet for tablet c1222e30bc4141b3ae686d80053c4ef5 (DEFAULT_TABLE table=default.Table_a63991c194724381a84c0ce2adab25da [id=31801defa6db4ca7a08fbf7f029b65c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:56.251108 1421 leader_election.cc:290] T bc1a1cf274b64b52aba58811e9594330 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:51:56.251902 521 master_service.cc:954] ReplaceTablet: received request to replace tablet 4b76c4408a7340e59b8506171622175f from {username='slave'} at 127.0.0.1:35314
I20251029 02:51:56.252228 1166 tablet_bootstrap.cc:492] T 7c041f4909274cd5a03b9fd78353faf9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:51:56.252914 1166 ts_tablet_manager.cc:1403] T 7c041f4909274cd5a03b9fd78353faf9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.155s user 0.001s sys 0.011s
W20251029 02:51:56.252911 849 leader_election.cc:343] T bc1a1cf274b64b52aba58811e9594330 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Not found: Tablet not found: bc1a1cf274b64b52aba58811e9594330
I20251029 02:51:56.253791 1166 raft_consensus.cc:359] T 7c041f4909274cd5a03b9fd78353faf9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:56.254019 1166 raft_consensus.cc:385] T 7c041f4909274cd5a03b9fd78353faf9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:56.254115 1166 raft_consensus.cc:740] T 7c041f4909274cd5a03b9fd78353faf9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:51:56.254312 1174 log.cc:1199] T ba014ecfec7f4602ae1c09d931bac824 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/ba014ecfec7f4602ae1c09d931bac824
I20251029 02:51:56.254626 1166 consensus_queue.cc:260] T 7c041f4909274cd5a03b9fd78353faf9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:56.255278 1166 ts_tablet_manager.cc:1434] T 7c041f4909274cd5a03b9fd78353faf9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 02:51:56.255669 1166 tablet_bootstrap.cc:492] T 6f93812578a940a3b7378015e2a46f95 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:51:56.257546 1166 tablet_bootstrap.cc:654] T 6f93812578a940a3b7378015e2a46f95 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:56.257982 1174 ts_tablet_manager.cc:1950] T ba014ecfec7f4602ae1c09d931bac824 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
W20251029 02:51:56.259953 847 leader_election.cc:343] T bc1a1cf274b64b52aba58811e9594330 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: bc1a1cf274b64b52aba58811e9594330
I20251029 02:51:56.260136 847 leader_election.cc:304] T bc1a1cf274b64b52aba58811e9594330 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9
I20251029 02:51:56.260660 1421 raft_consensus.cc:2749] T bc1a1cf274b64b52aba58811e9594330 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:51:56.261742 1167 tablet_bootstrap.cc:492] T ab421dfaa3f84593842bb732a2693160 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:51:56.261955 1167 ts_tablet_manager.cc:1403] T ab421dfaa3f84593842bb732a2693160 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.326s user 0.007s sys 0.023s
I20251029 02:51:56.263069 1167 raft_consensus.cc:359] T ab421dfaa3f84593842bb732a2693160 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:56.263286 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1222e30bc4141b3ae686d80053c4ef5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:56.263357 1167 raft_consensus.cc:385] T ab421dfaa3f84593842bb732a2693160 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:56.263509 1167 raft_consensus.cc:740] T ab421dfaa3f84593842bb732a2693160 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:51:56.263839 1167 consensus_queue.cc:260] T ab421dfaa3f84593842bb732a2693160 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:56.264405 739 tablet_service.cc:1552] Processing DeleteTablet for tablet fd0f89180aba43eabb20f926abccbaa6 with delete_type TABLET_DATA_DELETED (replaced by tablet 9bb2166a590d4210b02dd779ef6654ad) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:56.264706 1167 ts_tablet_manager.cc:1434] T ab421dfaa3f84593842bb732a2693160 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251029 02:51:56.265228 1167 tablet_bootstrap.cc:492] T 64b0fb400e944ce2b2c7a49d12135e9c P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:51:56.265513 739 tablet_service.cc:1505] Processing CreateTablet for tablet adf6e736cb9c4920807ee58682e134ef (DEFAULT_TABLE table=default.Table_746054018e16496ea35decf9614e9f5c [id=64650da66d464074828f2dc955045f68]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:56.265901 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet adf6e736cb9c4920807ee58682e134ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:56.265991 753 tablet_service.cc:1552] Processing DeleteTablet for tablet af4860c6b72f4d988b206b5ef2f22906 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:56 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:56.266661 756 tablet_service.cc:1505] Processing CreateTablet for tablet 731c0acfbe204e8199ad246e38ae2ccd (DEFAULT_TABLE table=default.Table_094b4e33c9654fc7b1b4bf460be40fcd [id=93e27f360fbb436aa652b936c9046510]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:56.267084 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 731c0acfbe204e8199ad246e38ae2ccd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:56.267495 1167 tablet_bootstrap.cc:654] T 64b0fb400e944ce2b2c7a49d12135e9c P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:56.271689 747 tablet_service.cc:1552] Processing DeleteTablet for tablet 5e31e061d74146009d24d5cd57487a5e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:56 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:56.272218 1445 raft_consensus.cc:493] T 9e07291d6b5645feb4d5890f58e1ef74 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:56.272212 743 tablet_service.cc:1505] Processing CreateTablet for tablet 643e20d9974447daa78885b3401f3b77 (DEFAULT_TABLE table=default.Table_d1050ce4c18f47c89dab970ce5edc755 [id=b4fc453bf54d45c4b8dddc3c49bf48f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:56.272739 1445 raft_consensus.cc:515] T 9e07291d6b5645feb4d5890f58e1ef74 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:56.273312 748 tablet_service.cc:1552] Processing DeleteTablet for tablet 0d84dc8dc465496f9adec798b2e03e13 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:56 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:56.273969 524 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_75949df235e044029e69b130e96b8d27" } new_table_name: "default.Table_bd020e7c45a54c6e92211c4261293575" modify_external_catalogs: true
I20251029 02:51:56.274713 1445 leader_election.cc:290] T 9e07291d6b5645feb4d5890f58e1ef74 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
W20251029 02:51:56.275504 849 consensus_peers.cc:597] T 61c1fec3ab4c402eb29f767b9bb7409f P c756528642a543eeacd93cef3e335a7d -> Peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Couldn't send request to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 61c1fec3ab4c402eb29f767b9bb7409f. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:51:56.276255 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e07291d6b5645feb4d5890f58e1ef74" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:51:56.276274 1059 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e07291d6b5645feb4d5890f58e1ef74" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:51:56.277705 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 254) took 1912 ms. Trace:
I20251029 02:51:56.277830 1174 rpcz_store.cc:276] 1029 02:51:54.365051 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:54.592299 (+227248us) service_pool.cc:225] Handling call
1029 02:51:56.096288 (+1503989us) ts_tablet_manager.cc:1151] Deleting tablet ba014ecfec7f4602ae1c09d931bac824
1029 02:51:56.096301 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:56.138538 (+ 42237us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:56.247542 (+109004us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:56.277252 (+ 29710us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1337300}
I20251029 02:51:56.278496 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet ba014ecfec7f4602ae1c09d931bac824 (table default.Table_64e512f8ce034224aea9b887ed0d09dc [id=7bb8b246fbf1431bb71bbb7c5520e301]) successfully deleted
I20251029 02:51:56.278914 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 643e20d9974447daa78885b3401f3b77. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:56.279208 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 259) took 1878 ms. Trace:
W20251029 02:51:56.279495 711 leader_election.cc:343] T 9e07291d6b5645feb4d5890f58e1ef74 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:56.279655 1176 ts_tablet_manager.cc:1929] T 64b0fb400e944ce2b2c7a49d12135e9c P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:56.279799 1176 log.cc:1199] T 64b0fb400e944ce2b2c7a49d12135e9c P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/64b0fb400e944ce2b2c7a49d12135e9c
I20251029 02:51:56.280182 1176 ts_tablet_manager.cc:1950] T 64b0fb400e944ce2b2c7a49d12135e9c P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
W20251029 02:51:56.280128 712 leader_election.cc:343] T 9e07291d6b5645feb4d5890f58e1ef74 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:56.280292 712 leader_election.cc:304] T 9e07291d6b5645feb4d5890f58e1ef74 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:51:56.281208 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 64b0fb400e944ce2b2c7a49d12135e9c (table default.Table_91cd5889c90148e5bde6051f2847832a [id=e71aea48b9884bacb250412624ec4615]) successfully deleted
I20251029 02:51:56.281278 1445 raft_consensus.cc:2749] T 9e07291d6b5645feb4d5890f58e1ef74 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:51:56.282593 1176 tablet_replica.cc:333] T c4884e22cf2044a4a1070c1b4b759d9e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:51:56.282807 1176 raft_consensus.cc:2243] T c4884e22cf2044a4a1070c1b4b759d9e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:56.279923 1174 rpcz_store.cc:276] 1029 02:51:54.400577 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:54.759016 (+358439us) service_pool.cc:225] Handling call
1029 02:51:56.278606 (+1519590us) ts_tablet_manager.cc:1151] Deleting tablet 6f93812578a940a3b7378015e2a46f95
1029 02:51:56.278615 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:56.279145 (+ 530us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1518430}
W20251029 02:51:56.283407 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 6f93812578a940a3b7378015e2a46f95 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6f93812578a940a3b7378015e2a46f95 already in progress: creating tablet
I20251029 02:51:56.284291 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 256) took 1918 ms. Trace:
I20251029 02:51:56.284425 1174 rpcz_store.cc:276] 1029 02:51:54.365332 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:54.621158 (+255826us) service_pool.cc:225] Handling call
1029 02:51:56.284146 (+1662988us) ts_tablet_manager.cc:1151] Deleting tablet 6cd7009e06a646acad0802918c2b425e
1029 02:51:56.284156 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:56.284245 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1513350}
I20251029 02:51:56.285334 1167 tablet_bootstrap.cc:492] T 64b0fb400e944ce2b2c7a49d12135e9c P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:51:56.285670 1167 ts_tablet_manager.cc:1403] T 64b0fb400e944ce2b2c7a49d12135e9c P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.000s
W20251029 02:51:56.286283 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 6cd7009e06a646acad0802918c2b425e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6cd7009e06a646acad0802918c2b425e already in progress: creating tablet
I20251029 02:51:56.286455 1167 raft_consensus.cc:359] T 64b0fb400e944ce2b2c7a49d12135e9c P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:56.287287 1167 raft_consensus.cc:385] T 64b0fb400e944ce2b2c7a49d12135e9c P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:56.287441 1167 raft_consensus.cc:740] T 64b0fb400e944ce2b2c7a49d12135e9c P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:51:56.287758 1167 consensus_queue.cc:260] T 64b0fb400e944ce2b2c7a49d12135e9c P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:56.290019 1167 ts_tablet_manager.cc:1434] T 64b0fb400e944ce2b2c7a49d12135e9c P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251029 02:51:56.290853 1166 tablet_bootstrap.cc:492] T 6f93812578a940a3b7378015e2a46f95 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:51:56.291361 1167 tablet_bootstrap.cc:492] T 7c041f4909274cd5a03b9fd78353faf9 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:51:56.292091 1166 ts_tablet_manager.cc:1403] T 6f93812578a940a3b7378015e2a46f95 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.004s
I20251029 02:51:56.292448 525 master_service.cc:954] ReplaceTablet: received request to replace tablet dd6ad641fbda472fab07aa35c5e4523a from {username='slave'} at 127.0.0.1:35314
I20251029 02:51:56.292955 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 257) took 1927 ms. Trace:
I20251029 02:51:56.293294 1166 raft_consensus.cc:359] T 6f93812578a940a3b7378015e2a46f95 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:56.293504 1166 raft_consensus.cc:385] T 6f93812578a940a3b7378015e2a46f95 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:56.293591 1166 raft_consensus.cc:740] T 6f93812578a940a3b7378015e2a46f95 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:51:56.293804 1166 consensus_queue.cc:260] T 6f93812578a940a3b7378015e2a46f95 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:56.294323 1166 ts_tablet_manager.cc:1434] T 6f93812578a940a3b7378015e2a46f95 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251029 02:51:56.294638 1166 tablet_bootstrap.cc:492] T 6c22497760274380a832feff7c5a2fae P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:51:56.296442 1166 tablet_bootstrap.cc:654] T 6c22497760274380a832feff7c5a2fae P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:56.297669 1167 tablet_bootstrap.cc:654] T 7c041f4909274cd5a03b9fd78353faf9 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:56.298694 1166 tablet_bootstrap.cc:492] T 6c22497760274380a832feff7c5a2fae P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:51:56.298846 1166 ts_tablet_manager.cc:1403] T 6c22497760274380a832feff7c5a2fae P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.002s
I20251029 02:51:56.298908 1445 raft_consensus.cc:493] T 078ace5eada044c88d2462efb2b57aa8 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:56.299585 1166 raft_consensus.cc:359] T 6c22497760274380a832feff7c5a2fae P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:56.299777 1166 raft_consensus.cc:385] T 6c22497760274380a832feff7c5a2fae P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:56.299845 1166 raft_consensus.cc:740] T 6c22497760274380a832feff7c5a2fae P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:51:56.300064 1166 consensus_queue.cc:260] T 6c22497760274380a832feff7c5a2fae P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:56.300575 1166 ts_tablet_manager.cc:1434] T 6c22497760274380a832feff7c5a2fae P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251029 02:51:56.300896 1166 tablet_bootstrap.cc:492] T f71388036478404d878a52fc20ece216 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:51:56.283004 1176 raft_consensus.cc:2272] T c4884e22cf2044a4a1070c1b4b759d9e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:56.296375 1174 rpcz_store.cc:276] 1029 02:51:54.365438 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:54.622111 (+256673us) service_pool.cc:225] Handling call
1029 02:51:56.292791 (+1670680us) ts_tablet_manager.cc:1151] Deleting tablet b5d5b737c86844d58c51b2388c0b1705
1029 02:51:56.292801 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:56.292906 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1519741}
I20251029 02:51:56.300302 920 raft_consensus.cc:3060] T 5e31e061d74146009d24d5cd57487a5e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:51:56.304075 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_1cf1a43474a9433fb65ef47f07029232"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:56.304637 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1cf1a43474a9433fb65ef47f07029232 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:56.306077 878 tablet_service.cc:1552] Processing DeleteTablet for tablet ab421dfaa3f84593842bb732a2693160 with delete_type TABLET_DATA_DELETED (replaced by tablet c9e923724ca9402682f13b252dbc37aa) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:56.306668 1176 ts_tablet_manager.cc:1916] T c4884e22cf2044a4a1070c1b4b759d9e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:56.309000 1166 tablet_bootstrap.cc:654] T f71388036478404d878a52fc20ece216 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:56.309535 1445 raft_consensus.cc:515] T 078ace5eada044c88d2462efb2b57aa8 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:56.310261 1445 leader_election.cc:290] T 078ace5eada044c88d2462efb2b57aa8 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:51:56.311234 519 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 6bd3fac2502e479aa9f4ceaabbb39504
I20251029 02:51:56.311713 519 catalog_manager.cc:6719] ReplaceTablet: tablet 6bd3fac2502e479aa9f4ceaabbb39504 deleted and replaced by tablet b125cc9aaf4248b8bfa48fb152b832a6
I20251029 02:51:56.311892 519 master_service.cc:954] ReplaceTablet: received request to replace tablet daa05a34c0684c07a35966290fd7d44c from {username='slave'} at 127.0.0.1:35314
I20251029 02:51:56.312332 1059 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "078ace5eada044c88d2462efb2b57aa8" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:51:56.313035 523 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2f198c3259864f7ab3625cb7bfae20b0 consensus state has no leader
I20251029 02:51:56.313010 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "078ace5eada044c88d2462efb2b57aa8" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:51:56.313408 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_04fadd7c87d1460dbc72c2caedbef915"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:56.313992 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_04fadd7c87d1460dbc72c2caedbef915 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:56.315733 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 263) took 1869 ms. Trace:
W20251029 02:51:56.315958 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet b5d5b737c86844d58c51b2388c0b1705 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b5d5b737c86844d58c51b2388c0b1705 already in progress: creating tablet
I20251029 02:51:56.316089 878 tablet_service.cc:1552] Processing DeleteTablet for tablet 6bd3fac2502e479aa9f4ceaabbb39504 with delete_type TABLET_DATA_DELETED (replaced by tablet b125cc9aaf4248b8bfa48fb152b832a6) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:56.317181 1174 rpcz_store.cc:276] 1029 02:51:54.445983 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:54.773077 (+327094us) service_pool.cc:225] Handling call
1029 02:51:56.315451 (+1542374us) ts_tablet_manager.cc:1151] Deleting tablet 1bab90fe904944af948dab3db06b7e34
1029 02:51:56.315461 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:56.315680 (+ 219us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1542206}
I20251029 02:51:56.317849 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 258) took 1917 ms. Trace:
I20251029 02:51:56.318046 1174 rpcz_store.cc:276] 1029 02:51:54.400461 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:54.623134 (+222673us) service_pool.cc:225] Handling call
1029 02:51:56.317658 (+1694524us) ts_tablet_manager.cc:1151] Deleting tablet fd0f89180aba43eabb20f926abccbaa6
1029 02:51:56.317669 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:56.317749 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1543695}
I20251029 02:51:56.317952 881 tablet_service.cc:1552] Processing DeleteTablet for tablet 64b0fb400e944ce2b2c7a49d12135e9c with delete_type TABLET_DATA_DELETED (replaced by tablet 9e07291d6b5645feb4d5890f58e1ef74) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:56.320443 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet 6bd3fac2502e479aa9f4ceaabbb39504 with delete_type TABLET_DATA_DELETED (replaced by tablet b125cc9aaf4248b8bfa48fb152b832a6) from {username='slave'} at 127.0.0.1:37626
W20251029 02:51:56.320976 712 leader_election.cc:343] T 078ace5eada044c88d2462efb2b57aa8 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
W20251029 02:51:56.321533 711 leader_election.cc:343] T 078ace5eada044c88d2462efb2b57aa8 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:56.323524 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 269) took 1593 ms. Trace:
I20251029 02:51:56.324654 1176 ts_tablet_manager.cc:1929] T c4884e22cf2044a4a1070c1b4b759d9e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:56.324813 1176 log.cc:1199] T c4884e22cf2044a4a1070c1b4b759d9e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/c4884e22cf2044a4a1070c1b4b759d9e
I20251029 02:51:56.325218 1176 ts_tablet_manager.cc:1950] T c4884e22cf2044a4a1070c1b4b759d9e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:51:56.326686 1174 rpcz_store.cc:276] 1029 02:51:54.730102 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:54.876943 (+146841us) service_pool.cc:225] Handling call
1029 02:51:56.320875 (+1443932us) ts_tablet_manager.cc:1151] Deleting tablet 785733da86ff425a90ccfab2933fc89e
1029 02:51:56.320885 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:56.323461 (+ 2576us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1443418}
I20251029 02:51:56.326944 1172 tablet_bootstrap.cc:492] T 69597d48631d43b6a5b92abbf60d5ed3 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:51:56.328169 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 271) took 1570 ms. Trace:
W20251029 02:51:56.328227 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 785733da86ff425a90ccfab2933fc89e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 785733da86ff425a90ccfab2933fc89e already in progress: creating tablet
I20251029 02:51:56.328565 711 leader_election.cc:304] T 078ace5eada044c88d2462efb2b57aa8 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:51:56.329247 1172 ts_tablet_manager.cc:1403] T 69597d48631d43b6a5b92abbf60d5ed3 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.149s user 0.003s sys 0.005s
I20251029 02:51:56.332878 748 tablet_service.cc:1552] Processing DeleteTablet for tablet 6bd3fac2502e479aa9f4ceaabbb39504 with delete_type TABLET_DATA_DELETED (replaced by tablet b125cc9aaf4248b8bfa48fb152b832a6) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:56.333359 1445 raft_consensus.cc:2749] T 078ace5eada044c88d2462efb2b57aa8 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:51:56.334765 1172 raft_consensus.cc:359] T 69597d48631d43b6a5b92abbf60d5ed3 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:56.335644 1172 raft_consensus.cc:385] T 69597d48631d43b6a5b92abbf60d5ed3 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:56.335739 1172 raft_consensus.cc:740] T 69597d48631d43b6a5b92abbf60d5ed3 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:51:56.335665 518 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_0fc48a708ecc446990ad491afab2efdd" } new_table_name: "default.Table_966779a0d37a4dc4ab3f89d2bf7c6b06" modify_external_catalogs: true
I20251029 02:51:56.336664 1172 consensus_queue.cc:260] T 69597d48631d43b6a5b92abbf60d5ed3 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:56.337483 1172 ts_tablet_manager.cc:1434] T 69597d48631d43b6a5b92abbf60d5ed3 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20251029 02:51:56.338320 1166 tablet_bootstrap.cc:492] T f71388036478404d878a52fc20ece216 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:51:56.338459 1166 ts_tablet_manager.cc:1403] T f71388036478404d878a52fc20ece216 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.038s user 0.002s sys 0.001s
I20251029 02:51:56.339612 1166 raft_consensus.cc:359] T f71388036478404d878a52fc20ece216 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:56.339887 1166 raft_consensus.cc:385] T f71388036478404d878a52fc20ece216 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:56.340006 1166 raft_consensus.cc:740] T f71388036478404d878a52fc20ece216 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:51:56.340240 1172 tablet_bootstrap.cc:492] T 5695b5a7b1854c3983e2d4ab684898b0 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:51:56.340276 1166 consensus_queue.cc:260] T f71388036478404d878a52fc20ece216 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:56.342397 1445 raft_consensus.cc:493] T 4bf06476c87a4549a73f5994314bb75f P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:56.344352 1166 ts_tablet_manager.cc:1434] T f71388036478404d878a52fc20ece216 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20251029 02:51:56.344854 1166 tablet_bootstrap.cc:492] T d6e1170dbb594a5696a3899cde8da7c9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:51:56.345417 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet c4884e22cf2044a4a1070c1b4b759d9e (table default.Table_da0be17b8bdc4e579b6c3faff71d9c98 [id=6e06ac70c6814e7d89a008bbaddccc7e]) successfully deleted
I20251029 02:51:56.346143 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_b8095fca27a447f2a9ac0475373352c6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:56.346760 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b8095fca27a447f2a9ac0475373352c6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:56.347385 1166 tablet_bootstrap.cc:654] T d6e1170dbb594a5696a3899cde8da7c9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:56.348315 1395 raft_consensus.cc:493] T 6c22497760274380a832feff7c5a2fae P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:56.348474 1395 raft_consensus.cc:515] T 6c22497760274380a832feff7c5a2fae P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:56.348963 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet c519832030884a049839d309cc738c28 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:52 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:56.349555 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6c22497760274380a832feff7c5a2fae" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:51:56.349030 1395 leader_election.cc:290] T 6c22497760274380a832feff7c5a2fae P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
W20251029 02:51:56.349774 989 leader_election.cc:343] T 6c22497760274380a832feff7c5a2fae P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 6c22497760274380a832feff7c5a2fae
W20251029 02:51:56.350455 990 leader_election.cc:343] T 6c22497760274380a832feff7c5a2fae P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:56.350618 990 leader_election.cc:304] T 6c22497760274380a832feff7c5a2fae P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5; no voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d
I20251029 02:51:56.351053 1395 raft_consensus.cc:2749] T 6c22497760274380a832feff7c5a2fae P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:51:56.355552 1445 raft_consensus.cc:515] T 4bf06476c87a4549a73f5994314bb75f P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:56.356407 1445 leader_election.cc:290] T 4bf06476c87a4549a73f5994314bb75f P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:51:56.356678 1175 ts_tablet_manager.cc:1929] T b5d5b737c86844d58c51b2388c0b1705 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:56.356886 1175 log.cc:1199] T b5d5b737c86844d58c51b2388c0b1705 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/b5d5b737c86844d58c51b2388c0b1705
I20251029 02:51:56.357049 1176 tablet_replica.cc:333] T ab421dfaa3f84593842bb732a2693160 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:51:56.357304 1059 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4bf06476c87a4549a73f5994314bb75f" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:51:56.357517 1175 ts_tablet_manager.cc:1950] T b5d5b737c86844d58c51b2388c0b1705 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
W20251029 02:51:56.358014 712 leader_election.cc:343] T 4bf06476c87a4549a73f5994314bb75f P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:56.358043 1176 raft_consensus.cc:2243] T ab421dfaa3f84593842bb732a2693160 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:56.358464 1176 raft_consensus.cc:2272] T ab421dfaa3f84593842bb732a2693160 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251029 02:51:56.358862 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 012ce243fd07466ab5301d1c9ce4b413 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 012ce243fd07466ab5301d1c9ce4b413 already in progress: creating tablet
I20251029 02:51:56.359068 1176 ts_tablet_manager.cc:1916] T ab421dfaa3f84593842bb732a2693160 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:56.359354 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4bf06476c87a4549a73f5994314bb75f" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:51:56.359387 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet b6d4f7bf702f4834bc5f4ef2b50b92a7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b6d4f7bf702f4834bc5f4ef2b50b92a7 already in progress: creating tablet
I20251029 02:51:56.359879 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet b5d5b737c86844d58c51b2388c0b1705 (table default.Table_8b39acd343e94119804478e17baaa785 [id=0b43261c80a5422092e96d77c0d2ee9e]) successfully deleted
W20251029 02:51:56.359743 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet c764062d3ba04d4ba8b2e367ad8ed433 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c764062d3ba04d4ba8b2e367ad8ed433 already in progress: creating tablet
I20251029 02:51:56.365568 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet 6c22497760274380a832feff7c5a2fae with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:53 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:56.366164 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet f71388036478404d878a52fc20ece216 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:52 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:56.366750 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet 7c041f4909274cd5a03b9fd78353faf9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:52 UTC) from {username='slave'} at 127.0.0.1:37626
W20251029 02:51:56.359946 711 leader_election.cc:343] T 4bf06476c87a4549a73f5994314bb75f P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:56.367046 711 leader_election.cc:304] T 4bf06476c87a4549a73f5994314bb75f P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:51:56.367405 1445 raft_consensus.cc:2749] T 4bf06476c87a4549a73f5994314bb75f P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:51:56.358592 1175 tablet_replica.cc:333] T 9964ad774890422f8e14f6b99f28cb55 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:51:56.368256 1175 raft_consensus.cc:2243] T 9964ad774890422f8e14f6b99f28cb55 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:56.368511 1175 raft_consensus.cc:2272] T 9964ad774890422f8e14f6b99f28cb55 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:56.368850 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet 6f93812578a940a3b7378015e2a46f95 with delete_type TABLET_DATA_DELETED (replaced by tablet ba699862e6f646c6beaa39f080d0faba) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:56.369102 1175 ts_tablet_manager.cc:1916] T 9964ad774890422f8e14f6b99f28cb55 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:56.371412 1166 tablet_bootstrap.cc:492] T d6e1170dbb594a5696a3899cde8da7c9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:51:56.371655 1166 ts_tablet_manager.cc:1403] T d6e1170dbb594a5696a3899cde8da7c9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.027s user 0.006s sys 0.005s
I20251029 02:51:56.372562 1166 raft_consensus.cc:359] T d6e1170dbb594a5696a3899cde8da7c9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:56.372852 1166 raft_consensus.cc:385] T d6e1170dbb594a5696a3899cde8da7c9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:56.372979 1166 raft_consensus.cc:740] T d6e1170dbb594a5696a3899cde8da7c9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:51:56.373294 1166 consensus_queue.cc:260] T d6e1170dbb594a5696a3899cde8da7c9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:56.366816 1172 tablet_bootstrap.cc:654] T 5695b5a7b1854c3983e2d4ab684898b0 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:56.375823 1166 ts_tablet_manager.cc:1434] T d6e1170dbb594a5696a3899cde8da7c9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251029 02:51:56.375993 1174 rpcz_store.cc:276] 1029 02:51:54.758076 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:54.880851 (+122775us) service_pool.cc:225] Handling call
1029 02:51:56.328001 (+1447150us) ts_tablet_manager.cc:1151] Deleting tablet 74873a05ef634867a30aa63b31fcfcc7
1029 02:51:56.328011 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:56.328115 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1446231}
I20251029 02:51:56.376335 1166 tablet_bootstrap.cc:492] T aacb6f4b9284457bb6607e3a85d5c809 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:51:56.376600 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 272) took 1618 ms. Trace:
I20251029 02:51:56.376699 1174 rpcz_store.cc:276] 1029 02:51:54.758208 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:54.881790 (+123582us) service_pool.cc:225] Handling call
1029 02:51:56.376404 (+1494614us) ts_tablet_manager.cc:1151] Deleting tablet ccdca31db2834de4b6ed0f2d67a10535
1029 02:51:56.376417 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:56.376546 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1494371}
I20251029 02:51:56.377063 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 273) took 1618 ms. Trace:
I20251029 02:51:56.377159 1174 rpcz_store.cc:276] 1029 02:51:54.758282 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:54.882041 (+123759us) service_pool.cc:225] Handling call
1029 02:51:56.376918 (+1494877us) ts_tablet_manager.cc:1151] Deleting tablet dbc2092b63e0417997565c8fadb53ae0
1029 02:51:56.376927 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:56.377023 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1494638}
W20251029 02:51:56.377408 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 74873a05ef634867a30aa63b31fcfcc7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 74873a05ef634867a30aa63b31fcfcc7 already in progress: creating tablet
W20251029 02:51:56.377681 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet ccdca31db2834de4b6ed0f2d67a10535 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ccdca31db2834de4b6ed0f2d67a10535 already in progress: creating tablet
I20251029 02:51:56.378858 1166 tablet_bootstrap.cc:654] T aacb6f4b9284457bb6607e3a85d5c809 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:56.379789 526 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_8f1e631e578a4010927f77c77cc29cb0" } modify_external_catalogs: true
I20251029 02:51:56.379989 526 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_8f1e631e578a4010927f77c77cc29cb0" } modify_external_catalogs: true
W20251029 02:51:56.382957 990 consensus_peers.cc:597] T 24016ceaa84147eca0b94b3ad288e772 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 -> Peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Couldn't send request to peer c756528642a543eeacd93cef3e335a7d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:51:56.384896 1172 tablet_bootstrap.cc:492] T 5695b5a7b1854c3983e2d4ab684898b0 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:51:56.385988 521 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 4b76c4408a7340e59b8506171622175f
I20251029 02:51:56.386595 521 catalog_manager.cc:6719] ReplaceTablet: tablet 4b76c4408a7340e59b8506171622175f deleted and replaced by tablet 24f07b1efac64b099b0b5ed467463749
I20251029 02:51:56.387529 1172 ts_tablet_manager.cc:1403] T 5695b5a7b1854c3983e2d4ab684898b0 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.048s user 0.005s sys 0.000s
I20251029 02:51:56.388372 1172 raft_consensus.cc:359] T 5695b5a7b1854c3983e2d4ab684898b0 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:56.388552 1172 raft_consensus.cc:385] T 5695b5a7b1854c3983e2d4ab684898b0 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:56.388635 1172 raft_consensus.cc:740] T 5695b5a7b1854c3983e2d4ab684898b0 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:51:56.390792 1172 consensus_queue.cc:260] T 5695b5a7b1854c3983e2d4ab684898b0 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:56.390923 743 tablet_service.cc:1552] Processing DeleteTablet for tablet 4b76c4408a7340e59b8506171622175f with delete_type TABLET_DATA_DELETED (replaced by tablet 24f07b1efac64b099b0b5ed467463749) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:56.392670 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet 4b76c4408a7340e59b8506171622175f with delete_type TABLET_DATA_DELETED (replaced by tablet 24f07b1efac64b099b0b5ed467463749) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:56.392976 1176 ts_tablet_manager.cc:1929] T ab421dfaa3f84593842bb732a2693160 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:56.393128 1176 log.cc:1199] T ab421dfaa3f84593842bb732a2693160 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/ab421dfaa3f84593842bb732a2693160
I20251029 02:51:56.393554 1176 ts_tablet_manager.cc:1950] T ab421dfaa3f84593842bb732a2693160 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:51:56.394573 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet ab421dfaa3f84593842bb732a2693160 (table default.Table_b70ecfbeceed48a4bfdb94abe7c580f4 [id=03a25420ce6a46fc93a9fb753246bc7b]) successfully deleted
I20251029 02:51:56.377981 1167 tablet_bootstrap.cc:492] T 7c041f4909274cd5a03b9fd78353faf9 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:51:56.397913 1167 ts_tablet_manager.cc:1403] T 7c041f4909274cd5a03b9fd78353faf9 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.107s user 0.003s sys 0.002s
I20251029 02:51:56.391387 1172 ts_tablet_manager.cc:1434] T 5695b5a7b1854c3983e2d4ab684898b0 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251029 02:51:56.398603 1172 tablet_bootstrap.cc:492] T 6cd7009e06a646acad0802918c2b425e P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:51:56.398641 521 master_service.cc:954] ReplaceTablet: received request to replace tablet f246f6c202cf44d48c9f5e4f335202c5 from {username='slave'} at 127.0.0.1:35314
I20251029 02:51:56.399247 521 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_d6a9ff6b4cea442eb2394340301eac96" } modify_external_catalogs: true
I20251029 02:51:56.399550 521 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_d6a9ff6b4cea442eb2394340301eac96" } modify_external_catalogs: true
I20251029 02:51:56.400497 1172 tablet_bootstrap.cc:654] T 6cd7009e06a646acad0802918c2b425e P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:56.401273 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet d6e1170dbb594a5696a3899cde8da7c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:52 UTC) from {username='slave'} at 127.0.0.1:37626
W20251029 02:51:56.404330 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 6ca9bd37de9e4be388501b15af4bda31 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6ca9bd37de9e4be388501b15af4bda31 already in progress: creating tablet
W20251029 02:51:56.404800 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet f246f6c202cf44d48c9f5e4f335202c5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f246f6c202cf44d48c9f5e4f335202c5 already in progress: creating tablet
I20251029 02:51:56.406087 1167 raft_consensus.cc:359] T 7c041f4909274cd5a03b9fd78353faf9 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:56.406375 1167 raft_consensus.cc:385] T 7c041f4909274cd5a03b9fd78353faf9 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:56.406499 1176 tablet_replica.cc:333] T 11f4252b5e694312aaa142468f8e68b4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:51:56.406837 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 277) took 1598 ms. Trace:
I20251029 02:51:56.407001 1174 rpcz_store.cc:276] 1029 02:51:54.807999 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:54.943483 (+135484us) service_pool.cc:225] Handling call
1029 02:51:56.406643 (+1463160us) ts_tablet_manager.cc:1151] Deleting tablet 576c4e6e70e24b0399ac9597a30f0569
1029 02:51:56.406659 (+ 16us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:56.406776 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1354282}
I20251029 02:51:56.383885 1538 tablet.cc:1722] T 5e31e061d74146009d24d5cd57487a5e P c756528642a543eeacd93cef3e335a7d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20251029 02:51:56.407636 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 576c4e6e70e24b0399ac9597a30f0569 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 576c4e6e70e24b0399ac9597a30f0569 already in progress: creating tablet
I20251029 02:51:56.406471 1167 raft_consensus.cc:740] T 7c041f4909274cd5a03b9fd78353faf9 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:51:56.408085 1172 tablet_bootstrap.cc:492] T 6cd7009e06a646acad0802918c2b425e P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:51:56.408242 1172 ts_tablet_manager.cc:1403] T 6cd7009e06a646acad0802918c2b425e P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20251029 02:51:56.408187 1167 consensus_queue.cc:260] T 7c041f4909274cd5a03b9fd78353faf9 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:56.408685 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 280) took 1108 ms. Trace:
I20251029 02:51:56.408788 1174 rpcz_store.cc:276] 1029 02:51:55.299802 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:55.312247 (+ 12445us) service_pool.cc:225] Handling call
1029 02:51:56.408524 (+1096277us) ts_tablet_manager.cc:1151] Deleting tablet 7e6c4f30be624769961c9ec17c5d5dcb
1029 02:51:56.408537 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:56.408632 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1092938}
I20251029 02:51:56.409035 1172 raft_consensus.cc:359] T 6cd7009e06a646acad0802918c2b425e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:56.409243 1172 raft_consensus.cc:385] T 6cd7009e06a646acad0802918c2b425e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251029 02:51:56.409334 524 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d2990e9f626c40618c118266bc595684 consensus state has no leader
I20251029 02:51:56.409328 1172 raft_consensus.cc:740] T 6cd7009e06a646acad0802918c2b425e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:51:56.409566 1172 consensus_queue.cc:260] T 6cd7009e06a646acad0802918c2b425e P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:56.410040 1172 ts_tablet_manager.cc:1434] T 6cd7009e06a646acad0802918c2b425e P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251029 02:51:56.410341 1172 tablet_bootstrap.cc:492] T 3362e95063af41c8a24e935f547e3089 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
W20251029 02:51:56.410774 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 7e6c4f30be624769961c9ec17c5d5dcb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7e6c4f30be624769961c9ec17c5d5dcb already in progress: creating tablet
I20251029 02:51:56.411160 1176 raft_consensus.cc:2243] T 11f4252b5e694312aaa142468f8e68b4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:51:56.411541 1176 raft_consensus.cc:2272] T 11f4252b5e694312aaa142468f8e68b4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:56.412139 1176 ts_tablet_manager.cc:1916] T 11f4252b5e694312aaa142468f8e68b4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:56.394932 881 tablet_service.cc:1552] Processing DeleteTablet for tablet 4b76c4408a7340e59b8506171622175f with delete_type TABLET_DATA_DELETED (replaced by tablet 24f07b1efac64b099b0b5ed467463749) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:56.417033 524 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_ae16b33c42244fc393dd1f163867961a" } new_table_name: "default.Table_3cca74f150f549b7aef4a213cafe99e5" modify_external_catalogs: true
I20251029 02:51:56.423475 1172 tablet_bootstrap.cc:654] T 3362e95063af41c8a24e935f547e3089 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:56.424988 525 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet dd6ad641fbda472fab07aa35c5e4523a
I20251029 02:51:56.425554 525 catalog_manager.cc:6719] ReplaceTablet: tablet dd6ad641fbda472fab07aa35c5e4523a deleted and replaced by tablet 4f24ae08e2c34375abccb6fd187adbd4
I20251029 02:51:56.426894 743 tablet_service.cc:1552] Processing DeleteTablet for tablet dd6ad641fbda472fab07aa35c5e4523a with delete_type TABLET_DATA_DELETED (replaced by tablet 4f24ae08e2c34375abccb6fd187adbd4) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:56.429430 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet dd6ad641fbda472fab07aa35c5e4523a with delete_type TABLET_DATA_DELETED (replaced by tablet 4f24ae08e2c34375abccb6fd187adbd4) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:56.429667 881 tablet_service.cc:1552] Processing DeleteTablet for tablet dd6ad641fbda472fab07aa35c5e4523a with delete_type TABLET_DATA_DELETED (replaced by tablet 4f24ae08e2c34375abccb6fd187adbd4) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:56.432392 525 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_3126aa57075d44dc8b73f08fe21aabdf" } modify_external_catalogs: true
I20251029 02:51:56.432929 525 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_3126aa57075d44dc8b73f08fe21aabdf" } modify_external_catalogs: true
I20251029 02:51:56.433998 881 tablet_service.cc:1552] Processing DeleteTablet for tablet 7c041f4909274cd5a03b9fd78353faf9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:52 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:56.434904 1174 tablet_replica.cc:333] T 6dd4696ad4b44b6dab87a205f3f16ebd P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:51:56.435456 1174 raft_consensus.cc:2243] T 6dd4696ad4b44b6dab87a205f3f16ebd P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:51:56.435949 1174 pending_rounds.cc:70] T 6dd4696ad4b44b6dab87a205f3f16ebd P c756528642a543eeacd93cef3e335a7d: Trying to abort 1 pending ops.
I20251029 02:51:56.436105 1174 pending_rounds.cc:77] T 6dd4696ad4b44b6dab87a205f3f16ebd P c756528642a543eeacd93cef3e335a7d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215949066492227584 op_type: NO_OP noop_request { }
I20251029 02:51:56.436385 1174 raft_consensus.cc:2889] T 6dd4696ad4b44b6dab87a205f3f16ebd P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 02:51:56.436592 1174 raft_consensus.cc:2272] T 6dd4696ad4b44b6dab87a205f3f16ebd P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:56.437471 1174 ts_tablet_manager.cc:1916] T 6dd4696ad4b44b6dab87a205f3f16ebd P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:56.435166 1167 ts_tablet_manager.cc:1434] T 7c041f4909274cd5a03b9fd78353faf9 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.037s user 0.003s sys 0.000s
I20251029 02:51:56.440626 1167 tablet_bootstrap.cc:492] T c4884e22cf2044a4a1070c1b4b759d9e P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:51:56.441816 1176 ts_tablet_manager.cc:1929] T 11f4252b5e694312aaa142468f8e68b4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20251029 02:51:56.441987 1176 log.cc:1199] T 11f4252b5e694312aaa142468f8e68b4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/11f4252b5e694312aaa142468f8e68b4
I20251029 02:51:56.442461 1176 ts_tablet_manager.cc:1950] T 11f4252b5e694312aaa142468f8e68b4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:51:56.445968 1176 tablet_replica.cc:333] T 5e31e061d74146009d24d5cd57487a5e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:51:56.446216 1176 raft_consensus.cc:2243] T 5e31e061d74146009d24d5cd57487a5e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:51:56.446269 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 11f4252b5e694312aaa142468f8e68b4 (table default.Table_b93cf815f69a4fe895e2ce461148fc71 [id=d0cd3d1567c54463947443927e18169f]) successfully deleted
I20251029 02:51:56.446637 1176 raft_consensus.cc:2272] T 5e31e061d74146009d24d5cd57487a5e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:56.447497 520 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_29766c136dbc44e486d4256e7dbba851" } modify_external_catalogs: true
I20251029 02:51:56.447687 520 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_29766c136dbc44e486d4256e7dbba851" } modify_external_catalogs: true
I20251029 02:51:56.452087 1176 ts_tablet_manager.cc:1916] T 5e31e061d74146009d24d5cd57487a5e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:56.454732 519 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet daa05a34c0684c07a35966290fd7d44c
I20251029 02:51:56.454818 1167 tablet_bootstrap.cc:654] T c4884e22cf2044a4a1070c1b4b759d9e P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:56.455216 519 catalog_manager.cc:6719] ReplaceTablet: tablet daa05a34c0684c07a35966290fd7d44c deleted and replaced by tablet fcb3db8399aa4cf188b07dda220f7f5b
I20251029 02:51:56.460095 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet daa05a34c0684c07a35966290fd7d44c with delete_type TABLET_DATA_DELETED (replaced by tablet fcb3db8399aa4cf188b07dda220f7f5b) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:56.460124 881 tablet_service.cc:1552] Processing DeleteTablet for tablet daa05a34c0684c07a35966290fd7d44c with delete_type TABLET_DATA_DELETED (replaced by tablet fcb3db8399aa4cf188b07dda220f7f5b) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:56.462425 519 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_b3ad8b7021294ca484b65c1773165849" } new_table_name: "default.Table_1d6e8e70399347f4a102c28e953a6c1e" modify_external_catalogs: true
I20251029 02:51:56.466187 1167 tablet_bootstrap.cc:492] T c4884e22cf2044a4a1070c1b4b759d9e P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:51:56.466400 1167 ts_tablet_manager.cc:1403] T c4884e22cf2044a4a1070c1b4b759d9e P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.002s
I20251029 02:51:56.467572 1167 raft_consensus.cc:359] T c4884e22cf2044a4a1070c1b4b759d9e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:56.467829 1167 raft_consensus.cc:385] T c4884e22cf2044a4a1070c1b4b759d9e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:56.467952 1167 raft_consensus.cc:740] T c4884e22cf2044a4a1070c1b4b759d9e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:51:56.468879 1172 tablet_bootstrap.cc:492] T 3362e95063af41c8a24e935f547e3089 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:51:56.469864 1172 ts_tablet_manager.cc:1403] T 3362e95063af41c8a24e935f547e3089 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.059s user 0.003s sys 0.001s
I20251029 02:51:56.470042 743 tablet_service.cc:1552] Processing DeleteTablet for tablet daa05a34c0684c07a35966290fd7d44c with delete_type TABLET_DATA_DELETED (replaced by tablet fcb3db8399aa4cf188b07dda220f7f5b) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:56.470556 1175 ts_tablet_manager.cc:1929] T 9964ad774890422f8e14f6b99f28cb55 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:56.470727 1175 log.cc:1199] T 9964ad774890422f8e14f6b99f28cb55 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/9964ad774890422f8e14f6b99f28cb55
I20251029 02:51:56.470851 1172 raft_consensus.cc:359] T 3362e95063af41c8a24e935f547e3089 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:56.471046 1172 raft_consensus.cc:385] T 3362e95063af41c8a24e935f547e3089 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:56.471140 1172 raft_consensus.cc:740] T 3362e95063af41c8a24e935f547e3089 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:51:56.471184 1175 ts_tablet_manager.cc:1950] T 9964ad774890422f8e14f6b99f28cb55 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:51:56.471385 1172 consensus_queue.cc:260] T 3362e95063af41c8a24e935f547e3089 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:56.471943 1172 ts_tablet_manager.cc:1434] T 3362e95063af41c8a24e935f547e3089 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251029 02:51:56.473024 523 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_9306136142dd41cca83965ab5a92c5a0" } modify_external_catalogs: true
I20251029 02:51:56.473228 523 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_9306136142dd41cca83965ab5a92c5a0" } modify_external_catalogs: true
I20251029 02:51:56.474161 1172 tablet_bootstrap.cc:492] T c65469173762446e9725262ae998dbf3 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:51:56.476054 1172 tablet_bootstrap.cc:654] T c65469173762446e9725262ae998dbf3 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:56.476624 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 9964ad774890422f8e14f6b99f28cb55 (table default.Table_da0be17b8bdc4e579b6c3faff71d9c98 [id=6e06ac70c6814e7d89a008bbaddccc7e]) successfully deleted
W20251029 02:51:56.476935 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 6ca9bd37de9e4be388501b15af4bda31 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6ca9bd37de9e4be388501b15af4bda31 already in progress: creating tablet
I20251029 02:51:56.477080 1175 tablet_replica.cc:333] T 44784a3a760946e0be24743bc2fc8138 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
W20251029 02:51:56.477232 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet f246f6c202cf44d48c9f5e4f335202c5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f246f6c202cf44d48c9f5e4f335202c5 already in progress: creating tablet
I20251029 02:51:56.477293 1175 raft_consensus.cc:2243] T 44784a3a760946e0be24743bc2fc8138 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:56.477483 1175 raft_consensus.cc:2272] T 44784a3a760946e0be24743bc2fc8138 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:56.478070 1175 ts_tablet_manager.cc:1916] T 44784a3a760946e0be24743bc2fc8138 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 02:51:56.481240 518 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d5741dde463a43d6a776ab92dcfb04bf consensus state has no leader
I20251029 02:51:56.482082 1172 tablet_bootstrap.cc:492] T c65469173762446e9725262ae998dbf3 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:51:56.482244 1172 ts_tablet_manager.cc:1403] T c65469173762446e9725262ae998dbf3 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20251029 02:51:56.483058 1172 raft_consensus.cc:359] T c65469173762446e9725262ae998dbf3 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:56.483253 1172 raft_consensus.cc:385] T c65469173762446e9725262ae998dbf3 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:56.482414 1167 consensus_queue.cc:260] T c4884e22cf2044a4a1070c1b4b759d9e P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:56.483431 1172 raft_consensus.cc:740] T c65469173762446e9725262ae998dbf3 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:51:56.484027 1167 ts_tablet_manager.cc:1434] T c4884e22cf2044a4a1070c1b4b759d9e P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
I20251029 02:51:56.484057 1176 ts_tablet_manager.cc:1929] T 5e31e061d74146009d24d5cd57487a5e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251029 02:51:56.484225 1176 log.cc:1199] T 5e31e061d74146009d24d5cd57487a5e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/5e31e061d74146009d24d5cd57487a5e
I20251029 02:51:56.484638 1167 tablet_bootstrap.cc:492] T 24016ceaa84147eca0b94b3ad288e772 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:51:56.484752 1176 ts_tablet_manager.cc:1950] T 5e31e061d74146009d24d5cd57487a5e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:51:56.484751 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_982d01dccabe47bfaac7a582282c5d54"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:56.485387 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_982d01dccabe47bfaac7a582282c5d54 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:56.485852 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 5e31e061d74146009d24d5cd57487a5e (table default.Table_f2bc74933428449197f8c28182c8698f [id=8e17d6b87b984d878787802bd6f627f2]) successfully deleted
I20251029 02:51:56.488020 1166 tablet_bootstrap.cc:492] T aacb6f4b9284457bb6607e3a85d5c809 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:51:56.488162 1166 ts_tablet_manager.cc:1403] T aacb6f4b9284457bb6607e3a85d5c809 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.112s user 0.038s sys 0.049s
I20251029 02:51:56.488986 1166 raft_consensus.cc:359] T aacb6f4b9284457bb6607e3a85d5c809 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:56.489193 1166 raft_consensus.cc:385] T aacb6f4b9284457bb6607e3a85d5c809 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:56.489297 1166 raft_consensus.cc:740] T aacb6f4b9284457bb6607e3a85d5c809 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:51:56.491019 1172 consensus_queue.cc:260] T c65469173762446e9725262ae998dbf3 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:56.489535 1166 consensus_queue.cc:260] T aacb6f4b9284457bb6607e3a85d5c809 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
W20251029 02:51:56.494642 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 0d84dc8dc465496f9adec798b2e03e13 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0d84dc8dc465496f9adec798b2e03e13 already in progress: creating tablet
I20251029 02:51:56.494879 1166 ts_tablet_manager.cc:1434] T aacb6f4b9284457bb6607e3a85d5c809 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20251029 02:51:56.495144 1166 tablet_bootstrap.cc:492] T 8d9722c764ac4b27a540595e9c6aafcd P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:51:56.497484 1166 tablet_bootstrap.cc:654] T 8d9722c764ac4b27a540595e9c6aafcd P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:56.498136 1167 tablet_bootstrap.cc:654] T 24016ceaa84147eca0b94b3ad288e772 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:56.500312 1172 ts_tablet_manager.cc:1434] T c65469173762446e9725262ae998dbf3 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.018s user 0.002s sys 0.000s
I20251029 02:51:56.502336 881 tablet_service.cc:1552] Processing DeleteTablet for tablet c4884e22cf2044a4a1070c1b4b759d9e with delete_type TABLET_DATA_DELETED (replaced by tablet 9964ad774890422f8e14f6b99f28cb55) from {username='slave'} at 127.0.0.1:35702
W20251029 02:51:56.503396 847 consensus_peers.cc:597] T 61c1fec3ab4c402eb29f767b9bb7409f P c756528642a543eeacd93cef3e335a7d -> Peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Couldn't send request to peer b495a49349324f7d81908f2da700f0d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 61c1fec3ab4c402eb29f767b9bb7409f. This is attempt 1: this message will repeat every 5th retry.
W20251029 02:51:56.504980 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet af4860c6b72f4d988b206b5ef2f22906 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet af4860c6b72f4d988b206b5ef2f22906 already in progress: creating tablet
W20251029 02:51:56.505895 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 6bd3fac2502e479aa9f4ceaabbb39504 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6bd3fac2502e479aa9f4ceaabbb39504 already in progress: creating tablet
I20251029 02:51:56.498958 1174 ts_tablet_manager.cc:1929] T 6dd4696ad4b44b6dab87a205f3f16ebd P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:51:56.506191 1174 log.cc:1199] T 6dd4696ad4b44b6dab87a205f3f16ebd P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/6dd4696ad4b44b6dab87a205f3f16ebd
W20251029 02:51:56.506466 524 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ba699862e6f646c6beaa39f080d0faba consensus state has no leader
I20251029 02:51:56.506462 1176 tablet_replica.cc:333] T c519832030884a049839d309cc738c28 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:51:56.506799 1174 ts_tablet_manager.cc:1950] T 6dd4696ad4b44b6dab87a205f3f16ebd P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:51:56.507032 1176 raft_consensus.cc:2243] T c519832030884a049839d309cc738c28 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:56.506944 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_8fee569eca38413dbac56707725c6a46"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:56.507236 1176 raft_consensus.cc:2272] T c519832030884a049839d309cc738c28 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251029 02:51:56.507582 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8fee569eca38413dbac56707725c6a46 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:56.507815 1176 ts_tablet_manager.cc:1916] T c519832030884a049839d309cc738c28 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:56.508505 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 6dd4696ad4b44b6dab87a205f3f16ebd (table default.Table_69af0d3b63c44157abd3263a9deb02da [id=90dfeabf0e8945c9bd86472d68884221]) successfully deleted
I20251029 02:51:56.508867 521 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet f187dbd8a55b45b292ea61db43caf47d
W20251029 02:51:56.509266 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 777992acb1bd4765886b802940db74bb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 777992acb1bd4765886b802940db74bb already in progress: creating tablet
I20251029 02:51:56.510047 881 tablet_service.cc:1552] Processing DeleteTablet for tablet f187dbd8a55b45b292ea61db43caf47d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:56 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:56.510113 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_4b9147e899c043ad9023bf0d8d555a0c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:56.510717 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4b9147e899c043ad9023bf0d8d555a0c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:56.512640 743 tablet_service.cc:1552] Processing DeleteTablet for tablet f187dbd8a55b45b292ea61db43caf47d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:56 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:56.514467 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet f187dbd8a55b45b292ea61db43caf47d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:56 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:56.516984 520 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 4f24ae08e2c34375abccb6fd187adbd4
I20251029 02:51:56.517411 1175 ts_tablet_manager.cc:1929] T 44784a3a760946e0be24743bc2fc8138 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:56.517627 1175 log.cc:1199] T 44784a3a760946e0be24743bc2fc8138 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/44784a3a760946e0be24743bc2fc8138
I20251029 02:51:56.518119 1175 ts_tablet_manager.cc:1950] T 44784a3a760946e0be24743bc2fc8138 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
W20251029 02:51:56.519555 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet dfa8b1b9a80e4e8689e0acb44e16346a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dfa8b1b9a80e4e8689e0acb44e16346a already in progress: creating tablet
I20251029 02:51:56.519521 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 44784a3a760946e0be24743bc2fc8138 (table default.Table_3e28d37b4b6146bf95edf245d3f9d0cf [id=915b76276e0940618b7180ebc06cd464]) successfully deleted
W20251029 02:51:56.520593 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 9bb2166a590d4210b02dd779ef6654ad because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9bb2166a590d4210b02dd779ef6654ad already in progress: creating tablet
I20251029 02:51:56.520910 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_757229f7dede492d8da98328eaabc0dd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:56.521435 1174 tablet_replica.cc:333] T 61c1fec3ab4c402eb29f767b9bb7409f P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
W20251029 02:51:56.521502 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_757229f7dede492d8da98328eaabc0dd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:56.521690 1174 raft_consensus.cc:2243] T 61c1fec3ab4c402eb29f767b9bb7409f P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:51:56.522178 1174 pending_rounds.cc:70] T 61c1fec3ab4c402eb29f767b9bb7409f P c756528642a543eeacd93cef3e335a7d: Trying to abort 1 pending ops.
I20251029 02:51:56.522248 1174 pending_rounds.cc:77] T 61c1fec3ab4c402eb29f767b9bb7409f P c756528642a543eeacd93cef3e335a7d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215949068075061248 op_type: NO_OP noop_request { }
I20251029 02:51:56.522363 1174 raft_consensus.cc:2889] T 61c1fec3ab4c402eb29f767b9bb7409f P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 02:51:56.522464 1174 raft_consensus.cc:2272] T 61c1fec3ab4c402eb29f767b9bb7409f P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:56.524943 1176 ts_tablet_manager.cc:1929] T c519832030884a049839d309cc738c28 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:56.525084 1176 log.cc:1199] T c519832030884a049839d309cc738c28 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/c519832030884a049839d309cc738c28
I20251029 02:51:56.525475 1176 ts_tablet_manager.cc:1950] T c519832030884a049839d309cc738c28 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:51:56.531774 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet c519832030884a049839d309cc738c28 (table default.Table_e95a0439ea534f538683392bbc8e5219 [id=8ad554168c7e4b86a373a3c4579e6211]) successfully deleted
I20251029 02:51:56.539597 1176 tablet_replica.cc:333] T 6c22497760274380a832feff7c5a2fae P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:51:56.539916 1176 raft_consensus.cc:2243] T 6c22497760274380a832feff7c5a2fae P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:56.540118 1176 raft_consensus.cc:2272] T 6c22497760274380a832feff7c5a2fae P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:56.541740 1176 ts_tablet_manager.cc:1916] T 6c22497760274380a832feff7c5a2fae P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:56.542126 881 tablet_service.cc:1505] Processing CreateTablet for tablet d041e87ceb11441983d0e38b7929c3a4 (DEFAULT_TABLE table=default.Table_eb12e18b799a47d98ae1f3e8ebfa3588 [id=c54b92541c3047c0887c36d31743e71c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:56.542610 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d041e87ceb11441983d0e38b7929c3a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:56.543330 517 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet 69597d48631d43b6a5b92abbf60d5ed3 (table default.Table_cec9500b890447609a5feb7346bcf7a0 [id=ffce392867b14b6ea9db44f4bdee57c3]). Expected version 2 got 0
I20251029 02:51:56.543980 1395 raft_consensus.cc:493] T 59af906061d049c99ae2153945597007 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:56.544148 1395 raft_consensus.cc:515] T 59af906061d049c99ae2153945597007 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:56.544862 1395 leader_election.cc:290] T 59af906061d049c99ae2153945597007 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:51:56.546020 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59af906061d049c99ae2153945597007" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:51:56.546283 776 raft_consensus.cc:2468] T 59af906061d049c99ae2153945597007 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 0.
I20251029 02:51:56.546948 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59af906061d049c99ae2153945597007" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:51:56.547143 989 leader_election.cc:304] T 59af906061d049c99ae2153945597007 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters:
W20251029 02:51:56.547890 990 leader_election.cc:343] T 59af906061d049c99ae2153945597007 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:56.547912 1395 raft_consensus.cc:2804] T 59af906061d049c99ae2153945597007 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:51:56.548522 1395 raft_consensus.cc:493] T 59af906061d049c99ae2153945597007 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:51:56.548637 1395 raft_consensus.cc:3060] T 59af906061d049c99ae2153945597007 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:51:56.553148 1395 raft_consensus.cc:515] T 59af906061d049c99ae2153945597007 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:56.555217 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59af906061d049c99ae2153945597007" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
I20251029 02:51:56.555847 1167 tablet_bootstrap.cc:492] T 24016ceaa84147eca0b94b3ad288e772 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
W20251029 02:51:56.555874 990 leader_election.cc:343] T 59af906061d049c99ae2153945597007 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:56.556022 1167 ts_tablet_manager.cc:1403] T 24016ceaa84147eca0b94b3ad288e772 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.071s user 0.005s sys 0.000s
I20251029 02:51:56.556885 1167 raft_consensus.cc:359] T 24016ceaa84147eca0b94b3ad288e772 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:56.556959 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59af906061d049c99ae2153945597007" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:51:56.557096 1167 raft_consensus.cc:385] T 24016ceaa84147eca0b94b3ad288e772 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:56.557163 776 raft_consensus.cc:3060] T 59af906061d049c99ae2153945597007 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:51:56.557171 1167 raft_consensus.cc:740] T 24016ceaa84147eca0b94b3ad288e772 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:51:56.557420 1167 consensus_queue.cc:260] T 24016ceaa84147eca0b94b3ad288e772 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:56.557894 1167 ts_tablet_manager.cc:1434] T 24016ceaa84147eca0b94b3ad288e772 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251029 02:51:56.560101 776 raft_consensus.cc:2468] T 59af906061d049c99ae2153945597007 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 1.
I20251029 02:51:56.560393 774 raft_consensus.cc:1275] T dd90fb9eb167478cbc3ec9363af86295 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Refusing update from remote peer 0cfd4bee16f84b7c8b0ff3ce14db20d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251029 02:51:56.561106 989 leader_election.cc:304] T 59af906061d049c99ae2153945597007 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters: c756528642a543eeacd93cef3e335a7d
I20251029 02:51:56.561553 1465 raft_consensus.cc:2804] T 59af906061d049c99ae2153945597007 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:51:56.562646 1350 consensus_queue.cc:1048] T dd90fb9eb167478cbc3ec9363af86295 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:51:56.565066 1395 leader_election.cc:290] T 59af906061d049c99ae2153945597007 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Requested vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:51:56.567337 517 master_service.cc:954] ReplaceTablet: received request to replace tablet 6d830d30b26b4ba799785fec8a6f0769 from {username='slave'} at 127.0.0.1:35314
I20251029 02:51:56.570358 1172 tablet_bootstrap.cc:492] T 1bab90fe904944af948dab3db06b7e34 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:51:56.570358 1176 ts_tablet_manager.cc:1929] T 6c22497760274380a832feff7c5a2fae P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:56.570572 1176 log.cc:1199] T 6c22497760274380a832feff7c5a2fae P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/6c22497760274380a832feff7c5a2fae
I20251029 02:51:56.570971 1176 ts_tablet_manager.cc:1950] T 6c22497760274380a832feff7c5a2fae P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:51:56.572216 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 6c22497760274380a832feff7c5a2fae (table default.Table_0fa5f3e932c846168255e7540c35a515 [id=c2475b97464f45da8990e004660c42b3]) successfully deleted
I20251029 02:51:56.572243 1445 raft_consensus.cc:493] T c65469173762446e9725262ae998dbf3 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:56.573392 1445 raft_consensus.cc:515] T c65469173762446e9725262ae998dbf3 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:56.574401 1445 leader_election.cc:290] T c65469173762446e9725262ae998dbf3 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:51:56.575307 743 tablet_service.cc:1505] Processing CreateTablet for tablet d041e87ceb11441983d0e38b7929c3a4 (DEFAULT_TABLE table=default.Table_eb12e18b799a47d98ae1f3e8ebfa3588 [id=c54b92541c3047c0887c36d31743e71c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:56.575722 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d041e87ceb11441983d0e38b7929c3a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:56.576145 1172 tablet_bootstrap.cc:654] T 1bab90fe904944af948dab3db06b7e34 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:56.577544 1027 tablet_service.cc:1505] Processing CreateTablet for tablet a9e0fcc759a64679b3f59a6c964877b6 (DEFAULT_TABLE table=default.Table_4903da2131c140bca97524eda175ce3c [id=657afc87656d4250b4ca87b473fbb752]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:56.578073 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9e0fcc759a64679b3f59a6c964877b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:56.580629 1041 tablet_service.cc:1505] Processing CreateTablet for tablet 59eee86f249e4d6a9d09d30276dcd659 (DEFAULT_TABLE table=default.Table_8f1e631e578a4010927f77c77cc29cb0 [id=edac8dce6fa747bda675efdf71adb1d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:56.581210 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59eee86f249e4d6a9d09d30276dcd659. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:56.581912 738 tablet_service.cc:1505] Processing CreateTablet for tablet a9e0fcc759a64679b3f59a6c964877b6 (DEFAULT_TABLE table=default.Table_4903da2131c140bca97524eda175ce3c [id=657afc87656d4250b4ca87b473fbb752]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:56.582307 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9e0fcc759a64679b3f59a6c964877b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:56.583874 739 tablet_service.cc:1505] Processing CreateTablet for tablet 59eee86f249e4d6a9d09d30276dcd659 (DEFAULT_TABLE table=default.Table_8f1e631e578a4010927f77c77cc29cb0 [id=edac8dce6fa747bda675efdf71adb1d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:56.584091 1059 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c65469173762446e9725262ae998dbf3" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:51:56.584275 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59eee86f249e4d6a9d09d30276dcd659. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:51:56.589658 712 leader_election.cc:343] T c65469173762446e9725262ae998dbf3 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:56.595458 1350 consensus_queue.cc:1048] T 24016ceaa84147eca0b94b3ad288e772 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:51:56.595748 1042 tablet_service.cc:1505] Processing CreateTablet for tablet d041e87ceb11441983d0e38b7929c3a4 (DEFAULT_TABLE table=default.Table_eb12e18b799a47d98ae1f3e8ebfa3588 [id=c54b92541c3047c0887c36d31743e71c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:56.596287 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d041e87ceb11441983d0e38b7929c3a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:56.605808 1176 tablet_replica.cc:333] T f71388036478404d878a52fc20ece216 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:51:56.606035 1176 raft_consensus.cc:2243] T f71388036478404d878a52fc20ece216 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:56.606231 1176 raft_consensus.cc:2272] T f71388036478404d878a52fc20ece216 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:56.608389 1041 tablet_service.cc:1505] Processing CreateTablet for tablet 67651306f3b44e7eb14dcee9a2007a81 (DEFAULT_TABLE table=default.Table_8cbb25deb1a043959698717a9af07e52 [id=95bbd9adfe994fecab10c9f23c0bedb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:56.608858 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67651306f3b44e7eb14dcee9a2007a81. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:56.629668 518 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_e461f80fc20349ec9ab63da15b9fb436" } new_table_name: "default.Table_5580c367c15f4f3a87626302cfde455f" modify_external_catalogs: true
I20251029 02:51:56.631028 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_3250d60b958a4c37bf654e8b9c5c25c1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:56.631280 1175 tablet_replica.cc:333] T 65fdec7cf5d143208d78d4c3806b1352 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:51:56.631506 1175 raft_consensus.cc:2243] T 65fdec7cf5d143208d78d4c3806b1352 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
W20251029 02:51:56.631636 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3250d60b958a4c37bf654e8b9c5c25c1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:56.631690 1175 raft_consensus.cc:2272] T 65fdec7cf5d143208d78d4c3806b1352 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:56.631632 520 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_0badd99ae4d346928005ec415468a5a1" } new_table_name: "default.Table_f2a72498cf4540949baff93eba1f5ef0" modify_external_catalogs: true
I20251029 02:51:56.632578 1175 ts_tablet_manager.cc:1916] T 65fdec7cf5d143208d78d4c3806b1352 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:56.633072 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_b276bd83a0514185a89941ada499686c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:56.633612 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b276bd83a0514185a89941ada499686c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:56.639735 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_6ed43c1a2617493882d7cb6c568d21b7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:56.640300 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6ed43c1a2617493882d7cb6c568d21b7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:56.640357 1176 ts_tablet_manager.cc:1916] T f71388036478404d878a52fc20ece216 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:56.644141 525 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 1bc45078a6a942c38ff3350c15885eff
I20251029 02:51:56.645023 523 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 1fc87e2d7b4240e08f2b4ac7277293ff
I20251029 02:51:56.645642 1027 tablet_service.cc:1552] Processing DeleteTablet for tablet 1bc45078a6a942c38ff3350c15885eff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:56 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:56.645749 526 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 59eee86f249e4d6a9d09d30276dcd659
I20251029 02:51:56.645864 1038 tablet_service.cc:1552] Processing DeleteTablet for tablet 1fc87e2d7b4240e08f2b4ac7277293ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:56 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:56.646956 739 tablet_service.cc:1552] Processing DeleteTablet for tablet 1fc87e2d7b4240e08f2b4ac7277293ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:56 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:56.647433 738 tablet_service.cc:1552] Processing DeleteTablet for tablet 59eee86f249e4d6a9d09d30276dcd659 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:56 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:56.647722 743 tablet_service.cc:1552] Processing DeleteTablet for tablet 1bc45078a6a942c38ff3350c15885eff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:56 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:56.647935 1038 tablet_service.cc:1552] Processing DeleteTablet for tablet 59eee86f249e4d6a9d09d30276dcd659 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:56 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:56.649485 1465 raft_consensus.cc:697] T 59af906061d049c99ae2153945597007 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Becoming Leader. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Running, Role: LEADER
I20251029 02:51:56.649852 1465 consensus_queue.cc:237] T 59af906061d049c99ae2153945597007 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:56.651592 1167 tablet_bootstrap.cc:492] T fa5127bcfcda40a7b5ad4ef016af072a P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:51:56.657951 1421 raft_consensus.cc:493] T 64b0fb400e944ce2b2c7a49d12135e9c P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:56.658138 1421 raft_consensus.cc:515] T 64b0fb400e944ce2b2c7a49d12135e9c P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:56.660183 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_3e8689e7e89543da830ba312d234ef3f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:56.660753 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3e8689e7e89543da830ba312d234ef3f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:56.662573 523 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet 24016ceaa84147eca0b94b3ad288e772 (table default.Table_f06003da6e324504ade193bbbb11d36a [id=bd279b3ce623477ca20e8d35e3b7c0df]). Expected version 1 got 0
I20251029 02:51:56.665262 525 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet dd90fb9eb167478cbc3ec9363af86295 (table default.Table_1d6e8e70399347f4a102c28e953a6c1e [id=052abf3dcc904bdcb869d5b885723dab]). Expected version 3 got 0
I20251029 02:51:56.666774 743 tablet_service.cc:1552] Processing DeleteTablet for tablet 6cd7009e06a646acad0802918c2b425e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:54 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:56.667460 738 tablet_service.cc:1552] Processing DeleteTablet for tablet 3362e95063af41c8a24e935f547e3089 with delete_type TABLET_DATA_DELETED (replaced by tablet da2a21c3236a4bd99872b89b12a36508) from {username='slave'} at 127.0.0.1:44580
W20251029 02:51:56.668099 990 consensus_peers.cc:597] T dd90fb9eb167478cbc3ec9363af86295 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 -> Peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Couldn't send request to peer c756528642a543eeacd93cef3e335a7d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:51:56.668411 738 tablet_service.cc:1552] Processing DeleteTablet for tablet c65469173762446e9725262ae998dbf3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:52 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:56.668699 743 tablet_service.cc:1552] Processing DeleteTablet for tablet 5695b5a7b1854c3983e2d4ab684898b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:52 UTC) from {username='slave'} at 127.0.0.1:44580
W20251029 02:51:56.670243 849 leader_election.cc:343] T 64b0fb400e944ce2b2c7a49d12135e9c P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Not found: Tablet not found: 64b0fb400e944ce2b2c7a49d12135e9c
W20251029 02:51:56.670571 847 leader_election.cc:343] T 64b0fb400e944ce2b2c7a49d12135e9c P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 64b0fb400e944ce2b2c7a49d12135e9c
I20251029 02:51:56.671995 523 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet 24016ceaa84147eca0b94b3ad288e772 (table default.Table_f06003da6e324504ade193bbbb11d36a [id=bd279b3ce623477ca20e8d35e3b7c0df]). Expected version 1 got 0
I20251029 02:51:56.673338 525 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet 24016ceaa84147eca0b94b3ad288e772 (table default.Table_f06003da6e324504ade193bbbb11d36a [id=bd279b3ce623477ca20e8d35e3b7c0df]). Expected version 1 got 0
I20251029 02:51:56.675436 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c65469173762446e9725262ae998dbf3" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:51:56.675527 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_875a426b444646729b1097ec39577eee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:56.676056 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_875a426b444646729b1097ec39577eee in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:51:56.676157 711 leader_election.cc:343] T c65469173762446e9725262ae998dbf3 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:56.676322 711 leader_election.cc:304] T c65469173762446e9725262ae998dbf3 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:51:56.676788 1445 raft_consensus.cc:2749] T c65469173762446e9725262ae998dbf3 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:51:56.677958 1175 ts_tablet_manager.cc:1929] T 65fdec7cf5d143208d78d4c3806b1352 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:56.678624 1421 leader_election.cc:290] T 64b0fb400e944ce2b2c7a49d12135e9c P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:51:56.678831 878 tablet_service.cc:1505] Processing CreateTablet for tablet a9e0fcc759a64679b3f59a6c964877b6 (DEFAULT_TABLE table=default.Table_4903da2131c140bca97524eda175ce3c [id=657afc87656d4250b4ca87b473fbb752]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:56.679265 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9e0fcc759a64679b3f59a6c964877b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:56.678889 1175 log.cc:1199] T 65fdec7cf5d143208d78d4c3806b1352 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/65fdec7cf5d143208d78d4c3806b1352
I20251029 02:51:56.679649 517 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 6d830d30b26b4ba799785fec8a6f0769
I20251029 02:51:56.680253 517 catalog_manager.cc:6719] ReplaceTablet: tablet 6d830d30b26b4ba799785fec8a6f0769 deleted and replaced by tablet abf18e06df9b482a8b25c078b9495272
I20251029 02:51:56.681143 743 tablet_service.cc:1552] Processing DeleteTablet for tablet 6d830d30b26b4ba799785fec8a6f0769 with delete_type TABLET_DATA_DELETED (replaced by tablet abf18e06df9b482a8b25c078b9495272) from {username='slave'} at 127.0.0.1:44580
W20251029 02:51:56.681298 518 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 28f7ceb91a34450cad4b5b45f5285b41 consensus state has no leader
I20251029 02:51:56.682289 1166 tablet_bootstrap.cc:492] T 8d9722c764ac4b27a540595e9c6aafcd P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:51:56.682765 1166 ts_tablet_manager.cc:1403] T 8d9722c764ac4b27a540595e9c6aafcd P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.188s user 0.013s sys 0.019s
I20251029 02:51:56.683755 883 tablet_service.cc:1505] Processing CreateTablet for tablet 59eee86f249e4d6a9d09d30276dcd659 (DEFAULT_TABLE table=default.Table_8f1e631e578a4010927f77c77cc29cb0 [id=edac8dce6fa747bda675efdf71adb1d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:56.684340 1042 tablet_service.cc:1552] Processing DeleteTablet for tablet 6d830d30b26b4ba799785fec8a6f0769 with delete_type TABLET_DATA_DELETED (replaced by tablet abf18e06df9b482a8b25c078b9495272) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:56.684482 1166 raft_consensus.cc:359] T 8d9722c764ac4b27a540595e9c6aafcd P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:56.684844 1166 raft_consensus.cc:385] T 8d9722c764ac4b27a540595e9c6aafcd P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:56.684978 887 tablet_service.cc:1505] Processing CreateTablet for tablet 67651306f3b44e7eb14dcee9a2007a81 (DEFAULT_TABLE table=default.Table_8cbb25deb1a043959698717a9af07e52 [id=95bbd9adfe994fecab10c9f23c0bedb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:56.685014 1166 raft_consensus.cc:740] T 8d9722c764ac4b27a540595e9c6aafcd P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:51:56.685371 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67651306f3b44e7eb14dcee9a2007a81. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:56.687757 847 leader_election.cc:304] T 64b0fb400e944ce2b2c7a49d12135e9c P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9
I20251029 02:51:56.687588 1176 ts_tablet_manager.cc:1929] T f71388036478404d878a52fc20ece216 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:56.687798 1175 ts_tablet_manager.cc:1950] T 65fdec7cf5d143208d78d4c3806b1352 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:51:56.688498 1421 raft_consensus.cc:2749] T 64b0fb400e944ce2b2c7a49d12135e9c P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:51:56.688714 1176 log.cc:1199] T f71388036478404d878a52fc20ece216 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/f71388036478404d878a52fc20ece216
I20251029 02:51:56.689088 1174 ts_tablet_manager.cc:1916] T 61c1fec3ab4c402eb29f767b9bb7409f P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:56.687160 1166 consensus_queue.cc:260] T 8d9722c764ac4b27a540595e9c6aafcd P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:56.689280 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59eee86f249e4d6a9d09d30276dcd659. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:56.689826 518 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_0c02e8daa2e74175bb2fdbdba0ec0eab" } new_table_name: "default.Table_39c3746edf0e467b9919c9326bbd8f5d" modify_external_catalogs: true
I20251029 02:51:56.691767 896 tablet_service.cc:1552] Processing DeleteTablet for tablet 1fc87e2d7b4240e08f2b4ac7277293ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:56 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:56.692209 890 tablet_service.cc:1552] Processing DeleteTablet for tablet 59eee86f249e4d6a9d09d30276dcd659 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:56 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:56.692353 898 tablet_service.cc:1552] Processing DeleteTablet for tablet 6d830d30b26b4ba799785fec8a6f0769 with delete_type TABLET_DATA_DELETED (replaced by tablet abf18e06df9b482a8b25c078b9495272) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:56.693392 1176 ts_tablet_manager.cc:1950] T f71388036478404d878a52fc20ece216 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:51:56.693917 1580 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.60:41233 every 8 connection(s)
I20251029 02:51:56.695039 1215 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.60:41233
I20251029 02:51:56.697746 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet f71388036478404d878a52fc20ece216 (table default.Table_03f7d6f02b064939ba7464c5173e02e3 [id=9898cffc2b964059ba3163ee4d1395c2]) successfully deleted
I20251029 02:51:56.696946 1166 ts_tablet_manager.cc:1434] T 8d9722c764ac4b27a540595e9c6aafcd P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.014s user 0.001s sys 0.002s
I20251029 02:51:56.698763 1215 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/master-2/data/info.pb
I20251029 02:51:56.698871 881 tablet_service.cc:1552] Processing DeleteTablet for tablet 1bc45078a6a942c38ff3350c15885eff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:56 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:56.701788 1176 tablet_replica.cc:333] T 7c041f4909274cd5a03b9fd78353faf9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:51:56.702132 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 65fdec7cf5d143208d78d4c3806b1352 (table default.Table_d1a375b674704cddb1c3bb13c566729e [id=909ecde47c7245b2b84dcd665e779b1e]) successfully deleted
I20251029 02:51:56.703534 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskYVokEG/build/asan/bin/kudu as pid 1215
I20251029 02:51:56.706235 1166 tablet_bootstrap.cc:492] T 012ce243fd07466ab5301d1c9ce4b413 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:51:56.714810 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_6e96405b009143d2bd1c0942cb98c033"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:56.715607 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6e96405b009143d2bd1c0942cb98c033 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:56.717505 523 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_5902a11255a049be9dd07acf8b50adda" } modify_external_catalogs: true
I20251029 02:51:56.717733 523 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_5902a11255a049be9dd07acf8b50adda" } modify_external_catalogs: true
I20251029 02:51:56.718954 1584 tablet.cc:1722] T 4b76c4408a7340e59b8506171622175f P b495a49349324f7d81908f2da700f0d9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 02:51:56.722360 522 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_9f5273b57e0b4d95af3e737377f031b6" } new_table_name: "default.Table_2ec9c2215f7147adab6bbfa2fd06d968" modify_external_catalogs: true
I20251029 02:51:56.724084 1166 tablet_bootstrap.cc:654] T 012ce243fd07466ab5301d1c9ce4b413 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:56.706317 1175 tablet_replica.cc:333] T 11f4252b5e694312aaa142468f8e68b4 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:51:56.725638 1167 tablet_bootstrap.cc:654] T fa5127bcfcda40a7b5ad4ef016af072a P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:56.727746 1175 raft_consensus.cc:2243] T 11f4252b5e694312aaa142468f8e68b4 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:56.729557 1175 raft_consensus.cc:2272] T 11f4252b5e694312aaa142468f8e68b4 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:56.733876 1175 ts_tablet_manager.cc:1916] T 11f4252b5e694312aaa142468f8e68b4 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:56.740445 1172 tablet_bootstrap.cc:492] T 1bab90fe904944af948dab3db06b7e34 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:51:56.740695 1172 ts_tablet_manager.cc:1403] T 1bab90fe904944af948dab3db06b7e34 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.220s user 0.003s sys 0.002s
I20251029 02:51:56.742198 743 tablet_service.cc:1505] Processing CreateTablet for tablet 67651306f3b44e7eb14dcee9a2007a81 (DEFAULT_TABLE table=default.Table_8cbb25deb1a043959698717a9af07e52 [id=95bbd9adfe994fecab10c9f23c0bedb0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:56.721046 1176 raft_consensus.cc:2243] T 7c041f4909274cd5a03b9fd78353faf9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:56.742568 1176 raft_consensus.cc:2272] T 7c041f4909274cd5a03b9fd78353faf9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:56.742678 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67651306f3b44e7eb14dcee9a2007a81. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:56.743387 1176 ts_tablet_manager.cc:1916] T 7c041f4909274cd5a03b9fd78353faf9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:56.751639 1588 tablet.cc:1722] T 4b76c4408a7340e59b8506171622175f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 02:51:56.755348 1585 sys_catalog.cc:263] Verifying existing consensus state
I20251029 02:51:56.760026 1166 tablet_bootstrap.cc:492] T 012ce243fd07466ab5301d1c9ce4b413 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:51:56.762491 878 tablet_service.cc:1505] Processing CreateTablet for tablet f281191e7c6143f983f11d87f1c9b64e (DEFAULT_TABLE table=default.Table_0c02e8daa2e74175bb2fdbdba0ec0eab [id=cc3e1f3858f04176acb8c39c65396a5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:56.763132 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f281191e7c6143f983f11d87f1c9b64e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:56.763859 1172 raft_consensus.cc:359] T 1bab90fe904944af948dab3db06b7e34 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:56.764148 1172 raft_consensus.cc:385] T 1bab90fe904944af948dab3db06b7e34 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:56.764338 1172 raft_consensus.cc:740] T 1bab90fe904944af948dab3db06b7e34 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:51:56.764672 1172 consensus_queue.cc:260] T 1bab90fe904944af948dab3db06b7e34 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:56.769459 822 heartbeater.cc:344] Connected to a master server at 127.0.104.60:41233
I20251029 02:51:56.770394 1172 ts_tablet_manager.cc:1434] T 1bab90fe904944af948dab3db06b7e34 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.024s user 0.001s sys 0.001s
I20251029 02:51:56.769950 1108 heartbeater.cc:344] Connected to a master server at 127.0.104.60:41233
I20251029 02:51:56.771876 1585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5e5222b26fe4145948620e51edf53fb: Bootstrap starting.
I20251029 02:51:56.770028 1042 tablet_service.cc:1505] Processing CreateTablet for tablet f281191e7c6143f983f11d87f1c9b64e (DEFAULT_TABLE table=default.Table_0c02e8daa2e74175bb2fdbdba0ec0eab [id=cc3e1f3858f04176acb8c39c65396a5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:56.773468 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f281191e7c6143f983f11d87f1c9b64e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:56.780725 883 tablet_service.cc:1505] Processing CreateTablet for tablet 378f549ff7364d018d8b9a5ef01ac878 (DEFAULT_TABLE table=default.Table_644024f7ee3a44ce988f60dedec9419b [id=0e6e6f5dc35241c5a04c9d6cf4c39b26]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:56.781237 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 378f549ff7364d018d8b9a5ef01ac878. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:56.783466 743 tablet_service.cc:1505] Processing CreateTablet for tablet f281191e7c6143f983f11d87f1c9b64e (DEFAULT_TABLE table=default.Table_0c02e8daa2e74175bb2fdbdba0ec0eab [id=cc3e1f3858f04176acb8c39c65396a5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:56.783941 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f281191e7c6143f983f11d87f1c9b64e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:56.784868 1172 tablet_bootstrap.cc:492] T 3cded156380f45cc95893825c71ba0e9 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:51:56.786733 1172 tablet_bootstrap.cc:654] T 3cded156380f45cc95893825c71ba0e9 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:56.787514 1585 log.cc:826] T 00000000000000000000000000000000 P e5e5222b26fe4145948620e51edf53fb: Log is configured to *not* fsync() on all Append() calls
I20251029 02:51:56.782004 1166 ts_tablet_manager.cc:1403] T 012ce243fd07466ab5301d1c9ce4b413 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.079s user 0.002s sys 0.003s
I20251029 02:51:56.792241 1568 tablet.cc:1722] T dd90fb9eb167478cbc3ec9363af86295 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251029 02:51:56.795217 1570 tablet.cc:1722] T dd90fb9eb167478cbc3ec9363af86295 P b495a49349324f7d81908f2da700f0d9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251029 02:51:56.796411 1166 raft_consensus.cc:359] T 012ce243fd07466ab5301d1c9ce4b413 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:56.797199 1166 raft_consensus.cc:385] T 012ce243fd07466ab5301d1c9ce4b413 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:56.797763 1166 raft_consensus.cc:740] T 012ce243fd07466ab5301d1c9ce4b413 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:51:56.798923 1166 consensus_queue.cc:260] T 012ce243fd07466ab5301d1c9ce4b413 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:56.805141 1166 ts_tablet_manager.cc:1434] T 012ce243fd07466ab5301d1c9ce4b413 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.014s user 0.000s sys 0.003s
I20251029 02:51:56.808413 1166 tablet_bootstrap.cc:492] T 14698af6348f41e384f10569392d9b23 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
W20251029 02:51:56.817939 522 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 59d03a190b784ab18a3e63a572675891 consensus state has no leader
W20251029 02:51:56.818914 520 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8048506ced4540a5aee9f79bfc4599bf consensus state has no leader
I20251029 02:51:56.818881 522 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_5265805d6c3746209f9466df88b164b9" } modify_external_catalogs: true
I20251029 02:51:56.819146 522 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_5265805d6c3746209f9466df88b164b9" } modify_external_catalogs: true
I20251029 02:51:56.819321 520 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_4903da2131c140bca97524eda175ce3c" } new_table_name: "default.Table_dc394a25546b48d2a96a1d97091c12f4" modify_external_catalogs: true
I20251029 02:51:56.820159 521 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet dd90fb9eb167478cbc3ec9363af86295 (table default.Table_1d6e8e70399347f4a102c28e953a6c1e [id=052abf3dcc904bdcb869d5b885723dab]). Expected version 3 got 0
I20251029 02:51:56.820981 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_0bb20faeb14644ca8e2d5acd4af09f85"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:56.821270 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_0b95eba70e97459eb847fecfac513686"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:56.825152 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0b95eba70e97459eb847fecfac513686 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:56.822031 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_d477b4de8ad4449aa5d53db48d45273f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:56.826858 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d477b4de8ad4449aa5d53db48d45273f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:51:56.823007 518 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f281191e7c6143f983f11d87f1c9b64e consensus state has no leader
I20251029 02:51:56.823603 523 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 0f80b840ed954230bfbfe7e40127cd2f
I20251029 02:51:56.829660 1175 ts_tablet_manager.cc:1929] T 11f4252b5e694312aaa142468f8e68b4 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20251029 02:51:56.831314 1175 log.cc:1199] T 11f4252b5e694312aaa142468f8e68b4 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/11f4252b5e694312aaa142468f8e68b4
I20251029 02:51:56.833216 738 tablet_service.cc:1552] Processing DeleteTablet for tablet 1bab90fe904944af948dab3db06b7e34 with delete_type TABLET_DATA_DELETED (replaced by tablet b6d4f7bf702f4834bc5f4ef2b50b92a7) from {username='slave'} at 127.0.0.1:44580
W20251029 02:51:56.833263 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0bb20faeb14644ca8e2d5acd4af09f85 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:56.833496 1166 tablet_bootstrap.cc:654] T 14698af6348f41e384f10569392d9b23 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:56.836359 743 tablet_service.cc:1552] Processing DeleteTablet for tablet 0f80b840ed954230bfbfe7e40127cd2f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:56 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:56.837314 1175 ts_tablet_manager.cc:1950] T 11f4252b5e694312aaa142468f8e68b4 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:51:56.836447 887 tablet_service.cc:1552] Processing DeleteTablet for tablet 0f80b840ed954230bfbfe7e40127cd2f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:56 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:56.843784 1042 tablet_service.cc:1552] Processing DeleteTablet for tablet 0f80b840ed954230bfbfe7e40127cd2f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:56 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:56.846119 523 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_ee95bb03ab1246e8826f28083411050d" } modify_external_catalogs: true
I20251029 02:51:56.846643 523 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_ee95bb03ab1246e8826f28083411050d" } modify_external_catalogs: true
I20251029 02:51:56.847373 518 catalog_manager.cc:5649] T 59af906061d049c99ae2153945597007 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } health_report { overall_health: UNKNOWN } } }
I20251029 02:51:56.847927 518 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet 59af906061d049c99ae2153945597007 (table default.Table_3bc6a1815ce646fb9a89e91fdda46ae0 [id=25f16d95843d41c981d6a07ec0d4b48a]). Expected version 1 got 0
I20251029 02:51:56.849118 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_ee6dec2fc7834c78914ddc9f13890c55"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:56.850031 1175 tablet_replica.cc:333] T fd0f89180aba43eabb20f926abccbaa6 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
W20251029 02:51:56.850445 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ee6dec2fc7834c78914ddc9f13890c55 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:56.850752 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 11f4252b5e694312aaa142468f8e68b4 (table default.Table_b93cf815f69a4fe895e2ce461148fc71 [id=d0cd3d1567c54463947443927e18169f]) successfully deleted
I20251029 02:51:56.850826 1175 raft_consensus.cc:2243] T fd0f89180aba43eabb20f926abccbaa6 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:56.851485 525 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_4e899dcc04fe4cb1b96ced1fa1b03585" } modify_external_catalogs: true
I20251029 02:51:56.851686 525 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_4e899dcc04fe4cb1b96ced1fa1b03585" } modify_external_catalogs: true
I20251029 02:51:56.851490 1175 raft_consensus.cc:2272] T fd0f89180aba43eabb20f926abccbaa6 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:56.852908 1167 tablet_bootstrap.cc:492] T fa5127bcfcda40a7b5ad4ef016af072a P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:51:56.853297 526 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_da4961f316764505bb9e0f191a62a327" } modify_external_catalogs: true
I20251029 02:51:56.853484 526 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_da4961f316764505bb9e0f191a62a327" } modify_external_catalogs: true
I20251029 02:51:56.853789 1175 ts_tablet_manager.cc:1916] T fd0f89180aba43eabb20f926abccbaa6 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:56.856065 1167 ts_tablet_manager.cc:1403] T fa5127bcfcda40a7b5ad4ef016af072a P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.298s user 0.003s sys 0.002s
I20251029 02:51:56.858649 1042 tablet_service.cc:1505] Processing CreateTablet for tablet 378f549ff7364d018d8b9a5ef01ac878 (DEFAULT_TABLE table=default.Table_644024f7ee3a44ce988f60dedec9419b [id=0e6e6f5dc35241c5a04c9d6cf4c39b26]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:56.859120 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 378f549ff7364d018d8b9a5ef01ac878. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:56.865309 1445 raft_consensus.cc:493] T 6cd7009e06a646acad0802918c2b425e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:56.865504 1445 raft_consensus.cc:515] T 6cd7009e06a646acad0802918c2b425e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:56.866278 1445 leader_election.cc:290] T 6cd7009e06a646acad0802918c2b425e P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:51:56.867065 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6cd7009e06a646acad0802918c2b425e" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:51:56.867827 711 leader_election.cc:343] T 6cd7009e06a646acad0802918c2b425e P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:56.868485 1059 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6cd7009e06a646acad0802918c2b425e" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:51:56.869069 712 leader_election.cc:343] T 6cd7009e06a646acad0802918c2b425e P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:56.869217 712 leader_election.cc:304] T 6cd7009e06a646acad0802918c2b425e P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:51:56.869637 1445 raft_consensus.cc:2749] T 6cd7009e06a646acad0802918c2b425e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:51:56.871053 743 tablet_service.cc:1505] Processing CreateTablet for tablet 378f549ff7364d018d8b9a5ef01ac878 (DEFAULT_TABLE table=default.Table_644024f7ee3a44ce988f60dedec9419b [id=0e6e6f5dc35241c5a04c9d6cf4c39b26]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:56.872017 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 378f549ff7364d018d8b9a5ef01ac878. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:56.876581 887 tablet_service.cc:1505] Processing CreateTablet for tablet 7bb85a12f573494cbb3118d1947eda27 (DEFAULT_TABLE table=default.Table_0100f32eabf24d88a739a3cdac3597e1 [id=f22450cafadd404d8d224cdee646b5a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:56.877094 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7bb85a12f573494cbb3118d1947eda27. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:56.877097 1041 tablet_service.cc:1505] Processing CreateTablet for tablet 7bb85a12f573494cbb3118d1947eda27 (DEFAULT_TABLE table=default.Table_0100f32eabf24d88a739a3cdac3597e1 [id=f22450cafadd404d8d224cdee646b5a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:56.877518 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7bb85a12f573494cbb3118d1947eda27. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:56.856911 1167 raft_consensus.cc:359] T fa5127bcfcda40a7b5ad4ef016af072a P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:56.879346 1167 raft_consensus.cc:385] T fa5127bcfcda40a7b5ad4ef016af072a P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:56.879474 1167 raft_consensus.cc:740] T fa5127bcfcda40a7b5ad4ef016af072a P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:51:56.888031 743 tablet_service.cc:1505] Processing CreateTablet for tablet 7bb85a12f573494cbb3118d1947eda27 (DEFAULT_TABLE table=default.Table_0100f32eabf24d88a739a3cdac3597e1 [id=f22450cafadd404d8d224cdee646b5a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:56.888832 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7bb85a12f573494cbb3118d1947eda27. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:56.893374 1167 consensus_queue.cc:260] T fa5127bcfcda40a7b5ad4ef016af072a P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:56.897066 1167 ts_tablet_manager.cc:1434] T fa5127bcfcda40a7b5ad4ef016af072a P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.041s user 0.001s sys 0.001s
I20251029 02:51:56.897418 1167 tablet_bootstrap.cc:492] T 59af906061d049c99ae2153945597007 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:51:56.900950 522 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 1275e192871440408a8345fef9b56260
I20251029 02:51:56.902598 522 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet fa5127bcfcda40a7b5ad4ef016af072a (table default.Table_5113d1a33e8e4bd2af5bc51e689f304f [id=ae5d56d40fa941a68f2b10d5e6e9ac83]). Expected version 2 got 0
I20251029 02:51:56.902904 1445 raft_consensus.cc:493] T 1bab90fe904944af948dab3db06b7e34 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:56.902383 524 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_fb891e36b29c4a3ab6f1eef96dde937d" } new_table_name: "default.Table_96ea715e50e74cc48759dda025a46318" modify_external_catalogs: true
I20251029 02:51:56.903069 1445 raft_consensus.cc:515] T 1bab90fe904944af948dab3db06b7e34 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:56.903784 1445 leader_election.cc:290] T 1bab90fe904944af948dab3db06b7e34 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:51:56.904992 1059 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1bab90fe904944af948dab3db06b7e34" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:51:56.905236 738 tablet_service.cc:1552] Processing DeleteTablet for tablet 1275e192871440408a8345fef9b56260 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:56 UTC) from {username='slave'} at 127.0.0.1:44580
W20251029 02:51:56.905718 712 leader_election.cc:343] T 1bab90fe904944af948dab3db06b7e34 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:56.906153 887 tablet_service.cc:1552] Processing DeleteTablet for tablet 1275e192871440408a8345fef9b56260 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:56 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:56.906368 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1bab90fe904944af948dab3db06b7e34" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:51:56.906718 966 heartbeater.cc:344] Connected to a master server at 127.0.104.60:41233
I20251029 02:51:56.910735 526 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 8f7c746f8b6948b58676298c2c2ea221
W20251029 02:51:56.911221 520 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a9e0fcc759a64679b3f59a6c964877b6 consensus state has no leader
W20251029 02:51:56.912045 711 leader_election.cc:343] T 1bab90fe904944af948dab3db06b7e34 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:56.912196 711 leader_election.cc:304] T 1bab90fe904944af948dab3db06b7e34 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:51:56.912694 738 tablet_service.cc:1552] Processing DeleteTablet for tablet 8f7c746f8b6948b58676298c2c2ea221 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:56 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:56.913177 1445 raft_consensus.cc:2749] T 1bab90fe904944af948dab3db06b7e34 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:51:56.913584 887 tablet_service.cc:1552] Processing DeleteTablet for tablet 8f7c746f8b6948b58676298c2c2ea221 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:56 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:56.916035 1027 tablet_service.cc:1552] Processing DeleteTablet for tablet 8f7c746f8b6948b58676298c2c2ea221 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:56 UTC) from {username='slave'} at 127.0.0.1:37626
W20251029 02:51:56.919974 510 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet faa835de84ff45b19525d1d603678987 (table default.Table_9dcb4bbb802b48b198c74e95f343200d [id=a184f139921d4864b1627057a7359810], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet faa835de84ff45b19525d1d603678987 consensus state has no leader
I20251029 02:51:56.921144 523 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 51effc9324034ae79e2b25cb57a996c1
I20251029 02:51:56.922262 887 tablet_service.cc:1552] Processing DeleteTablet for tablet 51effc9324034ae79e2b25cb57a996c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:56 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:56.922315 518 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet fa5127bcfcda40a7b5ad4ef016af072a (table default.Table_5113d1a33e8e4bd2af5bc51e689f304f [id=ae5d56d40fa941a68f2b10d5e6e9ac83]). Expected version 2 got 0
I20251029 02:51:56.923832 738 tablet_service.cc:1552] Processing DeleteTablet for tablet 51effc9324034ae79e2b25cb57a996c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:56 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:56.924870 1027 tablet_service.cc:1552] Processing DeleteTablet for tablet 51effc9324034ae79e2b25cb57a996c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:56 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:56.926047 525 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 71068739b4384badbf305313d902f0f5
I20251029 02:51:56.927435 1027 tablet_service.cc:1552] Processing DeleteTablet for tablet 71068739b4384badbf305313d902f0f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:56 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:56.927448 887 tablet_service.cc:1552] Processing DeleteTablet for tablet 71068739b4384badbf305313d902f0f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:56 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:56.927655 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet 1275e192871440408a8345fef9b56260 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:56 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:56.927992 743 tablet_service.cc:1552] Processing DeleteTablet for tablet 71068739b4384badbf305313d902f0f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:56 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:56.929394 1175 ts_tablet_manager.cc:1929] T fd0f89180aba43eabb20f926abccbaa6 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:56.930495 1175 log.cc:1199] T fd0f89180aba43eabb20f926abccbaa6 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/fd0f89180aba43eabb20f926abccbaa6
I20251029 02:51:56.933212 1175 ts_tablet_manager.cc:1950] T fd0f89180aba43eabb20f926abccbaa6 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:51:56.933951 1465 raft_consensus.cc:493] T 6f93812578a940a3b7378015e2a46f95 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:56.934116 1465 raft_consensus.cc:515] T 6f93812578a940a3b7378015e2a46f95 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:56.934877 1465 leader_election.cc:290] T 6f93812578a940a3b7378015e2a46f95 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:51:56.935635 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f93812578a940a3b7378015e2a46f95" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:51:56.936311 990 leader_election.cc:343] T 6f93812578a940a3b7378015e2a46f95 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
W20251029 02:51:56.937471 989 leader_election.cc:343] T 6f93812578a940a3b7378015e2a46f95 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 6f93812578a940a3b7378015e2a46f95
I20251029 02:51:56.937600 989 leader_election.cc:304] T 6f93812578a940a3b7378015e2a46f95 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5; no voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d
I20251029 02:51:56.938179 1465 raft_consensus.cc:2749] T 6f93812578a940a3b7378015e2a46f95 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:51:56.942229 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet fd0f89180aba43eabb20f926abccbaa6 (table default.Table_50252b3b3cc4403da5e259f9f91be915 [id=ff64a323de354792997beb99bae4a52c]) successfully deleted
I20251029 02:51:56.942858 887 tablet_service.cc:1505] Processing CreateTablet for tablet 58dde9397b434b089ade9968f369006a (DEFAULT_TABLE table=default.Table_b9f2a2e3d9f8446fbd46ca6c48155ccf [id=233cd457a662483a84d93d4243a978b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:56.943140 1028 tablet_service.cc:1505] Processing CreateTablet for tablet 58dde9397b434b089ade9968f369006a (DEFAULT_TABLE table=default.Table_b9f2a2e3d9f8446fbd46ca6c48155ccf [id=233cd457a662483a84d93d4243a978b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:56.943403 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58dde9397b434b089ade9968f369006a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:56.943569 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58dde9397b434b089ade9968f369006a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:56.943611 1175 tablet_replica.cc:333] T 5e31e061d74146009d24d5cd57487a5e P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:51:56.945115 1175 raft_consensus.cc:2243] T 5e31e061d74146009d24d5cd57487a5e P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:56.946229 1175 raft_consensus.cc:2272] T 5e31e061d74146009d24d5cd57487a5e P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251029 02:51:56.947392 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet af4860c6b72f4d988b206b5ef2f22906 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet af4860c6b72f4d988b206b5ef2f22906 already in progress: creating tablet
I20251029 02:51:56.949517 1175 ts_tablet_manager.cc:1916] T 5e31e061d74146009d24d5cd57487a5e P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 02:51:56.950392 524 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 95f342d1cfb84c4eb31bd3d15ea3bdc2 consensus state has no leader
I20251029 02:51:56.950630 1465 raft_consensus.cc:493] T aacb6f4b9284457bb6607e3a85d5c809 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:56.950788 1465 raft_consensus.cc:515] T aacb6f4b9284457bb6607e3a85d5c809 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:56.951484 1465 leader_election.cc:290] T aacb6f4b9284457bb6607e3a85d5c809 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:51:56.951522 517 catalog_manager.cc:6194] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: default.Table_1d6e8e70399347f4a102c28e953a6c1e [id=052abf3dcc904bdcb869d5b885723dab] alter complete (version 3)
I20251029 02:51:56.952181 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aacb6f4b9284457bb6607e3a85d5c809" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:51:56.952739 990 leader_election.cc:343] T aacb6f4b9284457bb6607e3a85d5c809 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:56.953505 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aacb6f4b9284457bb6607e3a85d5c809" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:51:56.953809 776 raft_consensus.cc:2468] T aacb6f4b9284457bb6607e3a85d5c809 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 0.
I20251029 02:51:56.954267 989 leader_election.cc:304] T aacb6f4b9284457bb6607e3a85d5c809 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters: c756528642a543eeacd93cef3e335a7d
I20251029 02:51:56.955320 1038 tablet_service.cc:1552] Processing DeleteTablet for tablet 012ce243fd07466ab5301d1c9ce4b413 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:55 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:56.955633 1027 tablet_service.cc:1552] Processing DeleteTablet for tablet 8d9722c764ac4b27a540595e9c6aafcd with delete_type TABLET_DATA_DELETED (replaced by tablet 51effc9324034ae79e2b25cb57a996c1) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:56.955909 1465 raft_consensus.cc:2804] T aacb6f4b9284457bb6607e3a85d5c809 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:51:56.956027 1465 raft_consensus.cc:493] T aacb6f4b9284457bb6607e3a85d5c809 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:51:56.956104 1465 raft_consensus.cc:3060] T aacb6f4b9284457bb6607e3a85d5c809 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:51:56.959420 1465 raft_consensus.cc:515] T aacb6f4b9284457bb6607e3a85d5c809 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:56.960664 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aacb6f4b9284457bb6607e3a85d5c809" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
W20251029 02:51:56.961257 990 leader_election.cc:343] T aacb6f4b9284457bb6607e3a85d5c809 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:56.961639 1465 leader_election.cc:290] T aacb6f4b9284457bb6607e3a85d5c809 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Requested vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:51:56.962314 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aacb6f4b9284457bb6607e3a85d5c809" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:51:56.962577 774 raft_consensus.cc:3060] T aacb6f4b9284457bb6607e3a85d5c809 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:51:56.880784 1174 ts_tablet_manager.cc:1929] T 61c1fec3ab4c402eb29f767b9bb7409f P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:51:56.963414 1174 log.cc:1199] T 61c1fec3ab4c402eb29f767b9bb7409f P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/61c1fec3ab4c402eb29f767b9bb7409f
I20251029 02:51:56.964257 1174 ts_tablet_manager.cc:1950] T 61c1fec3ab4c402eb29f767b9bb7409f P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:51:56.965667 774 raft_consensus.cc:2468] T aacb6f4b9284457bb6607e3a85d5c809 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 1.
I20251029 02:51:56.966414 989 leader_election.cc:304] T aacb6f4b9284457bb6607e3a85d5c809 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters: c756528642a543eeacd93cef3e335a7d
I20251029 02:51:56.968880 1176 ts_tablet_manager.cc:1929] T 7c041f4909274cd5a03b9fd78353faf9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:56.981442 743 tablet_service.cc:1505] Processing CreateTablet for tablet 58dde9397b434b089ade9968f369006a (DEFAULT_TABLE table=default.Table_b9f2a2e3d9f8446fbd46ca6c48155ccf [id=233cd457a662483a84d93d4243a978b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:56.981931 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58dde9397b434b089ade9968f369006a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:56.984125 1176 log.cc:1199] T 7c041f4909274cd5a03b9fd78353faf9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/7c041f4909274cd5a03b9fd78353faf9
I20251029 02:51:56.984565 1176 ts_tablet_manager.cc:1950] T 7c041f4909274cd5a03b9fd78353faf9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:51:56.966287 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 286) took 1318 ms. Trace:
I20251029 02:51:56.986105 1176 tablet_replica.cc:333] T 6f93812578a940a3b7378015e2a46f95 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:51:56.986093 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 7c041f4909274cd5a03b9fd78353faf9 (table default.Table_83d3fe1030e0461895fc2437732b4ff8 [id=e11618f2bd094526ad359c65bee6a9ba]) successfully deleted
I20251029 02:51:56.986341 1176 raft_consensus.cc:2243] T 6f93812578a940a3b7378015e2a46f95 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:56.986994 1176 raft_consensus.cc:2272] T 6f93812578a940a3b7378015e2a46f95 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:56.987692 1176 ts_tablet_manager.cc:1916] T 6f93812578a940a3b7378015e2a46f95 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:56.989770 1465 raft_consensus.cc:2804] T aacb6f4b9284457bb6607e3a85d5c809 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:51:56.992069 1465 raft_consensus.cc:697] T aacb6f4b9284457bb6607e3a85d5c809 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Becoming Leader. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Running, Role: LEADER
I20251029 02:51:56.992648 1465 consensus_queue.cc:237] T aacb6f4b9284457bb6607e3a85d5c809 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:56.990415 887 tablet_service.cc:1505] Processing CreateTablet for tablet 40e844fd7ad74680bcbe7bef936c5b9d (DEFAULT_TABLE table=default.Table_a66eb4754b854a998e458627d738338b [id=c47f0d4579d74e31be083ac638480e38]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:56.998965 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40e844fd7ad74680bcbe7bef936c5b9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:56.995030 1395 raft_consensus.cc:493] T d6e1170dbb594a5696a3899cde8da7c9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:57.001508 1395 raft_consensus.cc:515] T d6e1170dbb594a5696a3899cde8da7c9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
W20251029 02:51:57.004758 989 leader_election.cc:343] T d6e1170dbb594a5696a3899cde8da7c9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: d6e1170dbb594a5696a3899cde8da7c9
I20251029 02:51:57.005146 1395 leader_election.cc:290] T d6e1170dbb594a5696a3899cde8da7c9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:51:57.010047 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d6e1170dbb594a5696a3899cde8da7c9" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:51:57.012809 743 tablet_service.cc:1505] Processing CreateTablet for tablet 40e844fd7ad74680bcbe7bef936c5b9d (DEFAULT_TABLE table=default.Table_a66eb4754b854a998e458627d738338b [id=c47f0d4579d74e31be083ac638480e38]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 02:51:57.013942 990 leader_election.cc:343] T d6e1170dbb594a5696a3899cde8da7c9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:57.014986 1174 rpcz_store.cc:276] 1029 02:51:55.648128 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:55.648748 (+ 620us) service_pool.cc:225] Handling call
1029 02:51:56.521347 (+872599us) ts_tablet_manager.cc:1151] Deleting tablet 61c1fec3ab4c402eb29f767b9bb7409f
1029 02:51:56.521358 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:56.730958 (+209600us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:56.880661 (+149703us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:56.966169 (+ 85508us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":869039}
I20251029 02:51:57.015985 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 61c1fec3ab4c402eb29f767b9bb7409f (table default.Table_8cf076425649436289d327edc92d4642 [id=c124cfa700724f6db0f1a12fe495c2af]) successfully deleted
I20251029 02:51:57.016511 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40e844fd7ad74680bcbe7bef936c5b9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:57.023759 524 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_eb12e18b799a47d98ae1f3e8ebfa3588" } new_table_name: "default.Table_6a78c7ef966c490aaa5e73f56d6d194e" modify_external_catalogs: true
I20251029 02:51:57.025028 990 leader_election.cc:304] T d6e1170dbb594a5696a3899cde8da7c9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5; no voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d
I20251029 02:51:57.025612 518 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_39c3746edf0e467b9919c9326bbd8f5d" } new_table_name: "default.Table_e5ebaa623385480b9cd7be9694ff024f" modify_external_catalogs: true
I20251029 02:51:57.026221 1176 ts_tablet_manager.cc:1929] T 6f93812578a940a3b7378015e2a46f95 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:57.027196 1038 tablet_service.cc:1505] Processing CreateTablet for tablet 40e844fd7ad74680bcbe7bef936c5b9d (DEFAULT_TABLE table=default.Table_a66eb4754b854a998e458627d738338b [id=c47f0d4579d74e31be083ac638480e38]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:57.027622 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 40e844fd7ad74680bcbe7bef936c5b9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:57.033962 1558 raft_consensus.cc:493] T 8d9722c764ac4b27a540595e9c6aafcd P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:57.034117 1558 raft_consensus.cc:515] T 8d9722c764ac4b27a540595e9c6aafcd P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:57.034663 1350 raft_consensus.cc:2749] T d6e1170dbb594a5696a3899cde8da7c9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:51:57.033711 1175 ts_tablet_manager.cc:1929] T 5e31e061d74146009d24d5cd57487a5e P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251029 02:51:57.036688 1175 log.cc:1199] T 5e31e061d74146009d24d5cd57487a5e P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/5e31e061d74146009d24d5cd57487a5e
I20251029 02:51:57.037282 1558 leader_election.cc:290] T 8d9722c764ac4b27a540595e9c6aafcd P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
W20251029 02:51:57.040877 989 leader_election.cc:343] T 8d9722c764ac4b27a540595e9c6aafcd P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 8d9722c764ac4b27a540595e9c6aafcd
I20251029 02:51:57.041923 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_e0854491deb34525bce9198a673fb943"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:57.041750 1175 ts_tablet_manager.cc:1950] T 5e31e061d74146009d24d5cd57487a5e P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
W20251029 02:51:57.042759 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e0854491deb34525bce9198a673fb943 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:57.047339 521 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_9dcb4bbb802b48b198c74e95f343200d" } new_table_name: "default.Table_d9e164af67334c98a5d0821f2d80a35e" modify_external_catalogs: true
I20251029 02:51:57.047463 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_08fcf9684b844572a63b823f6a4d2302"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:57.047648 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_559b8d2aa7c04ba68f184405d44898d6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:57.048048 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_559b8d2aa7c04ba68f184405d44898d6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:57.049502 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_a6c1519740ac4bbcbfd00671719b07a0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:57.050033 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a6c1519740ac4bbcbfd00671719b07a0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:57.049955 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_b15248a339e14b85944e649464498b84"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:57.050557 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b15248a339e14b85944e649464498b84 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:57.051206 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_040794c7b9064642a98d202e77a60ac5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:57.051699 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_040794c7b9064642a98d202e77a60ac5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:57.052148 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 5e31e061d74146009d24d5cd57487a5e (table default.Table_f2bc74933428449197f8c28182c8698f [id=8e17d6b87b984d878787802bd6f627f2]) successfully deleted
I20251029 02:51:57.052800 519 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_2ec9c2215f7147adab6bbfa2fd06d968" } modify_external_catalogs: true
W20251029 02:51:57.052968 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 0d84dc8dc465496f9adec798b2e03e13 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0d84dc8dc465496f9adec798b2e03e13 already in progress: creating tablet
I20251029 02:51:57.053011 519 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_2ec9c2215f7147adab6bbfa2fd06d968" } modify_external_catalogs: true
W20251029 02:51:57.053411 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_08fcf9684b844572a63b823f6a4d2302 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:51:57.055261 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 6bd3fac2502e479aa9f4ceaabbb39504 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6bd3fac2502e479aa9f4ceaabbb39504 already in progress: creating tablet
W20251029 02:51:57.055498 524 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d041e87ceb11441983d0e38b7929c3a4 consensus state has no leader
I20251029 02:51:57.055805 1176 log.cc:1199] T 6f93812578a940a3b7378015e2a46f95 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/6f93812578a940a3b7378015e2a46f95
I20251029 02:51:57.056198 1445 raft_consensus.cc:493] T 5695b5a7b1854c3983e2d4ab684898b0 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:57.056886 1445 raft_consensus.cc:515] T 5695b5a7b1854c3983e2d4ab684898b0 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:57.057541 1445 leader_election.cc:290] T 5695b5a7b1854c3983e2d4ab684898b0 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:51:57.058063 1175 tablet_replica.cc:333] T 4b76c4408a7340e59b8506171622175f P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
W20251029 02:51:57.059131 518 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f281191e7c6143f983f11d87f1c9b64e consensus state has no leader
I20251029 02:51:57.059573 1059 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5695b5a7b1854c3983e2d4ab684898b0" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:51:57.060158 712 leader_election.cc:343] T 5695b5a7b1854c3983e2d4ab684898b0 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:57.061712 1175 raft_consensus.cc:2243] T 4b76c4408a7340e59b8506171622175f P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:57.062808 1175 raft_consensus.cc:2272] T 4b76c4408a7340e59b8506171622175f P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:57.070094 1176 ts_tablet_manager.cc:1950] T 6f93812578a940a3b7378015e2a46f95 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:51:57.071815 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 6f93812578a940a3b7378015e2a46f95 (table default.Table_3cca74f150f549b7aef4a213cafe99e5 [id=feb1f56ae44249c6adce65637017bb43]) successfully deleted
I20251029 02:51:57.072662 1176 tablet_replica.cc:333] T 4b76c4408a7340e59b8506171622175f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:51:57.073195 1176 raft_consensus.cc:2243] T 4b76c4408a7340e59b8506171622175f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:51:57.073753 1176 raft_consensus.cc:2272] T 4b76c4408a7340e59b8506171622175f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251029 02:51:57.079658 521 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet faa835de84ff45b19525d1d603678987 consensus state has no leader
I20251029 02:51:57.089839 1108 heartbeater.cc:344] Connected to a master server at 127.0.104.60:41233
I20251029 02:51:57.092981 525 catalog_manager.cc:5649] T aacb6f4b9284457bb6607e3a85d5c809 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } health_report { overall_health: UNKNOWN } } }
I20251029 02:51:57.106827 1568 tablet.cc:1717] T dd90fb9eb167478cbc3ec9363af86295 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Skipping requested alter to schema version 3, tablet already version 3
I20251029 02:51:57.108101 517 master_service.cc:954] ReplaceTablet: received request to replace tablet 7a1866afc7564217ad777e7ce53786c5 from {username='slave'} at 127.0.0.1:35314
I20251029 02:51:57.113121 520 master_service.cc:954] ReplaceTablet: received request to replace tablet e1fa4980f2624321b617b84c68b90113 from {username='slave'} at 127.0.0.1:35314
I20251029 02:51:57.117064 1166 tablet_bootstrap.cc:492] T 14698af6348f41e384f10569392d9b23 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:51:57.117902 1166 ts_tablet_manager.cc:1403] T 14698af6348f41e384f10569392d9b23 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.311s user 0.006s sys 0.014s
I20251029 02:51:57.119391 1166 raft_consensus.cc:359] T 14698af6348f41e384f10569392d9b23 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:57.120314 1166 raft_consensus.cc:385] T 14698af6348f41e384f10569392d9b23 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:57.120550 1166 raft_consensus.cc:740] T 14698af6348f41e384f10569392d9b23 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:51:57.120388 1570 tablet.cc:1717] T dd90fb9eb167478cbc3ec9363af86295 P b495a49349324f7d81908f2da700f0d9: Skipping requested alter to schema version 3, tablet already version 3
I20251029 02:51:57.119534 1176 ts_tablet_manager.cc:1916] T 4b76c4408a7340e59b8506171622175f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:57.123740 738 tablet_service.cc:1505] Processing CreateTablet for tablet 32519fbffcef44a890161e5f3043806c (DEFAULT_TABLE table=default.Table_a6a9f0fa8983438db6cece4c2111a3c9 [id=b95fe1c7d6ff4519b90f993f1fa961fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:57.123822 739 tablet_service.cc:1505] Processing CreateTablet for tablet 372d93914abc41b38a702742b7db0bc2 (DEFAULT_TABLE table=default.Table_bce83bae13a94cd5bf6733b53e6a1e9e [id=761ebdd0095c414488aa221cbc99d3bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:57.124717 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 372d93914abc41b38a702742b7db0bc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:57.124758 523 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_c7daf9eda79e470186f5774de2b824d3" } modify_external_catalogs: true
I20251029 02:51:57.124948 523 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_c7daf9eda79e470186f5774de2b824d3" } modify_external_catalogs: true
I20251029 02:51:57.125195 1166 consensus_queue.cc:260] T 14698af6348f41e384f10569392d9b23 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:57.125726 1166 ts_tablet_manager.cc:1434] T 14698af6348f41e384f10569392d9b23 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20251029 02:51:57.126039 1166 tablet_bootstrap.cc:492] T aa776cde2f494fa5810539edf9f289a1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:51:57.126693 518 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_1cf1a43474a9433fb65ef47f07029232" } new_table_name: "default.Table_663bb3557fe64d8092e0a19dfa14d1b4" modify_external_catalogs: true
I20251029 02:51:57.127353 756 tablet_service.cc:1505] Processing CreateTablet for tablet 27b08f88b99643d28ac609e45bf8104e (DEFAULT_TABLE table=default.Table_a6472497e2e940c3b6a79f15665aa636 [id=74a2a41c908146ec8e16c6a3227b94f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:57.127589 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32519fbffcef44a890161e5f3043806c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:57.127642 747 tablet_service.cc:1505] Processing CreateTablet for tablet 6789bc16664e4b37b86702449ab27855 (DEFAULT_TABLE table=default.Table_19ec7789551742df938b5c458a04326d [id=ab8aeaf90f6c4bc2ad5a98780945c40a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:57.127353 748 tablet_service.cc:1505] Processing CreateTablet for tablet 20a762344c5f46f88481c4ac7ad6e43b (DEFAULT_TABLE table=default.Table_3ca4a6512b174cb897b142dfe1259549 [id=a862192912a04942a8f682fc9825b014]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:57.128077 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6789bc16664e4b37b86702449ab27855. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:57.128314 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20a762344c5f46f88481c4ac7ad6e43b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:57.128535 1025 tablet_service.cc:1505] Processing CreateTablet for tablet 6789bc16664e4b37b86702449ab27855 (DEFAULT_TABLE table=default.Table_19ec7789551742df938b5c458a04326d [id=ab8aeaf90f6c4bc2ad5a98780945c40a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:57.128914 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6789bc16664e4b37b86702449ab27855. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:57.130813 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27b08f88b99643d28ac609e45bf8104e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:57.130867 1027 tablet_service.cc:1505] Processing CreateTablet for tablet 27b08f88b99643d28ac609e45bf8104e (DEFAULT_TABLE table=default.Table_a6472497e2e940c3b6a79f15665aa636 [id=74a2a41c908146ec8e16c6a3227b94f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:57.131219 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27b08f88b99643d28ac609e45bf8104e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:57.134934 524 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_3e8689e7e89543da830ba312d234ef3f" } modify_external_catalogs: true
I20251029 02:51:57.138890 524 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_3e8689e7e89543da830ba312d234ef3f" } modify_external_catalogs: true
I20251029 02:51:57.140069 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_30a991be88ad4c199ac8ce370ee76f17"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:57.140764 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_30a991be88ad4c199ac8ce370ee76f17 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:57.142908 1042 tablet_service.cc:1505] Processing CreateTablet for tablet 20a762344c5f46f88481c4ac7ad6e43b (DEFAULT_TABLE table=default.Table_3ca4a6512b174cb897b142dfe1259549 [id=a862192912a04942a8f682fc9825b014]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:57.143373 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20a762344c5f46f88481c4ac7ad6e43b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:57.146967 743 tablet_service.cc:1505] Processing CreateTablet for tablet 46cd75ff18b14212b556b9bcb3012dd9 (DEFAULT_TABLE table=default.Table_2ccb14f87f774da3b62823c798b31a7c [id=0829f9f13eea4c2081a9545b98e928bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:57.147414 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46cd75ff18b14212b556b9bcb3012dd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:57.149432 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_4619a2287cb14edbb32bcbec581eb4d9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:57.149952 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4619a2287cb14edbb32bcbec581eb4d9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:57.150080 1038 tablet_service.cc:1505] Processing CreateTablet for tablet 32519fbffcef44a890161e5f3043806c (DEFAULT_TABLE table=default.Table_a6a9f0fa8983438db6cece4c2111a3c9 [id=b95fe1c7d6ff4519b90f993f1fa961fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:57.150647 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32519fbffcef44a890161e5f3043806c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:57.151261 519 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 59d03a190b784ab18a3e63a572675891
I20251029 02:51:57.151893 1421 raft_consensus.cc:493] T c4884e22cf2044a4a1070c1b4b759d9e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:57.151996 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_c020d31b6d054efaad87057733e683b6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:57.152091 1421 raft_consensus.cc:515] T c4884e22cf2044a4a1070c1b4b759d9e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
W20251029 02:51:57.152451 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c020d31b6d054efaad87057733e683b6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:57.152735 1421 leader_election.cc:290] T c4884e22cf2044a4a1070c1b4b759d9e P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:51:57.152877 1421 raft_consensus.cc:493] T ab421dfaa3f84593842bb732a2693160 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:57.153002 1421 raft_consensus.cc:515] T ab421dfaa3f84593842bb732a2693160 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:57.153508 1421 leader_election.cc:290] T ab421dfaa3f84593842bb732a2693160 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:51:57.154188 526 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_cec9500b890447609a5feb7346bcf7a0" } modify_external_catalogs: true
I20251029 02:51:57.154356 526 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_cec9500b890447609a5feb7346bcf7a0" } modify_external_catalogs: true
I20251029 02:51:57.154623 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5695b5a7b1854c3983e2d4ab684898b0" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:51:57.154460 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 283) took 1498 ms. Trace:
I20251029 02:51:57.156622 738 tablet_service.cc:1552] Processing DeleteTablet for tablet 59d03a190b784ab18a3e63a572675891 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:57 UTC) from {username='slave'} at 127.0.0.1:44580
W20251029 02:51:57.157065 849 leader_election.cc:343] T c4884e22cf2044a4a1070c1b4b759d9e P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Not found: Tablet not found: c4884e22cf2044a4a1070c1b4b759d9e
I20251029 02:51:57.157790 917 raft_consensus.cc:3060] T 4b76c4408a7340e59b8506171622175f P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:51:57.159461 878 tablet_service.cc:1505] Processing CreateTablet for tablet 46cd75ff18b14212b556b9bcb3012dd9 (DEFAULT_TABLE table=default.Table_2ccb14f87f774da3b62823c798b31a7c [id=0829f9f13eea4c2081a9545b98e928bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:57.159953 1028 tablet_service.cc:1505] Processing CreateTablet for tablet 372d93914abc41b38a702742b7db0bc2 (DEFAULT_TABLE table=default.Table_bce83bae13a94cd5bf6733b53e6a1e9e [id=761ebdd0095c414488aa221cbc99d3bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:57.160305 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 372d93914abc41b38a702742b7db0bc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:57.160804 1585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5e5222b26fe4145948620e51edf53fb: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251029 02:51:57.161077 1175 ts_tablet_manager.cc:1916] T 4b76c4408a7340e59b8506171622175f P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:57.161439 1585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5e5222b26fe4145948620e51edf53fb: Bootstrap complete.
I20251029 02:51:57.163892 1027 tablet_service.cc:1552] Processing DeleteTablet for tablet 59d03a190b784ab18a3e63a572675891 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:57 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:57.165673 523 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 9be9add0274d41b688edd6292a253a06
W20251029 02:51:57.166500 847 leader_election.cc:343] T ab421dfaa3f84593842bb732a2693160 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: ab421dfaa3f84593842bb732a2693160
I20251029 02:51:57.166535 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_736690fbc69640d9be26e1662e3fa63e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:57.167047 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_736690fbc69640d9be26e1662e3fa63e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:51:57.167580 847 leader_election.cc:343] T c4884e22cf2044a4a1070c1b4b759d9e P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: c4884e22cf2044a4a1070c1b4b759d9e
I20251029 02:51:57.167694 847 leader_election.cc:304] T c4884e22cf2044a4a1070c1b4b759d9e P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9
I20251029 02:51:57.166283 1174 rpcz_store.cc:276] 1029 02:51:55.647859 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:55.648677 (+ 818us) service_pool.cc:225] Handling call
1029 02:51:57.016487 (+1367810us) ts_tablet_manager.cc:1151] Deleting tablet 5db519edc75d4bd0985b32322825ffae
1029 02:51:57.016533 (+ 46us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:57.145958 (+129425us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1363853}
I20251029 02:51:57.168921 520 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet e1fa4980f2624321b617b84c68b90113
I20251029 02:51:57.169341 520 catalog_manager.cc:6719] ReplaceTablet: tablet e1fa4980f2624321b617b84c68b90113 deleted and replaced by tablet e6e3e6a82b014d2b8de4e109b379461e
I20251029 02:51:57.169448 1027 tablet_service.cc:1552] Processing DeleteTablet for tablet 9be9add0274d41b688edd6292a253a06 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:57 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:57.169667 520 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_8a5ae35851664e0f85324ec1ca672031" } modify_external_catalogs: true
I20251029 02:51:57.169816 520 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_8a5ae35851664e0f85324ec1ca672031" } modify_external_catalogs: true
I20251029 02:51:57.170218 1027 tablet_service.cc:1552] Processing DeleteTablet for tablet e1fa4980f2624321b617b84c68b90113 with delete_type TABLET_DATA_DELETED (replaced by tablet e6e3e6a82b014d2b8de4e109b379461e) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:57.170969 738 tablet_service.cc:1552] Processing DeleteTablet for tablet 9be9add0274d41b688edd6292a253a06 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:57 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:57.171506 739 tablet_service.cc:1552] Processing DeleteTablet for tablet e1fa4980f2624321b617b84c68b90113 with delete_type TABLET_DATA_DELETED (replaced by tablet e6e3e6a82b014d2b8de4e109b379461e) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:57.172279 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 288) took 1412 ms. Trace:
I20251029 02:51:57.172392 1174 rpcz_store.cc:276] 1029 02:51:55.756365 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:55.756777 (+ 412us) service_pool.cc:225] Handling call
1029 02:51:57.168305 (+1411528us) ts_tablet_manager.cc:1151] Deleting tablet 2e97d1bc06a24e8d8ba11acb6fd280d8
1029 02:51:57.168313 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:57.168445 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1411290}
I20251029 02:51:57.172909 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 289) took 1349 ms. Trace:
I20251029 02:51:57.173007 1174 rpcz_store.cc:276] 1029 02:51:55.822982 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:55.823227 (+ 245us) service_pool.cc:225] Handling call
1029 02:51:57.172642 (+1349415us) ts_tablet_manager.cc:1151] Deleting tablet d61beda662d844cdaf67fd9a2e382ea2
1029 02:51:57.172652 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:57.172747 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1334177}
W20251029 02:51:57.173550 849 leader_election.cc:343] T ab421dfaa3f84593842bb732a2693160 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Not found: Tablet not found: ab421dfaa3f84593842bb732a2693160
I20251029 02:51:57.173745 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 290) took 1313 ms. Trace:
I20251029 02:51:57.173830 1174 rpcz_store.cc:276] 1029 02:51:55.859471 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:55.859539 (+ 68us) service_pool.cc:225] Handling call
1029 02:51:57.173217 (+1313678us) ts_tablet_manager.cc:1151] Deleting tablet 012ce243fd07466ab5301d1c9ce4b413
1029 02:51:57.173228 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:57.173311 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1310038}
I20251029 02:51:57.174104 1041 tablet_service.cc:1505] Processing CreateTablet for tablet 46cd75ff18b14212b556b9bcb3012dd9 (DEFAULT_TABLE table=default.Table_2ccb14f87f774da3b62823c798b31a7c [id=0829f9f13eea4c2081a9545b98e928bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:57.174527 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 293) took 1313 ms. Trace:
I20251029 02:51:57.174638 1174 rpcz_store.cc:276] 1029 02:51:55.861158 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:55.862042 (+ 884us) service_pool.cc:225] Handling call
1029 02:51:57.174053 (+1312011us) ts_tablet_manager.cc:1151] Deleting tablet cb55b3d5bc63405f96cdc122f5a612cb
1029 02:51:57.174062 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:57.174128 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1310728}
I20251029 02:51:57.175092 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 291) took 1314 ms. Trace:
I20251029 02:51:57.174276 1585 raft_consensus.cc:359] T 00000000000000000000000000000000 P e5e5222b26fe4145948620e51edf53fb [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9672114a6a742e2864c16ff60ba246e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42343 } } peers { permanent_uuid: "e9261eb85ea443d5a3beb36bd0c549f0" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34239 } } peers { permanent_uuid: "e5e5222b26fe4145948620e51edf53fb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41233 } }
I20251029 02:51:57.175207 1174 rpcz_store.cc:276] 1029 02:51:55.860180 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:55.860274 (+ 94us) service_pool.cc:225] Handling call
1029 02:51:57.174815 (+1314541us) ts_tablet_manager.cc:1151] Deleting tablet b6d4f7bf702f4834bc5f4ef2b50b92a7
1029 02:51:57.174823 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:57.174895 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1311090}
I20251029 02:51:57.175292 1585 raft_consensus.cc:740] T 00000000000000000000000000000000 P e5e5222b26fe4145948620e51edf53fb [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5e5222b26fe4145948620e51edf53fb, State: Initialized, Role: FOLLOWER
I20251029 02:51:57.176052 1585 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5e5222b26fe4145948620e51edf53fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9672114a6a742e2864c16ff60ba246e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42343 } } peers { permanent_uuid: "e9261eb85ea443d5a3beb36bd0c549f0" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34239 } } peers { permanent_uuid: "e5e5222b26fe4145948620e51edf53fb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41233 } }
W20251029 02:51:57.176663 518 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f4729ecd182042c2a10f67ca25224101 has no consensus state
I20251029 02:51:57.177170 518 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_875a426b444646729b1097ec39577eee" } new_table_name: "default.Table_ab94a2a8b29d498a9f201f39228d7bfe" modify_external_catalogs: true
I20251029 02:51:57.177369 896 tablet_service.cc:1505] Processing CreateTablet for tablet 6789bc16664e4b37b86702449ab27855 (DEFAULT_TABLE table=default.Table_19ec7789551742df938b5c458a04326d [id=ab8aeaf90f6c4bc2ad5a98780945c40a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:57.177839 893 tablet_service.cc:1552] Processing DeleteTablet for tablet 9be9add0274d41b688edd6292a253a06 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:57 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:57.178059 882 tablet_service.cc:1552] Processing DeleteTablet for tablet e1fa4980f2624321b617b84c68b90113 with delete_type TABLET_DATA_DELETED (replaced by tablet e6e3e6a82b014d2b8de4e109b379461e) from {username='slave'} at 127.0.0.1:35702
W20251029 02:51:57.178141 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 5db519edc75d4bd0985b32322825ffae because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5db519edc75d4bd0985b32322825ffae already in progress: creating tablet
I20251029 02:51:57.178491 890 tablet_service.cc:1505] Processing CreateTablet for tablet 27b08f88b99643d28ac609e45bf8104e (DEFAULT_TABLE table=default.Table_a6472497e2e940c3b6a79f15665aa636 [id=74a2a41c908146ec8e16c6a3227b94f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:57.178812 916 raft_consensus.cc:3060] T 24016ceaa84147eca0b94b3ad288e772 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:51:57.178900 881 tablet_service.cc:1505] Processing CreateTablet for tablet 372d93914abc41b38a702742b7db0bc2 (DEFAULT_TABLE table=default.Table_bce83bae13a94cd5bf6733b53e6a1e9e [id=761ebdd0095c414488aa221cbc99d3bc]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 02:51:57.178457 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 2e97d1bc06a24e8d8ba11acb6fd280d8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2e97d1bc06a24e8d8ba11acb6fd280d8 already in progress: creating tablet
W20251029 02:51:57.179404 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet d61beda662d844cdaf67fd9a2e382ea2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d61beda662d844cdaf67fd9a2e382ea2 already in progress: creating tablet
W20251029 02:51:57.179620 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 012ce243fd07466ab5301d1c9ce4b413 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 012ce243fd07466ab5301d1c9ce4b413 already in progress: creating tablet
I20251029 02:51:57.179667 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46cd75ff18b14212b556b9bcb3012dd9. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:51:57.179833 512 catalog_manager.cc:4952] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet cb55b3d5bc63405f96cdc122f5a612cb because the tablet was not found. No further retry: Not found: Tablet not found: cb55b3d5bc63405f96cdc122f5a612cb
W20251029 02:51:57.180109 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet b6d4f7bf702f4834bc5f4ef2b50b92a7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b6d4f7bf702f4834bc5f4ef2b50b92a7 already in progress: creating tablet
I20251029 02:51:57.184887 524 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet ec953c5b2a844aa99ef47bbcd7216c7c
I20251029 02:51:57.184991 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 292) took 1314 ms. Trace:
I20251029 02:51:57.185237 1421 raft_consensus.cc:2749] T c4884e22cf2044a4a1070c1b4b759d9e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:51:57.187647 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46cd75ff18b14212b556b9bcb3012dd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:57.185214 1174 rpcz_store.cc:276] 1029 02:51:55.861066 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:55.862019 (+ 953us) service_pool.cc:225] Handling call
1029 02:51:57.175467 (+1313448us) ts_tablet_manager.cc:1151] Deleting tablet c764062d3ba04d4ba8b2e367ad8ed433
1029 02:51:57.175478 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:57.175558 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1311226}
I20251029 02:51:57.191921 898 tablet_service.cc:1505] Processing CreateTablet for tablet 20a762344c5f46f88481c4ac7ad6e43b (DEFAULT_TABLE table=default.Table_3ca4a6512b174cb897b142dfe1259549 [id=a862192912a04942a8f682fc9825b014]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:57.192957 849 leader_election.cc:304] T ab421dfaa3f84593842bb732a2693160 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9
I20251029 02:51:57.193027 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d9722c764ac4b27a540595e9c6aafcd" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:51:57.193326 1174 tablet_replica.cc:333] T bc1a1cf274b64b52aba58811e9594330 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:51:57.193538 1174 raft_consensus.cc:2243] T bc1a1cf274b64b52aba58811e9594330 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:57.193727 1174 raft_consensus.cc:2272] T bc1a1cf274b64b52aba58811e9594330 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:57.193786 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27b08f88b99643d28ac609e45bf8104e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:57.194088 521 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet 24016ceaa84147eca0b94b3ad288e772 (table default.Table_f06003da6e324504ade193bbbb11d36a [id=bd279b3ce623477ca20e8d35e3b7c0df]). Expected version 1 got 0
I20251029 02:51:57.194285 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6789bc16664e4b37b86702449ab27855. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:51:57.194810 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet c764062d3ba04d4ba8b2e367ad8ed433 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c764062d3ba04d4ba8b2e367ad8ed433 already in progress: creating tablet
I20251029 02:51:57.198773 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20a762344c5f46f88481c4ac7ad6e43b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:57.199456 1421 raft_consensus.cc:2749] T ab421dfaa3f84593842bb732a2693160 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 02:51:57.199458 990 leader_election.cc:343] T 8d9722c764ac4b27a540595e9c6aafcd P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:57.199640 990 leader_election.cc:304] T 8d9722c764ac4b27a540595e9c6aafcd P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5; no voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d
I20251029 02:51:57.199709 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 372d93914abc41b38a702742b7db0bc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:57.199801 517 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 7a1866afc7564217ad777e7ce53786c5
I20251029 02:51:57.200332 1174 ts_tablet_manager.cc:1916] T bc1a1cf274b64b52aba58811e9594330 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:57.200503 517 catalog_manager.cc:6719] ReplaceTablet: tablet 7a1866afc7564217ad777e7ce53786c5 deleted and replaced by tablet dc33fd3404ec41c3935b02c84a12971b
I20251029 02:51:57.201248 1558 raft_consensus.cc:2749] T 8d9722c764ac4b27a540595e9c6aafcd P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:51:57.203274 920 raft_consensus.cc:3060] T fa5127bcfcda40a7b5ad4ef016af072a P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:51:57.203567 739 tablet_service.cc:1552] Processing DeleteTablet for tablet 7a1866afc7564217ad777e7ce53786c5 with delete_type TABLET_DATA_DELETED (replaced by tablet dc33fd3404ec41c3935b02c84a12971b) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:57.204218 1027 tablet_service.cc:1552] Processing DeleteTablet for tablet 14698af6348f41e384f10569392d9b23 with delete_type TABLET_DATA_DELETED (replaced by tablet 7309d3190cfa415281c5253a634fa7f3) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:57.204454 1025 tablet_service.cc:1552] Processing DeleteTablet for tablet 7a1866afc7564217ad777e7ce53786c5 with delete_type TABLET_DATA_DELETED (replaced by tablet dc33fd3404ec41c3935b02c84a12971b) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:57.208921 1172 tablet_bootstrap.cc:492] T 3cded156380f45cc95893825c71ba0e9 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:51:57.211236 519 master_service.cc:954] ReplaceTablet: received request to replace tablet 13ee62fc2339410b9db70dfd3d3c6fa6 from {username='slave'} at 127.0.0.1:35314
I20251029 02:51:57.213553 879 tablet_service.cc:1552] Processing DeleteTablet for tablet 59d03a190b784ab18a3e63a572675891 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:57 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:57.209112 1172 ts_tablet_manager.cc:1403] T 3cded156380f45cc95893825c71ba0e9 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.424s user 0.008s sys 0.034s
I20251029 02:51:57.214627 882 tablet_service.cc:1552] Processing DeleteTablet for tablet 7a1866afc7564217ad777e7ce53786c5 with delete_type TABLET_DATA_DELETED (replaced by tablet dc33fd3404ec41c3935b02c84a12971b) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:57.218706 1167 tablet_bootstrap.cc:654] T 59af906061d049c99ae2153945597007 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:57.218806 526 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 69597d48631d43b6a5b92abbf60d5ed3
I20251029 02:51:57.219585 1172 raft_consensus.cc:359] T 3cded156380f45cc95893825c71ba0e9 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:57.219857 1172 raft_consensus.cc:385] T 3cded156380f45cc95893825c71ba0e9 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:57.219964 1172 raft_consensus.cc:740] T 3cded156380f45cc95893825c71ba0e9 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:51:57.220795 739 tablet_service.cc:1552] Processing DeleteTablet for tablet 69597d48631d43b6a5b92abbf60d5ed3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:57 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:57.221195 520 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet e6e3e6a82b014d2b8de4e109b379461e
I20251029 02:51:57.221802 1025 tablet_service.cc:1552] Processing DeleteTablet for tablet 69597d48631d43b6a5b92abbf60d5ed3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:57 UTC) from {username='slave'} at 127.0.0.1:37626
W20251029 02:51:57.225039 518 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 23b9b2ea814645eba63e5f6b12cfb966 has no consensus state
I20251029 02:51:57.225718 882 tablet_service.cc:1552] Processing DeleteTablet for tablet 69597d48631d43b6a5b92abbf60d5ed3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:57 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:57.239516 1166 tablet_bootstrap.cc:654] T aa776cde2f494fa5810539edf9f289a1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:57.242480 883 tablet_service.cc:1505] Processing CreateTablet for tablet 32519fbffcef44a890161e5f3043806c (DEFAULT_TABLE table=default.Table_a6a9f0fa8983438db6cece4c2111a3c9 [id=b95fe1c7d6ff4519b90f993f1fa961fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:57.255112 519 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 13ee62fc2339410b9db70dfd3d3c6fa6
I20251029 02:51:57.255395 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32519fbffcef44a890161e5f3043806c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:57.255745 519 catalog_manager.cc:6719] ReplaceTablet: tablet 13ee62fc2339410b9db70dfd3d3c6fa6 deleted and replaced by tablet 35d9b8dfd61e4d8db441427a49fb9dca
I20251029 02:51:57.256984 739 tablet_service.cc:1552] Processing DeleteTablet for tablet 13ee62fc2339410b9db70dfd3d3c6fa6 with delete_type TABLET_DATA_DELETED (replaced by tablet 35d9b8dfd61e4d8db441427a49fb9dca) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:57.261214 739 tablet_service.cc:1505] Processing CreateTablet for tablet 6142018e44af4d32818adefa2ee43d02 (DEFAULT_TABLE table=default.Table_e0854491deb34525bce9198a673fb943 [id=b3e01f7d5a5f4675a998eaad985347ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:57.261765 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6142018e44af4d32818adefa2ee43d02. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:57.263324 1025 tablet_service.cc:1505] Processing CreateTablet for tablet ae4231daa2594258aa51419c97a2f50a (DEFAULT_TABLE table=default.Table_b15248a339e14b85944e649464498b84 [id=f4e028bdd23f4eea975f940d0b2c73f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:57.263759 1042 tablet_service.cc:1505] Processing CreateTablet for tablet 6142018e44af4d32818adefa2ee43d02 (DEFAULT_TABLE table=default.Table_e0854491deb34525bce9198a673fb943 [id=b3e01f7d5a5f4675a998eaad985347ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:57.263859 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae4231daa2594258aa51419c97a2f50a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:57.264165 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6142018e44af4d32818adefa2ee43d02. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:57.268064 1027 tablet_service.cc:1505] Processing CreateTablet for tablet 72f9d4408f384e7093316d85f092e1b2 (DEFAULT_TABLE table=default.Table_a6c1519740ac4bbcbfd00671719b07a0 [id=e5c07dcc74d1493fa9e1ef153b10d87b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:57.268730 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72f9d4408f384e7093316d85f092e1b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:57.269374 519 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_a6c1519740ac4bbcbfd00671719b07a0" } new_table_name: "default.Table_6d866cb3e90a40dcb068990637194ac5" modify_external_catalogs: true
I20251029 02:51:57.269651 518 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_c90ddb84f3e044ba96f01a51fd463d6e" } new_table_name: "default.Table_943114eb062441f6bdbf5cdd16c62101" modify_external_catalogs: true
I20251029 02:51:57.274850 738 tablet_service.cc:1505] Processing CreateTablet for tablet ae4231daa2594258aa51419c97a2f50a (DEFAULT_TABLE table=default.Table_b15248a339e14b85944e649464498b84 [id=f4e028bdd23f4eea975f940d0b2c73f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:57.275298 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae4231daa2594258aa51419c97a2f50a. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:51:57.276376 518 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 02e4588b03144ea2850e13a1f8f29f06 consensus state has no leader
I20251029 02:51:57.279448 893 tablet_service.cc:1505] Processing CreateTablet for tablet 72f9d4408f384e7093316d85f092e1b2 (DEFAULT_TABLE table=default.Table_a6c1519740ac4bbcbfd00671719b07a0 [id=e5c07dcc74d1493fa9e1ef153b10d87b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:57.279623 879 tablet_service.cc:1505] Processing CreateTablet for tablet ae4231daa2594258aa51419c97a2f50a (DEFAULT_TABLE table=default.Table_b15248a339e14b85944e649464498b84 [id=f4e028bdd23f4eea975f940d0b2c73f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:57.279901 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72f9d4408f384e7093316d85f092e1b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:57.280126 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae4231daa2594258aa51419c97a2f50a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:57.280787 882 tablet_service.cc:1505] Processing CreateTablet for tablet 6142018e44af4d32818adefa2ee43d02 (DEFAULT_TABLE table=default.Table_e0854491deb34525bce9198a673fb943 [id=b3e01f7d5a5f4675a998eaad985347ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:57.281208 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6142018e44af4d32818adefa2ee43d02. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:57.287736 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_a4fcd46aef6146e986d4b8d3599d66f4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:57.287936 526 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_3250d60b958a4c37bf654e8b9c5c25c1" } new_table_name: "default.Table_5a915593503d490fa12b10fccd76ac3e" modify_external_catalogs: true
W20251029 02:51:57.288403 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a4fcd46aef6146e986d4b8d3599d66f4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:57.291237 1172 consensus_queue.cc:260] T 3cded156380f45cc95893825c71ba0e9 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:57.291807 1172 ts_tablet_manager.cc:1434] T 3cded156380f45cc95893825c71ba0e9 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.077s user 0.000s sys 0.001s
I20251029 02:51:57.292160 1172 tablet_bootstrap.cc:492] T 6feec2c8e030447fae9c292a626dcd1d P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:51:57.294217 822 heartbeater.cc:344] Connected to a master server at 127.0.104.60:41233
I20251029 02:51:57.295531 1038 tablet_service.cc:1552] Processing DeleteTablet for tablet 13ee62fc2339410b9db70dfd3d3c6fa6 with delete_type TABLET_DATA_DELETED (replaced by tablet 35d9b8dfd61e4d8db441427a49fb9dca) from {username='slave'} at 127.0.0.1:37626
W20251029 02:51:57.297308 526 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d31e548feae24b739087c1e211d514f9 consensus state has no leader
I20251029 02:51:57.299898 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_c5b59d221b994982a9867326266b7bcd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:57.300472 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c5b59d221b994982a9867326266b7bcd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:57.300560 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_9c1124b16dc44ec5b2696d567f1c52f0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:57.301003 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9c1124b16dc44ec5b2696d567f1c52f0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:57.301682 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_fbf0f51798e54f2fbb2c3e333c586319"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:57.302127 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fbf0f51798e54f2fbb2c3e333c586319 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:57.306553 747 tablet_service.cc:1552] Processing DeleteTablet for tablet 3cded156380f45cc95893825c71ba0e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:53 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:57.311021 1166 tablet_bootstrap.cc:492] T aa776cde2f494fa5810539edf9f289a1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:51:57.311410 1166 ts_tablet_manager.cc:1403] T aa776cde2f494fa5810539edf9f289a1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.185s user 0.003s sys 0.001s
I20251029 02:51:57.319268 890 tablet_service.cc:1552] Processing DeleteTablet for tablet 13ee62fc2339410b9db70dfd3d3c6fa6 with delete_type TABLET_DATA_DELETED (replaced by tablet 35d9b8dfd61e4d8db441427a49fb9dca) from {username='slave'} at 127.0.0.1:35702
W20251029 02:51:57.320897 519 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 72f9d4408f384e7093316d85f092e1b2 consensus state has no leader
I20251029 02:51:57.322556 1174 ts_tablet_manager.cc:1929] T bc1a1cf274b64b52aba58811e9594330 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:57.322554 747 tablet_service.cc:1505] Processing CreateTablet for tablet 72f9d4408f384e7093316d85f092e1b2 (DEFAULT_TABLE table=default.Table_a6c1519740ac4bbcbfd00671719b07a0 [id=e5c07dcc74d1493fa9e1ef153b10d87b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:57.322752 1174 log.cc:1199] T bc1a1cf274b64b52aba58811e9594330 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/bc1a1cf274b64b52aba58811e9594330
I20251029 02:51:57.323177 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72f9d4408f384e7093316d85f092e1b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:57.323177 1174 ts_tablet_manager.cc:1950] T bc1a1cf274b64b52aba58811e9594330 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:51:57.323488 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 295) took 1348 ms. Trace:
I20251029 02:51:57.323484 1025 tablet_service.cc:1505] Processing CreateTablet for tablet 70b008011d5547bbaf3f45685394139f (DEFAULT_TABLE table=default.Table_0bb20faeb14644ca8e2d5acd4af09f85 [id=d39d7d159c7d46b496caa09226366e1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:57.323593 1174 rpcz_store.cc:276] 1029 02:51:55.974906 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:55.974961 (+ 55us) service_pool.cc:225] Handling call
1029 02:51:57.192046 (+1217085us) ts_tablet_manager.cc:1151] Deleting tablet bc1a1cf274b64b52aba58811e9594330
1029 02:51:57.192058 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:57.250569 (+ 58511us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:57.322386 (+ 71817us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:57.323452 (+ 1066us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1197430}
I20251029 02:51:57.323886 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70b008011d5547bbaf3f45685394139f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:57.324029 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 304) took 1216 ms. Trace:
I20251029 02:51:57.324139 1174 rpcz_store.cc:276] 1029 02:51:56.107363 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:56.107424 (+ 61us) service_pool.cc:225] Handling call
1029 02:51:57.323887 (+1216463us) ts_tablet_manager.cc:1151] Deleting tablet 6ca9bd37de9e4be388501b15af4bda31
1029 02:51:57.323897 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:57.323994 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1216255}
I20251029 02:51:57.324461 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 306) took 1207 ms. Trace:
I20251029 02:51:57.324537 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet bc1a1cf274b64b52aba58811e9594330 (table default.Table_98034c2933664cefab94c26cb8349dc4 [id=efb8f381fea74cf48842af6b1a4d5a81]) successfully deleted
I20251029 02:51:57.324564 1174 rpcz_store.cc:276] 1029 02:51:56.116801 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:56.116910 (+ 109us) service_pool.cc:225] Handling call
1029 02:51:57.324348 (+1207438us) ts_tablet_manager.cc:1151] Deleting tablet f246f6c202cf44d48c9f5e4f335202c5
1029 02:51:57.324356 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:57.324431 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1207216}
I20251029 02:51:57.324819 1174 tablet_replica.cc:333] T 11f4252b5e694312aaa142468f8e68b4 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:51:57.325033 1174 raft_consensus.cc:2243] T 11f4252b5e694312aaa142468f8e68b4 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus shutting down.
W20251029 02:51:57.325075 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 6ca9bd37de9e4be388501b15af4bda31 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6ca9bd37de9e4be388501b15af4bda31 already in progress: creating tablet
I20251029 02:51:57.325256 1174 raft_consensus.cc:2272] T 11f4252b5e694312aaa142468f8e68b4 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
W20251029 02:51:57.325302 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet f246f6c202cf44d48c9f5e4f335202c5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f246f6c202cf44d48c9f5e4f335202c5 already in progress: creating tablet
I20251029 02:51:57.325843 1174 ts_tablet_manager.cc:1916] T 11f4252b5e694312aaa142468f8e68b4 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:57.333196 756 tablet_service.cc:1505] Processing CreateTablet for tablet eca016aed05e4ce09e1a2bf96e2286f5 (DEFAULT_TABLE table=default.Table_ee6dec2fc7834c78914ddc9f13890c55 [id=2faf0479a2a6477fa1d70d30094ef1db]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:57.333582 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eca016aed05e4ce09e1a2bf96e2286f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:57.337735 1172 tablet_bootstrap.cc:654] T 6feec2c8e030447fae9c292a626dcd1d P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:57.341112 1030 tablet_service.cc:1505] Processing CreateTablet for tablet d31e548feae24b739087c1e211d514f9 (DEFAULT_TABLE table=default.Table_5a915593503d490fa12b10fccd76ac3e [id=6fd26d78f4904f30a2cb76e9a4bec2db]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:57.342659 1038 tablet_service.cc:1505] Processing CreateTablet for tablet 6dff036b32904ddb9598be4b79504bb6 (DEFAULT_TABLE table=default.Table_b276bd83a0514185a89941ada499686c [id=65ec61bf400a4628b98e64e7d43c025c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:57.345278 748 tablet_service.cc:1505] Processing CreateTablet for tablet d31e548feae24b739087c1e211d514f9 (DEFAULT_TABLE table=default.Table_5a915593503d490fa12b10fccd76ac3e [id=6fd26d78f4904f30a2cb76e9a4bec2db]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:57.348726 739 tablet_service.cc:1505] Processing CreateTablet for tablet 23b9b2ea814645eba63e5f6b12cfb966 (DEFAULT_TABLE table=default.Table_ab94a2a8b29d498a9f201f39228d7bfe [id=4876e3df916248d2a64ba3cad736b5ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:57.349659 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d31e548feae24b739087c1e211d514f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:57.349619 738 tablet_service.cc:1505] Processing CreateTablet for tablet 6dff036b32904ddb9598be4b79504bb6 (DEFAULT_TABLE table=default.Table_b276bd83a0514185a89941ada499686c [id=65ec61bf400a4628b98e64e7d43c025c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:57.350024 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6dff036b32904ddb9598be4b79504bb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:57.350335 1029 tablet_service.cc:1505] Processing CreateTablet for tablet d29cc2a0c8b84e3ebadc7f571d666847 (DEFAULT_TABLE table=default.Table_8fee569eca38413dbac56707725c6a46 [id=50de770b3b9b4d8c85546014a959da24]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:57.350932 743 tablet_service.cc:1505] Processing CreateTablet for tablet 70b008011d5547bbaf3f45685394139f (DEFAULT_TABLE table=default.Table_0bb20faeb14644ca8e2d5acd4af09f85 [id=d39d7d159c7d46b496caa09226366e1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:57.351285 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70b008011d5547bbaf3f45685394139f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:57.352836 753 tablet_service.cc:1505] Processing CreateTablet for tablet 748777c6cd6e4f49ab49912b719e48bd (DEFAULT_TABLE table=default.Table_6ed43c1a2617493882d7cb6c568d21b7 [id=a2f3b36dc78a4e94bd054a6a5712b384]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:57.353178 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 748777c6cd6e4f49ab49912b719e48bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:57.354301 746 tablet_service.cc:1505] Processing CreateTablet for tablet d29cc2a0c8b84e3ebadc7f571d666847 (DEFAULT_TABLE table=default.Table_8fee569eca38413dbac56707725c6a46 [id=50de770b3b9b4d8c85546014a959da24]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:57.354691 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d29cc2a0c8b84e3ebadc7f571d666847. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:57.355945 755 tablet_service.cc:1505] Processing CreateTablet for tablet ad214c870ee649279ed7321c5b69cbee (DEFAULT_TABLE table=default.Table_982d01dccabe47bfaac7a582282c5d54 [id=033c1db6a2b44a2e99361183fd18c8f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:57.356745 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad214c870ee649279ed7321c5b69cbee. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:57.358619 1039 tablet_service.cc:1505] Processing CreateTablet for tablet cd597c319af645909b8f3d1212fcb702 (DEFAULT_TABLE table=default.Table_d9cfb3211dcd4bb295955b8750600b5d [id=e4c9177afbae4d00bc98fe741494dbdd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:57.360268 886 tablet_service.cc:1505] Processing CreateTablet for tablet 6f8cf09f23984b879b60ab6b27c7d52e (DEFAULT_TABLE table=default.Table_04fadd7c87d1460dbc72c2caedbef915 [id=98f28a15eced4860ac13a60abdc8e0e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:57.360980 1036 tablet_service.cc:1505] Processing CreateTablet for tablet defb795026a44e12afb2defb9c9c6db2 (DEFAULT_TABLE table=default.Table_eae5a69444004b54bf8bf18ea29fe73f [id=499cbddf4ef64aa28594d85252774611]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:57.361217 888 tablet_service.cc:1505] Processing CreateTablet for tablet ad214c870ee649279ed7321c5b69cbee (DEFAULT_TABLE table=default.Table_982d01dccabe47bfaac7a582282c5d54 [id=033c1db6a2b44a2e99361183fd18c8f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:57.361963 892 tablet_service.cc:1505] Processing CreateTablet for tablet 589a521f5cc24e6289c37b9cf256bce5 (DEFAULT_TABLE table=default.Table_b8095fca27a447f2a9ac0475373352c6 [id=56b57a7a90624bee92fa43661728c0f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:57.362684 1031 tablet_service.cc:1505] Processing CreateTablet for tablet 84993b2f63e7421290a161dde4bb0607 (DEFAULT_TABLE table=default.Table_93b405a4f257481289518c5ff28832a0 [id=da230f99a9894ab8aa2fdb4b167169ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:57.364198 1421 raft_consensus.cc:493] T 7c041f4909274cd5a03b9fd78353faf9 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:57.364372 1421 raft_consensus.cc:515] T 7c041f4909274cd5a03b9fd78353faf9 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:57.364742 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_163ca13775fd4ae28ec7ead8297929c8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:57.365343 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_163ca13775fd4ae28ec7ead8297929c8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:57.366238 879 tablet_service.cc:1505] Processing CreateTablet for tablet 6dff036b32904ddb9598be4b79504bb6 (DEFAULT_TABLE table=default.Table_b276bd83a0514185a89941ada499686c [id=65ec61bf400a4628b98e64e7d43c025c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:57.366289 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_5069071caedd4971a57a438fba4fac4f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:57.366773 900 tablet_service.cc:1505] Processing CreateTablet for tablet d29cc2a0c8b84e3ebadc7f571d666847 (DEFAULT_TABLE table=default.Table_8fee569eca38413dbac56707725c6a46 [id=50de770b3b9b4d8c85546014a959da24]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 02:51:57.366871 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5069071caedd4971a57a438fba4fac4f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:51:57.366247 849 leader_election.cc:343] T 7c041f4909274cd5a03b9fd78353faf9 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Not found: Tablet not found: 7c041f4909274cd5a03b9fd78353faf9
I20251029 02:51:57.367671 896 tablet_service.cc:1505] Processing CreateTablet for tablet 23b9b2ea814645eba63e5f6b12cfb966 (DEFAULT_TABLE table=default.Table_ab94a2a8b29d498a9f201f39228d7bfe [id=4876e3df916248d2a64ba3cad736b5ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:57.371026 520 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_fbf0f51798e54f2fbb2c3e333c586319" } modify_external_catalogs: true
I20251029 02:51:57.371224 520 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_fbf0f51798e54f2fbb2c3e333c586319" } modify_external_catalogs: true
I20251029 02:51:57.371693 526 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_b8095fca27a447f2a9ac0475373352c6" } modify_external_catalogs: true
I20251029 02:51:57.371863 526 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_b8095fca27a447f2a9ac0475373352c6" } modify_external_catalogs: true
I20251029 02:51:57.378244 891 tablet_service.cc:1505] Processing CreateTablet for tablet 84993b2f63e7421290a161dde4bb0607 (DEFAULT_TABLE table=default.Table_93b405a4f257481289518c5ff28832a0 [id=da230f99a9894ab8aa2fdb4b167169ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:57.379782 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f8cf09f23984b879b60ab6b27c7d52e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:57.380040 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6dff036b32904ddb9598be4b79504bb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:57.380187 894 tablet_service.cc:1505] Processing CreateTablet for tablet cd597c319af645909b8f3d1212fcb702 (DEFAULT_TABLE table=default.Table_d9cfb3211dcd4bb295955b8750600b5d [id=e4c9177afbae4d00bc98fe741494dbdd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:57.380573 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd597c319af645909b8f3d1212fcb702. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:57.381630 884 tablet_service.cc:1505] Processing CreateTablet for tablet 748777c6cd6e4f49ab49912b719e48bd (DEFAULT_TABLE table=default.Table_6ed43c1a2617493882d7cb6c568d21b7 [id=a2f3b36dc78a4e94bd054a6a5712b384]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:57.381848 885 tablet_service.cc:1505] Processing CreateTablet for tablet d31e548feae24b739087c1e211d514f9 (DEFAULT_TABLE table=default.Table_5a915593503d490fa12b10fccd76ac3e [id=6fd26d78f4904f30a2cb76e9a4bec2db]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:57.381943 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84993b2f63e7421290a161dde4bb0607. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:57.382321 890 tablet_service.cc:1505] Processing CreateTablet for tablet eca016aed05e4ce09e1a2bf96e2286f5 (DEFAULT_TABLE table=default.Table_ee6dec2fc7834c78914ddc9f13890c55 [id=2faf0479a2a6477fa1d70d30094ef1db]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:57.382683 518 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_1281926bf1224c0cb4541bf561102021" } new_table_name: "default.Table_21b8ae2debb241bda65fe387a4a3190f" modify_external_catalogs: true
I20251029 02:51:57.383234 1421 leader_election.cc:290] T 7c041f4909274cd5a03b9fd78353faf9 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:51:57.383301 893 tablet_service.cc:1505] Processing CreateTablet for tablet defb795026a44e12afb2defb9c9c6db2 (DEFAULT_TABLE table=default.Table_eae5a69444004b54bf8bf18ea29fe73f [id=499cbddf4ef64aa28594d85252774611]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:57.390664 520 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet edcdd897db0941c3b87c2d0c35dd171b
I20251029 02:51:57.392036 526 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 589a521f5cc24e6289c37b9cf256bce5
I20251029 02:51:57.393211 519 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_d9e164af67334c98a5d0821f2d80a35e" } modify_external_catalogs: true
I20251029 02:51:57.393456 519 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_d9e164af67334c98a5d0821f2d80a35e" } modify_external_catalogs: true
I20251029 02:51:57.395231 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_e0a153971f6f4af98ee02f8bbc615620"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:57.379191 881 tablet_service.cc:1505] Processing CreateTablet for tablet 70b008011d5547bbaf3f45685394139f (DEFAULT_TABLE table=default.Table_0bb20faeb14644ca8e2d5acd4af09f85 [id=d39d7d159c7d46b496caa09226366e1f]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 02:51:57.395908 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e0a153971f6f4af98ee02f8bbc615620 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:57.401748 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d29cc2a0c8b84e3ebadc7f571d666847. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:57.405311 895 tablet_service.cc:1552] Processing DeleteTablet for tablet 589a521f5cc24e6289c37b9cf256bce5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:57 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:57.408703 1027 tablet_service.cc:1505] Processing CreateTablet for tablet 23b9b2ea814645eba63e5f6b12cfb966 (DEFAULT_TABLE table=default.Table_ab94a2a8b29d498a9f201f39228d7bfe [id=4876e3df916248d2a64ba3cad736b5ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:57.408799 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_5876d3fefb8144e2a1a76782912ab03b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:57.409166 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d31e548feae24b739087c1e211d514f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:57.409425 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84993b2f63e7421290a161dde4bb0607. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:51:57.409519 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5876d3fefb8144e2a1a76782912ab03b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:57.409601 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23b9b2ea814645eba63e5f6b12cfb966. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:57.411042 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d29cc2a0c8b84e3ebadc7f571d666847. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:57.413168 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet defb795026a44e12afb2defb9c9c6db2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:57.415447 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_4f9515ed04dc4e9e8480232c53fbac2e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:57.416249 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4f9515ed04dc4e9e8480232c53fbac2e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:57.418694 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd597c319af645909b8f3d1212fcb702. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:51:57.421502 518 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 24f07b1efac64b099b0b5ed467463749 consensus state has no leader
I20251029 02:51:57.422983 522 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_3d94f8b10e884dc0b338dcfbb9c82309" } modify_external_catalogs: true
I20251029 02:51:57.423167 522 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_3d94f8b10e884dc0b338dcfbb9c82309" } modify_external_catalogs: true
I20251029 02:51:57.424760 517 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_e168a30d3a1e48ed8053847b304036e9" } modify_external_catalogs: true
I20251029 02:51:57.424927 517 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_e168a30d3a1e48ed8053847b304036e9" } modify_external_catalogs: true
I20251029 02:51:57.426213 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_2ed982f44d384784b875b8a49394305e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:57.426931 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2ed982f44d384784b875b8a49394305e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:57.427361 521 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_a0b78d19117a4bd69361d316b318decf" } new_table_name: "default.Table_b22826191d2448e780eb6865d1cd7d05" modify_external_catalogs: true
I20251029 02:51:57.430315 519 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet faa835de84ff45b19525d1d603678987
I20251029 02:51:57.432292 1032 tablet_service.cc:1552] Processing DeleteTablet for tablet faa835de84ff45b19525d1d603678987 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:57 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:57.441516 1037 tablet_service.cc:1505] Processing CreateTablet for tablet 748777c6cd6e4f49ab49912b719e48bd (DEFAULT_TABLE table=default.Table_6ed43c1a2617493882d7cb6c568d21b7 [id=a2f3b36dc78a4e94bd054a6a5712b384]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:57.441975 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 748777c6cd6e4f49ab49912b719e48bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:57.442603 1024 tablet_service.cc:1505] Processing CreateTablet for tablet ad214c870ee649279ed7321c5b69cbee (DEFAULT_TABLE table=default.Table_982d01dccabe47bfaac7a582282c5d54 [id=033c1db6a2b44a2e99361183fd18c8f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:57.443044 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad214c870ee649279ed7321c5b69cbee. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:57.443092 1023 tablet_service.cc:1505] Processing CreateTablet for tablet 589a521f5cc24e6289c37b9cf256bce5 (DEFAULT_TABLE table=default.Table_b8095fca27a447f2a9ac0475373352c6 [id=56b57a7a90624bee92fa43661728c0f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:57.443295 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70b008011d5547bbaf3f45685394139f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:57.443603 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 589a521f5cc24e6289c37b9cf256bce5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:57.444444 1028 tablet_service.cc:1505] Processing CreateTablet for tablet eca016aed05e4ce09e1a2bf96e2286f5 (DEFAULT_TABLE table=default.Table_ee6dec2fc7834c78914ddc9f13890c55 [id=2faf0479a2a6477fa1d70d30094ef1db]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:57.445562 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eca016aed05e4ce09e1a2bf96e2286f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:57.451666 1026 tablet_service.cc:1505] Processing CreateTablet for tablet c081d2eacfac4ca8a0048e2a1c4f8f83 (DEFAULT_TABLE table=default.Table_dbf2e434d7964eac80709c205f61191e [id=b6afde5f860040be9ede53221978f44d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:57.452059 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet 589a521f5cc24e6289c37b9cf256bce5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:57 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:57.452348 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6dff036b32904ddb9598be4b79504bb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:57.452862 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c081d2eacfac4ca8a0048e2a1c4f8f83. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:57.456818 1025 tablet_service.cc:1505] Processing CreateTablet for tablet 6f8cf09f23984b879b60ab6b27c7d52e (DEFAULT_TABLE table=default.Table_04fadd7c87d1460dbc72c2caedbef915 [id=98f28a15eced4860ac13a60abdc8e0e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:57.457340 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f8cf09f23984b879b60ab6b27c7d52e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:57.469731 895 tablet_service.cc:1552] Processing DeleteTablet for tablet faa835de84ff45b19525d1d603678987 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:57 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:57.470932 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 748777c6cd6e4f49ab49912b719e48bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:57.476459 1611 tablet.cc:1722] T fa5127bcfcda40a7b5ad4ef016af072a P c756528642a543eeacd93cef3e335a7d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 02:51:57.480083 523 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_91cd5889c90148e5bde6051f2847832a" } new_table_name: "default.Table_0edd4b07469b4d24b302ecdb6859d1d1" modify_external_catalogs: true
I20251029 02:51:57.479995 1166 raft_consensus.cc:359] T aa776cde2f494fa5810539edf9f289a1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:57.482164 1166 raft_consensus.cc:385] T aa776cde2f494fa5810539edf9f289a1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:57.482865 1166 raft_consensus.cc:740] T aa776cde2f494fa5810539edf9f289a1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:51:57.484640 1166 consensus_queue.cc:260] T aa776cde2f494fa5810539edf9f289a1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:57.487607 522 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet c32c28b0e74a4af3a0e69bbf24ab1a1f
I20251029 02:51:57.488942 895 tablet_service.cc:1552] Processing DeleteTablet for tablet c32c28b0e74a4af3a0e69bbf24ab1a1f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:57 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:57.494650 1030 tablet_service.cc:1552] Processing DeleteTablet for tablet aa776cde2f494fa5810539edf9f289a1 with delete_type TABLET_DATA_DELETED (replaced by tablet c9bd3f2deb5d4af09b7918269a98509f) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:57.495813 1176 ts_tablet_manager.cc:1929] T 4b76c4408a7340e59b8506171622175f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251029 02:51:57.496327 1176 log.cc:1199] T 4b76c4408a7340e59b8506171622175f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/4b76c4408a7340e59b8506171622175f
I20251029 02:51:57.496634 895 tablet_service.cc:1505] Processing CreateTablet for tablet c081d2eacfac4ca8a0048e2a1c4f8f83 (DEFAULT_TABLE table=default.Table_dbf2e434d7964eac80709c205f61191e [id=b6afde5f860040be9ede53221978f44d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:57.499364 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_73bc152b9f224b07ae2524e0e18f7df0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:57.500012 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_73bc152b9f224b07ae2524e0e18f7df0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:57.501502 522 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_a6a9f0fa8983438db6cece4c2111a3c9" } new_table_name: "default.Table_66370c05265c48dbbe5f1e6ebf0a940b" modify_external_catalogs: true
I20251029 02:51:57.504089 517 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 4aa78f4ac38e4919864c924e50f1aaf4
I20251029 02:51:57.505513 882 tablet_service.cc:1552] Processing DeleteTablet for tablet 4aa78f4ac38e4919864c924e50f1aaf4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:57 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:57.510082 517 master_service.cc:954] ReplaceTablet: received request to replace tablet 2f198c3259864f7ab3625cb7bfae20b0 from {username='slave'} at 127.0.0.1:35314
I20251029 02:51:57.510351 1030 tablet_service.cc:1505] Processing CreateTablet for tablet f4729ecd182042c2a10f67ca25224101 (DEFAULT_TABLE table=default.Table_663bb3557fe64d8092e0a19dfa14d1b4 [id=1dd2d5488bc647c8b82c53886583b28d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:57.510810 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet defb795026a44e12afb2defb9c9c6db2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:57.511098 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4729ecd182042c2a10f67ca25224101. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:57.511511 1023 tablet_service.cc:1505] Processing CreateTablet for tablet fcb3db8399aa4cf188b07dda220f7f5b (DEFAULT_TABLE table=default.Table_33eacbf00740465185d70b46a3782893 [id=22e43c4483ae488b81fb94a7d221bdce]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:57.511968 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fcb3db8399aa4cf188b07dda220f7f5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:57.514219 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_8462690958454b098e0531b1086e60f7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:57.515043 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8462690958454b098e0531b1086e60f7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:57.516738 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_e8072943e31241f683bed508cb4f2029"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:57.516937 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c081d2eacfac4ca8a0048e2a1c4f8f83. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:51:57.517544 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e8072943e31241f683bed508cb4f2029 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:57.518682 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_02e6720794e845a6b77d20376060dd15"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:57.519155 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_02e6720794e845a6b77d20376060dd15 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:57.521088 894 tablet_service.cc:1505] Processing CreateTablet for tablet fcb3db8399aa4cf188b07dda220f7f5b (DEFAULT_TABLE table=default.Table_33eacbf00740465185d70b46a3782893 [id=22e43c4483ae488b81fb94a7d221bdce]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:57.526929 524 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_3bc6a1815ce646fb9a89e91fdda46ae0" } new_table_name: "default.Table_6da557988d044647b5e68f09b3f00407" modify_external_catalogs: true
I20251029 02:51:57.528790 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad214c870ee649279ed7321c5b69cbee. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:57.529300 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 589a521f5cc24e6289c37b9cf256bce5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:57.530283 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet 4aa78f4ac38e4919864c924e50f1aaf4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:57 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:57.530731 1027 tablet_service.cc:1552] Processing DeleteTablet for tablet c32c28b0e74a4af3a0e69bbf24ab1a1f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:57 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:57.533064 879 tablet_service.cc:1505] Processing CreateTablet for tablet f4729ecd182042c2a10f67ca25224101 (DEFAULT_TABLE table=default.Table_663bb3557fe64d8092e0a19dfa14d1b4 [id=1dd2d5488bc647c8b82c53886583b28d]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 02:51:57.535753 521 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b125cc9aaf4248b8bfa48fb152b832a6 consensus state has no leader
I20251029 02:51:57.536470 1176 ts_tablet_manager.cc:1950] T 4b76c4408a7340e59b8506171622175f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:51:57.537529 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 334) took 1144 ms. Trace:
I20251029 02:51:57.537811 1176 rpcz_store.cc:276] 1029 02:51:56.392508 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:56.392567 (+ 59us) service_pool.cc:225] Handling call
1029 02:51:57.072537 (+679970us) ts_tablet_manager.cc:1151] Deleting tablet 4b76c4408a7340e59b8506171622175f
1029 02:51:57.072547 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:57.218651 (+146104us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:57.495523 (+276872us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:57.537392 (+ 41869us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":679733}
I20251029 02:51:57.538856 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 4b76c4408a7340e59b8506171622175f (table default.Table_21b8ae2debb241bda65fe387a4a3190f [id=f3fa2dbbef7e47359492d5a5eef9b8fd]) successfully deleted
I20251029 02:51:57.539654 1176 tablet_replica.cc:333] T d6e1170dbb594a5696a3899cde8da7c9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:51:57.543318 886 tablet_service.cc:1505] Processing CreateTablet for tablet 055184c641ba45a588bca611fb24d130 (DEFAULT_TABLE table=default.Table_757229f7dede492d8da98328eaabc0dd [id=aec3db766dee438c89fb2f1939ee7b7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:57.546252 1167 tablet_bootstrap.cc:492] T 59af906061d049c99ae2153945597007 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:51:57.546497 1167 ts_tablet_manager.cc:1403] T 59af906061d049c99ae2153945597007 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.649s user 0.010s sys 0.007s
I20251029 02:51:57.547498 1167 raft_consensus.cc:359] T 59af906061d049c99ae2153945597007 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:57.547852 1167 raft_consensus.cc:385] T 59af906061d049c99ae2153945597007 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:57.547983 1167 raft_consensus.cc:740] T 59af906061d049c99ae2153945597007 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:51:57.548394 1167 consensus_queue.cc:260] T 59af906061d049c99ae2153945597007 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:57.549188 1167 ts_tablet_manager.cc:1434] T 59af906061d049c99ae2153945597007 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 02:51:57.549623 1167 tablet_bootstrap.cc:492] T d6e1170dbb594a5696a3899cde8da7c9 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:51:57.549973 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d31e548feae24b739087c1e211d514f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:57.550359 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 055184c641ba45a588bca611fb24d130. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:57.559556 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23b9b2ea814645eba63e5f6b12cfb966. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:57.562780 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eca016aed05e4ce09e1a2bf96e2286f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:57.571558 751 tablet_service.cc:1505] Processing CreateTablet for tablet 589a521f5cc24e6289c37b9cf256bce5 (DEFAULT_TABLE table=default.Table_b8095fca27a447f2a9ac0475373352c6 [id=56b57a7a90624bee92fa43661728c0f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:57.574362 521 master_service.cc:954] ReplaceTablet: received request to replace tablet 2f198c3259864f7ab3625cb7bfae20b0 from {username='slave'} at 127.0.0.1:35314
W20251029 02:51:57.574872 523 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9e07291d6b5645feb4d5890f58e1ef74 consensus state has no leader
W20251029 02:51:57.575914 711 leader_election.cc:343] T 5695b5a7b1854c3983e2d4ab684898b0 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:57.580144 1445 raft_consensus.cc:493] T 3362e95063af41c8a24e935f547e3089 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:57.580335 1445 raft_consensus.cc:515] T 3362e95063af41c8a24e935f547e3089 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:57.581190 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23b9b2ea814645eba63e5f6b12cfb966. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:57.581733 1445 leader_election.cc:290] T 3362e95063af41c8a24e935f547e3089 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:51:57.571405 1605 raft_consensus.cc:493] T 69597d48631d43b6a5b92abbf60d5ed3 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:57.583976 711 leader_election.cc:304] T 5695b5a7b1854c3983e2d4ab684898b0 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
W20251029 02:51:57.585698 522 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 32519fbffcef44a890161e5f3043806c consensus state has no leader
I20251029 02:51:57.588882 1445 raft_consensus.cc:2749] T 5695b5a7b1854c3983e2d4ab684898b0 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:51:57.589177 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_fdcb71c680504418a7f89d81feb372b0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:57.589845 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fdcb71c680504418a7f89d81feb372b0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:57.587754 1605 raft_consensus.cc:515] T 69597d48631d43b6a5b92abbf60d5ed3 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:57.587921 744 tablet_service.cc:1505] Processing CreateTablet for tablet 6f8cf09f23984b879b60ab6b27c7d52e (DEFAULT_TABLE table=default.Table_04fadd7c87d1460dbc72c2caedbef915 [id=98f28a15eced4860ac13a60abdc8e0e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:57.593103 1605 leader_election.cc:290] T 69597d48631d43b6a5b92abbf60d5ed3 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:51:57.593461 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_3d0a6e32dc914ba2b0263680b2587277"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:57.594013 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3d0a6e32dc914ba2b0263680b2587277 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:57.594033 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 589a521f5cc24e6289c37b9cf256bce5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:57.595953 518 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_0edd4b07469b4d24b302ecdb6859d1d1" } modify_external_catalogs: true
I20251029 02:51:57.596128 518 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_0edd4b07469b4d24b302ecdb6859d1d1" } modify_external_catalogs: true
I20251029 02:51:57.597739 1585 sys_catalog.cc:565] T 00000000000000000000000000000000 P e5e5222b26fe4145948620e51edf53fb [sys.catalog]: configured and running, proceeding with master startup.
I20251029 02:51:57.599823 520 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_ab94a2a8b29d498a9f201f39228d7bfe" } modify_external_catalogs: true
I20251029 02:51:57.600019 517 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 2f198c3259864f7ab3625cb7bfae20b0
I20251029 02:51:57.600211 520 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_ab94a2a8b29d498a9f201f39228d7bfe" } modify_external_catalogs: true
I20251029 02:51:57.597856 1601 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5e5222b26fe4145948620e51edf53fb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9672114a6a742e2864c16ff60ba246e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42343 } } peers { permanent_uuid: "e9261eb85ea443d5a3beb36bd0c549f0" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34239 } } peers { permanent_uuid: "e5e5222b26fe4145948620e51edf53fb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41233 } } }
I20251029 02:51:57.600483 517 catalog_manager.cc:6719] ReplaceTablet: tablet 2f198c3259864f7ab3625cb7bfae20b0 deleted and replaced by tablet 877e278deb4a4b01ac66b06be0890baf
I20251029 02:51:57.600603 1601 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5e5222b26fe4145948620e51edf53fb [sys.catalog]: This master's current role is: FOLLOWER
I20251029 02:51:57.601930 1172 tablet_bootstrap.cc:492] T 6feec2c8e030447fae9c292a626dcd1d P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:51:57.602291 1172 ts_tablet_manager.cc:1403] T 6feec2c8e030447fae9c292a626dcd1d P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.310s user 0.011s sys 0.006s
I20251029 02:51:57.608439 1175 ts_tablet_manager.cc:1929] T 4b76c4408a7340e59b8506171622175f P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251029 02:51:57.608681 1175 log.cc:1199] T 4b76c4408a7340e59b8506171622175f P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/4b76c4408a7340e59b8506171622175f
I20251029 02:51:57.609179 1175 ts_tablet_manager.cc:1950] T 4b76c4408a7340e59b8506171622175f P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:51:57.610075 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 320) took 1219 ms. Trace:
I20251029 02:51:57.610190 1175 rpcz_store.cc:276] 1029 02:51:56.390669 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:56.390739 (+ 70us) service_pool.cc:225] Handling call
1029 02:51:57.054899 (+664160us) ts_tablet_manager.cc:1151] Deleting tablet 4b76c4408a7340e59b8506171622175f
1029 02:51:57.055007 (+ 108us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:57.291599 (+236592us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:57.603559 (+311960us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:57.610012 (+ 6453us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":657931}
I20251029 02:51:57.610756 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 321) took 1184 ms. Trace:
I20251029 02:51:57.610852 1175 rpcz_store.cc:276] 1029 02:51:56.426698 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:56.426770 (+ 72us) service_pool.cc:225] Handling call
1029 02:51:57.610552 (+1183782us) ts_tablet_manager.cc:1151] Deleting tablet dd6ad641fbda472fab07aa35c5e4523a
1029 02:51:57.610566 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:57.610698 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1183225}
I20251029 02:51:57.611222 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 322) took 1146 ms. Trace:
I20251029 02:51:57.611310 1175 rpcz_store.cc:276] 1029 02:51:56.465154 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:56.465240 (+ 86us) service_pool.cc:225] Handling call
1029 02:51:57.611081 (+1145841us) ts_tablet_manager.cc:1151] Deleting tablet daa05a34c0684c07a35966290fd7d44c
1029 02:51:57.611093 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:57.611173 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1140867}
I20251029 02:51:57.611646 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 323) took 1099 ms. Trace:
I20251029 02:51:57.611725 1175 rpcz_store.cc:276] 1029 02:51:56.512446 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:56.512556 (+ 110us) service_pool.cc:225] Handling call
1029 02:51:57.611497 (+1098941us) ts_tablet_manager.cc:1151] Deleting tablet f187dbd8a55b45b292ea61db43caf47d
1029 02:51:57.611505 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:57.611584 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1098706}
I20251029 02:51:57.626827 522 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_08fcf9684b844572a63b823f6a4d2302" } new_table_name: "default.Table_44b56ca77fed49489c18e54dffb4b108" modify_external_catalogs: true
I20251029 02:51:57.632558 518 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 9e07291d6b5645feb4d5890f58e1ef74
I20251029 02:51:57.633720 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_243752c5b95a4421a86f6510f801e0b9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:57.634327 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_243752c5b95a4421a86f6510f801e0b9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:57.635643 518 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_d4caf12f7662463c824210f3c9cd7e0f" } modify_external_catalogs: true
I20251029 02:51:57.635833 518 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_d4caf12f7662463c824210f3c9cd7e0f" } modify_external_catalogs: true
I20251029 02:51:57.621824 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f8cf09f23984b879b60ab6b27c7d52e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:57.652519 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_b1d1e274a2ae4a639d80fb46e27bca52"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:57.653314 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b1d1e274a2ae4a639d80fb46e27bca52 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:51:57.654973 522 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c64d6761ef924ccaa9c2579f8ba47e33 has no consensus state
I20251029 02:51:57.656105 520 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 23b9b2ea814645eba63e5f6b12cfb966
I20251029 02:51:57.612489 1175 tablet_replica.cc:333] T 6cd7009e06a646acad0802918c2b425e P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:51:57.671522 1175 raft_consensus.cc:2243] T 6cd7009e06a646acad0802918c2b425e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:57.672127 1175 raft_consensus.cc:2272] T 6cd7009e06a646acad0802918c2b425e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:57.673000 1175 ts_tablet_manager.cc:1916] T 6cd7009e06a646acad0802918c2b425e P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:57.678989 518 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet afb19ba39399410bb90b6225d7b5b5b1
I20251029 02:51:57.608489 1172 raft_consensus.cc:359] T 6feec2c8e030447fae9c292a626dcd1d P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:57.679450 1172 raft_consensus.cc:385] T 6feec2c8e030447fae9c292a626dcd1d P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:57.679548 1172 raft_consensus.cc:740] T 6feec2c8e030447fae9c292a626dcd1d P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:51:57.680195 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_59a5648999114c6f8b2a58e938c8e49a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:57.680912 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_59a5648999114c6f8b2a58e938c8e49a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:57.682502 1172 consensus_queue.cc:260] T 6feec2c8e030447fae9c292a626dcd1d P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:57.687273 1172 ts_tablet_manager.cc:1434] T 6feec2c8e030447fae9c292a626dcd1d P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.085s user 0.002s sys 0.000s
I20251029 02:51:57.687640 1172 tablet_bootstrap.cc:492] T faa835de84ff45b19525d1d603678987 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:51:57.689785 1172 tablet_bootstrap.cc:654] T faa835de84ff45b19525d1d603678987 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:57.695461 520 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_5113d1a33e8e4bd2af5bc51e689f304f" } modify_external_catalogs: true
I20251029 02:51:57.695899 520 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_5113d1a33e8e4bd2af5bc51e689f304f" } modify_external_catalogs: true
I20251029 02:51:57.701424 1176 raft_consensus.cc:2243] T d6e1170dbb594a5696a3899cde8da7c9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:57.704916 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fcb3db8399aa4cf188b07dda220f7f5b. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:51:57.706621 847 leader_election.cc:343] T 7c041f4909274cd5a03b9fd78353faf9 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 7c041f4909274cd5a03b9fd78353faf9
I20251029 02:51:57.706766 847 leader_election.cc:304] T 7c041f4909274cd5a03b9fd78353faf9 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9
I20251029 02:51:57.700409 1041 tablet_service.cc:1505] Processing CreateTablet for tablet 055184c641ba45a588bca611fb24d130 (DEFAULT_TABLE table=default.Table_757229f7dede492d8da98328eaabc0dd [id=aec3db766dee438c89fb2f1939ee7b7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:57.707405 1421 raft_consensus.cc:2749] T 7c041f4909274cd5a03b9fd78353faf9 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:51:57.708105 1605 raft_consensus.cc:493] T 3cded156380f45cc95893825c71ba0e9 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:57.708279 1605 raft_consensus.cc:515] T 3cded156380f45cc95893825c71ba0e9 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:57.708963 1605 leader_election.cc:290] T 3cded156380f45cc95893825c71ba0e9 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:51:57.709430 1025 tablet_service.cc:1505] Processing CreateTablet for tablet 5815626b05204b578cc4a110f246bf10 (DEFAULT_TABLE table=default.Table_4b9147e899c043ad9023bf0d8d555a0c [id=45dc803b4e1041e78f40a3e2e1369041]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:57.709905 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4729ecd182042c2a10f67ca25224101. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:57.712038 523 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_02e6720794e845a6b77d20376060dd15" } modify_external_catalogs: true
I20251029 02:51:57.712328 523 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_02e6720794e845a6b77d20376060dd15" } modify_external_catalogs: true
I20251029 02:51:57.712944 1350 raft_consensus.cc:493] T 14698af6348f41e384f10569392d9b23 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:57.713307 1350 raft_consensus.cc:515] T 14698af6348f41e384f10569392d9b23 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:57.714033 1350 leader_election.cc:290] T 14698af6348f41e384f10569392d9b23 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:51:57.727857 895 tablet_service.cc:1505] Processing CreateTablet for tablet 5815626b05204b578cc4a110f246bf10 (DEFAULT_TABLE table=default.Table_4b9147e899c043ad9023bf0d8d555a0c [id=45dc803b4e1041e78f40a3e2e1369041]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:57.728858 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_93bc8cfb59f440849d8ae4d6529e6a01"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:57.729516 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_93bc8cfb59f440849d8ae4d6529e6a01 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:57.731487 520 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet fa5127bcfcda40a7b5ad4ef016af072a
I20251029 02:51:57.734653 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5815626b05204b578cc4a110f246bf10. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:57.740690 884 tablet_service.cc:1505] Processing CreateTablet for tablet 628a7d0419cb434f8171faf500d585d8 (DEFAULT_TABLE table=default.Table_31aa9d49c83b4ab29823acea7480bc96 [id=d6b287a41f2c46868c3f207b7d4359ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:57.742228 918 raft_consensus.cc:3060] T 59af906061d049c99ae2153945597007 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:51:57.743835 1167 tablet_bootstrap.cc:654] T d6e1170dbb594a5696a3899cde8da7c9 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:57.744660 1172 tablet_bootstrap.cc:492] T faa835de84ff45b19525d1d603678987 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:51:57.746467 1172 ts_tablet_manager.cc:1403] T faa835de84ff45b19525d1d603678987 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.059s user 0.002s sys 0.004s
I20251029 02:51:57.747490 520 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_644024f7ee3a44ce988f60dedec9419b" } new_table_name: "default.Table_8c5d559a2d7f427094b719d1c0b6cff4" modify_external_catalogs: true
I20251029 02:51:57.749953 523 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet e7c228f70c8f4e0a97f4da07e6f567e7
I20251029 02:51:57.758492 1172 raft_consensus.cc:359] T faa835de84ff45b19525d1d603678987 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:57.761333 1172 raft_consensus.cc:385] T faa835de84ff45b19525d1d603678987 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:57.762568 1172 raft_consensus.cc:740] T faa835de84ff45b19525d1d603678987 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:51:57.743927 1612 tablet.cc:1722] T 4b76c4408a7340e59b8506171622175f P c756528642a543eeacd93cef3e335a7d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 02:51:57.766409 891 tablet_service.cc:1505] Processing CreateTablet for tablet 24f07b1efac64b099b0b5ed467463749 (DEFAULT_TABLE table=default.Table_21b8ae2debb241bda65fe387a4a3190f [id=f3fa2dbbef7e47359492d5a5eef9b8fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:57.765910 878 tablet_service.cc:1505] Processing CreateTablet for tablet f93b0c39d7174dabbe018b1546dbc198 (DEFAULT_TABLE table=default.Table_559b8d2aa7c04ba68f184405d44898d6 [id=61fac9a7d5b049e08485ce8695ba5a48]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:57.767731 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 628a7d0419cb434f8171faf500d585d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:57.766031 1172 consensus_queue.cc:260] T faa835de84ff45b19525d1d603678987 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:57.769425 887 tablet_service.cc:1505] Processing CreateTablet for tablet b0fec06ad37d4c12a7045d424b88cc5e (DEFAULT_TABLE table=default.Table_c4333d8c61d346b1ac1a809d3ccb1915 [id=6d576d3499e649d78000c31fc5dab58c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:57.772079 1175 ts_tablet_manager.cc:1929] T 6cd7009e06a646acad0802918c2b425e P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:57.773550 1175 log.cc:1199] T 6cd7009e06a646acad0802918c2b425e P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/6cd7009e06a646acad0802918c2b425e
I20251029 02:51:57.774289 898 tablet_service.cc:1552] Processing DeleteTablet for tablet 2f198c3259864f7ab3625cb7bfae20b0 with delete_type TABLET_DATA_DELETED (replaced by tablet 877e278deb4a4b01ac66b06be0890baf) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:57.774784 882 tablet_service.cc:1505] Processing CreateTablet for tablet abf18e06df9b482a8b25c078b9495272 (DEFAULT_TABLE table=default.Table_19165ff9f7f24a97bf835e72701455d3 [id=ae6032816e6c4b80aa294478cbd49460]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:57.775815 1172 ts_tablet_manager.cc:1434] T faa835de84ff45b19525d1d603678987 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.028s user 0.004s sys 0.000s
I20251029 02:51:57.776577 518 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_746054018e16496ea35decf9614e9f5c" } new_table_name: "default.Table_18daac68b3444f3aaf2f2e45618ef8f4" modify_external_catalogs: true
I20251029 02:51:57.776536 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_19039883faf6464387f65003d0d638a3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:57.777568 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24f07b1efac64b099b0b5ed467463749. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:51:57.777846 520 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 378f549ff7364d018d8b9a5ef01ac878 consensus state has no leader
I20251029 02:51:57.778923 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0fec06ad37d4c12a7045d424b88cc5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:57.778474 1175 ts_tablet_manager.cc:1950] T 6cd7009e06a646acad0802918c2b425e P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:51:57.780149 1172 tablet_bootstrap.cc:492] T 392a36a6b59a42f78e3ac7c8fb63214b P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:51:57.780905 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abf18e06df9b482a8b25c078b9495272. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:57.781805 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f93b0c39d7174dabbe018b1546dbc198. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:57.783670 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "14698af6348f41e384f10569392d9b23" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:51:57.786650 518 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet adf6e736cb9c4920807ee58682e134ef consensus state has no leader
I20251029 02:51:57.789594 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 330) took 1122 ms. Trace:
I20251029 02:51:57.791507 898 tablet_service.cc:1505] Processing CreateTablet for tablet 7816556521144c928d7a02694e22e79f (DEFAULT_TABLE table=default.Table_0b95eba70e97459eb847fecfac513686 [id=67fc9a53c89144399f905c2f48800e17]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:57.792125 518 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_9b6f49f9513d4a09a4782893f8f16e5c" } new_table_name: "default.Table_d897a86961894f1aaad67274b687080e" modify_external_catalogs: true
I20251029 02:51:57.792526 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7816556521144c928d7a02694e22e79f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:57.791483 1175 rpcz_store.cc:276] 1029 02:51:56.666648 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:56.666703 (+ 55us) service_pool.cc:225] Handling call
1029 02:51:57.612415 (+945712us) ts_tablet_manager.cc:1151] Deleting tablet 6cd7009e06a646acad0802918c2b425e
1029 02:51:57.612421 (+ 6us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:57.723327 (+110906us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:57.770449 (+ 47122us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:57.789158 (+ 18709us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":945504}
I20251029 02:51:57.796356 1175 tablet_replica.cc:333] T 3362e95063af41c8a24e935f547e3089 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:51:57.798926 899 tablet_service.cc:1505] Processing CreateTablet for tablet b125cc9aaf4248b8bfa48fb152b832a6 (DEFAULT_TABLE table=default.Table_b22826191d2448e780eb6865d1cd7d05 [id=3513c69f6b3941849ac5b8ae0a5846d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:57.799726 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b125cc9aaf4248b8bfa48fb152b832a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:57.799317 1175 raft_consensus.cc:2243] T 3362e95063af41c8a24e935f547e3089 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
W20251029 02:51:57.802019 518 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dabb8d760aae4aa09eb83cdb586c9788 consensus state has no leader
I20251029 02:51:57.803346 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_fd1d23f31ebf47b884897c68eac9004d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:57.801275 1175 raft_consensus.cc:2272] T 3362e95063af41c8a24e935f547e3089 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251029 02:51:57.807242 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_19039883faf6464387f65003d0d638a3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:57.807808 1172 tablet_bootstrap.cc:654] T 392a36a6b59a42f78e3ac7c8fb63214b P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
W20251029 02:51:57.808670 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fd1d23f31ebf47b884897c68eac9004d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:57.813158 520 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_4e701fdc7b034edc80ccc5ae0c9c4439" } new_table_name: "default.Table_8cf5614390ae4a32a856fce51f3edc08" modify_external_catalogs: true
I20251029 02:51:57.814342 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_d0a24cc3c6b244b0b43c78742dab0877"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:57.814548 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_6e1283a0c161498db405cf1e9b2c93d7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:57.814906 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d0a24cc3c6b244b0b43c78742dab0877 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:51:57.814927 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6e1283a0c161498db405cf1e9b2c93d7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:57.815057 1175 ts_tablet_manager.cc:1916] T 3362e95063af41c8a24e935f547e3089 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:57.813158 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_cd55951068e2462ab1b5b075a5b40767"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:57.816747 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cd55951068e2462ab1b5b075a5b40767 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:57.819918 900 tablet_service.cc:1505] Processing CreateTablet for tablet 506f8c112a3b43e1a3d6bbafb8cb81e8 (DEFAULT_TABLE table=default.Table_e8e977ce21ca40d4a22d0434d73ef70b [id=9107fd53f5c94d2d80be50ef1f7c8e9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:57.822576 888 tablet_service.cc:1505] Processing CreateTablet for tablet 1df0d944717d487a98ea83a8ec04bd46 (DEFAULT_TABLE table=default.Table_d477b4de8ad4449aa5d53db48d45273f [id=513d017e315a49d0a0fbe1fb12649f82]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:57.829272 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1df0d944717d487a98ea83a8ec04bd46. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:57.833480 1172 tablet_bootstrap.cc:492] T 392a36a6b59a42f78e3ac7c8fb63214b P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:51:57.834499 523 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_11de4e6789214efca8b43cbcdae81a93" } modify_external_catalogs: true
I20251029 02:51:57.834748 523 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_11de4e6789214efca8b43cbcdae81a93" } modify_external_catalogs: true
I20251029 02:51:57.835331 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_e280ab7143204fbd8daa4f2c898617f9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:57.835178 1172 ts_tablet_manager.cc:1403] T 392a36a6b59a42f78e3ac7c8fb63214b P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.056s user 0.004s sys 0.004s
W20251029 02:51:57.835840 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e280ab7143204fbd8daa4f2c898617f9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:57.840828 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 506f8c112a3b43e1a3d6bbafb8cb81e8. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:51:57.845394 520 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9269622ca92645d5aa8e73341201ccbb consensus state has no leader
I20251029 02:51:57.846695 1605 raft_consensus.cc:493] T faa835de84ff45b19525d1d603678987 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:57.846853 1605 raft_consensus.cc:515] T faa835de84ff45b19525d1d603678987 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:57.845346 1172 raft_consensus.cc:359] T 392a36a6b59a42f78e3ac7c8fb63214b P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:57.846246 492 master_runner.cc:186] Error getting master registration for 127.0.104.60:41233: OK, instance_id { permanent_uuid: "e5e5222b26fe4145948620e51edf53fb" instance_seqno: 1761706316385283 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20251029 02:51:57.847275 492 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251029 02:51:57.847497 1605 leader_election.cc:290] T faa835de84ff45b19525d1d603678987 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:51:57.847720 1172 raft_consensus.cc:385] T 392a36a6b59a42f78e3ac7c8fb63214b P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:57.848996 1172 raft_consensus.cc:740] T 392a36a6b59a42f78e3ac7c8fb63214b P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:51:57.851852 1172 consensus_queue.cc:260] T 392a36a6b59a42f78e3ac7c8fb63214b P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:57.857227 520 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_7f609ea711d84843b92af7fc7c328804" } new_table_name: "default.Table_b89b7b9f911e4c308272ed13a311ac9b" modify_external_catalogs: true
I20251029 02:51:57.859616 525 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_8cf076425649436289d327edc92d4642" } modify_external_catalogs: true
I20251029 02:51:57.859805 525 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_8cf076425649436289d327edc92d4642" } modify_external_catalogs: true
I20251029 02:51:57.862010 1172 ts_tablet_manager.cc:1434] T 392a36a6b59a42f78e3ac7c8fb63214b P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.026s user 0.004s sys 0.000s
I20251029 02:51:57.866221 893 tablet_service.cc:1552] Processing DeleteTablet for tablet 9e07291d6b5645feb4d5890f58e1ef74 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:57 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:57.867512 1172 tablet_bootstrap.cc:492] T a5c3a15510cb4321bbb80d61fabd0c0a P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:51:57.847678 890 tablet_service.cc:1505] Processing CreateTablet for tablet cc802bda2f6a4e439b605f2cafa37967 (DEFAULT_TABLE table=default.Table_6e96405b009143d2bd1c0942cb98c033 [id=67905be94563417c8713eadd51eb94cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:57.870340 1166 ts_tablet_manager.cc:1434] T aa776cde2f494fa5810539edf9f289a1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.387s user 0.001s sys 0.004s
W20251029 02:51:57.871085 988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251029 02:51:57.874042 883 tablet_service.cc:1552] Processing DeleteTablet for tablet 23b9b2ea814645eba63e5f6b12cfb966 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:57 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:57.874734 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_88289be99f39482d837c84fecf31efb1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:57.879019 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_88289be99f39482d837c84fecf31efb1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:57.881122 522 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_6e96405b009143d2bd1c0942cb98c033" } modify_external_catalogs: true
I20251029 02:51:57.881332 522 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_6e96405b009143d2bd1c0942cb98c033" } modify_external_catalogs: true
I20251029 02:51:57.883344 1027 tablet_service.cc:1505] Processing CreateTablet for tablet 628a7d0419cb434f8171faf500d585d8 (DEFAULT_TABLE table=default.Table_31aa9d49c83b4ab29823acea7480bc96 [id=d6b287a41f2c46868c3f207b7d4359ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:57.883550 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3362e95063af41c8a24e935f547e3089" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:51:57.884693 1166 ts_tablet_manager.cc:1467] T aa776cde2f494fa5810539edf9f289a1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Tablet startup took 558ms
I20251029 02:51:57.884744 1059 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "69597d48631d43b6a5b92abbf60d5ed3" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:51:57.885999 1350 raft_consensus.cc:493] T aa776cde2f494fa5810539edf9f289a1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:57.887481 1465 raft_consensus.cc:493] T 012ce243fd07466ab5301d1c9ce4b413 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251029 02:51:57.886528 1166 ts_tablet_manager.cc:1469] T aa776cde2f494fa5810539edf9f289a1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Trace:
1029 02:51:52.116049 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:52.117740 (+ 1691us) service_pool.cc:225] Handling call
1029 02:51:52.118229 (+ 489us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1029 02:51:52.118274 (+ 45us) ts_tablet_manager.cc:682] Creating new metadata...
1029 02:51:52.154248 (+ 35974us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:52.157979 (+ 3731us) tablet_replica.cc:206] Creating consensus instance
1029 02:51:52.158138 (+ 159us) inbound_call.cc:173] Queueing success response
1029 02:51:57.125910 (+4967772us) ts_tablet_manager.cc:1379] Bootstrapping tablet
1029 02:51:57.311600 (+185690us) ts_tablet_manager.cc:1435] Starting tablet replica
1029 02:51:57.311946 (+ 346us) tablet_replica.cc:261] Starting instrumentation
1029 02:51:57.312148 (+ 202us) tablet_replica.cc:280] Starting consensus
Metrics: {"raft.queue_time_us":155,"raft.run_cpu_time_us":79,"raft.run_wall_time_us":80,"tablet-open.queue_time_us":4967816}
I20251029 02:51:57.890400 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 055184c641ba45a588bca611fb24d130. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:57.893373 893 tablet_service.cc:1552] Processing DeleteTablet for tablet fa5127bcfcda40a7b5ad4ef016af072a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:57 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:57.898434 1172 tablet_bootstrap.cc:654] T a5c3a15510cb4321bbb80d61fabd0c0a P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:57.901558 1042 tablet_service.cc:1505] Processing CreateTablet for tablet 506f8c112a3b43e1a3d6bbafb8cb81e8 (DEFAULT_TABLE table=default.Table_e8e977ce21ca40d4a22d0434d73ef70b [id=9107fd53f5c94d2d80be50ef1f7c8e9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:57.901618 1350 raft_consensus.cc:515] T aa776cde2f494fa5810539edf9f289a1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:57.910235 1062 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cded156380f45cc95893825c71ba0e9" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:51:57.910634 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5815626b05204b578cc4a110f246bf10. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:57.912779 1166 tablet_bootstrap.cc:492] T d2990e9f626c40618c118266bc595684 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:51:57.912830 894 tablet_service.cc:1552] Processing DeleteTablet for tablet afb19ba39399410bb90b6225d7b5b5b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:57 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:57.913017 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc802bda2f6a4e439b605f2cafa37967. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:57.914415 1174 ts_tablet_manager.cc:1929] T 11f4252b5e694312aaa142468f8e68b4 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20251029 02:51:57.914669 1174 log.cc:1199] T 11f4252b5e694312aaa142468f8e68b4 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/11f4252b5e694312aaa142468f8e68b4
I20251029 02:51:57.915050 1350 leader_election.cc:290] T aa776cde2f494fa5810539edf9f289a1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:51:57.915246 1174 ts_tablet_manager.cc:1950] T 11f4252b5e694312aaa142468f8e68b4 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:51:57.915208 1465 raft_consensus.cc:515] T 012ce243fd07466ab5301d1c9ce4b413 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:57.915589 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 309) took 1765 ms. Trace:
I20251029 02:51:57.915704 1174 rpcz_store.cc:276] 1029 02:51:56.150597 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:56.150649 (+ 52us) service_pool.cc:225] Handling call
1029 02:51:57.324757 (+1174108us) ts_tablet_manager.cc:1151] Deleting tablet 11f4252b5e694312aaa142468f8e68b4
1029 02:51:57.324765 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:57.728616 (+403851us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:57.866097 (+137481us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:57.915534 (+ 49437us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1173896}
I20251029 02:51:57.916123 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 11f4252b5e694312aaa142468f8e68b4 (table default.Table_b93cf815f69a4fe895e2ce461148fc71 [id=d0cd3d1567c54463947443927e18169f]) successfully deleted
I20251029 02:51:57.917193 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_b1c04ce867284dc5bf55b02f25bf66ae"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:57.917698 1029 tablet_service.cc:1505] Processing CreateTablet for tablet f93b0c39d7174dabbe018b1546dbc198 (DEFAULT_TABLE table=default.Table_559b8d2aa7c04ba68f184405d44898d6 [id=61fac9a7d5b049e08485ce8695ba5a48]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 02:51:57.917877 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b1c04ce867284dc5bf55b02f25bf66ae in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:57.918059 1026 tablet_service.cc:1505] Processing CreateTablet for tablet 24f07b1efac64b099b0b5ed467463749 (DEFAULT_TABLE table=default.Table_21b8ae2debb241bda65fe387a4a3190f [id=f3fa2dbbef7e47359492d5a5eef9b8fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:57.918572 1035 tablet_service.cc:1505] Processing CreateTablet for tablet abf18e06df9b482a8b25c078b9495272 (DEFAULT_TABLE table=default.Table_19165ff9f7f24a97bf835e72701455d3 [id=ae6032816e6c4b80aa294478cbd49460]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:57.919253 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 628a7d0419cb434f8171faf500d585d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:57.919638 1033 tablet_service.cc:1552] Processing DeleteTablet for tablet 23b9b2ea814645eba63e5f6b12cfb966 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:57 UTC) from {username='slave'} at 127.0.0.1:37626
W20251029 02:51:57.920106 513 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet adf6e736cb9c4920807ee58682e134ef (table default.Table_18daac68b3444f3aaf2f2e45618ef8f4 [id=64650da66d464074828f2dc955045f68], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet adf6e736cb9c4920807ee58682e134ef consensus state has no leader
I20251029 02:51:57.920352 1039 tablet_service.cc:1505] Processing CreateTablet for tablet b125cc9aaf4248b8bfa48fb152b832a6 (DEFAULT_TABLE table=default.Table_b22826191d2448e780eb6865d1cd7d05 [id=3513c69f6b3941849ac5b8ae0a5846d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:57.920718 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b125cc9aaf4248b8bfa48fb152b832a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:57.921476 1032 tablet_service.cc:1505] Processing CreateTablet for tablet 1df0d944717d487a98ea83a8ec04bd46 (DEFAULT_TABLE table=default.Table_d477b4de8ad4449aa5d53db48d45273f [id=513d017e315a49d0a0fbe1fb12649f82]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:57.921985 1034 tablet_service.cc:1505] Processing CreateTablet for tablet cc802bda2f6a4e439b605f2cafa37967 (DEFAULT_TABLE table=default.Table_6e96405b009143d2bd1c0942cb98c033 [id=67905be94563417c8713eadd51eb94cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:57.922319 1040 tablet_service.cc:1552] Processing DeleteTablet for tablet 9e07291d6b5645feb4d5890f58e1ef74 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:57 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:57.924479 1024 tablet_service.cc:1505] Processing CreateTablet for tablet b0fec06ad37d4c12a7045d424b88cc5e (DEFAULT_TABLE table=default.Table_c4333d8c61d346b1ac1a809d3ccb1915 [id=6d576d3499e649d78000c31fc5dab58c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:57.924786 1037 tablet_service.cc:1552] Processing DeleteTablet for tablet 2f198c3259864f7ab3625cb7bfae20b0 with delete_type TABLET_DATA_DELETED (replaced by tablet 877e278deb4a4b01ac66b06be0890baf) from {username='slave'} at 127.0.0.1:37626
W20251029 02:51:57.932579 520 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1bdb67125493430398ff5c80b73de2ac consensus state has no leader
I20251029 02:51:57.932838 1350 consensus_queue.cc:1048] T 59af906061d049c99ae2153945597007 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20251029 02:51:57.934191 990 leader_election.cc:343] T 14698af6348f41e384f10569392d9b23 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:57.934420 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_86ad8c0a1e9c4fd6b12b9f59a195e951"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:57.936107 526 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_af1ab75f523d41338d4471824997f935" } modify_external_catalogs: true
I20251029 02:51:57.936337 526 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_af1ab75f523d41338d4471824997f935" } modify_external_catalogs: true
I20251029 02:51:57.937180 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "012ce243fd07466ab5301d1c9ce4b413" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:51:57.937676 918 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "aa776cde2f494fa5810539edf9f289a1" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:51:57.938781 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_86ad8c0a1e9c4fd6b12b9f59a195e951 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:57.937263 1465 leader_election.cc:290] T 012ce243fd07466ab5301d1c9ce4b413 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:51:57.945480 1031 tablet_service.cc:1505] Processing CreateTablet for tablet 7816556521144c928d7a02694e22e79f (DEFAULT_TABLE table=default.Table_0b95eba70e97459eb847fecfac513686 [id=67fc9a53c89144399f905c2f48800e17]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:57.946264 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1df0d944717d487a98ea83a8ec04bd46. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:51:57.948130 712 leader_election.cc:343] T 3362e95063af41c8a24e935f547e3089 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
W20251029 02:51:57.948407 712 leader_election.cc:343] T 69597d48631d43b6a5b92abbf60d5ed3 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
W20251029 02:51:57.948578 712 leader_election.cc:343] T 3cded156380f45cc95893825c71ba0e9 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:57.948812 1040 tablet_service.cc:1552] Processing DeleteTablet for tablet afb19ba39399410bb90b6225d7b5b5b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:57 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:57.961222 524 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_93b405a4f257481289518c5ff28832a0" } modify_external_catalogs: true
I20251029 02:51:57.961464 524 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_93b405a4f257481289518c5ff28832a0" } modify_external_catalogs: true
I20251029 02:51:57.964749 1174 tablet_replica.cc:333] T 5e31e061d74146009d24d5cd57487a5e P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:51:57.967875 1166 tablet_bootstrap.cc:654] T d2990e9f626c40618c118266bc595684 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:57.968331 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24f07b1efac64b099b0b5ed467463749. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:57.972079 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_293c6e92af8a4707be8e2596399fd6a2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:57.972851 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_293c6e92af8a4707be8e2596399fd6a2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:57.977316 1062 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "faa835de84ff45b19525d1d603678987" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:51:57.982671 989 leader_election.cc:343] T 14698af6348f41e384f10569392d9b23 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 14698af6348f41e384f10569392d9b23
I20251029 02:51:57.982946 1176 raft_consensus.cc:2272] T d6e1170dbb594a5696a3899cde8da7c9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:57.982870 989 leader_election.cc:304] T 14698af6348f41e384f10569392d9b23 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5; no voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d
I20251029 02:51:57.986781 1033 tablet_service.cc:1552] Processing DeleteTablet for tablet fa5127bcfcda40a7b5ad4ef016af072a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:57 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:57.986980 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f93b0c39d7174dabbe018b1546dbc198. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:57.987563 1465 consensus_queue.cc:1048] T 59af906061d049c99ae2153945597007 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.039s
I20251029 02:51:57.988488 1176 ts_tablet_manager.cc:1916] T d6e1170dbb594a5696a3899cde8da7c9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:57.992015 521 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_1d6e8e70399347f4a102c28e953a6c1e" } new_table_name: "default.Table_8b6217e43e8d467693c104ef84e78911" modify_external_catalogs: true
I20251029 02:51:57.993719 519 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_f914dd26d0b846f8923187c2c8edf27e" } new_table_name: "default.Table_10befcd256124bc187c2f0ed7887d711" modify_external_catalogs: true
W20251029 02:51:57.994567 989 leader_election.cc:343] T aa776cde2f494fa5810539edf9f289a1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: aa776cde2f494fa5810539edf9f289a1
I20251029 02:51:57.995889 1628 raft_consensus.cc:2749] T 14698af6348f41e384f10569392d9b23 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:51:57.997876 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abf18e06df9b482a8b25c078b9495272. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.006357 895 tablet_service.cc:1505] Processing CreateTablet for tablet 786d9148fb1d417a8c2c8b8c53665899 (DEFAULT_TABLE table=default.Table_3d0a6e32dc914ba2b0263680b2587277 [id=0809df5782144d1da622b5726db932f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.008931 518 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_5c973aa3bd7f4a9ebf8435ed761552a5" } new_table_name: "default.Table_3b4bdfaf4ddf43cfa2d2287594c57816" modify_external_catalogs: true
I20251029 02:51:58.008983 1174 raft_consensus.cc:2243] T 5e31e061d74146009d24d5cd57487a5e P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus shutting down.
W20251029 02:51:58.009991 698 kernel_stack_watchdog.cc:198] Thread 711 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 193ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7f11784da980 (unknown) at ??:0
@ 0x93d797 __sanitizer::internal_sched_yield() at ??:?
@ 0x8a2e4d __interceptor_pthread_create at ??:?
@ 0x7f1174ec3be1 kudu::Thread::StartThread() at ??:0
@ 0xa75107 kudu::Thread::Create() at ??:?
@ 0x7f1174ee3a2b kudu::ThreadPool::CreateThread() at ??:0
@ 0x7f1174edf44d kudu::ThreadPool::DoSubmit() at ??:0
@ 0x7f1174ede89f kudu::ThreadPoolToken::Submit() at ??:0
@ 0x7f1180c472e8 kudu::consensus::Peer::ProcessResponse() at ??:0
@ 0x7f1180c47b15 kudu::consensus::Peer::ProcessSingleResponse() at ??:0
@ 0x7f1180c4f6a1 kudu::consensus::Peer::SendNextRequest()::$_6::operator()() at ??:0
@ 0x7f1180c4f4ae std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa8867d std::function<>::operator()() at ??:?
@ 0x7f117bdace0f kudu::rpc::Proxy::AsyncRequest()::$_1::operator()() at ??:0
@ 0x7f117bdac9ee std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa8867d std::function<>::operator()() at ??:?
I20251029 02:51:58.014348 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 786d9148fb1d417a8c2c8b8c53665899. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.018716 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7816556521144c928d7a02694e22e79f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.019227 526 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 49d9ce32e4f04f11be2ecefc5d523a8b
I20251029 02:51:58.021064 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 506f8c112a3b43e1a3d6bbafb8cb81e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.025457 1174 raft_consensus.cc:2272] T 5e31e061d74146009d24d5cd57487a5e P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
W20251029 02:51:58.025561 990 leader_election.cc:343] T 012ce243fd07466ab5301d1c9ce4b413 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:58.026109 1174 ts_tablet_manager.cc:1916] T 5e31e061d74146009d24d5cd57487a5e P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 02:51:58.026109 989 leader_election.cc:343] T 012ce243fd07466ab5301d1c9ce4b413 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 012ce243fd07466ab5301d1c9ce4b413
I20251029 02:51:58.027047 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0fec06ad37d4c12a7045d424b88cc5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.028911 989 leader_election.cc:304] T 012ce243fd07466ab5301d1c9ce4b413 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5; no voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d
I20251029 02:51:58.030107 524 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 84993b2f63e7421290a161dde4bb0607
W20251029 02:51:58.030320 712 leader_election.cc:343] T faa835de84ff45b19525d1d603678987 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:58.034634 886 tablet_service.cc:1552] Processing DeleteTablet for tablet 84993b2f63e7421290a161dde4bb0607 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:57 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:58.035877 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc802bda2f6a4e439b605f2cafa37967. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.038262 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_2e760e5f5f1a4009b5d7c227df182a1e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:58.038642 1628 raft_consensus.cc:2749] T 012ce243fd07466ab5301d1c9ce4b413 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 02:51:58.038919 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2e760e5f5f1a4009b5d7c227df182a1e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:58.040156 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_03569ce890a341d88f5ff48f32013ea0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:58.040776 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_03569ce890a341d88f5ff48f32013ea0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:58.042619 885 tablet_service.cc:1552] Processing DeleteTablet for tablet 49d9ce32e4f04f11be2ecefc5d523a8b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:57 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:58.042802 891 tablet_service.cc:1505] Processing CreateTablet for tablet 9492c1027aba4455ad8f539addaed5a7 (DEFAULT_TABLE table=default.Table_e8072943e31241f683bed508cb4f2029 [id=63e946b11bce40028e4c6f5c072bdd75]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.043191 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9492c1027aba4455ad8f539addaed5a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.044799 885 tablet_service.cc:1505] Processing CreateTablet for tablet c45ffc304e73490387ce800b702b5881 (DEFAULT_TABLE table=default.Table_8462690958454b098e0531b1086e60f7 [id=97e8edf0d4df452d96c9e13e55aa0736]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.045236 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c45ffc304e73490387ce800b702b5881. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.046440 1033 tablet_service.cc:1505] Processing CreateTablet for tablet 9492c1027aba4455ad8f539addaed5a7 (DEFAULT_TABLE table=default.Table_e8072943e31241f683bed508cb4f2029 [id=63e946b11bce40028e4c6f5c072bdd75]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.047096 526 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_a63991c194724381a84c0ce2adab25da" } new_table_name: "default.Table_4e576f3e5a1a4b99af528a20793e1956" modify_external_catalogs: true
I20251029 02:51:58.050803 1038 tablet_service.cc:1505] Processing CreateTablet for tablet c45ffc304e73490387ce800b702b5881 (DEFAULT_TABLE table=default.Table_8462690958454b098e0531b1086e60f7 [id=97e8edf0d4df452d96c9e13e55aa0736]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.052163 525 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet a5c3a15510cb4321bbb80d61fabd0c0a
I20251029 02:51:58.055512 886 tablet_service.cc:1552] Processing DeleteTablet for tablet a5c3a15510cb4321bbb80d61fabd0c0a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:57 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:58.057498 525 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet 59af906061d049c99ae2153945597007 (table default.Table_6da557988d044647b5e68f09b3f00407 [id=25f16d95843d41c981d6a07ec0d4b48a]). Expected version 2 got 0
W20251029 02:51:58.057780 990 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:41701, user_credentials={real_user=slave}} blocked reactor thread for 108354us
I20251029 02:51:58.060585 1039 tablet_service.cc:1552] Processing DeleteTablet for tablet 84993b2f63e7421290a161dde4bb0607 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:57 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:58.061455 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet 49d9ce32e4f04f11be2ecefc5d523a8b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:57 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:58.063261 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9492c1027aba4455ad8f539addaed5a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.065160 1030 tablet_service.cc:1505] Processing CreateTablet for tablet 786d9148fb1d417a8c2c8b8c53665899 (DEFAULT_TABLE table=default.Table_3d0a6e32dc914ba2b0263680b2587277 [id=0809df5782144d1da622b5726db932f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.065725 525 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet 24016ceaa84147eca0b94b3ad288e772 (table default.Table_f06003da6e324504ade193bbbb11d36a [id=bd279b3ce623477ca20e8d35e3b7c0df]). Expected version 1 got 0
I20251029 02:51:58.071463 525 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet 24016ceaa84147eca0b94b3ad288e772 (table default.Table_f06003da6e324504ade193bbbb11d36a [id=bd279b3ce623477ca20e8d35e3b7c0df]). Expected version 1 got 0
I20251029 02:51:58.071689 525 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet 59af906061d049c99ae2153945597007 (table default.Table_6da557988d044647b5e68f09b3f00407 [id=25f16d95843d41c981d6a07ec0d4b48a]). Expected version 2 got 0
I20251029 02:51:58.073701 886 tablet_service.cc:1505] Processing CreateTablet for tablet 2b278db5b2b04344b864797055a5f665 (DEFAULT_TABLE table=default.Table_73bc152b9f224b07ae2524e0e18f7df0 [id=61b689e02284411a9d7cd0795682791a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.075644 523 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet bfec87f690a640c39c58c64e16cc67e7
I20251029 02:51:58.077634 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b278db5b2b04344b864797055a5f665. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.080693 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_2d8b100ebfb8416a994557617d07af62"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:58.081038 888 tablet_service.cc:1552] Processing DeleteTablet for tablet bfec87f690a640c39c58c64e16cc67e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:57 UTC) from {username='slave'} at 127.0.0.1:35702
W20251029 02:51:58.081334 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2d8b100ebfb8416a994557617d07af62 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:58.082556 1039 tablet_service.cc:1552] Processing DeleteTablet for tablet a5c3a15510cb4321bbb80d61fabd0c0a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:57 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:58.083056 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_f05d33ecadba4313a2c9e46b27409196"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:58.083710 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f05d33ecadba4313a2c9e46b27409196 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:58.090458 888 tablet_service.cc:1505] Processing CreateTablet for tablet 688af6580c4d4637a39f817b4e324618 (DEFAULT_TABLE table=default.Table_4f9515ed04dc4e9e8480232c53fbac2e [id=0dc6f8e6205d4e3997946747bbc86e2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.090972 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 688af6580c4d4637a39f817b4e324618. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.094841 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet bfec87f690a640c39c58c64e16cc67e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:57 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:58.096150 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_114cbdf1ef0f493f8f8c834735271053"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:58.096810 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_114cbdf1ef0f493f8f8c834735271053 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:58.099138 1039 tablet_service.cc:1505] Processing CreateTablet for tablet 688af6580c4d4637a39f817b4e324618 (DEFAULT_TABLE table=default.Table_4f9515ed04dc4e9e8480232c53fbac2e [id=0dc6f8e6205d4e3997946747bbc86e2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.105996 887 tablet_service.cc:1505] Processing CreateTablet for tablet 7fb5b908db77453da5cfb32e2c955f41 (DEFAULT_TABLE table=default.Table_163ca13775fd4ae28ec7ead8297929c8 [id=09a60dfda8994fb0b36f3b4a55e58d09]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.106514 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7fb5b908db77453da5cfb32e2c955f41. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.108721 898 tablet_service.cc:1505] Processing CreateTablet for tablet e3000183230e4350b47f5085952f2db6 (DEFAULT_TABLE table=default.Table_5069071caedd4971a57a438fba4fac4f [id=79d22e7fddb24e68a0db1aeacbdfb6e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.109102 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3000183230e4350b47f5085952f2db6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.109455 1040 tablet_service.cc:1505] Processing CreateTablet for tablet 2b278db5b2b04344b864797055a5f665 (DEFAULT_TABLE table=default.Table_73bc152b9f224b07ae2524e0e18f7df0 [id=61b689e02284411a9d7cd0795682791a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.113349 522 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet cc802bda2f6a4e439b605f2cafa37967
W20251029 02:51:58.114215 518 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 97b2ecae572a4b7ea1b9595074556708 consensus state has no leader
W20251029 02:51:58.115538 526 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c1222e30bc4141b3ae686d80053c4ef5 consensus state has no leader
W20251029 02:51:58.116433 519 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b3de6670357c4ae59eb77469b2fb434c consensus state has no leader
I20251029 02:51:58.117327 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c45ffc304e73490387ce800b702b5881. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.119329 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_5c0d5b3233fe48f3a817f437c161967c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:58.120141 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5c0d5b3233fe48f3a817f437c161967c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:58.121418 884 tablet_service.cc:1552] Processing DeleteTablet for tablet cc802bda2f6a4e439b605f2cafa37967 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:57 UTC) from {username='slave'} at 127.0.0.1:35702
W20251029 02:51:58.121701 990 leader_election.cc:343] T aa776cde2f494fa5810539edf9f289a1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:58.122131 884 tablet_service.cc:1505] Processing CreateTablet for tablet 08371b3daa184a0e9f4eb3d313b52c56 (DEFAULT_TABLE table=default.Table_736690fbc69640d9be26e1662e3fa63e [id=40f1beecaad24a64b4348663c9ae9afd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.122509 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08371b3daa184a0e9f4eb3d313b52c56. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.123688 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 786d9148fb1d417a8c2c8b8c53665899. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.123932 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b278db5b2b04344b864797055a5f665. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.127337 899 tablet_service.cc:1505] Processing CreateTablet for tablet 294afb9b012c40c0b05b73795d6f0ac8 (DEFAULT_TABLE table=default.Table_4619a2287cb14edbb32bcbec581eb4d9 [id=6213d8ae6fb943568919256d7922513f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.128024 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 294afb9b012c40c0b05b73795d6f0ac8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.129626 1166 tablet_bootstrap.cc:492] T d2990e9f626c40618c118266bc595684 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:51:58.135016 526 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_528aa3f8d5e84754bc56f86ca7a480aa" } new_table_name: "default.Table_755825b17834457cb94b05163ae3e26d" modify_external_catalogs: true
I20251029 02:51:58.142647 882 tablet_service.cc:1505] Processing CreateTablet for tablet 023a4094b424464c8f49d6eb9cc15da4 (DEFAULT_TABLE table=default.Table_c020d31b6d054efaad87057733e683b6 [id=b00ec70f267a4fff98f12cc029eb3e04]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.142838 990 leader_election.cc:304] T aa776cde2f494fa5810539edf9f289a1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5; no voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d
I20251029 02:51:58.143126 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 023a4094b424464c8f49d6eb9cc15da4. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:51:58.149708 990 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:41701, user_credentials={real_user=slave}} blocked reactor thread for 91480.5us
I20251029 02:51:58.149843 1465 raft_consensus.cc:2749] T aa776cde2f494fa5810539edf9f289a1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:51:58.149992 899 tablet_service.cc:1505] Processing CreateTablet for tablet ff4c38150df04a75855e0e60ff417ddb (DEFAULT_TABLE table=default.Table_040794c7b9064642a98d202e77a60ac5 [id=b4e3ba45194b4fcd8f14d1a8cb4365dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.150396 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff4c38150df04a75855e0e60ff417ddb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.155107 882 tablet_service.cc:1505] Processing CreateTablet for tablet da904606986f452cadd06187fff06785 (DEFAULT_TABLE table=default.Table_a4fcd46aef6146e986d4b8d3599d66f4 [id=24f8ffe28d81427d9e9c5beb9c938d02]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.156119 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da904606986f452cadd06187fff06785. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.156509 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 688af6580c4d4637a39f817b4e324618. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.158552 1024 tablet_service.cc:1505] Processing CreateTablet for tablet c7b78fd458e4414b9fc3aa064a80b206 (DEFAULT_TABLE table=default.Table_5876d3fefb8144e2a1a76782912ab03b [id=9b6e3e9eef094d9bb64fb22f512bb3d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.158907 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7b78fd458e4414b9fc3aa064a80b206. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.161612 1034 tablet_service.cc:1505] Processing CreateTablet for tablet e3000183230e4350b47f5085952f2db6 (DEFAULT_TABLE table=default.Table_5069071caedd4971a57a438fba4fac4f [id=79d22e7fddb24e68a0db1aeacbdfb6e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.165513 1035 tablet_service.cc:1505] Processing CreateTablet for tablet f65fe9a5137b48669afef51ad7dd76ae (DEFAULT_TABLE table=default.Table_9c1124b16dc44ec5b2696d567f1c52f0 [id=c60923f7053642eb95881163d039644f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.165956 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_fd6ca61b22e442fabd966ea848f4389e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:58.165966 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_3d101961029143a882eeafb119560e5d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:58.166572 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fd6ca61b22e442fabd966ea848f4389e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:51:58.166690 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3d101961029143a882eeafb119560e5d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:58.167136 1027 tablet_service.cc:1505] Processing CreateTablet for tablet 15137fc1c70347b381ffdfe5653a5f22 (DEFAULT_TABLE table=default.Table_fdcb71c680504418a7f89d81feb372b0 [id=4bacb12c51d34f78bf4b1aba964c38ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.170035 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_a15f5c838c7d4600b91a7368321fff38"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:58.170624 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a15f5c838c7d4600b91a7368321fff38 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:58.171319 1032 tablet_service.cc:1505] Processing CreateTablet for tablet 7fb5b908db77453da5cfb32e2c955f41 (DEFAULT_TABLE table=default.Table_163ca13775fd4ae28ec7ead8297929c8 [id=09a60dfda8994fb0b36f3b4a55e58d09]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.175748 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3000183230e4350b47f5085952f2db6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.178665 1042 tablet_service.cc:1505] Processing CreateTablet for tablet 023a4094b424464c8f49d6eb9cc15da4 (DEFAULT_TABLE table=default.Table_c020d31b6d054efaad87057733e683b6 [id=b00ec70f267a4fff98f12cc029eb3e04]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 02:51:58.183802 711 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.596s user 0.011s sys 0.025s
W20251029 02:51:58.184370 711 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.597s user 0.011s sys 0.025s
W20251029 02:51:58.184772 711 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:41701, user_credentials={real_user=slave}} blocked reactor thread for 597086us
I20251029 02:51:58.185467 753 tablet_service.cc:1505] Processing CreateTablet for tablet cd597c319af645909b8f3d1212fcb702 (DEFAULT_TABLE table=default.Table_d9cfb3211dcd4bb295955b8750600b5d [id=e4c9177afbae4d00bc98fe741494dbdd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.185896 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd597c319af645909b8f3d1212fcb702. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.187492 892 tablet_service.cc:1505] Processing CreateTablet for tablet e3ac6597d52441569a7aeef051087ad2 (DEFAULT_TABLE table=default.Table_c5b59d221b994982a9867326266b7bcd [id=242159edd3bc4771ac206d4e450b9475]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.189034 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3ac6597d52441569a7aeef051087ad2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.191817 900 tablet_service.cc:1505] Processing CreateTablet for tablet c7b78fd458e4414b9fc3aa064a80b206 (DEFAULT_TABLE table=default.Table_5876d3fefb8144e2a1a76782912ab03b [id=9b6e3e9eef094d9bb64fb22f512bb3d7]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 02:51:58.192150 526 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cc74e83833754f0ca19046031a24ebb9 consensus state has no leader
I20251029 02:51:58.197036 884 tablet_service.cc:1505] Processing CreateTablet for tablet 15137fc1c70347b381ffdfe5653a5f22 (DEFAULT_TABLE table=default.Table_fdcb71c680504418a7f89d81feb372b0 [id=4bacb12c51d34f78bf4b1aba964c38ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.197690 887 tablet_service.cc:1505] Processing CreateTablet for tablet f65fe9a5137b48669afef51ad7dd76ae (DEFAULT_TABLE table=default.Table_9c1124b16dc44ec5b2696d567f1c52f0 [id=c60923f7053642eb95881163d039644f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.197681 1025 tablet_service.cc:1505] Processing CreateTablet for tablet ff4c38150df04a75855e0e60ff417ddb (DEFAULT_TABLE table=default.Table_040794c7b9064642a98d202e77a60ac5 [id=b4e3ba45194b4fcd8f14d1a8cb4365dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.207031 1445 raft_consensus.cc:493] T 6feec2c8e030447fae9c292a626dcd1d P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:58.207258 1445 raft_consensus.cc:515] T 6feec2c8e030447fae9c292a626dcd1d P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:58.208091 1445 leader_election.cc:290] T 6feec2c8e030447fae9c292a626dcd1d P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:51:58.209676 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_e5a5de26cf4041fca4bfbc384115b9b3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:58.210232 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e5a5de26cf4041fca4bfbc384115b9b3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:58.211457 751 tablet_service.cc:1505] Processing CreateTablet for tablet 84993b2f63e7421290a161dde4bb0607 (DEFAULT_TABLE table=default.Table_93b405a4f257481289518c5ff28832a0 [id=da230f99a9894ab8aa2fdb4b167169ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.211925 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84993b2f63e7421290a161dde4bb0607. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.214476 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7b78fd458e4414b9fc3aa064a80b206. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.217543 898 tablet_service.cc:1505] Processing CreateTablet for tablet 68b1c4f88aee4e4795397d1c796ff948 (DEFAULT_TABLE table=default.Table_2ed982f44d384784b875b8a49394305e [id=f2d457ba2bcf44b5bcd44dea5e2aa299]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.219636 886 tablet_service.cc:1505] Processing CreateTablet for tablet 35d9b8dfd61e4d8db441427a49fb9dca (DEFAULT_TABLE table=default.Table_73eec04729fd4870aca2779a5738d772 [id=b54d7930e083434680f3934fc4023653]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.220718 518 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet 59af906061d049c99ae2153945597007 (table default.Table_6da557988d044647b5e68f09b3f00407 [id=25f16d95843d41c981d6a07ec0d4b48a]). Expected version 2 got 0
I20251029 02:51:58.221573 519 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_0f40dc35be6d46b3a991e93384fa1e1f" } modify_external_catalogs: true
I20251029 02:51:58.221794 519 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_0f40dc35be6d46b3a991e93384fa1e1f" } modify_external_catalogs: true
I20251029 02:51:58.222590 753 tablet_service.cc:1505] Processing CreateTablet for tablet defb795026a44e12afb2defb9c9c6db2 (DEFAULT_TABLE table=default.Table_eae5a69444004b54bf8bf18ea29fe73f [id=499cbddf4ef64aa28594d85252774611]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.222960 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet defb795026a44e12afb2defb9c9c6db2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.230340 892 tablet_service.cc:1505] Processing CreateTablet for tablet 877e278deb4a4b01ac66b06be0890baf (DEFAULT_TABLE table=default.Table_a3377614e4a54266a4457944d48b0060 [id=991a151cc5d84421bf50858b1fa864e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.235635 1028 tablet_service.cc:1505] Processing CreateTablet for tablet 294afb9b012c40c0b05b73795d6f0ac8 (DEFAULT_TABLE table=default.Table_4619a2287cb14edbb32bcbec581eb4d9 [id=6213d8ae6fb943568919256d7922513f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.236186 1040 tablet_service.cc:1505] Processing CreateTablet for tablet 08371b3daa184a0e9f4eb3d313b52c56 (DEFAULT_TABLE table=default.Table_736690fbc69640d9be26e1662e3fa63e [id=40f1beecaad24a64b4348663c9ae9afd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.236641 1026 tablet_service.cc:1552] Processing DeleteTablet for tablet cc802bda2f6a4e439b605f2cafa37967 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:57 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:58.236778 523 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_435f2dfe42134760abc9e8880fa19f32" } modify_external_catalogs: true
I20251029 02:51:58.237190 1026 tablet_service.cc:1505] Processing CreateTablet for tablet e3ac6597d52441569a7aeef051087ad2 (DEFAULT_TABLE table=default.Table_c5b59d221b994982a9867326266b7bcd [id=242159edd3bc4771ac206d4e450b9475]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.237639 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15137fc1c70347b381ffdfe5653a5f22. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.238835 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08371b3daa184a0e9f4eb3d313b52c56. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.239459 523 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_435f2dfe42134760abc9e8880fa19f32" } modify_external_catalogs: true
I20251029 02:51:58.240037 1166 ts_tablet_manager.cc:1403] T d2990e9f626c40618c118266bc595684 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.238s user 0.007s sys 0.004s
I20251029 02:51:58.243626 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 294afb9b012c40c0b05b73795d6f0ac8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.247236 743 tablet_service.cc:1552] Processing DeleteTablet for tablet faa835de84ff45b19525d1d603678987 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:57 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:58.247908 743 tablet_service.cc:1552] Processing DeleteTablet for tablet c32c28b0e74a4af3a0e69bbf24ab1a1f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:57 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:58.248633 743 tablet_service.cc:1505] Processing CreateTablet for tablet c081d2eacfac4ca8a0048e2a1c4f8f83 (DEFAULT_TABLE table=default.Table_dbf2e434d7964eac80709c205f61191e [id=b6afde5f860040be9ede53221978f44d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.249104 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c081d2eacfac4ca8a0048e2a1c4f8f83. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.250710 1174 ts_tablet_manager.cc:1929] T 5e31e061d74146009d24d5cd57487a5e P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251029 02:51:58.250873 1174 log.cc:1199] T 5e31e061d74146009d24d5cd57487a5e P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/5e31e061d74146009d24d5cd57487a5e
I20251029 02:51:58.256764 518 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_2d8b100ebfb8416a994557617d07af62" } modify_external_catalogs: true
I20251029 02:51:58.257001 518 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_2d8b100ebfb8416a994557617d07af62" } modify_external_catalogs: true
I20251029 02:51:58.257546 751 tablet_service.cc:1552] Processing DeleteTablet for tablet 589a521f5cc24e6289c37b9cf256bce5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:57 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:57.907666 1175 ts_tablet_manager.cc:1929] T 3362e95063af41c8a24e935f547e3089 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:58.258054 1175 log.cc:1199] T 3362e95063af41c8a24e935f547e3089 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/3362e95063af41c8a24e935f547e3089
I20251029 02:51:58.258474 1175 ts_tablet_manager.cc:1950] T 3362e95063af41c8a24e935f547e3089 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:51:58.259011 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 332) took 1591 ms. Trace:
I20251029 02:51:58.259093 1175 rpcz_store.cc:276] 1029 02:51:56.667220 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:56.667343 (+ 123us) service_pool.cc:225] Handling call
1029 02:51:57.795334 (+1127991us) ts_tablet_manager.cc:1151] Deleting tablet 3362e95063af41c8a24e935f547e3089
1029 02:51:57.795539 (+ 205us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:57.856635 (+ 61096us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:57.906184 (+ 49549us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:58.258956 (+352772us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1127573}
I20251029 02:51:58.259455 1175 tablet_replica.cc:333] T c65469173762446e9725262ae998dbf3 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:51:58.259574 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15137fc1c70347b381ffdfe5653a5f22. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.259644 1175 raft_consensus.cc:2243] T c65469173762446e9725262ae998dbf3 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:58.259824 1175 raft_consensus.cc:2272] T c65469173762446e9725262ae998dbf3 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:58.260366 1175 ts_tablet_manager.cc:1916] T c65469173762446e9725262ae998dbf3 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:58.260706 881 tablet_service.cc:1505] Processing CreateTablet for tablet c64d6761ef924ccaa9c2579f8ba47e33 (DEFAULT_TABLE table=default.Table_44b56ca77fed49489c18e54dffb4b108 [id=86861bd15ecd481a8579fb71ee4b9ef1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.261054 890 tablet_service.cc:1505] Processing CreateTablet for tablet 3a2b95997454452bb985c51fc02d77e2 (DEFAULT_TABLE table=default.Table_30a991be88ad4c199ac8ce370ee76f17 [id=0c3d955bdfd34db2ac137dd6f5d84a2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.261438 885 tablet_service.cc:1505] Processing CreateTablet for tablet d1866d6dcb0349bba5db86163375cada (DEFAULT_TABLE table=default.Table_e0a153971f6f4af98ee02f8bbc615620 [id=9bf3236a9242474b9f0c75b3e2f3d02d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.262001 888 tablet_service.cc:1505] Processing CreateTablet for tablet dc33fd3404ec41c3935b02c84a12971b (DEFAULT_TABLE table=default.Table_78b81f8805aa49b8a07a940c8d886de5 [id=09ee821707fa4082889bfccb8667b0b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.262629 1174 ts_tablet_manager.cc:1950] T 5e31e061d74146009d24d5cd57487a5e P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:51:58.264189 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 312) took 2077 ms. Trace:
I20251029 02:51:58.264387 1174 rpcz_store.cc:276] 1029 02:51:56.186437 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:56.186501 (+ 64us) service_pool.cc:225] Handling call
1029 02:51:57.953811 (+1767310us) ts_tablet_manager.cc:1151] Deleting tablet 5e31e061d74146009d24d5cd57487a5e
1029 02:51:57.953827 (+ 16us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:58.139448 (+185621us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:58.192961 (+ 53513us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:58.264125 (+ 71164us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1767004}
I20251029 02:51:58.264928 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68b1c4f88aee4e4795397d1c796ff948. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.265210 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c64d6761ef924ccaa9c2579f8ba47e33. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:51:58.265235 990 consensus_peers.cc:597] T aacb6f4b9284457bb6607e3a85d5c809 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 -> Peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Couldn't send request to peer c756528642a543eeacd93cef3e335a7d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:51:58.266528 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a2b95997454452bb985c51fc02d77e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.267267 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f65fe9a5137b48669afef51ad7dd76ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.267887 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 877e278deb4a4b01ac66b06be0890baf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.268504 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35d9b8dfd61e4d8db441427a49fb9dca. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.268692 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3ac6597d52441569a7aeef051087ad2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.269168 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1866d6dcb0349bba5db86163375cada. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.269593 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff4c38150df04a75855e0e60ff417ddb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.270211 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7fb5b908db77453da5cfb32e2c955f41. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.275991 1029 tablet_service.cc:1505] Processing CreateTablet for tablet da904606986f452cadd06187fff06785 (DEFAULT_TABLE table=default.Table_a4fcd46aef6146e986d4b8d3599d66f4 [id=24f8ffe28d81427d9e9c5beb9c938d02]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.276445 519 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 061f3d38c4604fa4b57e74ff6f6fc7da
I20251029 02:51:58.276551 895 tablet_service.cc:1505] Processing CreateTablet for tablet 7e9d7ee2601640d88c0ddd0603e8a6e5 (DEFAULT_TABLE table=default.Table_5c0d5b3233fe48f3a817f437c161967c [id=e895f8b2327f4b838540d72e60fc06d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.277808 891 tablet_service.cc:1552] Processing DeleteTablet for tablet 061f3d38c4604fa4b57e74ff6f6fc7da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:58 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:58.279748 519 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_2e760e5f5f1a4009b5d7c227df182a1e" } modify_external_catalogs: true
I20251029 02:51:58.279874 891 tablet_service.cc:1505] Processing CreateTablet for tablet f18b0ca5a6a44175a912642e9eea1778 (DEFAULT_TABLE table=default.Table_f05d33ecadba4313a2c9e46b27409196 [id=4cf725c11fbb4ab3981000b3a41e8496]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.279955 519 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_2e760e5f5f1a4009b5d7c227df182a1e" } modify_external_catalogs: true
I20251029 02:51:58.280782 523 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 30d03e1e42d34242bd354f9e9ec4f127
I20251029 02:51:58.281849 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f65fe9a5137b48669afef51ad7dd76ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.282024 879 tablet_service.cc:1552] Processing DeleteTablet for tablet 30d03e1e42d34242bd354f9e9ec4f127 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:58 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:58.282444 518 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 897a9a75d13f493e90526da4ca7f54d2
I20251029 02:51:58.285686 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc33fd3404ec41c3935b02c84a12971b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.286152 879 tablet_service.cc:1552] Processing DeleteTablet for tablet 897a9a75d13f493e90526da4ca7f54d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:58 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:58.287743 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e9d7ee2601640d88c0ddd0603e8a6e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.289332 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f18b0ca5a6a44175a912642e9eea1778. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.289831 519 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet aee42fd9ca894319b3759beb2bae342e
I20251029 02:51:58.290138 879 tablet_service.cc:1505] Processing CreateTablet for tablet 897a9a75d13f493e90526da4ca7f54d2 (DEFAULT_TABLE table=default.Table_2d8b100ebfb8416a994557617d07af62 [id=1f176d1a5f044322b9a693421a43ed5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.291373 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 897a9a75d13f493e90526da4ca7f54d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.292124 1172 tablet_bootstrap.cc:492] T a5c3a15510cb4321bbb80d61fabd0c0a P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:51:58.292292 1172 ts_tablet_manager.cc:1403] T a5c3a15510cb4321bbb80d61fabd0c0a P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.426s user 0.003s sys 0.003s
I20251029 02:51:58.294183 739 tablet_service.cc:1505] Processing CreateTablet for tablet fcb3db8399aa4cf188b07dda220f7f5b (DEFAULT_TABLE table=default.Table_33eacbf00740465185d70b46a3782893 [id=22e43c4483ae488b81fb94a7d221bdce]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.294651 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fcb3db8399aa4cf188b07dda220f7f5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.295329 893 tablet_service.cc:1552] Processing DeleteTablet for tablet aee42fd9ca894319b3759beb2bae342e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:58 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:58.295477 894 tablet_service.cc:1505] Processing CreateTablet for tablet 2b0988f736a7402997ca3ba323a59e18 (DEFAULT_TABLE table=default.Table_03569ce890a341d88f5ff48f32013ea0 [id=dbb0cf4e24ca4fe1b989c9795dc4da82]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.295941 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b0988f736a7402997ca3ba323a59e18. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.297472 1172 raft_consensus.cc:359] T a5c3a15510cb4321bbb80d61fabd0c0a P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:58.297807 1172 raft_consensus.cc:385] T a5c3a15510cb4321bbb80d61fabd0c0a P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:58.297953 1172 raft_consensus.cc:740] T a5c3a15510cb4321bbb80d61fabd0c0a P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:51:58.298406 1172 consensus_queue.cc:260] T a5c3a15510cb4321bbb80d61fabd0c0a P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:58.300653 893 tablet_service.cc:1505] Processing CreateTablet for tablet 38f83ebcb2a440bc8e6a8d36607e6666 (DEFAULT_TABLE table=default.Table_86ad8c0a1e9c4fd6b12b9f59a195e951 [id=d3cffce7aefa4c028f8e369c875fc19f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.300886 519 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_30a991be88ad4c199ac8ce370ee76f17" } new_table_name: "default.Table_10d0ed02cd464576877b041b08d9af61" modify_external_catalogs: true
I20251029 02:51:58.301354 751 tablet_service.cc:1552] Processing DeleteTablet for tablet 4aa78f4ac38e4919864c924e50f1aaf4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:57 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:58.301399 878 tablet_service.cc:1505] Processing CreateTablet for tablet d24fe99a1fb54476b90da2c22ff5d383 (DEFAULT_TABLE table=default.Table_e280ab7143204fbd8daa4f2c898617f9 [id=3c055a359301472cac54dd72a37520c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.301858 883 tablet_service.cc:1505] Processing CreateTablet for tablet e74eda48dca647efacabe525ef316944 (DEFAULT_TABLE table=default.Table_6e1283a0c161498db405cf1e9b2c93d7 [id=4de1990274fe4b8487506a86872fe976]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.301899 743 tablet_service.cc:1505] Processing CreateTablet for tablet f4729ecd182042c2a10f67ca25224101 (DEFAULT_TABLE table=default.Table_663bb3557fe64d8092e0a19dfa14d1b4 [id=1dd2d5488bc647c8b82c53886583b28d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.302235 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4729ecd182042c2a10f67ca25224101. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.302850 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38f83ebcb2a440bc8e6a8d36607e6666. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.305423 1172 ts_tablet_manager.cc:1434] T a5c3a15510cb4321bbb80d61fabd0c0a P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.013s user 0.001s sys 0.002s
I20251029 02:51:58.306974 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e74eda48dca647efacabe525ef316944. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:51:58.307904 519 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3a2b95997454452bb985c51fc02d77e2 consensus state has no leader
I20251029 02:51:58.308292 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d24fe99a1fb54476b90da2c22ff5d383. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.310621 1628 consensus_queue.cc:1048] T aacb6f4b9284457bb6607e3a85d5c809 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.026s
I20251029 02:51:58.311350 896 tablet_service.cc:1505] Processing CreateTablet for tablet c172417d1d444c439e9cda611a2ed551 (DEFAULT_TABLE table=default.Table_293c6e92af8a4707be8e2596399fd6a2 [id=d7a06dc089ab4371813f376ef37261ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.312675 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da904606986f452cadd06187fff06785. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.312692 1172 tablet_bootstrap.cc:492] T 1fc87e2d7b4240e08f2b4ac7277293ff P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:51:58.314247 746 tablet_service.cc:1505] Processing CreateTablet for tablet 628a7d0419cb434f8171faf500d585d8 (DEFAULT_TABLE table=default.Table_31aa9d49c83b4ab29823acea7480bc96 [id=d6b287a41f2c46868c3f207b7d4359ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.314678 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 628a7d0419cb434f8171faf500d585d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.315326 1172 tablet_bootstrap.cc:654] T 1fc87e2d7b4240e08f2b4ac7277293ff P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:58.316275 519 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_19039883faf6464387f65003d0d638a3" } modify_external_catalogs: true
I20251029 02:51:58.316475 519 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_19039883faf6464387f65003d0d638a3" } modify_external_catalogs: true
I20251029 02:51:58.316572 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c172417d1d444c439e9cda611a2ed551. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.319741 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_dbf30facf56548e1b20d6fec42080e05"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:58.319866 525 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_b276bd83a0514185a89941ada499686c" } new_table_name: "default.Table_47c159158e26411c934a19f853eb19e9" modify_external_catalogs: true
I20251029 02:51:58.320101 518 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_ed30c800e4eb4970a4b03299c936e7ed" } new_table_name: "default.Table_2611211ae1424e22a985b944e78f4843" modify_external_catalogs: true
W20251029 02:51:58.320287 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dbf30facf56548e1b20d6fec42080e05 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:58.321405 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_2474d13afb604655864254dd55221051"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:58.321966 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2474d13afb604655864254dd55221051 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:58.324472 751 tablet_service.cc:1505] Processing CreateTablet for tablet 055184c641ba45a588bca611fb24d130 (DEFAULT_TABLE table=default.Table_757229f7dede492d8da98328eaabc0dd [id=aec3db766dee438c89fb2f1939ee7b7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.324921 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 055184c641ba45a588bca611fb24d130. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.321431 520 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_dc394a25546b48d2a96a1d97091c12f4" } new_table_name: "default.Table_455fef4acf064ba799a55df3d87a94ad" modify_external_catalogs: true
I20251029 02:51:58.330314 739 tablet_service.cc:1505] Processing CreateTablet for tablet 5815626b05204b578cc4a110f246bf10 (DEFAULT_TABLE table=default.Table_4b9147e899c043ad9023bf0d8d555a0c [id=45dc803b4e1041e78f40a3e2e1369041]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.330780 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5815626b05204b578cc4a110f246bf10. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.339572 1176 ts_tablet_manager.cc:1929] T d6e1170dbb594a5696a3899cde8da7c9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:58.340049 1041 tablet_service.cc:1505] Processing CreateTablet for tablet d1866d6dcb0349bba5db86163375cada (DEFAULT_TABLE table=default.Table_e0a153971f6f4af98ee02f8bbc615620 [id=9bf3236a9242474b9f0c75b3e2f3d02d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.340373 1062 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6feec2c8e030447fae9c292a626dcd1d" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:51:58.341043 518 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 217144957930495aa3e9dfcfde268e71 consensus state has no leader
W20251029 02:51:58.342603 525 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6dff036b32904ddb9598be4b79504bb6 consensus state has no leader
W20251029 02:51:58.343019 520 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a9e0fcc759a64679b3f59a6c964877b6 consensus state has no leader
I20251029 02:51:58.347946 519 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 5ae5f33567324d90b7a5a6e0f1b5fb67
I20251029 02:51:58.349592 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 023a4094b424464c8f49d6eb9cc15da4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.373698 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 314) took 2159 ms. Trace:
I20251029 02:51:58.413094 424 master_runner.cc:186] Error getting master registration for 127.0.104.60:41233: OK, instance_id { permanent_uuid: "e5e5222b26fe4145948620e51edf53fb" instance_seqno: 1761706316385283 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20251029 02:51:58.465368 424 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251029 02:51:58.352018 1176 log.cc:1199] T d6e1170dbb594a5696a3899cde8da7c9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/d6e1170dbb594a5696a3899cde8da7c9
I20251029 02:51:58.465919 1176 ts_tablet_manager.cc:1950] T d6e1170dbb594a5696a3899cde8da7c9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:51:58.466310 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 336) took 2065 ms. Trace:
W20251029 02:51:58.398470 1585 thread.cc:641] auto-rebalancer (catalog manager) Time spent creating pthread: real 0.658s user 0.041s sys 0.099s
W20251029 02:51:58.466673 1585 thread.cc:608] auto-rebalancer (catalog manager) Time spent starting thread: real 0.726s user 0.041s sys 0.099s
I20251029 02:51:58.466485 1176 rpcz_store.cc:276] 1029 02:51:56.401128 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:56.401180 (+ 52us) service_pool.cc:225] Handling call
1029 02:51:57.539324 (+1138144us) ts_tablet_manager.cc:1151] Deleting tablet d6e1170dbb594a5696a3899cde8da7c9
1029 02:51:57.539450 (+ 126us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:58.079887 (+540437us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:58.281691 (+201804us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:58.466270 (+184579us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":155698,"tablet-delete.queue_time_us":1137919}
I20251029 02:51:58.467108 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 337) took 2037 ms. Trace:
I20251029 02:51:58.467221 1176 rpcz_store.cc:276] 1029 02:51:56.429227 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:56.429290 (+ 63us) service_pool.cc:225] Handling call
1029 02:51:58.466943 (+2037653us) ts_tablet_manager.cc:1151] Deleting tablet dd6ad641fbda472fab07aa35c5e4523a
1029 02:51:58.466956 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:58.467076 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2037364}
I20251029 02:51:58.467674 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 338) took 2007 ms. Trace:
I20251029 02:51:58.376657 881 tablet_service.cc:1505] Processing CreateTablet for tablet 5ef87f7a3dff4f00a0e7273eaec98dff (DEFAULT_TABLE table=default.Table_fd1d23f31ebf47b884897c68eac9004d [id=1c99ebd287ff47d08e817853a628256c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.467789 1176 rpcz_store.cc:276] 1029 02:51:56.459939 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:56.460005 (+ 66us) service_pool.cc:225] Handling call
1029 02:51:58.467487 (+2007482us) ts_tablet_manager.cc:1151] Deleting tablet daa05a34c0684c07a35966290fd7d44c
1029 02:51:58.467497 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:58.467630 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2007243}
I20251029 02:51:58.468155 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 339) took 1953 ms. Trace:
I20251029 02:51:58.353228 755 tablet_service.cc:1505] Processing CreateTablet for tablet f93b0c39d7174dabbe018b1546dbc198 (DEFAULT_TABLE table=default.Table_559b8d2aa7c04ba68f184405d44898d6 [id=61fac9a7d5b049e08485ce8695ba5a48]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.353844 748 tablet_service.cc:1505] Processing CreateTablet for tablet 24f07b1efac64b099b0b5ed467463749 (DEFAULT_TABLE table=default.Table_21b8ae2debb241bda65fe387a4a3190f [id=f3fa2dbbef7e47359492d5a5eef9b8fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.354346 746 tablet_service.cc:1505] Processing CreateTablet for tablet 506f8c112a3b43e1a3d6bbafb8cb81e8 (DEFAULT_TABLE table=default.Table_e8e977ce21ca40d4a22d0434d73ef70b [id=9107fd53f5c94d2d80be50ef1f7c8e9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.468274 1176 rpcz_store.cc:276] 1029 02:51:56.514285 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:56.514345 (+ 60us) service_pool.cc:225] Handling call
1029 02:51:58.468030 (+1953685us) ts_tablet_manager.cc:1151] Deleting tablet f187dbd8a55b45b292ea61db43caf47d
1029 02:51:58.468040 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:58.468124 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1953390}
I20251029 02:51:58.354751 738 tablet_service.cc:1505] Processing CreateTablet for tablet b0fec06ad37d4c12a7045d424b88cc5e (DEFAULT_TABLE table=default.Table_c4333d8c61d346b1ac1a809d3ccb1915 [id=6d576d3499e649d78000c31fc5dab58c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.355376 744 tablet_service.cc:1505] Processing CreateTablet for tablet abf18e06df9b482a8b25c078b9495272 (DEFAULT_TABLE table=default.Table_19165ff9f7f24a97bf835e72701455d3 [id=ae6032816e6c4b80aa294478cbd49460]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.356108 756 tablet_service.cc:1552] Processing DeleteTablet for tablet 2f198c3259864f7ab3625cb7bfae20b0 with delete_type TABLET_DATA_DELETED (replaced by tablet 877e278deb4a4b01ac66b06be0890baf) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:58.414353 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1866d6dcb0349bba5db86163375cada. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.468724 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f93b0c39d7174dabbe018b1546dbc198. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.468951 756 tablet_service.cc:1552] Processing DeleteTablet for tablet 84993b2f63e7421290a161dde4bb0607 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:57 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:58.469398 756 tablet_service.cc:1505] Processing CreateTablet for tablet c45ffc304e73490387ce800b702b5881 (DEFAULT_TABLE table=default.Table_8462690958454b098e0531b1086e60f7 [id=97e8edf0d4df452d96c9e13e55aa0736]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.356755 747 tablet_service.cc:1505] Processing CreateTablet for tablet 7816556521144c928d7a02694e22e79f (DEFAULT_TABLE table=default.Table_0b95eba70e97459eb847fecfac513686 [id=67fc9a53c89144399f905c2f48800e17]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.470479 1033 tablet_service.cc:1505] Processing CreateTablet for tablet 877e278deb4a4b01ac66b06be0890baf (DEFAULT_TABLE table=default.Table_a3377614e4a54266a4457944d48b0060 [id=991a151cc5d84421bf50858b1fa864e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.414808 1037 tablet_service.cc:1505] Processing CreateTablet for tablet 68b1c4f88aee4e4795397d1c796ff948 (DEFAULT_TABLE table=default.Table_2ed982f44d384784b875b8a49394305e [id=f2d457ba2bcf44b5bcd44dea5e2aa299]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.418426 1166 raft_consensus.cc:359] T d2990e9f626c40618c118266bc595684 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:58.418799 1036 tablet_service.cc:1505] Processing CreateTablet for tablet 35d9b8dfd61e4d8db441427a49fb9dca (DEFAULT_TABLE table=default.Table_73eec04729fd4870aca2779a5738d772 [id=b54d7930e083434680f3934fc4023653]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.473309 1166 raft_consensus.cc:385] T d2990e9f626c40618c118266bc595684 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:58.473449 1166 raft_consensus.cc:740] T d2990e9f626c40618c118266bc595684 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:51:58.362138 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 5e31e061d74146009d24d5cd57487a5e (table default.Table_f2bc74933428449197f8c28182c8698f [id=8e17d6b87b984d878787802bd6f627f2]) successfully deleted
W20251029 02:51:58.474673 512 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.2:41701, user_credentials={real_user=slave}} blocked reactor thread for 112570us
W20251029 02:51:58.402920 1514 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.806s user 0.000s sys 0.046s
W20251029 02:51:58.476751 1514 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.879s user 0.000s sys 0.046s
I20251029 02:51:58.477120 1166 consensus_queue.cc:260] T d2990e9f626c40618c118266bc595684 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:58.477753 1166 ts_tablet_manager.cc:1434] T d2990e9f626c40618c118266bc595684 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.235s user 0.003s sys 0.000s
I20251029 02:51:58.389961 1167 tablet_bootstrap.cc:492] T d6e1170dbb594a5696a3899cde8da7c9 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:51:58.355537 1172 tablet_bootstrap.cc:492] T 1fc87e2d7b4240e08f2b4ac7277293ff P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:51:58.357403 745 tablet_service.cc:1505] Processing CreateTablet for tablet b125cc9aaf4248b8bfa48fb152b832a6 (DEFAULT_TABLE table=default.Table_b22826191d2448e780eb6865d1cd7d05 [id=3513c69f6b3941849ac5b8ae0a5846d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.358065 752 tablet_service.cc:1505] Processing CreateTablet for tablet 1df0d944717d487a98ea83a8ec04bd46 (DEFAULT_TABLE table=default.Table_d477b4de8ad4449aa5d53db48d45273f [id=513d017e315a49d0a0fbe1fb12649f82]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.478943 1172 ts_tablet_manager.cc:1403] T 1fc87e2d7b4240e08f2b4ac7277293ff P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.166s user 0.008s sys 0.007s
I20251029 02:51:58.358734 740 tablet_service.cc:1505] Processing CreateTablet for tablet cc802bda2f6a4e439b605f2cafa37967 (DEFAULT_TABLE table=default.Table_6e96405b009143d2bd1c0942cb98c033 [id=67905be94563417c8713eadd51eb94cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.359265 754 tablet_service.cc:1552] Processing DeleteTablet for tablet 9e07291d6b5645feb4d5890f58e1ef74 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:57 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:58.479364 754 tablet_service.cc:1552] Processing DeleteTablet for tablet a5c3a15510cb4321bbb80d61fabd0c0a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:57 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:58.479769 1172 raft_consensus.cc:359] T 1fc87e2d7b4240e08f2b4ac7277293ff P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:58.479952 1172 raft_consensus.cc:385] T 1fc87e2d7b4240e08f2b4ac7277293ff P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:58.479890 754 tablet_service.cc:1505] Processing CreateTablet for tablet 2b278db5b2b04344b864797055a5f665 (DEFAULT_TABLE table=default.Table_73bc152b9f224b07ae2524e0e18f7df0 [id=61b689e02284411a9d7cd0795682791a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.480032 1172 raft_consensus.cc:740] T 1fc87e2d7b4240e08f2b4ac7277293ff P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:51:58.359723 750 tablet_service.cc:1552] Processing DeleteTablet for tablet 23b9b2ea814645eba63e5f6b12cfb966 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:57 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:58.480233 1172 consensus_queue.cc:260] T 1fc87e2d7b4240e08f2b4ac7277293ff P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:58.480346 750 tablet_service.cc:1552] Processing DeleteTablet for tablet bfec87f690a640c39c58c64e16cc67e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:57 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:58.480732 1172 ts_tablet_manager.cc:1434] T 1fc87e2d7b4240e08f2b4ac7277293ff P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 02:51:58.374030 519 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet 59af906061d049c99ae2153945597007 (table default.Table_6da557988d044647b5e68f09b3f00407 [id=25f16d95843d41c981d6a07ec0d4b48a]). Expected version 2 got 0
I20251029 02:51:58.480751 750 tablet_service.cc:1505] Processing CreateTablet for tablet 688af6580c4d4637a39f817b4e324618 (DEFAULT_TABLE table=default.Table_4f9515ed04dc4e9e8480232c53fbac2e [id=0dc6f8e6205d4e3997946747bbc86e2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.481758 519 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_b1c04ce867284dc5bf55b02f25bf66ae" } modify_external_catalogs: true
I20251029 02:51:58.481940 519 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_b1c04ce867284dc5bf55b02f25bf66ae" } modify_external_catalogs: true
I20251029 02:51:58.393909 894 tablet_service.cc:1505] Processing CreateTablet for tablet 5ae5f33567324d90b7a5a6e0f1b5fb67 (DEFAULT_TABLE table=default.Table_19039883faf6464387f65003d0d638a3 [id=1d031141cc2c44eface6277fd55922cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.374924 742 tablet_service.cc:1552] Processing DeleteTablet for tablet 6feec2c8e030447fae9c292a626dcd1d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:52 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:58.483212 742 tablet_service.cc:1505] Processing CreateTablet for tablet c7b78fd458e4414b9fc3aa064a80b206 (DEFAULT_TABLE table=default.Table_5876d3fefb8144e2a1a76782912ab03b [id=9b6e3e9eef094d9bb64fb22f512bb3d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.483251 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 4b76c4408a7340e59b8506171622175f (table default.Table_21b8ae2debb241bda65fe387a4a3190f [id=f3fa2dbbef7e47359492d5a5eef9b8fd]) successfully deleted
I20251029 02:51:58.392658 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_72f0d7d8e4d34e47b90c8aa8130ca61d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:58.483861 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet daa05a34c0684c07a35966290fd7d44c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet daa05a34c0684c07a35966290fd7d44c already in progress: creating tablet
W20251029 02:51:58.483982 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_72f0d7d8e4d34e47b90c8aa8130ca61d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:51:58.484231 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet f187dbd8a55b45b292ea61db43caf47d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f187dbd8a55b45b292ea61db43caf47d already in progress: creating tablet
W20251029 02:51:58.484598 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 1fc87e2d7b4240e08f2b4ac7277293ff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1fc87e2d7b4240e08f2b4ac7277293ff already in progress: creating tablet
W20251029 02:51:58.484920 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 59eee86f249e4d6a9d09d30276dcd659 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 59eee86f249e4d6a9d09d30276dcd659 already in progress: creating tablet
I20251029 02:51:58.394496 1174 rpcz_store.cc:276] 1029 02:51:56.206072 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:56.215452 (+ 9380us) service_pool.cc:225] Handling call
1029 02:51:58.354172 (+2138720us) ts_tablet_manager.cc:1151] Deleting tablet 0d84dc8dc465496f9adec798b2e03e13
1029 02:51:58.354185 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:58.365757 (+ 11572us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2135241}
I20251029 02:51:58.375003 737 tablet_service.cc:1552] Processing DeleteTablet for tablet fa5127bcfcda40a7b5ad4ef016af072a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:57 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:58.435551 1023 tablet_service.cc:1505] Processing CreateTablet for tablet dc33fd3404ec41c3935b02c84a12971b (DEFAULT_TABLE table=default.Table_78b81f8805aa49b8a07a940c8d886de5 [id=09ee821707fa4082889bfccb8667b0b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.440183 1027 tablet_service.cc:1505] Processing CreateTablet for tablet 7e9d7ee2601640d88c0ddd0603e8a6e5 (DEFAULT_TABLE table=default.Table_5c0d5b3233fe48f3a817f437c161967c [id=e895f8b2327f4b838540d72e60fc06d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.485561 737 tablet_service.cc:1505] Processing CreateTablet for tablet e3000183230e4350b47f5085952f2db6 (DEFAULT_TABLE table=default.Table_5069071caedd4971a57a438fba4fac4f [id=79d22e7fddb24e68a0db1aeacbdfb6e3]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 02:51:58.442642 1637 log.cc:927] Time spent T 59af906061d049c99ae2153945597007 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Append to log took a long time: real 0.074s user 0.000s sys 0.000s
I20251029 02:51:58.454718 1038 tablet_service.cc:1505] Processing CreateTablet for tablet c64d6761ef924ccaa9c2579f8ba47e33 (DEFAULT_TABLE table=default.Table_44b56ca77fed49489c18e54dffb4b108 [id=86861bd15ecd481a8579fb71ee4b9ef1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.458798 1031 tablet_service.cc:1505] Processing CreateTablet for tablet 3a2b95997454452bb985c51fc02d77e2 (DEFAULT_TABLE table=default.Table_30a991be88ad4c199ac8ce370ee76f17 [id=0c3d955bdfd34db2ac137dd6f5d84a2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.393126 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_1b9efccf77ba4f8c91c0136ca2433af8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:58.488320 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1b9efccf77ba4f8c91c0136ca2433af8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:51:58.489192 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 1bc45078a6a942c38ff3350c15885eff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1bc45078a6a942c38ff3350c15885eff already in progress: creating tablet
I20251029 02:51:58.415001 899 tablet_service.cc:1505] Processing CreateTablet for tablet a530fe8de8ae4be5b0fd9a5f7e94ea9e (DEFAULT_TABLE table=default.Table_59a5648999114c6f8b2a58e938c8e49a [id=d34f7b094b6c48bea8a767a1e913d74f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.375327 749 tablet_service.cc:1552] Processing DeleteTablet for tablet 392a36a6b59a42f78e3ac7c8fb63214b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:54 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:58.489862 749 tablet_service.cc:1505] Processing CreateTablet for tablet 7fb5b908db77453da5cfb32e2c955f41 (DEFAULT_TABLE table=default.Table_163ca13775fd4ae28ec7ead8297929c8 [id=09a60dfda8994fb0b36f3b4a55e58d09]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.393603 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_a45edb83b6254a4bb8fb610220e5cf65"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:58.491259 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a45edb83b6254a4bb8fb610220e5cf65 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:58.417019 878 tablet_service.cc:1552] Processing DeleteTablet for tablet 5ae5f33567324d90b7a5a6e0f1b5fb67 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:58 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:58.375959 741 tablet_service.cc:1505] Processing CreateTablet for tablet 786d9148fb1d417a8c2c8b8c53665899 (DEFAULT_TABLE table=default.Table_3d0a6e32dc914ba2b0263680b2587277 [id=0809df5782144d1da622b5726db932f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.393137 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_2e41802f8663436e83915f560cf8f94b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:58.492971 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2e41802f8663436e83915f560cf8f94b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:58.431078 893 tablet_service.cc:1505] Processing CreateTablet for tablet 7b0f7d6c1c9842789c6ce2cecaa3d5b0 (DEFAULT_TABLE table=default.Table_b1d1e274a2ae4a639d80fb46e27bca52 [id=6a5cb859f2f64e26b39723468a768027]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.394109 523 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_4c16a32e4f364fecb17b3c79b3a5b98c" } new_table_name: "default.Table_f35ae25e36f64d2886752bb3b1d66710" modify_external_catalogs: true
I20251029 02:51:58.404608 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_013611a621b2452c8ffb8d9161725af6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:58.494894 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_013611a621b2452c8ffb8d9161725af6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:58.495040 1166 tablet_bootstrap.cc:492] T 75236b2fa82f4f28b82efae9a2c5160d P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:51:58.405076 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_58fd9561b8e04989a95f097e29a79613"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:58.496110 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_58fd9561b8e04989a95f097e29a79613 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:58.405732 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_d497c064fe99434faf6566e6c5b6836c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:58.426159 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_ad59ec000b0c4dbcb80dd9a5d9648ec6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:58.497264 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d497c064fe99434faf6566e6c5b6836c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:51:58.497403 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ad59ec000b0c4dbcb80dd9a5d9648ec6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:58.497808 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 506f8c112a3b43e1a3d6bbafb8cb81e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.500871 520 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_93bc8cfb59f440849d8ae4d6529e6a01" } modify_external_catalogs: true
I20251029 02:51:58.501057 520 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_93bc8cfb59f440849d8ae4d6529e6a01" } modify_external_catalogs: true
I20251029 02:51:58.503237 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0fec06ad37d4c12a7045d424b88cc5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.506850 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet abf18e06df9b482a8b25c078b9495272. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.507112 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7816556521144c928d7a02694e22e79f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.510864 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b125cc9aaf4248b8bfa48fb152b832a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.515058 521 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_f05d33ecadba4313a2c9e46b27409196" } modify_external_catalogs: true
I20251029 02:51:58.515264 521 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_f05d33ecadba4313a2c9e46b27409196" } modify_external_catalogs: true
I20251029 02:51:58.515671 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1df0d944717d487a98ea83a8ec04bd46. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.518848 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc802bda2f6a4e439b605f2cafa37967. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.524379 523 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_77a5f27bd0ae4eb68a30c883268d2f9c" } modify_external_catalogs: true
I20251029 02:51:58.524611 523 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_77a5f27bd0ae4eb68a30c883268d2f9c" } modify_external_catalogs: true
I20251029 02:51:58.526247 517 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_663bb3557fe64d8092e0a19dfa14d1b4" } new_table_name: "default.Table_8e3f488e93f6452d8736d4c582cdf82c" modify_external_catalogs: true
I20251029 02:51:58.526340 878 tablet_service.cc:1505] Processing CreateTablet for tablet 94bdbb9394f7490e99916bd818986208 (DEFAULT_TABLE table=default.Table_243752c5b95a4421a86f6510f801e0b9 [id=bb5e87a155a949dba4636207700a30e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.526501 918 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "69597d48631d43b6a5b92abbf60d5ed3" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:51:58.527179 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b278db5b2b04344b864797055a5f665. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.527305 1172 tablet_bootstrap.cc:492] T 8afc01d52d784f2fa559cab7ce07c8be P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:51:58.528321 882 tablet_service.cc:1505] Processing CreateTablet for tablet 998b9cbe76514665be76c6c387dc75b3 (DEFAULT_TABLE table=default.Table_cd55951068e2462ab1b5b075a5b40767 [id=476be76ffbd643b49556539c2ffb653b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.530718 1172 tablet_bootstrap.cc:654] T 8afc01d52d784f2fa559cab7ce07c8be P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:58.531422 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 688af6580c4d4637a39f817b4e324618. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.531713 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7b78fd458e4414b9fc3aa064a80b206. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.533178 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ef87f7a3dff4f00a0e7273eaec98dff. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:51:58.533574 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 0d84dc8dc465496f9adec798b2e03e13 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0d84dc8dc465496f9adec798b2e03e13 already in progress: creating tablet
I20251029 02:51:58.534633 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c45ffc304e73490387ce800b702b5881. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.535100 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3000183230e4350b47f5085952f2db6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.536873 879 tablet_service.cc:1505] Processing CreateTablet for tablet c0672a948e5e41c5bbfe7e8199025986 (DEFAULT_TABLE table=default.Table_88289be99f39482d837c84fecf31efb1 [id=20c79a2035154fb79385b5149ca53eb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.537173 1167 ts_tablet_manager.cc:1403] T d6e1170dbb594a5696a3899cde8da7c9 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.929s user 0.009s sys 0.029s
I20251029 02:51:58.538638 896 tablet_service.cc:1505] Processing CreateTablet for tablet 1878d7eef14a4578bc929da68872cee4 (DEFAULT_TABLE table=default.Table_d0a24cc3c6b244b0b43c78742dab0877 [id=78d1faddf0ca494ea722c99ca3566abe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.538970 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 786d9148fb1d417a8c2c8b8c53665899. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.538933 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 316) took 2282 ms. Trace:
I20251029 02:51:58.539507 1167 raft_consensus.cc:359] T d6e1170dbb594a5696a3899cde8da7c9 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:58.539733 1167 raft_consensus.cc:385] T d6e1170dbb594a5696a3899cde8da7c9 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:58.540115 1167 raft_consensus.cc:740] T d6e1170dbb594a5696a3899cde8da7c9 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:51:58.540340 1167 consensus_queue.cc:260] T d6e1170dbb594a5696a3899cde8da7c9 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:58.541770 1167 ts_tablet_manager.cc:1434] T d6e1170dbb594a5696a3899cde8da7c9 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20251029 02:51:58.542109 1167 tablet_bootstrap.cc:492] T 6f93812578a940a3b7378015e2a46f95 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:51:58.542153 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_51aa8265801a4f8d9c0330cbf2bc72a4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:58.542716 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_51aa8265801a4f8d9c0330cbf2bc72a4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:58.542901 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24f07b1efac64b099b0b5ed467463749. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.542902 883 tablet_service.cc:1505] Processing CreateTablet for tablet aee42fd9ca894319b3759beb2bae342e (DEFAULT_TABLE table=default.Table_2e760e5f5f1a4009b5d7c227df182a1e [id=51655edb7c684ca38744e37d5384be48]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.544821 522 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_9c88c372f29d4f0980baa173608086f2" } new_table_name: "default.Table_e48aadc66a0e48658d45d0812c29cbdf" modify_external_catalogs: true
I20251029 02:51:58.546123 526 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_b70ecfbeceed48a4bfdb94abe7c580f4" } new_table_name: "default.Table_73e59d41837947c4a4b59f6639ad6f06" modify_external_catalogs: true
I20251029 02:51:58.547278 525 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_2611211ae1424e22a985b944e78f4843" } new_table_name: "default.Table_15cde6ff49784e5082de7519ce61c81d" modify_external_catalogs: true
I20251029 02:51:58.548326 1174 rpcz_store.cc:276] 1029 02:51:56.240885 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:56.241009 (+ 124us) service_pool.cc:225] Handling call
1029 02:51:58.522766 (+2281757us) ts_tablet_manager.cc:1151] Deleting tablet af4860c6b72f4d988b206b5ef2f22906
1029 02:51:58.522780 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:58.522905 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2281528}
W20251029 02:51:58.550151 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet af4860c6b72f4d988b206b5ef2f22906 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet af4860c6b72f4d988b206b5ef2f22906 already in progress: creating tablet
I20251029 02:51:58.550755 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a530fe8de8ae4be5b0fd9a5f7e94ea9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.552363 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 347) took 1845 ms. Trace:
I20251029 02:51:58.556795 895 tablet_service.cc:1505] Processing CreateTablet for tablet 0f96af1cb0ea4b46bd846ae0739f4947 (DEFAULT_TABLE table=default.Table_114cbdf1ef0f493f8f8c834735271053 [id=5107b378d9c24778ba7bf30c33b5516f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.561487 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7fb5b908db77453da5cfb32e2c955f41. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.563608 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3362e95063af41c8a24e935f547e3089" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:51:58.569074 1174 tablet_replica.cc:333] T ab421dfaa3f84593842bb732a2693160 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:51:58.571738 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_99567b9d8c154303bae22fda47db1d0a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:58.572432 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_99567b9d8c154303bae22fda47db1d0a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:51:58.574059 517 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f4729ecd182042c2a10f67ca25224101 consensus state has no leader
I20251029 02:51:58.574492 517 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_ee6dec2fc7834c78914ddc9f13890c55" } modify_external_catalogs: true
I20251029 02:51:58.577567 517 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_ee6dec2fc7834c78914ddc9f13890c55" } modify_external_catalogs: true
I20251029 02:51:58.575382 519 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet c8389c8bfead4ff7a23f363c9c6770bf
I20251029 02:51:58.578863 1174 raft_consensus.cc:2243] T ab421dfaa3f84593842bb732a2693160 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:58.579075 1174 raft_consensus.cc:2272] T ab421dfaa3f84593842bb732a2693160 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:58.575944 523 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 86fc22bb84794149b598b97df41eb881
I20251029 02:51:58.580770 1174 ts_tablet_manager.cc:1916] T ab421dfaa3f84593842bb732a2693160 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:58.582783 521 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet f18b0ca5a6a44175a912642e9eea1778
I20251029 02:51:58.585994 520 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 358bdfbcb4ee440c899fb6eedef68f44
I20251029 02:51:58.587321 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b0f7d6c1c9842789c6ce2cecaa3d5b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.597340 1613 raft_consensus.cc:493] T a5c3a15510cb4321bbb80d61fabd0c0a P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:58.597560 1613 raft_consensus.cc:515] T a5c3a15510cb4321bbb80d61fabd0c0a P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:58.598220 1613 leader_election.cc:290] T a5c3a15510cb4321bbb80d61fabd0c0a P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:51:58.601703 740 tablet_service.cc:1505] Processing CreateTablet for tablet 294afb9b012c40c0b05b73795d6f0ac8 (DEFAULT_TABLE table=default.Table_4619a2287cb14edbb32bcbec581eb4d9 [id=6213d8ae6fb943568919256d7922513f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.603117 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 294afb9b012c40c0b05b73795d6f0ac8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.603873 892 tablet_service.cc:1505] Processing CreateTablet for tablet 358bdfbcb4ee440c899fb6eedef68f44 (DEFAULT_TABLE table=default.Table_93bc8cfb59f440849d8ae4d6529e6a01 [id=38f2afef718b46828519c300d6b592b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.606467 884 tablet_service.cc:1505] Processing CreateTablet for tablet c8389c8bfead4ff7a23f363c9c6770bf (DEFAULT_TABLE table=default.Table_b1c04ce867284dc5bf55b02f25bf66ae [id=7d8b9c11844d47f6a2eecd698765f050]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 02:51:58.607302 526 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c9e923724ca9402682f13b252dbc37aa consensus state has no leader
W20251029 02:51:58.607741 525 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 217144957930495aa3e9dfcfde268e71 consensus state has no leader
I20251029 02:51:58.608661 737 tablet_service.cc:1505] Processing CreateTablet for tablet ff4c38150df04a75855e0e60ff417ddb (DEFAULT_TABLE table=default.Table_040794c7b9064642a98d202e77a60ac5 [id=b4e3ba45194b4fcd8f14d1a8cb4365dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.609211 887 tablet_service.cc:1505] Processing CreateTablet for tablet 5842e9bba4fa4295ba21e90d645a56df (DEFAULT_TABLE table=default.Table_a15f5c838c7d4600b91a7368321fff38 [id=67b92918d6fc4c1fa795908bc20bd707]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.609658 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 6cd7009e06a646acad0802918c2b425e (table default.Table_198644e9ee1f45ecb6ad95a759b54607 [id=7810f54ddc4849a8b13bb2928eb761a7]) successfully deleted
I20251029 02:51:58.610809 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 3362e95063af41c8a24e935f547e3089 (table default.Table_db3418c43cde44f8a505e27220d0b5fd [id=fb42af08ec1f48498ee5e91cce62333e]) successfully deleted
I20251029 02:51:58.611057 900 tablet_service.cc:1505] Processing CreateTablet for tablet 78662eeae70b4148a4ad0d47be5c566d (DEFAULT_TABLE table=default.Table_fd6ca61b22e442fabd966ea848f4389e [id=d45dc85e3f3244ed8024eceb15d3e765]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.616580 886 tablet_service.cc:1505] Processing CreateTablet for tablet 7d1408dc544944669de519a1553d9bb7 (DEFAULT_TABLE table=default.Table_3d101961029143a882eeafb119560e5d [id=4d483223ce624eea98e68a9375900497]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.388712 751 tablet_service.cc:1552] Processing DeleteTablet for tablet 49d9ce32e4f04f11be2ecefc5d523a8b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:57 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:58.427775 753 tablet_service.cc:1505] Processing CreateTablet for tablet 9492c1027aba4455ad8f539addaed5a7 (DEFAULT_TABLE table=default.Table_e8072943e31241f683bed508cb4f2029 [id=63e946b11bce40028e4c6f5c072bdd75]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.431116 1445 raft_consensus.cc:493] T 392a36a6b59a42f78e3ac7c8fb63214b P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:58.430796 743 tablet_service.cc:1552] Processing DeleteTablet for tablet afb19ba39399410bb90b6225d7b5b5b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:57 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:58.440488 739 tablet_service.cc:1505] Processing CreateTablet for tablet f18b0ca5a6a44175a912642e9eea1778 (DEFAULT_TABLE table=default.Table_f05d33ecadba4313a2c9e46b27409196 [id=4cf725c11fbb4ab3981000b3a41e8496]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.453677 1175 ts_tablet_manager.cc:1929] T c65469173762446e9725262ae998dbf3 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:58.633982 756 tablet_service.cc:1505] Processing CreateTablet for tablet f65fe9a5137b48669afef51ad7dd76ae (DEFAULT_TABLE table=default.Table_9c1124b16dc44ec5b2696d567f1c52f0 [id=c60923f7053642eb95881163d039644f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.634541 755 tablet_service.cc:1552] Processing DeleteTablet for tablet cc802bda2f6a4e439b605f2cafa37967 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:57 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:58.637699 919 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a5c3a15510cb4321bbb80d61fabd0c0a" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:51:58.637825 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6feec2c8e030447fae9c292a626dcd1d" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:51:58.639168 517 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet eca016aed05e4ce09e1a2bf96e2286f5
I20251029 02:51:58.639720 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cded156380f45cc95893825c71ba0e9" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:51:58.639748 918 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "faa835de84ff45b19525d1d603678987" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:51:58.644867 898 tablet_service.cc:1505] Processing CreateTablet for tablet 5ae35704b3944b36bc5199f9ae0546f6 (DEFAULT_TABLE table=default.Table_e5a5de26cf4041fca4bfbc384115b9b3 [id=163c34d030c94e21b43295fd6e457d0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.647081 743 tablet_service.cc:1505] Processing CreateTablet for tablet 023a4094b424464c8f49d6eb9cc15da4 (DEFAULT_TABLE table=default.Table_c020d31b6d054efaad87057733e683b6 [id=b00ec70f267a4fff98f12cc029eb3e04]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.647483 891 tablet_service.cc:1505] Processing CreateTablet for tablet e5336762c90e43278db7978dda295e40 (DEFAULT_TABLE table=default.Table_2474d13afb604655864254dd55221051 [id=3d9f1bcd08c74e65b3b04c50e4613c74]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.647761 1175 log.cc:1199] T c65469173762446e9725262ae998dbf3 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/c65469173762446e9725262ae998dbf3
W20251029 02:51:58.647814 711 leader_election.cc:343] T 69597d48631d43b6a5b92abbf60d5ed3 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:58.651989 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff4c38150df04a75855e0e60ff417ddb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.653050 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ae5f33567324d90b7a5a6e0f1b5fb67. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.653959 890 tablet_service.cc:1505] Processing CreateTablet for tablet 75aa2dd1ed1b49649f3257df5d45cb32 (DEFAULT_TABLE table=default.Table_dbf30facf56548e1b20d6fec42080e05 [id=e03d91a588f0492ab73c4f3f6d4022e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.654656 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f65fe9a5137b48669afef51ad7dd76ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.654661 1445 raft_consensus.cc:515] T 392a36a6b59a42f78e3ac7c8fb63214b P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:58.655735 1445 leader_election.cc:290] T 392a36a6b59a42f78e3ac7c8fb63214b P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:51:58.657143 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 023a4094b424464c8f49d6eb9cc15da4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.663957 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9492c1027aba4455ad8f539addaed5a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.665436 738 tablet_service.cc:1505] Processing CreateTablet for tablet 08371b3daa184a0e9f4eb3d313b52c56 (DEFAULT_TABLE table=default.Table_736690fbc69640d9be26e1662e3fa63e [id=40f1beecaad24a64b4348663c9ae9afd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.670830 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 998b9cbe76514665be76c6c387dc75b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.680856 885 tablet_service.cc:1552] Processing DeleteTablet for tablet d6e1170dbb594a5696a3899cde8da7c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:52 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:58.682693 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f18b0ca5a6a44175a912642e9eea1778. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.688475 711 leader_election.cc:304] T 69597d48631d43b6a5b92abbf60d5ed3 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
W20251029 02:51:58.688845 711 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:41701, user_credentials={real_user=slave}} blocked reactor thread for 55125.1us
W20251029 02:51:58.689106 711 leader_election.cc:343] T 3362e95063af41c8a24e935f547e3089 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:58.689217 711 leader_election.cc:304] T 3362e95063af41c8a24e935f547e3089 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
W20251029 02:51:58.689393 711 raft_consensus.cc:2686] T 3362e95063af41c8a24e935f547e3089 P b495a49349324f7d81908f2da700f0d9: unable to run election callback: Service unavailable: Thread pool token was shut down
I20251029 02:51:58.690193 1175 ts_tablet_manager.cc:1950] T c65469173762446e9725262ae998dbf3 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:51:58.690551 1613 raft_consensus.cc:2749] T 69597d48631d43b6a5b92abbf60d5ed3 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:51:58.690693 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 877e278deb4a4b01ac66b06be0890baf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.691215 755 tablet_service.cc:1505] Processing CreateTablet for tablet da904606986f452cadd06187fff06785 (DEFAULT_TABLE table=default.Table_a4fcd46aef6146e986d4b8d3599d66f4 [id=24f8ffe28d81427d9e9c5beb9c938d02]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.695107 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 333) took 2025 ms. Trace:
I20251029 02:51:58.696768 1175 rpcz_store.cc:276] 1029 02:51:56.667737 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:56.668313 (+ 576us) service_pool.cc:225] Handling call
1029 02:51:58.259360 (+1591047us) ts_tablet_manager.cc:1151] Deleting tablet c65469173762446e9725262ae998dbf3
1029 02:51:58.259372 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:58.379977 (+120605us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:58.451446 (+ 71469us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:58.693382 (+241936us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1590838}
I20251029 02:51:58.697126 1175 tablet_replica.cc:333] T 5695b5a7b1854c3983e2d4ab684898b0 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:51:58.699707 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08371b3daa184a0e9f4eb3d313b52c56. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.700001 1176 rpcz_store.cc:276] 1029 02:51:56.645249 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:56.645535 (+ 286us) service_pool.cc:225] Handling call
1029 02:51:58.475995 (+1830460us) ts_tablet_manager.cc:1151] Deleting tablet 1bc45078a6a942c38ff3350c15885eff
1029 02:51:58.476058 (+ 63us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:58.490909 (+ 14851us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1830143}
I20251029 02:51:58.700920 966 heartbeater.cc:344] Connected to a master server at 127.0.104.60:41233
I20251029 02:51:58.702122 750 tablet_service.cc:1505] Processing CreateTablet for tablet 3a2b95997454452bb985c51fc02d77e2 (DEFAULT_TABLE table=default.Table_30a991be88ad4c199ac8ce370ee76f17 [id=0c3d955bdfd34db2ac137dd6f5d84a2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.702560 746 tablet_service.cc:1505] Processing CreateTablet for tablet d1866d6dcb0349bba5db86163375cada (DEFAULT_TABLE table=default.Table_e0a153971f6f4af98ee02f8bbc615620 [id=9bf3236a9242474b9f0c75b3e2f3d02d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.702783 885 tablet_service.cc:1552] Processing DeleteTablet for tablet c8389c8bfead4ff7a23f363c9c6770bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:58 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:58.702850 1175 raft_consensus.cc:2243] T 5695b5a7b1854c3983e2d4ab684898b0 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:58.703058 1175 raft_consensus.cc:2272] T 5695b5a7b1854c3983e2d4ab684898b0 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:58.703202 885 tablet_service.cc:1552] Processing DeleteTablet for tablet 86fc22bb84794149b598b97df41eb881 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:58 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:58.703658 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da904606986f452cadd06187fff06785. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.704221 748 tablet_service.cc:1505] Processing CreateTablet for tablet 68b1c4f88aee4e4795397d1c796ff948 (DEFAULT_TABLE table=default.Table_2ed982f44d384784b875b8a49394305e [id=f2d457ba2bcf44b5bcd44dea5e2aa299]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.705590 747 tablet_service.cc:1505] Processing CreateTablet for tablet 35d9b8dfd61e4d8db441427a49fb9dca (DEFAULT_TABLE table=default.Table_73eec04729fd4870aca2779a5738d772 [id=b54d7930e083434680f3934fc4023653]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.706038 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1866d6dcb0349bba5db86163375cada. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.706745 741 tablet_service.cc:1505] Processing CreateTablet for tablet e3ac6597d52441569a7aeef051087ad2 (DEFAULT_TABLE table=default.Table_c5b59d221b994982a9867326266b7bcd [id=242159edd3bc4771ac206d4e450b9475]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.707341 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_fd89990c08d142ffa08457a5802e5e69"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:58.707523 1445 raft_consensus.cc:493] T 1fc87e2d7b4240e08f2b4ac7277293ff P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:58.707674 1445 raft_consensus.cc:515] T 1fc87e2d7b4240e08f2b4ac7277293ff P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
W20251029 02:51:58.707952 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fd89990c08d142ffa08457a5802e5e69 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:58.709017 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc33fd3404ec41c3935b02c84a12971b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.709075 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68b1c4f88aee4e4795397d1c796ff948. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.712745 1175 ts_tablet_manager.cc:1916] T 5695b5a7b1854c3983e2d4ab684898b0 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:58.713747 1445 leader_election.cc:290] T 1fc87e2d7b4240e08f2b4ac7277293ff P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:51:58.714471 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94bdbb9394f7490e99916bd818986208. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.714895 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3ac6597d52441569a7aeef051087ad2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.713059 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 348) took 2063 ms. Trace:
I20251029 02:51:58.717149 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aee42fd9ca894319b3759beb2bae342e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.717332 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35d9b8dfd61e4d8db441427a49fb9dca. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.718220 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_68075467476c41f2b5f28dcfa7633b10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:58.718675 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a2b95997454452bb985c51fc02d77e2. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:51:58.718853 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_68075467476c41f2b5f28dcfa7633b10 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:58.720666 751 tablet_service.cc:1505] Processing CreateTablet for tablet 15137fc1c70347b381ffdfe5653a5f22 (DEFAULT_TABLE table=default.Table_fdcb71c680504418a7f89d81feb372b0 [id=4bacb12c51d34f78bf4b1aba964c38ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.721592 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15137fc1c70347b381ffdfe5653a5f22. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.725757 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a2b95997454452bb985c51fc02d77e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.727524 885 tablet_service.cc:1552] Processing DeleteTablet for tablet f18b0ca5a6a44175a912642e9eea1778 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:58 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:58.727814 885 tablet_service.cc:1552] Processing DeleteTablet for tablet eca016aed05e4ce09e1a2bf96e2286f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:58 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:58.730737 742 tablet_service.cc:1505] Processing CreateTablet for tablet c64d6761ef924ccaa9c2579f8ba47e33 (DEFAULT_TABLE table=default.Table_44b56ca77fed49489c18e54dffb4b108 [id=86861bd15ecd481a8579fb71ee4b9ef1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.732448 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_691548aa805841a281c6b0bbff6af9cc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:58.733003 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_691548aa805841a281c6b0bbff6af9cc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:58.733073 744 tablet_service.cc:1505] Processing CreateTablet for tablet 877e278deb4a4b01ac66b06be0890baf (DEFAULT_TABLE table=default.Table_a3377614e4a54266a4457944d48b0060 [id=991a151cc5d84421bf50858b1fa864e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.733749 888 tablet_service.cc:1552] Processing DeleteTablet for tablet 358bdfbcb4ee440c899fb6eedef68f44 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:58 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:58.734622 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1878d7eef14a4578bc929da68872cee4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.735033 524 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_736690fbc69640d9be26e1662e3fa63e" } new_table_name: "default.Table_1453b5168e64406688903f1d6927f549" modify_external_catalogs: true
I20251029 02:51:58.736255 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_c0a7b43b54fb41c6ab5347fdb4ea3cf2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:58.736737 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c0a7b43b54fb41c6ab5347fdb4ea3cf2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:58.737841 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_0f0a1e0ac81a4b3492d304bef28ee22d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:58.738268 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0f0a1e0ac81a4b3492d304bef28ee22d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:58.739706 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_5263ec7f27d74f73892b0ca252e9fa83"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:58.739976 888 tablet_service.cc:1505] Processing CreateTablet for tablet 4cede3b183494845af91dca39a2096c4 (DEFAULT_TABLE table=default.Table_ad59ec000b0c4dbcb80dd9a5d9648ec6 [id=7f651f203c294c549a87a3c0dba15bb8]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 02:51:58.740152 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5263ec7f27d74f73892b0ca252e9fa83 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:58.741523 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_a111747c600544a3a67b954b834e4039"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:58.741956 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a111747c600544a3a67b954b834e4039 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:58.742774 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c64d6761ef924ccaa9c2579f8ba47e33. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.744805 754 tablet_service.cc:1505] Processing CreateTablet for tablet dc33fd3404ec41c3935b02c84a12971b (DEFAULT_TABLE table=default.Table_78b81f8805aa49b8a07a940c8d886de5 [id=09ee821707fa4082889bfccb8667b0b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.745904 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75aa2dd1ed1b49649f3257df5d45cb32. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.746110 1018 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:37622 (request call id 88) took 1169 ms. Trace:
I20251029 02:51:58.746205 1018 rpcz_store.cc:276] 1029 02:51:57.576400 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:57.576443 (+ 43us) service_pool.cc:225] Handling call
1029 02:51:57.994709 (+418266us) spinlock_profiling.cc:243] Waited 23.2 ms on lock 0x6160001a2654. stack: 00007f34717829f6 00007f34717827b0 00007f347075e3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f347d61c200 00007f34814cc66f 00007f347b8f2c36 00007f347b8f2965 00007f347872127e 00007f347872015e 00007f3478726395 00007f3478726cbf 00007f3478726ae8 0000000000a8867c
1029 02:51:58.139107 (+144398us) spinlock_profiling.cc:243] Waited 872 us on lock 0x61800011088c. stack: 00007f34717829f6 00007f34717827b0 00007f347075e3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f347eba944e 00007f347e9a4ac1 00007f34814cc478 00007f347b8f2c36 00007f347b8f2965 00007f347872127e 00007f347872015e 00007f3478726395 00007f3478726cbf 00007f3478726ae8
1029 02:51:58.746051 (+606944us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":55443072}
I20251029 02:51:58.740746 1176 rpcz_store.cc:276] 1029 02:51:56.645677 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:56.645734 (+ 57us) service_pool.cc:225] Handling call
1029 02:51:58.700396 (+2054662us) ts_tablet_manager.cc:1151] Deleting tablet 1fc87e2d7b4240e08f2b4ac7277293ff
1029 02:51:58.700420 (+ 24us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:58.708889 (+ 8469us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2054390}
I20251029 02:51:58.751322 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 349) took 2103 ms. Trace:
I20251029 02:51:58.752277 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 877e278deb4a4b01ac66b06be0890baf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.752688 1176 rpcz_store.cc:276] 1029 02:51:56.647799 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:56.647849 (+ 50us) service_pool.cc:225] Handling call
1029 02:51:58.751152 (+2103303us) ts_tablet_manager.cc:1151] Deleting tablet 59eee86f249e4d6a9d09d30276dcd659
1029 02:51:58.751165 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:58.751285 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2103087}
I20251029 02:51:58.753177 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 350) took 2071 ms. Trace:
I20251029 02:51:58.753726 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78662eeae70b4148a4ad0d47be5c566d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.754365 1176 rpcz_store.cc:276] 1029 02:51:56.681814 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:56.684243 (+ 2429us) service_pool.cc:225] Handling call
1029 02:51:58.753002 (+2068759us) ts_tablet_manager.cc:1151] Deleting tablet 6d830d30b26b4ba799785fec8a6f0769
1029 02:51:58.753014 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:58.753132 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2068481}
I20251029 02:51:58.754758 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 352) took 1923 ms. Trace:
I20251029 02:51:58.756156 1176 rpcz_store.cc:276] 1029 02:51:56.831607 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:56.843652 (+ 12045us) service_pool.cc:225] Handling call
1029 02:51:58.754632 (+1910980us) ts_tablet_manager.cc:1151] Deleting tablet 0f80b840ed954230bfbfe7e40127cd2f
1029 02:51:58.754641 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:58.754728 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1910623}
I20251029 02:51:58.756300 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc33fd3404ec41c3935b02c84a12971b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.756539 1176 tablet_replica.cc:333] T 8f7c746f8b6948b58676298c2c2ea221 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:51:58.756763 1176 raft_consensus.cc:2243] T 8f7c746f8b6948b58676298c2c2ea221 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:58.756979 1176 raft_consensus.cc:2272] T 8f7c746f8b6948b58676298c2c2ea221 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251029 02:51:58.757155 524 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 08371b3daa184a0e9f4eb3d313b52c56 consensus state has no leader
I20251029 02:51:58.757711 1176 ts_tablet_manager.cc:1916] T 8f7c746f8b6948b58676298c2c2ea221 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:58.764240 739 tablet_service.cc:1505] Processing CreateTablet for tablet 7e9d7ee2601640d88c0ddd0603e8a6e5 (DEFAULT_TABLE table=default.Table_5c0d5b3233fe48f3a817f437c161967c [id=e895f8b2327f4b838540d72e60fc06d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.766477 885 tablet_service.cc:1505] Processing CreateTablet for tablet 27e26b5e812f4af0a4914bf9022d62d4 (DEFAULT_TABLE table=default.Table_72f0d7d8e4d34e47b90c8aa8130ca61d [id=37a2a807288046ba8cdc6ac961416990]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.773089 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0672a948e5e41c5bbfe7e8199025986. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.773804 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35d9b8dfd61e4d8db441427a49fb9dca. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.776098 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ae35704b3944b36bc5199f9ae0546f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.778666 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_cb67eff3fb7c4cd9bc24d6bf31c3cd8f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:58.779258 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cb67eff3fb7c4cd9bc24d6bf31c3cd8f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:58.779803 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c64d6761ef924ccaa9c2579f8ba47e33. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.783712 520 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_db3418c43cde44f8a505e27220d0b5fd" } new_table_name: "default.Table_5391971cdf974108a468e2d263d1cec1" modify_external_catalogs: true
I20251029 02:51:58.785075 526 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_040794c7b9064642a98d202e77a60ac5" } new_table_name: "default.Table_a13e1171ad12408fbc598714d7701865" modify_external_catalogs: true
I20251029 02:51:58.786240 525 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_0b95eba70e97459eb847fecfac513686" } modify_external_catalogs: true
I20251029 02:51:58.786394 525 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_0b95eba70e97459eb847fecfac513686" } modify_external_catalogs: true
I20251029 02:51:58.786911 517 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_243752c5b95a4421a86f6510f801e0b9" } modify_external_catalogs: true
I20251029 02:51:58.787065 517 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_243752c5b95a4421a86f6510f801e0b9" } modify_external_catalogs: true
I20251029 02:51:58.788869 1022 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:37622 (request call id 89) took 1211 ms. Trace:
I20251029 02:51:58.788911 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5336762c90e43278db7978dda295e40. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.788976 1022 rpcz_store.cc:276] 1029 02:51:57.577424 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:57.578547 (+ 1123us) service_pool.cc:225] Handling call
1029 02:51:58.390694 (+812147us) spinlock_profiling.cc:243] Waited 80.7 ms on lock 0x616000112430. stack: 00007f34717829f6 00007f34717827b0 00007f347075e3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f347e9a4829 00007f34814cc478 00007f347b8f2c36 00007f347b8f2965 00007f347872127e 00007f347872015e 00007f3478726395 00007f3478726cbf 00007f3478726ae8 0000000000a8867c
1029 02:51:58.788809 (+398115us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":185608448}
I20251029 02:51:58.795195 1166 tablet_bootstrap.cc:654] T 75236b2fa82f4f28b82efae9a2c5160d P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:58.797268 524 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_47a1a59db0444fa49f7274509385ff0c" } modify_external_catalogs: true
I20251029 02:51:58.797530 524 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_47a1a59db0444fa49f7274509385ff0c" } modify_external_catalogs: true
I20251029 02:51:58.798167 523 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_d0a24cc3c6b244b0b43c78742dab0877" } modify_external_catalogs: true
I20251029 02:51:58.798318 523 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_d0a24cc3c6b244b0b43c78742dab0877" } modify_external_catalogs: true
I20251029 02:51:58.800843 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68b1c4f88aee4e4795397d1c796ff948. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.803769 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f96af1cb0ea4b46bd846ae0739f4947. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:51:58.806967 526 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ff4c38150df04a75855e0e60ff417ddb consensus state has no leader
W20251029 02:51:58.807396 520 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet da2a21c3236a4bd99872b89b12a36508 consensus state has no leader
I20251029 02:51:58.808692 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8389c8bfead4ff7a23f363c9c6770bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.810911 526 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_c784291ac23b4f86ad8bd33ebd8f4755" } new_table_name: "default.Table_eef9ea2caa6e4d73a71eebd2ab5f5cde" modify_external_catalogs: true
I20251029 02:51:58.812201 520 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_6e1283a0c161498db405cf1e9b2c93d7" } new_table_name: "default.Table_1c932fc09555407cbd8c9c2d3ef20f7b" modify_external_catalogs: true
I20251029 02:51:58.773408 749 tablet_service.cc:1552] Processing DeleteTablet for tablet 30d03e1e42d34242bd354f9e9ec4f127 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:58 UTC) from {username='slave'} at 127.0.0.1:44580
W20251029 02:51:58.821538 520 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e74eda48dca647efacabe525ef316944 consensus state has no leader
I20251029 02:51:58.823218 1039 tablet_service.cc:1552] Processing DeleteTablet for tablet 897a9a75d13f493e90526da4ca7f54d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:58 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:58.828770 1024 tablet_service.cc:1552] Processing DeleteTablet for tablet 061f3d38c4604fa4b57e74ff6f6fc7da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:58 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:58.828970 1350 raft_consensus.cc:493] T d2990e9f626c40618c118266bc595684 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:58.829558 1028 tablet_service.cc:1505] Processing CreateTablet for tablet 897a9a75d13f493e90526da4ca7f54d2 (DEFAULT_TABLE table=default.Table_2d8b100ebfb8416a994557617d07af62 [id=1f176d1a5f044322b9a693421a43ed5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.829589 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_39a7937d49ce4ee290f72ed5295a2a3e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:58.829963 882 tablet_service.cc:1505] Processing CreateTablet for tablet 5bd2163bc4be467fb695629bfbb5a6ca (DEFAULT_TABLE table=default.Table_a45edb83b6254a4bb8fb610220e5cf65 [id=19f7af2387764bb8813321ddb6443d8d]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 02:51:58.830127 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_39a7937d49ce4ee290f72ed5295a2a3e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:58.830121 1024 tablet_service.cc:1505] Processing CreateTablet for tablet 38f83ebcb2a440bc8e6a8d36607e6666 (DEFAULT_TABLE table=default.Table_86ad8c0a1e9c4fd6b12b9f59a195e951 [id=d3cffce7aefa4c028f8e369c875fc19f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.830921 1350 raft_consensus.cc:515] T d2990e9f626c40618c118266bc595684 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:58.831252 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_a714017562834a6dbfaa057b9f9aae41"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:58.831648 1350 leader_election.cc:290] T d2990e9f626c40618c118266bc595684 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
W20251029 02:51:58.831722 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a714017562834a6dbfaa057b9f9aae41 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:58.834662 1039 tablet_service.cc:1552] Processing DeleteTablet for tablet aee42fd9ca894319b3759beb2bae342e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:58 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:58.835145 1025 tablet_service.cc:1505] Processing CreateTablet for tablet f18b0ca5a6a44175a912642e9eea1778 (DEFAULT_TABLE table=default.Table_f05d33ecadba4313a2c9e46b27409196 [id=4cf725c11fbb4ab3981000b3a41e8496]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.835554 1032 tablet_service.cc:1552] Processing DeleteTablet for tablet 30d03e1e42d34242bd354f9e9ec4f127 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:58 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:58.835927 1035 tablet_service.cc:1505] Processing CreateTablet for tablet 2b0988f736a7402997ca3ba323a59e18 (DEFAULT_TABLE table=default.Table_03569ce890a341d88f5ff48f32013ea0 [id=dbb0cf4e24ca4fe1b989c9795dc4da82]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.832439 1660 tablet.cc:1722] T 59af906061d049c99ae2153945597007 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 02:51:58.838001 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5842e9bba4fa4295ba21e90d645a56df. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.840224 1032 tablet_service.cc:1505] Processing CreateTablet for tablet d24fe99a1fb54476b90da2c22ff5d383 (DEFAULT_TABLE table=default.Table_e280ab7143204fbd8daa4f2c898617f9 [id=3c055a359301472cac54dd72a37520c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.840771 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 358bdfbcb4ee440c899fb6eedef68f44. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.841006 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e9d7ee2601640d88c0ddd0603e8a6e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.841933 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2990e9f626c40618c118266bc595684" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:51:58.842063 745 tablet_service.cc:1552] Processing DeleteTablet for tablet aee42fd9ca894319b3759beb2bae342e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:58 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:58.843353 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27e26b5e812f4af0a4914bf9022d62d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.842486 749 tablet_service.cc:1505] Processing CreateTablet for tablet 38f83ebcb2a440bc8e6a8d36607e6666 (DEFAULT_TABLE table=default.Table_86ad8c0a1e9c4fd6b12b9f59a195e951 [id=d3cffce7aefa4c028f8e369c875fc19f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.844484 526 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_c020d31b6d054efaad87057733e683b6" } modify_external_catalogs: true
I20251029 02:51:58.844686 526 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_c020d31b6d054efaad87057733e683b6" } modify_external_catalogs: true
I20251029 02:51:58.845136 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38f83ebcb2a440bc8e6a8d36607e6666. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.845906 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4cede3b183494845af91dca39a2096c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.846498 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d1408dc544944669de519a1553d9bb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.846760 1175 ts_tablet_manager.cc:1929] T 5695b5a7b1854c3983e2d4ab684898b0 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:58.846962 1175 log.cc:1199] T 5695b5a7b1854c3983e2d4ab684898b0 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/5695b5a7b1854c3983e2d4ab684898b0
I20251029 02:51:58.847111 1167 tablet_bootstrap.cc:654] T 6f93812578a940a3b7378015e2a46f95 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:58.847391 1175 ts_tablet_manager.cc:1950] T 5695b5a7b1854c3983e2d4ab684898b0 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:51:58.848237 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5bd2163bc4be467fb695629bfbb5a6ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.852222 884 tablet_service.cc:1505] Processing CreateTablet for tablet 4a55f22b0d3b4ebe98ec2e054d05b095 (DEFAULT_TABLE table=default.Table_2e41802f8663436e83915f560cf8f94b [id=4e2615cec64b4828b8311b37e4a568b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.858762 526 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 023a4094b424464c8f49d6eb9cc15da4
I20251029 02:51:58.866478 523 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 1878d7eef14a4578bc929da68872cee4
I20251029 02:51:58.877199 525 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 7816556521144c928d7a02694e22e79f
I20251029 02:51:58.781116 741 tablet_service.cc:1552] Processing DeleteTablet for tablet 897a9a75d13f493e90526da4ca7f54d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:58 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:58.804328 753 tablet_service.cc:1505] Processing CreateTablet for tablet 897a9a75d13f493e90526da4ca7f54d2 (DEFAULT_TABLE table=default.Table_2d8b100ebfb8416a994557617d07af62 [id=1f176d1a5f044322b9a693421a43ed5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.879278 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 897a9a75d13f493e90526da4ca7f54d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.881155 1172 tablet_bootstrap.cc:492] T 8afc01d52d784f2fa559cab7ce07c8be P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
W20251029 02:51:58.882818 990 leader_election.cc:343] T d2990e9f626c40618c118266bc595684 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:58.887228 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e9d7ee2601640d88c0ddd0603e8a6e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.887490 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f18b0ca5a6a44175a912642e9eea1778. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.887706 517 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 94bdbb9394f7490e99916bd818986208
I20251029 02:51:58.893983 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 897a9a75d13f493e90526da4ca7f54d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.896893 524 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet c9bd3f2deb5d4af09b7918269a98509f
I20251029 02:51:58.898681 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 38f83ebcb2a440bc8e6a8d36607e6666. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.900930 1445 raft_consensus.cc:493] T 0a47781d036149569e1164dd938a45e9 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:58.904469 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d24fe99a1fb54476b90da2c22ff5d383. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.905341 746 tablet_service.cc:1505] Processing CreateTablet for tablet e74eda48dca647efacabe525ef316944 (DEFAULT_TABLE table=default.Table_6e1283a0c161498db405cf1e9b2c93d7 [id=4de1990274fe4b8487506a86872fe976]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.905815 741 tablet_service.cc:1505] Processing CreateTablet for tablet c172417d1d444c439e9cda611a2ed551 (DEFAULT_TABLE table=default.Table_293c6e92af8a4707be8e2596399fd6a2 [id=d7a06dc089ab4371813f376ef37261ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.907002 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b0988f736a7402997ca3ba323a59e18. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.911470 1020 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:37622 (request call id 87) took 1335 ms. Trace:
I20251029 02:51:58.911587 1020 rpcz_store.cc:276] 1029 02:51:57.576243 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:57.576386 (+ 143us) service_pool.cc:225] Handling call
1029 02:51:58.393160 (+816774us) spinlock_profiling.cc:243] Waited 188 ms on lock 0x6160001a2654. stack: 00007f34717829f6 00007f34717827b0 00007f347075e3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f347d61c200 00007f34814cc66f 00007f347b8f2c36 00007f347b8f2965 00007f347872127e 00007f347872015e 00007f3478726395 00007f3478726cbf 00007f3478726ae8 0000000000a8867c
1029 02:51:58.911414 (+518254us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":433457280}
I20251029 02:51:58.915045 1172 ts_tablet_manager.cc:1403] T 8afc01d52d784f2fa559cab7ce07c8be P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.354s user 0.001s sys 0.012s
I20251029 02:51:58.811200 752 tablet_service.cc:1505] Processing CreateTablet for tablet 2b0988f736a7402997ca3ba323a59e18 (DEFAULT_TABLE table=default.Table_03569ce890a341d88f5ff48f32013ea0 [id=dbb0cf4e24ca4fe1b989c9795dc4da82]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.752063 743 tablet_service.cc:1552] Processing DeleteTablet for tablet 061f3d38c4604fa4b57e74ff6f6fc7da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:58 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:58.916558 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e74eda48dca647efacabe525ef316944. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.918742 738 tablet_service.cc:1552] Processing DeleteTablet for tablet 5ae5f33567324d90b7a5a6e0f1b5fb67 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:58 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:58.919565 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_203b46ee33744bf3bfe75190ab006c13"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:58.920161 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_203b46ee33744bf3bfe75190ab006c13 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:58.920300 1445 raft_consensus.cc:515] T 0a47781d036149569e1164dd938a45e9 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:58.920882 743 tablet_service.cc:1505] Processing CreateTablet for tablet a530fe8de8ae4be5b0fd9a5f7e94ea9e (DEFAULT_TABLE table=default.Table_59a5648999114c6f8b2a58e938c8e49a [id=d34f7b094b6c48bea8a767a1e913d74f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.921110 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 331) took 2219 ms. Trace:
I20251029 02:51:58.921200 1175 rpcz_store.cc:276] 1029 02:51:56.667101 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:56.668619 (+ 1518us) service_pool.cc:225] Handling call
1029 02:51:58.697074 (+2028455us) ts_tablet_manager.cc:1151] Deleting tablet 5695b5a7b1854c3983e2d4ab684898b0
1029 02:51:58.697082 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:58.769486 (+ 72404us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:58.846658 (+ 77172us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:58.886454 (+ 39796us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2028222}
W20251029 02:51:58.921396 512 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 08371b3daa184a0e9f4eb3d313b52c56 (table default.Table_1453b5168e64406688903f1d6927f549 [id=40f1beecaad24a64b4348663c9ae9afd], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 08371b3daa184a0e9f4eb3d313b52c56 consensus state has no leader
I20251029 02:51:58.922264 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a530fe8de8ae4be5b0fd9a5f7e94ea9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.923048 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 334) took 2242 ms. Trace:
I20251029 02:51:58.923151 1175 rpcz_store.cc:276] 1029 02:51:56.681010 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:56.681064 (+ 54us) service_pool.cc:225] Handling call
1029 02:51:58.921483 (+2240419us) ts_tablet_manager.cc:1151] Deleting tablet 6d830d30b26b4ba799785fec8a6f0769
1029 02:51:58.922865 (+ 1382us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:58.922988 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2240219}
I20251029 02:51:58.923444 1175 tablet_replica.cc:333] T 1bab90fe904944af948dab3db06b7e34 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:51:58.923631 1175 raft_consensus.cc:2243] T 1bab90fe904944af948dab3db06b7e34 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:58.923830 1175 raft_consensus.cc:2272] T 1bab90fe904944af948dab3db06b7e34 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:58.924363 1175 ts_tablet_manager.cc:1916] T 1bab90fe904944af948dab3db06b7e34 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:58.925096 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b0988f736a7402997ca3ba323a59e18. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.930080 1445 leader_election.cc:290] T 0a47781d036149569e1164dd938a45e9 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:51:58.930197 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_bc803f454de540c28f741096815c1ce2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:58.930789 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bc803f454de540c28f741096815c1ce2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:58.932850 1660 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:37626 (request call id 414) took 1013 ms. Trace:
I20251029 02:51:58.933303 740 tablet_service.cc:1505] Processing CreateTablet for tablet 5ef87f7a3dff4f00a0e7273eaec98dff (DEFAULT_TABLE table=default.Table_fd1d23f31ebf47b884897c68eac9004d [id=1c99ebd287ff47d08e817853a628256c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.932978 1660 rpcz_store.cc:276] 1029 02:51:57.918983 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:57.919624 (+ 641us) service_pool.cc:225] Handling call
1029 02:51:58.932804 (+1013180us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1029 02:51:58.278981 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
1029 02:51:58.279145 (+ 164us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
1029 02:51:58.279153 (+ 8us) alter_schema_op.cc:66] Acquired schema lock
1029 02:51:58.279160 (+ 7us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
1029 02:51:58.279200 (+ 40us) alter_schema_op.cc:128] START. Timestamp: 1761706318279131
1029 02:51:58.279202 (+ 2us) op_driver.cc:348] REPLICATION: starting
1029 02:51:58.417345 (+138143us) log.cc:844] Serialized 173 byte log entry
1029 02:51:58.432152 (+ 14807us) spinlock_profiling.cc:243] Waited 72.2 ms on lock 0x6160001a2654. stack: 00007f34717829f6 00007f34717827b0 00007f347075e3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f347d622862 00007f347ea052a5 00007f347ea03dc9 00007f347ea0c1f0 00007f347ea0bffd 0000000000a8867c 00007f34717c8d7f 00007f34717cc04b 00007f34717cbec8 0000000000a8867c
1029 02:51:58.494422 (+ 62270us) op_driver.cc:464] REPLICATION: finished
1029 02:51:58.806219 (+311797us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
1029 02:51:58.922733 (+116514us) log.cc:844] Serialized 5 byte log entry
1029 02:51:58.926012 (+ 3279us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:58.926308 (+ 296us) log.cc:844] Serialized 14 byte log entry
1029 02:51:58.932762 (+ 6454us) alter_schema_op.cc:73] Released schema lock
1029 02:51:58.932768 (+ 6us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":311708,"prepare.queue_time_us":358946,"prepare.run_cpu_time_us":2039,"prepare.run_wall_time_us":153783,"raft.queue_time_us":149,"raft.run_cpu_time_us":497,"raft.run_wall_time_us":497,"replication_time_us":215168,"spinlock_wait_cycles":165962624,"thread_start_us":756122,"threads_started":2}]]}
I20251029 02:51:58.933701 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c172417d1d444c439e9cda611a2ed551. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.937055 744 tablet_service.cc:1505] Processing CreateTablet for tablet 998b9cbe76514665be76c6c387dc75b3 (DEFAULT_TABLE table=default.Table_cd55951068e2462ab1b5b075a5b40767 [id=476be76ffbd643b49556539c2ffb653b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.934928 738 tablet_service.cc:1505] Processing CreateTablet for tablet 7b0f7d6c1c9842789c6ce2cecaa3d5b0 (DEFAULT_TABLE table=default.Table_b1d1e274a2ae4a639d80fb46e27bca52 [id=6a5cb859f2f64e26b39723468a768027]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.939823 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ef87f7a3dff4f00a0e7273eaec98dff. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.942093 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b0f7d6c1c9842789c6ce2cecaa3d5b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.945051 747 tablet_service.cc:1505] Processing CreateTablet for tablet 1878d7eef14a4578bc929da68872cee4 (DEFAULT_TABLE table=default.Table_d0a24cc3c6b244b0b43c78742dab0877 [id=78d1faddf0ca494ea722c99ca3566abe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.950703 754 tablet_service.cc:1505] Processing CreateTablet for tablet 94bdbb9394f7490e99916bd818986208 (DEFAULT_TABLE table=default.Table_243752c5b95a4421a86f6510f801e0b9 [id=bb5e87a155a949dba4636207700a30e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.955165 748 tablet_service.cc:1505] Processing CreateTablet for tablet c0672a948e5e41c5bbfe7e8199025986 (DEFAULT_TABLE table=default.Table_88289be99f39482d837c84fecf31efb1 [id=20c79a2035154fb79385b5149ca53eb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.952100 1172 raft_consensus.cc:359] T 8afc01d52d784f2fa559cab7ce07c8be P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:58.957813 750 tablet_service.cc:1505] Processing CreateTablet for tablet 5ae5f33567324d90b7a5a6e0f1b5fb67 (DEFAULT_TABLE table=default.Table_19039883faf6464387f65003d0d638a3 [id=1d031141cc2c44eface6277fd55922cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.959012 517 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_b22826191d2448e780eb6865d1cd7d05" } new_table_name: "default.Table_17185d86d94d40dc9d61f6f60d45bf51" modify_external_catalogs: true
I20251029 02:51:58.960143 742 tablet_service.cc:1505] Processing CreateTablet for tablet 0f96af1cb0ea4b46bd846ae0739f4947 (DEFAULT_TABLE table=default.Table_114cbdf1ef0f493f8f8c834735271053 [id=5107b378d9c24778ba7bf30c33b5516f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.961215 1172 raft_consensus.cc:385] T 8afc01d52d784f2fa559cab7ce07c8be P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:58.961314 1172 raft_consensus.cc:740] T 8afc01d52d784f2fa559cab7ce07c8be P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:51:58.962929 1172 consensus_queue.cc:260] T 8afc01d52d784f2fa559cab7ce07c8be P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:58.854432 878 tablet_service.cc:1505] Processing CreateTablet for tablet 7ff2399833e546448b85b9caeb5200fb (DEFAULT_TABLE table=default.Table_58fd9561b8e04989a95f097e29a79613 [id=9c994e4d21f042b2a5c1b86b27abfbb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.864198 900 tablet_service.cc:1505] Processing CreateTablet for tablet 6364601618314251b5601f30971ac61f (DEFAULT_TABLE table=default.Table_d497c064fe99434faf6566e6c5b6836c [id=5fa9ea29aa9c4175b68461684a57c420]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.867412 896 tablet_service.cc:1505] Processing CreateTablet for tablet ff3e9227b9674d31b874823cf4a62d9b (DEFAULT_TABLE table=default.Table_99567b9d8c154303bae22fda47db1d0a [id=2b8bd2d2c2664edaad536b1d7706096b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.878062 894 tablet_service.cc:1505] Processing CreateTablet for tablet eb78efbb100946a781056fab1d8bfaca (DEFAULT_TABLE table=default.Table_51aa8265801a4f8d9c0330cbf2bc72a4 [id=1efd18c0e7614e1da602a0d016391696]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.881387 899 tablet_service.cc:1505] Processing CreateTablet for tablet 5f52f7520ebc446f865aa7d999d9f7db (DEFAULT_TABLE table=default.Table_013611a621b2452c8ffb8d9161725af6 [id=02c02bd43c994c559655b1bee49893df]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.964373 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_8c062218c19c46169ae3652f572ed821"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:58.884979 881 tablet_service.cc:1552] Processing DeleteTablet for tablet 023a4094b424464c8f49d6eb9cc15da4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:58 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:58.885550 1667 tablet.cc:1722] T 59af906061d049c99ae2153945597007 P c756528642a543eeacd93cef3e335a7d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20251029 02:51:58.964918 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8c062218c19c46169ae3652f572ed821 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:51:58.909160 849 consensus_peers.cc:597] T 8f7c746f8b6948b58676298c2c2ea221 P c756528642a543eeacd93cef3e335a7d -> Peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Couldn't send request to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20251029 02:51:58.965571 849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:45003, user_credentials={real_user=slave}} blocked reactor thread for 56496us
W20251029 02:51:58.965962 711 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:41701, user_credentials={real_user=slave}} blocked reactor thread for 263092us
I20251029 02:51:58.966367 1167 tablet_bootstrap.cc:492] T 6f93812578a940a3b7378015e2a46f95 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:51:58.966563 1167 ts_tablet_manager.cc:1403] T 6f93812578a940a3b7378015e2a46f95 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.425s user 0.012s sys 0.003s
I20251029 02:51:58.967036 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 998b9cbe76514665be76c6c387dc75b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.967391 1167 raft_consensus.cc:359] T 6f93812578a940a3b7378015e2a46f95 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:58.967597 1167 raft_consensus.cc:385] T 6f93812578a940a3b7378015e2a46f95 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:58.967669 1167 raft_consensus.cc:740] T 6f93812578a940a3b7378015e2a46f95 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:51:58.967892 1167 consensus_queue.cc:260] T 6f93812578a940a3b7378015e2a46f95 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:58.968434 1167 ts_tablet_manager.cc:1434] T 6f93812578a940a3b7378015e2a46f95 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251029 02:51:58.932636 890 tablet_service.cc:1552] Processing DeleteTablet for tablet 1878d7eef14a4578bc929da68872cee4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:58 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:58.935798 1174 ts_tablet_manager.cc:1929] T ab421dfaa3f84593842bb732a2693160 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:58.969597 1174 log.cc:1199] T ab421dfaa3f84593842bb732a2693160 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/ab421dfaa3f84593842bb732a2693160
I20251029 02:51:58.970000 1174 ts_tablet_manager.cc:1950] T ab421dfaa3f84593842bb732a2693160 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:51:58.874176 883 tablet_service.cc:1505] Processing CreateTablet for tablet d9fe530823864f4e8c2643bf7fe6d383 (DEFAULT_TABLE table=default.Table_1b9efccf77ba4f8c91c0136ca2433af8 [id=31b2158fa0a54f798794379f76d55a23]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.939019 888 tablet_service.cc:1552] Processing DeleteTablet for tablet 7816556521144c928d7a02694e22e79f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:58 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:58.950224 879 tablet_service.cc:1552] Processing DeleteTablet for tablet c9bd3f2deb5d4af09b7918269a98509f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:58 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:58.950696 891 tablet_service.cc:1552] Processing DeleteTablet for tablet 94bdbb9394f7490e99916bd818986208 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:58 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:58.959697 895 tablet_service.cc:1505] Processing CreateTablet for tablet 845a69a8696e469a8548787a7645d41a (DEFAULT_TABLE table=default.Table_68075467476c41f2b5f28dcfa7633b10 [id=0c4e504854004a45be67fee7b396ff96]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.960208 893 tablet_service.cc:1505] Processing CreateTablet for tablet ead73d81d18a40a4b0aa9935f86421f8 (DEFAULT_TABLE table=default.Table_fd89990c08d142ffa08457a5802e5e69 [id=f3f342e7edc849fdbffea50a912b175f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.973209 525 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet 8afc01d52d784f2fa559cab7ce07c8be (table default.Table_2a10c9dfc91d4974845dbcd500d1d8be [id=ed7295f3f21848b79af75785d3983b67]). Expected version 1 got 0
I20251029 02:51:58.973451 525 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet 70e8cfe5eaa9492c85874dc3214a1e0b (table default.Table_eef9ea2caa6e4d73a71eebd2ab5f5cde [id=82df85af001340efb570040e047fa136]). Expected version 1 got 0
I20251029 02:51:58.974260 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a55f22b0d3b4ebe98ec2e054d05b095. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:51:58.975859 517 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b125cc9aaf4248b8bfa48fb152b832a6 consensus state has no leader
I20251029 02:51:58.976366 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_da40f10638dc4f96a028f73fd7f001f1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:58.976441 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_d901c631b88743509930ea084b31e7a8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:58.976929 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_da40f10638dc4f96a028f73fd7f001f1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:51:58.977010 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d901c631b88743509930ea084b31e7a8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:58.978183 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_fc1eba7107f64286916e3dfce5f828b6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:58.978696 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fc1eba7107f64286916e3dfce5f828b6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:58.979966 1176 ts_tablet_manager.cc:1929] T 8f7c746f8b6948b58676298c2c2ea221 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20251029 02:51:58.981387 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_724cce6ff005499cae9f0077c069e82a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:58.981873 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_724cce6ff005499cae9f0077c069e82a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:58.984265 891 tablet_service.cc:1505] Processing CreateTablet for tablet 30282e0f1b98401bb450e19965f037a5 (DEFAULT_TABLE table=default.Table_a714017562834a6dbfaa057b9f9aae41 [id=a6e350736cb6499a8d93cee9be338f08]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.984329 1660 tablet.cc:1722] T 24016ceaa84147eca0b94b3ad288e772 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 02:51:58.984583 890 tablet_service.cc:1505] Processing CreateTablet for tablet f2d47238761e4ed49c88abdbb677b18a (DEFAULT_TABLE table=default.Table_39a7937d49ce4ee290f72ed5295a2a3e [id=1571e82f4f9948089d81b0450dbfdee3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.985960 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb78efbb100946a781056fab1d8bfaca. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.988624 881 tablet_service.cc:1505] Processing CreateTablet for tablet 1fa8cc0c60cd43cdaa7a7646f70cfb97 (DEFAULT_TABLE table=default.Table_691548aa805841a281c6b0bbff6af9cc [id=2f66ded3048744238e4ab43d78520c2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.994129 755 tablet_service.cc:1505] Processing CreateTablet for tablet aee42fd9ca894319b3759beb2bae342e (DEFAULT_TABLE table=default.Table_2e760e5f5f1a4009b5d7c227df182a1e [id=51655edb7c684ca38744e37d5384be48]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:58.994410 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94bdbb9394f7490e99916bd818986208. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:58.997342 892 tablet_service.cc:1505] Processing CreateTablet for tablet 1cb0fdbd125446bf8d28080ea15e37ea (DEFAULT_TABLE table=default.Table_a111747c600544a3a67b954b834e4039 [id=29d9b8f74c26432c880e6ff411c1b7ad]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 02:51:58.994529 711 leader_election.cc:343] T 6feec2c8e030447fae9c292a626dcd1d P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
W20251029 02:51:58.998335 711 leader_election.cc:343] T a5c3a15510cb4321bbb80d61fabd0c0a P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:58.998704 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1878d7eef14a4578bc929da68872cee4. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:51:59.005879 711 leader_election.cc:343] T 3cded156380f45cc95893825c71ba0e9 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:59.006045 711 leader_election.cc:304] T 3cded156380f45cc95893825c71ba0e9 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:51:59.006381 1613 raft_consensus.cc:2749] T 3cded156380f45cc95893825c71ba0e9 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:51:59.006707 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ae5f33567324d90b7a5a6e0f1b5fb67. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:51:59.006906 711 leader_election.cc:343] T faa835de84ff45b19525d1d603678987 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:59.008150 711 leader_election.cc:304] T faa835de84ff45b19525d1d603678987 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:51:59.008461 1613 raft_consensus.cc:2749] T faa835de84ff45b19525d1d603678987 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:51:59.008692 1667 tablet.cc:1722] T 24016ceaa84147eca0b94b3ad288e772 P c756528642a543eeacd93cef3e335a7d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 02:51:59.013700 745 tablet_service.cc:1505] Processing CreateTablet for tablet d24fe99a1fb54476b90da2c22ff5d383 (DEFAULT_TABLE table=default.Table_e280ab7143204fbd8daa4f2c898617f9 [id=3c055a359301472cac54dd72a37520c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:59.013769 886 tablet_service.cc:1552] Processing DeleteTablet for tablet 6f93812578a940a3b7378015e2a46f95 with delete_type TABLET_DATA_DELETED (replaced by tablet ba699862e6f646c6beaa39f080d0faba) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:59.014145 1167 tablet_bootstrap.cc:492] T dd90fb9eb167478cbc3ec9363af86295 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:51:59.014210 898 tablet_service.cc:1505] Processing CreateTablet for tablet 85159079149241adae174fc0453ed0e7 (DEFAULT_TABLE table=default.Table_cb67eff3fb7c4cd9bc24d6bf31c3cd8f [id=3c7affb3187c4d1d987fdf2102c30a99]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:59.014955 879 tablet_service.cc:1505] Processing CreateTablet for tablet df669cf8da7f4c98a3f16423c55c15eb (DEFAULT_TABLE table=default.Table_c0a7b43b54fb41c6ab5347fdb4ea3cf2 [id=c480bda570c44f07a7c2a2664233619c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:59.015736 885 tablet_service.cc:1505] Processing CreateTablet for tablet 2089b058d0e94d99b40ad2718518353e (DEFAULT_TABLE table=default.Table_0f0a1e0ac81a4b3492d304bef28ee22d [id=b64a13ef78ef4005a4ca28599f3db270]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:59.016394 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f52f7520ebc446f865aa7d999d9f7db. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:59.018224 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1cb0fdbd125446bf8d28080ea15e37ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:59.022666 888 tablet_service.cc:1505] Processing CreateTablet for tablet fd1bf6b0f45349d0981f9ddd89269489 (DEFAULT_TABLE table=default.Table_5263ec7f27d74f73892b0ca252e9fa83 [id=18549ebdadaf443189d5f3eda682f3f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:59.022997 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85159079149241adae174fc0453ed0e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:59.022917 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_db8d41696cb54486aa9fa06d4396e499"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:59.023692 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_db8d41696cb54486aa9fa06d4396e499 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:59.026491 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9fe530823864f4e8c2643bf7fe6d383. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:59.027190 1167 tablet_bootstrap.cc:654] T dd90fb9eb167478cbc3ec9363af86295 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:59.030354 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff3e9227b9674d31b874823cf4a62d9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:59.033308 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd1bf6b0f45349d0981f9ddd89269489. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:59.034817 525 catalog_manager.cc:6194] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: default.Table_6da557988d044647b5e68f09b3f00407 [id=25f16d95843d41c981d6a07ec0d4b48a] alter complete (version 2)
I20251029 02:51:59.035027 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1fa8cc0c60cd43cdaa7a7646f70cfb97. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:59.035420 523 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_218b1046ed7c4b0d8b9c55a9542a6a2e" } new_table_name: "default.Table_683d67689da74097b69f31b3d9461fe2" modify_external_catalogs: true
I20251029 02:51:59.035960 524 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_04fadd7c87d1460dbc72c2caedbef915" } new_table_name: "default.Table_9aa86bf792a94cfeb454cf16575d139a" modify_external_catalogs: true
I20251029 02:51:59.037076 525 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_66370c05265c48dbbe5f1e6ebf0a940b" } new_table_name: "default.Table_d64a9ac717a94c40a98f826136089824" modify_external_catalogs: true
I20251029 02:51:59.038743 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6364601618314251b5601f30971ac61f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:59.041810 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f96af1cb0ea4b46bd846ae0739f4947. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:59.042665 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ff2399833e546448b85b9caeb5200fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:59.043196 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2d47238761e4ed49c88abdbb677b18a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:59.043716 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df669cf8da7f4c98a3f16423c55c15eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:59.043789 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_beeb4b56920e4aff85dfbb9758fa414c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:59.044180 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 845a69a8696e469a8548787a7645d41a. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:51:59.044374 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_beeb4b56920e4aff85dfbb9758fa414c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:59.045120 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ead73d81d18a40a4b0aa9935f86421f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:59.045497 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30282e0f1b98401bb450e19965f037a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:59.045822 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 318) took 2688 ms. Trace:
I20251029 02:51:59.045792 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_51e08a9cce7d4d9f9b358a693206462f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:59.045908 1174 rpcz_store.cc:276] 1029 02:51:56.305937 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:56.305986 (+ 49us) service_pool.cc:225] Handling call
1029 02:51:58.548688 (+2242702us) ts_tablet_manager.cc:1151] Deleting tablet ab421dfaa3f84593842bb732a2693160
1029 02:51:58.554176 (+ 5488us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:58.901294 (+347118us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:58.935685 (+ 34391us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:58.994550 (+ 58865us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2242489}
W20251029 02:51:59.046265 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_51e08a9cce7d4d9f9b358a693206462f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:59.046454 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 319) took 2730 ms. Trace:
I20251029 02:51:59.046571 1174 rpcz_store.cc:276] 1029 02:51:56.315964 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:56.316018 (+ 54us) service_pool.cc:225] Handling call
1029 02:51:59.046285 (+2730267us) ts_tablet_manager.cc:1151] Deleting tablet 6bd3fac2502e479aa9f4ceaabbb39504
1029 02:51:59.046298 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:59.046398 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2730064}
I20251029 02:51:59.046669 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet ab421dfaa3f84593842bb732a2693160 (table default.Table_73e59d41837947c4a4b59f6639ad6f06 [id=03a25420ce6a46fc93a9fb753246bc7b]) successfully deleted
W20251029 02:51:59.046959 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 6bd3fac2502e479aa9f4ceaabbb39504 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6bd3fac2502e479aa9f4ceaabbb39504 already in progress: creating tablet
I20251029 02:51:59.047077 1174 tablet_replica.cc:333] T 64b0fb400e944ce2b2c7a49d12135e9c P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:51:59.047253 1174 raft_consensus.cc:2243] T 64b0fb400e944ce2b2c7a49d12135e9c P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:59.043794 517 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_d37ad2580f614a6e9c469258b145525b" } new_table_name: "default.Table_570f159e617548ec974b41a654c1d800" modify_external_catalogs: true
I20251029 02:51:59.047426 1174 raft_consensus.cc:2272] T 64b0fb400e944ce2b2c7a49d12135e9c P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:59.047937 1174 ts_tablet_manager.cc:1916] T 64b0fb400e944ce2b2c7a49d12135e9c P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:59.048290 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2089b058d0e94d99b40ad2718518353e. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:51:59.048640 524 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6f8cf09f23984b879b60ab6b27c7d52e consensus state has no leader
W20251029 02:51:59.051396 523 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9e6eb099648549dbbf9969e59865f51f consensus state has no leader
W20251029 02:51:59.056604 525 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 32519fbffcef44a890161e5f3043806c consensus state has no leader
I20251029 02:51:59.059422 1172 ts_tablet_manager.cc:1434] T 8afc01d52d784f2fa559cab7ce07c8be P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.067s user 0.001s sys 0.001s
I20251029 02:51:59.070780 520 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_21b8ae2debb241bda65fe387a4a3190f" } modify_external_catalogs: true
I20251029 02:51:59.071010 520 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_21b8ae2debb241bda65fe387a4a3190f" } modify_external_catalogs: true
I20251029 02:51:59.072084 526 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_a66eb4754b854a998e458627d738338b" } modify_external_catalogs: true
I20251029 02:51:59.072240 526 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_a66eb4754b854a998e458627d738338b" } modify_external_catalogs: true
I20251029 02:51:59.074971 1176 log.cc:1199] T 8f7c746f8b6948b58676298c2c2ea221 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/8f7c746f8b6948b58676298c2c2ea221
I20251029 02:51:59.081265 1023 tablet_service.cc:1505] Processing CreateTablet for tablet 7b0f7d6c1c9842789c6ce2cecaa3d5b0 (DEFAULT_TABLE table=default.Table_b1d1e274a2ae4a639d80fb46e27bca52 [id=6a5cb859f2f64e26b39723468a768027]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 02:51:59.081514 517 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f529e17e97074dca91ec2f48954cfd32 consensus state has no leader
I20251029 02:51:59.081714 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b0f7d6c1c9842789c6ce2cecaa3d5b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:59.081763 1035 tablet_service.cc:1505] Processing CreateTablet for tablet a530fe8de8ae4be5b0fd9a5f7e94ea9e (DEFAULT_TABLE table=default.Table_59a5648999114c6f8b2a58e938c8e49a [id=d34f7b094b6c48bea8a767a1e913d74f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:59.084357 1166 tablet_bootstrap.cc:492] T 75236b2fa82f4f28b82efae9a2c5160d P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:51:59.087129 1167 tablet_bootstrap.cc:492] T dd90fb9eb167478cbc3ec9363af86295 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:51:59.087076 523 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_b9f2a2e3d9f8446fbd46ca6c48155ccf" } modify_external_catalogs: true
I20251029 02:51:59.087307 523 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_b9f2a2e3d9f8446fbd46ca6c48155ccf" } modify_external_catalogs: true
I20251029 02:51:59.087426 1167 ts_tablet_manager.cc:1403] T dd90fb9eb167478cbc3ec9363af86295 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.100s user 0.000s sys 0.004s
I20251029 02:51:59.090122 517 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_5580c367c15f4f3a87626302cfde455f" } modify_external_catalogs: true
I20251029 02:51:59.090291 517 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_5580c367c15f4f3a87626302cfde455f" } modify_external_catalogs: true
I20251029 02:51:59.088444 1167 raft_consensus.cc:359] T dd90fb9eb167478cbc3ec9363af86295 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:59.091437 1167 raft_consensus.cc:385] T dd90fb9eb167478cbc3ec9363af86295 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:59.091679 1167 raft_consensus.cc:740] T dd90fb9eb167478cbc3ec9363af86295 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:51:59.092592 1167 consensus_queue.cc:260] T dd90fb9eb167478cbc3ec9363af86295 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:59.093945 737 tablet_service.cc:1505] Processing CreateTablet for tablet 78662eeae70b4148a4ad0d47be5c566d (DEFAULT_TABLE table=default.Table_fd6ca61b22e442fabd966ea848f4389e [id=d45dc85e3f3244ed8024eceb15d3e765]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:59.096843 1167 ts_tablet_manager.cc:1434] T dd90fb9eb167478cbc3ec9363af86295 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20251029 02:51:59.097962 1172 tablet_bootstrap.cc:492] T 2242f5b2c20346d9b753d7cd29242441 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:51:59.098997 521 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_a15f5c838c7d4600b91a7368321fff38" } modify_external_catalogs: true
I20251029 02:51:59.099223 521 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_a15f5c838c7d4600b91a7368321fff38" } modify_external_catalogs: true
I20251029 02:51:59.100381 1167 tablet_bootstrap.cc:492] T c519832030884a049839d309cc738c28 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:51:59.109158 1166 ts_tablet_manager.cc:1403] T 75236b2fa82f4f28b82efae9a2c5160d P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.607s user 0.008s sys 0.007s
I20251029 02:51:59.109355 1031 tablet_service.cc:1505] Processing CreateTablet for tablet 5ae5f33567324d90b7a5a6e0f1b5fb67 (DEFAULT_TABLE table=default.Table_19039883faf6464387f65003d0d638a3 [id=1d031141cc2c44eface6277fd55922cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:59.109839 1038 tablet_service.cc:1552] Processing DeleteTablet for tablet 5ae5f33567324d90b7a5a6e0f1b5fb67 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:58 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:59.110700 1039 tablet_service.cc:1505] Processing CreateTablet for tablet aee42fd9ca894319b3759beb2bae342e (DEFAULT_TABLE table=default.Table_2e760e5f5f1a4009b5d7c227df182a1e [id=51655edb7c684ca38744e37d5384be48]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:59.111220 1036 tablet_service.cc:1505] Processing CreateTablet for tablet 998b9cbe76514665be76c6c387dc75b3 (DEFAULT_TABLE table=default.Table_cd55951068e2462ab1b5b075a5b40767 [id=476be76ffbd643b49556539c2ffb653b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:59.113590 524 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet 8afc01d52d784f2fa559cab7ce07c8be (table default.Table_2a10c9dfc91d4974845dbcd500d1d8be [id=ed7295f3f21848b79af75785d3983b67]). Expected version 1 got 0
I20251029 02:51:59.115808 1033 tablet_service.cc:1505] Processing CreateTablet for tablet 94bdbb9394f7490e99916bd818986208 (DEFAULT_TABLE table=default.Table_243752c5b95a4421a86f6510f801e0b9 [id=bb5e87a155a949dba4636207700a30e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:59.116163 1032 tablet_service.cc:1505] Processing CreateTablet for tablet c172417d1d444c439e9cda611a2ed551 (DEFAULT_TABLE table=default.Table_293c6e92af8a4707be8e2596399fd6a2 [id=d7a06dc089ab4371813f376ef37261ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:59.116490 1042 tablet_service.cc:1505] Processing CreateTablet for tablet 358bdfbcb4ee440c899fb6eedef68f44 (DEFAULT_TABLE table=default.Table_93bc8cfb59f440849d8ae4d6529e6a01 [id=38f2afef718b46828519c300d6b592b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:59.116858 1041 tablet_service.cc:1505] Processing CreateTablet for tablet c0672a948e5e41c5bbfe7e8199025986 (DEFAULT_TABLE table=default.Table_88289be99f39482d837c84fecf31efb1 [id=20c79a2035154fb79385b5149ca53eb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:59.117199 1026 tablet_service.cc:1505] Processing CreateTablet for tablet 1878d7eef14a4578bc929da68872cee4 (DEFAULT_TABLE table=default.Table_d0a24cc3c6b244b0b43c78742dab0877 [id=78d1faddf0ca494ea722c99ca3566abe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:59.119159 1176 ts_tablet_manager.cc:1950] T 8f7c746f8b6948b58676298c2c2ea221 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:51:59.124402 525 catalog_manager.cc:6194] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: default.Table_f06003da6e324504ade193bbbb11d36a [id=bd279b3ce623477ca20e8d35e3b7c0df] alter complete (version 1)
I20251029 02:51:59.125083 520 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 24f07b1efac64b099b0b5ed467463749
I20251029 02:51:59.136505 521 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 5842e9bba4fa4295ba21e90d645a56df
I20251029 02:51:59.137773 526 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 40e844fd7ad74680bcbe7bef936c5b9d
I20251029 02:51:59.138146 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0672a948e5e41c5bbfe7e8199025986. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:59.138743 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aee42fd9ca894319b3759beb2bae342e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:59.138897 517 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 28f7ceb91a34450cad4b5b45f5285b41
I20251029 02:51:59.139083 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78662eeae70b4148a4ad0d47be5c566d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:59.140658 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_62644549a49a4f039492b17deac6113d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:59.141031 1172 tablet_bootstrap.cc:654] T 2242f5b2c20346d9b753d7cd29242441 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
W20251029 02:51:59.141251 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_62644549a49a4f039492b17deac6113d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:59.141881 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_1ff672db4f064544b523c0ae9667d967"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:59.142185 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d24fe99a1fb54476b90da2c22ff5d383. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:51:59.142391 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1ff672db4f064544b523c0ae9667d967 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:59.147262 523 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 58dde9397b434b089ade9968f369006a
I20251029 02:51:59.152774 1167 tablet_bootstrap.cc:654] T c519832030884a049839d309cc738c28 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:59.161629 1027 tablet_service.cc:1505] Processing CreateTablet for tablet 5ae35704b3944b36bc5199f9ae0546f6 (DEFAULT_TABLE table=default.Table_e5a5de26cf4041fca4bfbc384115b9b3 [id=163c34d030c94e21b43295fd6e457d0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:59.166215 892 tablet_service.cc:1505] Processing CreateTablet for tablet 07316861d2da4f55bd41111e400677fe (DEFAULT_TABLE table=default.Table_fc1eba7107f64286916e3dfce5f828b6 [id=7a248a5faa8346e09038e9e1c892625b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:59.167922 1037 tablet_service.cc:1505] Processing CreateTablet for tablet 5842e9bba4fa4295ba21e90d645a56df (DEFAULT_TABLE table=default.Table_a15f5c838c7d4600b91a7368321fff38 [id=67b92918d6fc4c1fa795908bc20bd707]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:59.168213 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07316861d2da4f55bd41111e400677fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:59.169811 517 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet dd90fb9eb167478cbc3ec9363af86295 (table default.Table_8b6217e43e8d467693c104ef84e78911 [id=052abf3dcc904bdcb869d5b885723dab]). Expected version 4 got 0
I20251029 02:51:59.165865 1024 tablet_service.cc:1505] Processing CreateTablet for tablet 5ef87f7a3dff4f00a0e7273eaec98dff (DEFAULT_TABLE table=default.Table_fd1d23f31ebf47b884897c68eac9004d [id=1c99ebd287ff47d08e817853a628256c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:59.175020 884 tablet_service.cc:1505] Processing CreateTablet for tablet 15cadd681ceb4daba178a9cc2b7aec08 (DEFAULT_TABLE table=default.Table_bc803f454de540c28f741096815c1ce2 [id=f132223faea74eb2b424cecb26440aac]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:59.175168 878 tablet_service.cc:1552] Processing DeleteTablet for tablet 24f07b1efac64b099b0b5ed467463749 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:59 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:59.176014 896 tablet_service.cc:1552] Processing DeleteTablet for tablet 5842e9bba4fa4295ba21e90d645a56df with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:59 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:59.176688 881 tablet_service.cc:1505] Processing CreateTablet for tablet a945d64287f94ca697447e540fab39ce (DEFAULT_TABLE table=default.Table_d901c631b88743509930ea084b31e7a8 [id=74b92cc3eadb43ec96c77b178917a04d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:59.181773 517 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_6d866cb3e90a40dcb068990637194ac5" } new_table_name: "default.Table_8a277d5e05fb437096e36f29a6307d74" modify_external_catalogs: true
I20251029 02:51:59.185707 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_8ecb63fece7348379a7d341b1c4b5891"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:59.186293 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8ecb63fece7348379a7d341b1c4b5891 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:59.187924 1034 tablet_service.cc:1505] Processing CreateTablet for tablet 78662eeae70b4148a4ad0d47be5c566d (DEFAULT_TABLE table=default.Table_fd6ca61b22e442fabd966ea848f4389e [id=d45dc85e3f3244ed8024eceb15d3e765]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:59.187933 895 tablet_service.cc:1505] Processing CreateTablet for tablet 85f979babbd34eaeaa89ea5906ddab90 (DEFAULT_TABLE table=default.Table_8c062218c19c46169ae3652f572ed821 [id=8a5e8af85fb74b71b4fa8568c0606319]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:59.188344 1040 tablet_service.cc:1505] Processing CreateTablet for tablet 0f96af1cb0ea4b46bd846ae0739f4947 (DEFAULT_TABLE table=default.Table_114cbdf1ef0f493f8f8c834735271053 [id=5107b378d9c24778ba7bf30c33b5516f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:59.188443 1029 tablet_service.cc:1505] Processing CreateTablet for tablet 7d1408dc544944669de519a1553d9bb7 (DEFAULT_TABLE table=default.Table_3d101961029143a882eeafb119560e5d [id=4d483223ce624eea98e68a9375900497]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:59.188555 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_c4ce908a6d5c40e1b825533a63689d0a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:59.188366 1030 tablet_service.cc:1505] Processing CreateTablet for tablet c8389c8bfead4ff7a23f363c9c6770bf (DEFAULT_TABLE table=default.Table_b1c04ce867284dc5bf55b02f25bf66ae [id=7d8b9c11844d47f6a2eecd698765f050]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:59.188813 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 358) took 2215 ms. Trace:
I20251029 02:51:59.188906 1176 rpcz_store.cc:276] 1029 02:51:56.915408 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:56.915951 (+ 543us) service_pool.cc:225] Handling call
1029 02:51:58.756476 (+1840525us) ts_tablet_manager.cc:1151] Deleting tablet 8f7c746f8b6948b58676298c2c2ea221
1029 02:51:58.756486 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:58.915305 (+158819us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:58.969025 (+ 53720us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:59.130599 (+161574us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1840307}
W20251029 02:51:59.189050 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c4ce908a6d5c40e1b825533a63689d0a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:59.189067 1025 tablet_service.cc:1505] Processing CreateTablet for tablet e74eda48dca647efacabe525ef316944 (DEFAULT_TABLE table=default.Table_6e1283a0c161498db405cf1e9b2c93d7 [id=4de1990274fe4b8487506a86872fe976]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:59.189374 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 359) took 2264 ms. Trace:
I20251029 02:51:59.189482 1176 rpcz_store.cc:276] 1029 02:51:56.924701 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:56.924762 (+ 61us) service_pool.cc:225] Handling call
1029 02:51:59.189199 (+2264437us) ts_tablet_manager.cc:1151] Deleting tablet 51effc9324034ae79e2b25cb57a996c1
1029 02:51:59.189210 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:59.189318 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2264190}
I20251029 02:51:59.190419 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 360) took 2262 ms. Trace:
I20251029 02:51:59.190551 1176 rpcz_store.cc:276] 1029 02:51:56.927286 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:56.927340 (+ 54us) service_pool.cc:225] Handling call
1029 02:51:59.189745 (+2262405us) ts_tablet_manager.cc:1151] Deleting tablet 71068739b4384badbf305313d902f0f5
1029 02:51:59.189754 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:59.189840 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2262169}
I20251029 02:51:59.190960 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 356) took 2275 ms. Trace:
I20251029 02:51:59.191053 1176 rpcz_store.cc:276] 1029 02:51:56.915173 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:56.927583 (+ 12410us) service_pool.cc:225] Handling call
1029 02:51:59.190810 (+2263227us) ts_tablet_manager.cc:1151] Deleting tablet 1275e192871440408a8345fef9b56260
1029 02:51:59.190818 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:59.190904 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2262978}
I20251029 02:51:59.191370 1176 tablet_replica.cc:333] T 012ce243fd07466ab5301d1c9ce4b413 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:51:59.191579 1166 raft_consensus.cc:359] T 75236b2fa82f4f28b82efae9a2c5160d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:59.191784 1166 raft_consensus.cc:385] T 75236b2fa82f4f28b82efae9a2c5160d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:59.191881 1166 raft_consensus.cc:740] T 75236b2fa82f4f28b82efae9a2c5160d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:51:59.192117 1166 consensus_queue.cc:260] T 75236b2fa82f4f28b82efae9a2c5160d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:59.192602 1166 ts_tablet_manager.cc:1434] T 75236b2fa82f4f28b82efae9a2c5160d P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.083s user 0.001s sys 0.001s
I20251029 02:51:59.192932 1166 tablet_bootstrap.cc:492] T fd900eec3f0c43bf965c148ea90870a3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
W20251029 02:51:59.194092 517 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 72f9d4408f384e7093316d85f092e1b2 consensus state has no leader
I20251029 02:51:59.194631 1038 tablet_service.cc:1505] Processing CreateTablet for tablet 75aa2dd1ed1b49649f3257df5d45cb32 (DEFAULT_TABLE table=default.Table_dbf30facf56548e1b20d6fec42080e05 [id=e03d91a588f0492ab73c4f3f6d4022e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:59.195096 1028 tablet_service.cc:1505] Processing CreateTablet for tablet e5336762c90e43278db7978dda295e40 (DEFAULT_TABLE table=default.Table_2474d13afb604655864254dd55221051 [id=3d9f1bcd08c74e65b3b04c50e4613c74]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:59.196864 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15cadd681ceb4daba178a9cc2b7aec08. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:59.196815 899 tablet_service.cc:1505] Processing CreateTablet for tablet b41e114ed6e94f5895bdec9bee836d2d (DEFAULT_TABLE table=default.Table_203b46ee33744bf3bfe75190ab006c13 [id=825464f47f3a45f79eb319c3ca3e54e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:59.199870 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_9cd59d4e874c4239b50b475aa8def07c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:59.200461 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9cd59d4e874c4239b50b475aa8def07c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:59.200668 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_556d748de1214c5592a0f74d11d7a7d9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:59.203155 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_556d748de1214c5592a0f74d11d7a7d9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:59.206197 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_290338e0424d46cf9166ed27b1c765a1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:59.206874 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_290338e0424d46cf9166ed27b1c765a1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:59.212358 749 tablet_service.cc:1505] Processing CreateTablet for tablet 7d1408dc544944669de519a1553d9bb7 (DEFAULT_TABLE table=default.Table_3d101961029143a882eeafb119560e5d [id=4d483223ce624eea98e68a9375900497]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:59.213519 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b41e114ed6e94f5895bdec9bee836d2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:59.223803 746 tablet_service.cc:1505] Processing CreateTablet for tablet 358bdfbcb4ee440c899fb6eedef68f44 (DEFAULT_TABLE table=default.Table_93bc8cfb59f440849d8ae4d6529e6a01 [id=38f2afef718b46828519c300d6b592b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:59.225793 900 tablet_service.cc:1505] Processing CreateTablet for tablet 5171c8aa52ac497d9858bb6d0ea8dfee (DEFAULT_TABLE table=default.Table_724cce6ff005499cae9f0077c069e82a [id=def36e1439084a659906905bd1eb7993]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:59.226191 890 tablet_service.cc:1552] Processing DeleteTablet for tablet 40e844fd7ad74680bcbe7bef936c5b9d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:59 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:59.227632 879 tablet_service.cc:1505] Processing CreateTablet for tablet ea7b977224f24bf49dadfb83ffa28310 (DEFAULT_TABLE table=default.Table_da40f10638dc4f96a028f73fd7f001f1 [id=579d0cc01040411f9de03e699ff959e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:59.247478 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d1408dc544944669de519a1553d9bb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:59.247761 896 tablet_service.cc:1552] Processing DeleteTablet for tablet 58dde9397b434b089ade9968f369006a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:59 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:59.250329 1599 raft_consensus.cc:493] T dd90fb9eb167478cbc3ec9363af86295 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:59.250530 1599 raft_consensus.cc:515] T dd90fb9eb167478cbc3ec9363af86295 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:59.252357 520 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_86ad8c0a1e9c4fd6b12b9f59a195e951" } new_table_name: "default.Table_863cbc1c4f434b7d818bb0c8b168cb73" modify_external_catalogs: true
I20251029 02:51:59.252909 521 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_58fd9561b8e04989a95f097e29a79613" } new_table_name: "default.Table_220c535de1944595a5c5c101efaed2f3" modify_external_catalogs: true
I20251029 02:51:59.254400 878 tablet_service.cc:1552] Processing DeleteTablet for tablet 28f7ceb91a34450cad4b5b45f5285b41 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:59 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:59.254745 890 tablet_service.cc:1505] Processing CreateTablet for tablet a7685b0bed174ed0b25f99ef8a46fca3 (DEFAULT_TABLE table=default.Table_beeb4b56920e4aff85dfbb9758fa414c [id=213b1abd06cf4e22b05bfb35460bdd3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:59.254949 1599 leader_election.cc:290] T dd90fb9eb167478cbc3ec9363af86295 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:51:59.255205 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85f979babbd34eaeaa89ea5906ddab90. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:59.255594 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5171c8aa52ac497d9858bb6d0ea8dfee. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:59.255792 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7685b0bed174ed0b25f99ef8a46fca3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:59.255973 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea7b977224f24bf49dadfb83ffa28310. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:59.255947 1062 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd90fb9eb167478cbc3ec9363af86295" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:51:59.258769 517 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_fc1eba7107f64286916e3dfce5f828b6" } new_table_name: "default.Table_330b705cd45340578758a7fb532d14fa" modify_external_catalogs: true
I20251029 02:51:59.261113 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_c32e2f0d85f9486d8957cedb7e09dda5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:59.261679 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c32e2f0d85f9486d8957cedb7e09dda5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:59.262945 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_360a4b04c44345e0a36d04e6d90c698f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:59.263453 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_360a4b04c44345e0a36d04e6d90c698f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:59.268807 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a945d64287f94ca697447e540fab39ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:59.269629 898 tablet_service.cc:1505] Processing CreateTablet for tablet 9c6f4dc4b0c74ddd80d9dc762619b1ec (DEFAULT_TABLE table=default.Table_51e08a9cce7d4d9f9b358a693206462f [id=e1f53eaba1d64ef495c67d40be796e49]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:59.270278 883 tablet_service.cc:1505] Processing CreateTablet for tablet a04bc4d9605d4c049a151d985035bb63 (DEFAULT_TABLE table=default.Table_62644549a49a4f039492b17deac6113d [id=bb15ae87ebcd4a39a99982967fd215b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:59.270931 893 tablet_service.cc:1505] Processing CreateTablet for tablet c8b6afc8c77c4a7fbe1956cb30bba656 (DEFAULT_TABLE table=default.Table_db8d41696cb54486aa9fa06d4396e499 [id=f87557451c8c466c8f6ea573155aa684]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:59.272006 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 358bdfbcb4ee440c899fb6eedef68f44. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:59.275014 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd90fb9eb167478cbc3ec9363af86295" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:51:59.274994 751 tablet_service.cc:1505] Processing CreateTablet for tablet c8389c8bfead4ff7a23f363c9c6770bf (DEFAULT_TABLE table=default.Table_b1c04ce867284dc5bf55b02f25bf66ae [id=7d8b9c11844d47f6a2eecd698765f050]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:59.276010 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8389c8bfead4ff7a23f363c9c6770bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:59.276551 1172 tablet_bootstrap.cc:492] T 2242f5b2c20346d9b753d7cd29242441 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:51:59.277001 1172 ts_tablet_manager.cc:1403] T 2242f5b2c20346d9b753d7cd29242441 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.214s user 0.017s sys 0.008s
I20251029 02:51:59.278975 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a530fe8de8ae4be5b0fd9a5f7e94ea9e. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:51:59.281319 520 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 38f83ebcb2a440bc8e6a8d36607e6666 consensus state has no leader
W20251029 02:51:59.282565 521 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7ff2399833e546448b85b9caeb5200fb consensus state has no leader
I20251029 02:51:59.285784 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8b6afc8c77c4a7fbe1956cb30bba656. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:59.285830 847 leader_election.cc:304] T dd90fb9eb167478cbc3ec9363af86295 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9
I20251029 02:51:59.286306 1599 raft_consensus.cc:2749] T dd90fb9eb167478cbc3ec9363af86295 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:51:59.287504 521 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_2474d13afb604655864254dd55221051" } new_table_name: "default.Table_2f3d2d9f6b7a40edaff47555449e4a49" modify_external_catalogs: true
I20251029 02:51:59.289471 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a04bc4d9605d4c049a151d985035bb63. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:59.290302 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c6f4dc4b0c74ddd80d9dc762619b1ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:59.292488 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ae5f33567324d90b7a5a6e0f1b5fb67. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:59.293242 891 tablet_service.cc:1505] Processing CreateTablet for tablet 699ee74f39b342639ab84d0d65e53a70 (DEFAULT_TABLE table=default.Table_1ff672db4f064544b523c0ae9667d967 [id=60cf9a35c3c848d8a4e7cd7a8128b12f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:59.298224 1613 raft_consensus.cc:493] T 9e07291d6b5645feb4d5890f58e1ef74 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:59.298432 1613 raft_consensus.cc:515] T 9e07291d6b5645feb4d5890f58e1ef74 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:59.300575 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 699ee74f39b342639ab84d0d65e53a70. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:59.301057 737 tablet_service.cc:1505] Processing CreateTablet for tablet 75aa2dd1ed1b49649f3257df5d45cb32 (DEFAULT_TABLE table=default.Table_dbf30facf56548e1b20d6fec42080e05 [id=e03d91a588f0492ab73c4f3f6d4022e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:59.308498 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_4559c2fec511424f8d43a3d208c5ee61"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:59.309141 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4559c2fec511424f8d43a3d208c5ee61 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:59.309829 743 tablet_service.cc:1505] Processing CreateTablet for tablet 5842e9bba4fa4295ba21e90d645a56df (DEFAULT_TABLE table=default.Table_a15f5c838c7d4600b91a7368321fff38 [id=67b92918d6fc4c1fa795908bc20bd707]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:59.314065 753 tablet_service.cc:1505] Processing CreateTablet for tablet 5ae35704b3944b36bc5199f9ae0546f6 (DEFAULT_TABLE table=default.Table_e5a5de26cf4041fca4bfbc384115b9b3 [id=163c34d030c94e21b43295fd6e457d0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:59.315853 1613 leader_election.cc:290] T 9e07291d6b5645feb4d5890f58e1ef74 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:51:59.302381 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5842e9bba4fa4295ba21e90d645a56df. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:59.321242 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75aa2dd1ed1b49649f3257df5d45cb32. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:59.322286 756 tablet_service.cc:1505] Processing CreateTablet for tablet e5336762c90e43278db7978dda295e40 (DEFAULT_TABLE table=default.Table_2474d13afb604655864254dd55221051 [id=3d9f1bcd08c74e65b3b04c50e4613c74]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:59.326442 739 tablet_service.cc:1552] Processing DeleteTablet for tablet c8389c8bfead4ff7a23f363c9c6770bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:58 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:59.327564 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8389c8bfead4ff7a23f363c9c6770bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:59.327867 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ae35704b3944b36bc5199f9ae0546f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:59.327816 525 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_863cbc1c4f434b7d818bb0c8b168cb73" } modify_external_catalogs: true
I20251029 02:51:59.330116 1166 tablet_bootstrap.cc:654] T fd900eec3f0c43bf965c148ea90870a3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:59.331084 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78662eeae70b4148a4ad0d47be5c566d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:59.331550 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ef87f7a3dff4f00a0e7273eaec98dff. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:59.332060 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0f96af1cb0ea4b46bd846ae0739f4947. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:59.332361 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 358bdfbcb4ee440c899fb6eedef68f44. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:59.332872 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e74eda48dca647efacabe525ef316944. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:59.334914 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5ae35704b3944b36bc5199f9ae0546f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:59.337617 747 tablet_service.cc:1552] Processing DeleteTablet for tablet f18b0ca5a6a44175a912642e9eea1778 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:58 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:59.337926 738 tablet_service.cc:1552] Processing DeleteTablet for tablet eca016aed05e4ce09e1a2bf96e2286f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:58 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:59.338272 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 998b9cbe76514665be76c6c387dc75b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:59.341461 524 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_9c1124b16dc44ec5b2696d567f1c52f0" } modify_external_catalogs: true
I20251029 02:51:59.342002 518 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_99592a0735024ba7a0d8d6145b61f2e4" } modify_external_catalogs: true
I20251029 02:51:59.342872 519 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_68075467476c41f2b5f28dcfa7633b10" } modify_external_catalogs: true
I20251029 02:51:59.348435 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1878d7eef14a4578bc929da68872cee4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:59.349781 522 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_33eacbf00740465185d70b46a3782893" } modify_external_catalogs: true
I20251029 02:51:59.355373 518 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_99592a0735024ba7a0d8d6145b61f2e4" } modify_external_catalogs: true
W20251029 02:51:59.355518 517 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 07316861d2da4f55bd41111e400677fe consensus state has no leader
I20251029 02:51:59.357481 524 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_9c1124b16dc44ec5b2696d567f1c52f0" } modify_external_catalogs: true
I20251029 02:51:59.357751 522 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_33eacbf00740465185d70b46a3782893" } modify_external_catalogs: true
I20251029 02:51:59.357524 525 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_863cbc1c4f434b7d818bb0c8b168cb73" } modify_external_catalogs: true
I20251029 02:51:59.358790 519 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_68075467476c41f2b5f28dcfa7633b10" } modify_external_catalogs: true
I20251029 02:51:59.359200 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_d44327be26be4ff1b62a7be7dada45c0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:59.359910 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d44327be26be4ff1b62a7be7dada45c0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:59.364269 752 tablet_service.cc:1505] Processing CreateTablet for tablet 5bd2163bc4be467fb695629bfbb5a6ca (DEFAULT_TABLE table=default.Table_a45edb83b6254a4bb8fb610220e5cf65 [id=19f7af2387764bb8813321ddb6443d8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:59.366163 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_a8855248e99e4374aaaf0d4f66e9cad1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:59.356146 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d1408dc544944669de519a1553d9bb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:59.368840 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aee42fd9ca894319b3759beb2bae342e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:59.370667 1172 raft_consensus.cc:359] T 2242f5b2c20346d9b753d7cd29242441 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
W20251029 02:51:59.373766 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a8855248e99e4374aaaf0d4f66e9cad1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:59.374289 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5336762c90e43278db7978dda295e40. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:59.378713 522 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet fcb3db8399aa4cf188b07dda220f7f5b
I20251029 02:51:59.380770 1176 raft_consensus.cc:2243] T 012ce243fd07466ab5301d1c9ce4b413 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:59.382823 519 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 845a69a8696e469a8548787a7645d41a
I20251029 02:51:59.384379 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet d6e1170dbb594a5696a3899cde8da7c9 (table default.Table_4d2c99159e924a06858977465efa38a6 [id=ad583c111f524bb0a40dcf1eb4d5c527]) successfully deleted
I20251029 02:51:59.385511 525 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 38f83ebcb2a440bc8e6a8d36607e6666
W20251029 02:51:59.389523 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet f187dbd8a55b45b292ea61db43caf47d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f187dbd8a55b45b292ea61db43caf47d already in progress: creating tablet
I20251029 02:51:59.390413 1176 raft_consensus.cc:2272] T 012ce243fd07466ab5301d1c9ce4b413 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:59.391093 1176 ts_tablet_manager.cc:1916] T 012ce243fd07466ab5301d1c9ce4b413 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 02:51:59.392902 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 1bc45078a6a942c38ff3350c15885eff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1bc45078a6a942c38ff3350c15885eff already in progress: creating tablet
W20251029 02:51:59.398180 521 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e5336762c90e43278db7978dda295e40 consensus state has no leader
I20251029 02:51:59.398840 518 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 8df97119d0654d7d92bb5bd17c5d8562
I20251029 02:51:59.399210 524 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet f65fe9a5137b48669afef51ad7dd76ae
I20251029 02:51:59.401548 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94bdbb9394f7490e99916bd818986208. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:59.402992 517 master_service.cc:954] ReplaceTablet: received request to replace tablet 7bb85a12f573494cbb3118d1947eda27 from {username='slave'} at 127.0.0.1:35314
I20251029 02:51:59.403802 739 tablet_service.cc:1552] Processing DeleteTablet for tablet 86fc22bb84794149b598b97df41eb881 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:58 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:59.404347 878 tablet_service.cc:1552] Processing DeleteTablet for tablet 8df97119d0654d7d92bb5bd17c5d8562 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:59 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:59.404351 739 tablet_service.cc:1505] Processing CreateTablet for tablet 4a55f22b0d3b4ebe98ec2e054d05b095 (DEFAULT_TABLE table=default.Table_2e41802f8663436e83915f560cf8f94b [id=4e2615cec64b4828b8311b37e4a568b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:59.404934 885 tablet_service.cc:1552] Processing DeleteTablet for tablet 845a69a8696e469a8548787a7645d41a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:59 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:59.403836 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 1fc87e2d7b4240e08f2b4ac7277293ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:56 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:59.403802 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5842e9bba4fa4295ba21e90d645a56df. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:59.405467 892 tablet_service.cc:1552] Processing DeleteTablet for tablet f65fe9a5137b48669afef51ad7dd76ae with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:59 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:59.407817 1660 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:37626 (request call id 420) took 1461 ms. Trace:
W20251029 02:51:59.409224 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 1fc87e2d7b4240e08f2b4ac7277293ff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1fc87e2d7b4240e08f2b4ac7277293ff already in progress: creating tablet
W20251029 02:51:59.410101 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 59eee86f249e4d6a9d09d30276dcd659 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 59eee86f249e4d6a9d09d30276dcd659 already in progress: creating tablet
I20251029 02:51:59.410285 524 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_8cf5614390ae4a32a856fce51f3edc08" } new_table_name: "default.Table_4df27f8e1a5d4e1596b3c52a1f6ab58d" modify_external_catalogs: true
W20251029 02:51:59.410929 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 6d830d30b26b4ba799785fec8a6f0769 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6d830d30b26b4ba799785fec8a6f0769 already in progress: creating tablet
I20251029 02:51:59.407945 1660 rpcz_store.cc:276] 1029 02:51:57.935690 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:57.935761 (+ 71us) service_pool.cc:225] Handling call
1029 02:51:59.381207 (+1445446us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1029 02:51:58.362165 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
1029 02:51:58.430122 (+ 67957us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
1029 02:51:58.430132 (+ 10us) alter_schema_op.cc:66] Acquired schema lock
1029 02:51:58.430141 (+ 9us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
1029 02:51:58.430191 (+ 50us) alter_schema_op.cc:128] START. Timestamp: 1761706318430155
1029 02:51:58.430194 (+ 3us) op_driver.cc:348] REPLICATION: starting
1029 02:51:58.433425 (+ 3231us) log.cc:844] Serialized 173 byte log entry
1029 02:51:58.973443 (+540018us) op_driver.cc:464] REPLICATION: finished
1029 02:51:58.973557 (+ 114us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
1029 02:51:59.367709 (+394152us) log.cc:844] Serialized 5 byte log entry
1029 02:51:59.377184 (+ 9475us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:59.380947 (+ 3763us) log.cc:844] Serialized 14 byte log entry
1029 02:51:59.381162 (+ 215us) alter_schema_op.cc:73] Released schema lock
1029 02:51:59.381167 (+ 5us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":65,"prepare.queue_time_us":416405,"prepare.run_cpu_time_us":25655,"prepare.run_wall_time_us":444446,"raft.queue_time_us":449,"raft.run_cpu_time_us":634,"raft.run_wall_time_us":6858,"replication_time_us":543219,"spinlock_wait_cycles":20224,"thread_start_us":789349,"threads_started":2,"wal-append.queue_time_us":291516}]]}
I20251029 02:51:59.413360 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5336762c90e43278db7978dda295e40. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:51:59.411598 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 0f80b840ed954230bfbfe7e40127cd2f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0f80b840ed954230bfbfe7e40127cd2f already in progress: creating tablet
I20251029 02:51:59.415109 899 tablet_service.cc:1552] Processing DeleteTablet for tablet fcb3db8399aa4cf188b07dda220f7f5b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:59 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:59.418162 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_b0b0cdffdbbc4f89a6117b857e80df26"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:59.418903 1172 raft_consensus.cc:385] T 2242f5b2c20346d9b753d7cd29242441 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251029 02:51:59.418864 712 leader_election.cc:343] T 6feec2c8e030447fae9c292a626dcd1d P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:59.418484 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_877abcb0d221493c8875368e1c75382b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:59.419309 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b0b0cdffdbbc4f89a6117b857e80df26 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:51:59.419510 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_877abcb0d221493c8875368e1c75382b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:59.420850 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_240f7d6a65f04099abf3e204b2f47fb6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:59.421468 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_240f7d6a65f04099abf3e204b2f47fb6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:59.423255 890 tablet_service.cc:1552] Processing DeleteTablet for tablet 38f83ebcb2a440bc8e6a8d36607e6666 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:59 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:59.427018 747 tablet_service.cc:1552] Processing DeleteTablet for tablet 358bdfbcb4ee440c899fb6eedef68f44 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:58 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:59.427058 745 tablet_service.cc:1505] Processing CreateTablet for tablet 7ff2399833e546448b85b9caeb5200fb (DEFAULT_TABLE table=default.Table_58fd9561b8e04989a95f097e29a79613 [id=9c994e4d21f042b2a5c1b86b27abfbb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:59.427472 747 tablet_service.cc:1505] Processing CreateTablet for tablet 6364601618314251b5601f30971ac61f (DEFAULT_TABLE table=default.Table_d497c064fe99434faf6566e6c5b6836c [id=5fa9ea29aa9c4175b68461684a57c420]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 02:51:59.430608 524 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9269622ca92645d5aa8e73341201ccbb consensus state has no leader
I20251029 02:51:59.431355 1167 tablet_bootstrap.cc:492] T c519832030884a049839d309cc738c28 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:51:59.431736 1599 raft_consensus.cc:493] T d6e1170dbb594a5696a3899cde8da7c9 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:59.431965 1599 raft_consensus.cc:515] T d6e1170dbb594a5696a3899cde8da7c9 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:59.440075 1665 raft_consensus.cc:493] T 8afc01d52d784f2fa559cab7ce07c8be P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:59.440244 1665 raft_consensus.cc:515] T 8afc01d52d784f2fa559cab7ce07c8be P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:59.440618 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a55f22b0d3b4ebe98ec2e054d05b095. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:59.440876 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ff2399833e546448b85b9caeb5200fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:59.441016 1172 raft_consensus.cc:740] T 2242f5b2c20346d9b753d7cd29242441 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:51:59.441259 1172 consensus_queue.cc:260] T 2242f5b2c20346d9b753d7cd29242441 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:59.441829 1172 ts_tablet_manager.cc:1434] T 2242f5b2c20346d9b753d7cd29242441 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.164s user 0.003s sys 0.000s
I20251029 02:51:59.441690 712 leader_election.cc:304] T 6feec2c8e030447fae9c292a626dcd1d P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:51:59.442118 1172 tablet_bootstrap.cc:492] T 7309d3190cfa415281c5253a634fa7f3 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:51:59.439074 738 tablet_service.cc:1505] Processing CreateTablet for tablet 27e26b5e812f4af0a4914bf9022d62d4 (DEFAULT_TABLE table=default.Table_72f0d7d8e4d34e47b90c8aa8130ca61d [id=37a2a807288046ba8cdc6ac961416990]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:59.446319 1613 raft_consensus.cc:2749] T 6feec2c8e030447fae9c292a626dcd1d P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:51:59.444404 755 tablet_service.cc:1505] Processing CreateTablet for tablet 4cede3b183494845af91dca39a2096c4 (DEFAULT_TABLE table=default.Table_ad59ec000b0c4dbcb80dd9a5d9648ec6 [id=7f651f203c294c549a87a3c0dba15bb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:59.449506 750 tablet_service.cc:1505] Processing CreateTablet for tablet ff3e9227b9674d31b874823cf4a62d9b (DEFAULT_TABLE table=default.Table_99567b9d8c154303bae22fda47db1d0a [id=2b8bd2d2c2664edaad536b1d7706096b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:59.450332 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6364601618314251b5601f30971ac61f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:59.452318 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 75aa2dd1ed1b49649f3257df5d45cb32. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:59.453941 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_695c13e96c124cf0a3f4b49014203bdb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:59.454537 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_695c13e96c124cf0a3f4b49014203bdb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:59.454640 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_21064497551747b08fa4839601b7c680"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:59.455248 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_21064497551747b08fa4839601b7c680 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:59.459108 1665 leader_election.cc:290] T 8afc01d52d784f2fa559cab7ce07c8be P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:51:59.459897 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0672a948e5e41c5bbfe7e8199025986. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:59.460903 522 master_service.cc:954] ReplaceTablet: received request to replace tablet 785733da86ff425a90ccfab2933fc89e from {username='slave'} at 127.0.0.1:35314
I20251029 02:51:59.462002 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c172417d1d444c439e9cda611a2ed551. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:59.465297 1019 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:37622 (request call id 91) took 1153 ms. Trace:
I20251029 02:51:59.465466 1019 rpcz_store.cc:276] 1029 02:51:58.311568 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:58.311755 (+ 187us) service_pool.cc:225] Handling call
1029 02:51:58.805697 (+493942us) spinlock_profiling.cc:243] Waited 10.8 ms on lock 0x6160001a1a54. stack: 00007f34717829f6 00007f34717827b0 00007f347075e3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f347d61c200 00007f34814cc66f 00007f347b8f2c36 00007f347b8f2965 00007f347872127e 00007f347872015e 00007f3478726395 00007f3478726cbf 00007f3478726ae8 0000000000a8867c
1029 02:51:59.465099 (+659402us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":24806144}
I20251029 02:51:59.472960 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5bd2163bc4be467fb695629bfbb5a6ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:59.473331 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff3e9227b9674d31b874823cf4a62d9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:59.482589 518 master_service.cc:954] ReplaceTablet: received request to replace tablet c65469173762446e9725262ae998dbf3 from {username='slave'} at 127.0.0.1:35314
I20251029 02:51:59.484975 1167 ts_tablet_manager.cc:1403] T c519832030884a049839d309cc738c28 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.332s user 0.002s sys 0.006s
I20251029 02:51:59.484922 518 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_8cbb25deb1a043959698717a9af07e52" } new_table_name: "default.Table_ff796c36eb2c4bb1809f39e1cb68c223" modify_external_catalogs: true
I20251029 02:51:59.485864 1167 raft_consensus.cc:359] T c519832030884a049839d309cc738c28 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:59.486116 1167 raft_consensus.cc:385] T c519832030884a049839d309cc738c28 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:59.486184 1167 raft_consensus.cc:740] T c519832030884a049839d309cc738c28 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:51:59.486430 1167 consensus_queue.cc:260] T c519832030884a049839d309cc738c28 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:59.487104 522 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_8e3f488e93f6452d8736d4c582cdf82c" } new_table_name: "default.Table_57a0f7b13bf146d890469978e60ac4be" modify_external_catalogs: true
I20251029 02:51:59.492796 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27e26b5e812f4af0a4914bf9022d62d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:59.494632 525 master_service.cc:954] ReplaceTablet: received request to replace tablet 84993b2f63e7421290a161dde4bb0607 from {username='slave'} at 127.0.0.1:35314
I20251029 02:51:59.496240 754 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:44580 (request call id 449) took 1018 ms. Trace:
I20251029 02:51:59.476501 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2990e9f626c40618c118266bc595684" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:51:59.506335 772 raft_consensus.cc:2468] T d2990e9f626c40618c118266bc595684 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 0.
I20251029 02:51:59.506253 754 rpcz_store.cc:276] 1029 02:51:58.467877 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:58.950463 (+482586us) service_pool.cc:225] Handling call
1029 02:51:58.951036 (+ 573us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1029 02:51:58.951058 (+ 22us) ts_tablet_manager.cc:682] Creating new metadata...
1029 02:51:59.274339 (+323281us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:59.485758 (+211419us) tablet_replica.cc:206] Creating consensus instance
1029 02:51:59.485929 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251029 02:51:59.507193 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4cede3b183494845af91dca39a2096c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:59.507915 754 tablet_service.cc:1505] Processing CreateTablet for tablet d9fe530823864f4e8c2643bf7fe6d383 (DEFAULT_TABLE table=default.Table_1b9efccf77ba4f8c91c0136ca2433af8 [id=31b2158fa0a54f798794379f76d55a23]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:59.509140 740 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:44580 (request call id 442) took 1040 ms. Trace:
I20251029 02:51:59.510744 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet c65469173762446e9725262ae998dbf3 (table default.Table_06e7c4a46c50490196ec56306b13a2bb [id=859e6643e304437c8dc9c14973b21b88]) successfully deleted
I20251029 02:51:59.512539 1599 leader_election.cc:290] T d6e1170dbb594a5696a3899cde8da7c9 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:51:59.514467 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9fe530823864f4e8c2643bf7fe6d383. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:59.516187 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 5695b5a7b1854c3983e2d4ab684898b0 (table default.Table_e43a2189b18a43f9beb9f3ce25848374 [id=d151d895aac64712b2eeee989d19b878]) successfully deleted
W20251029 02:51:59.516474 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 6d830d30b26b4ba799785fec8a6f0769 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6d830d30b26b4ba799785fec8a6f0769 already in progress: creating tablet
I20251029 02:51:59.518802 740 rpcz_store.cc:276] 1029 02:51:58.453028 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:58.886200 (+433172us) service_pool.cc:225] Handling call
1029 02:51:58.933546 (+ 47346us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1029 02:51:58.933589 (+ 43us) ts_tablet_manager.cc:682] Creating new metadata...
1029 02:51:59.310967 (+377378us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:59.492265 (+181298us) tablet_replica.cc:206] Creating consensus instance
1029 02:51:59.492461 (+ 196us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251029 02:51:59.525429 918 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "392a36a6b59a42f78e3ac7c8fb63214b" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:51:59.528000 919 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0a47781d036149569e1164dd938a45e9" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:51:59.530831 1174 ts_tablet_manager.cc:1929] T 64b0fb400e944ce2b2c7a49d12135e9c P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:59.531175 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1fc87e2d7b4240e08f2b4ac7277293ff" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:51:59.531683 1174 log.cc:1199] T 64b0fb400e944ce2b2c7a49d12135e9c P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/64b0fb400e944ce2b2c7a49d12135e9c
I20251029 02:51:59.537271 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e07291d6b5645feb4d5890f58e1ef74" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:51:59.537777 918 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8afc01d52d784f2fa559cab7ce07c8be" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:51:59.538012 847 leader_election.cc:343] T d6e1170dbb594a5696a3899cde8da7c9 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: d6e1170dbb594a5696a3899cde8da7c9
I20251029 02:51:59.541385 1174 ts_tablet_manager.cc:1950] T 64b0fb400e944ce2b2c7a49d12135e9c P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:51:59.542471 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 317) took 3239 ms. Trace:
I20251029 02:51:59.542619 1174 rpcz_store.cc:276] 1029 02:51:56.303468 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:56.316250 (+ 12782us) service_pool.cc:225] Handling call
1029 02:51:59.046879 (+2730629us) ts_tablet_manager.cc:1151] Deleting tablet 64b0fb400e944ce2b2c7a49d12135e9c
1029 02:51:59.046888 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:59.339955 (+293067us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:59.506576 (+166621us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:59.542429 (+ 35853us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2728418}
I20251029 02:51:59.543457 1174 tablet_replica.cc:333] T 4b76c4408a7340e59b8506171622175f P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:51:59.543663 1174 raft_consensus.cc:2243] T 4b76c4408a7340e59b8506171622175f P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:59.543833 1174 raft_consensus.cc:2272] T 4b76c4408a7340e59b8506171622175f P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:59.544373 1174 ts_tablet_manager.cc:1916] T 4b76c4408a7340e59b8506171622175f P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:59.551003 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 8f7c746f8b6948b58676298c2c2ea221 (table default.Table_da4961f316764505bb9e0f191a62a327 [id=669ae9dd0a5749b1aa3557ab8183a057]) successfully deleted
W20251029 02:51:59.551280 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 51effc9324034ae79e2b25cb57a996c1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 51effc9324034ae79e2b25cb57a996c1 already in progress: creating tablet
W20251029 02:51:59.551493 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 71068739b4384badbf305313d902f0f5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 71068739b4384badbf305313d902f0f5 already in progress: creating tablet
W20251029 02:51:59.551716 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 1275e192871440408a8345fef9b56260 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1275e192871440408a8345fef9b56260 already in progress: creating tablet
W20251029 02:51:59.553350 849 leader_election.cc:343] T d6e1170dbb594a5696a3899cde8da7c9 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Not found: Tablet not found: d6e1170dbb594a5696a3899cde8da7c9
I20251029 02:51:59.553503 849 leader_election.cc:304] T d6e1170dbb594a5696a3899cde8da7c9 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9
I20251029 02:51:59.554416 1599 raft_consensus.cc:2749] T d6e1170dbb594a5696a3899cde8da7c9 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:51:59.558359 740 tablet_service.cc:1505] Processing CreateTablet for tablet eb78efbb100946a781056fab1d8bfaca (DEFAULT_TABLE table=default.Table_51aa8265801a4f8d9c0330cbf2bc72a4 [id=1efd18c0e7614e1da602a0d016391696]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:59.558887 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb78efbb100946a781056fab1d8bfaca. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:59.559144 1042 tablet_service.cc:1552] Processing DeleteTablet for tablet 86fc22bb84794149b598b97df41eb881 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:58 UTC) from {username='slave'} at 127.0.0.1:37626
W20251029 02:51:59.561769 711 leader_election.cc:343] T 392a36a6b59a42f78e3ac7c8fb63214b P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:59.562114 742 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:44580 (request call id 452) took 1075 ms. Trace:
I20251029 02:51:59.563007 741 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:44580 (request call id 441) took 1106 ms. Trace:
I20251029 02:51:59.570087 1174 ts_tablet_manager.cc:1929] T 4b76c4408a7340e59b8506171622175f P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251029 02:51:59.570240 1174 log.cc:1199] T 4b76c4408a7340e59b8506171622175f P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/4b76c4408a7340e59b8506171622175f
I20251029 02:51:59.572196 1174 ts_tablet_manager.cc:1950] T 4b76c4408a7340e59b8506171622175f P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:51:59.577020 1026 tablet_service.cc:1552] Processing DeleteTablet for tablet 358bdfbcb4ee440c899fb6eedef68f44 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:58 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:59.573097 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 320) took 3184 ms. Trace:
I20251029 02:51:59.577615 1174 rpcz_store.cc:276] 1029 02:51:56.388152 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:56.394864 (+ 6712us) service_pool.cc:225] Handling call
1029 02:51:59.543397 (+3148533us) ts_tablet_manager.cc:1151] Deleting tablet 4b76c4408a7340e59b8506171622175f
1029 02:51:59.543405 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:59.547971 (+ 4566us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:59.569958 (+ 21987us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:59.573042 (+ 3084us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3128505}
W20251029 02:51:59.578899 711 leader_election.cc:343] T 0a47781d036149569e1164dd938a45e9 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:59.582667 1027 tablet_service.cc:1505] Processing CreateTablet for tablet 27e26b5e812f4af0a4914bf9022d62d4 (DEFAULT_TABLE table=default.Table_72f0d7d8e4d34e47b90c8aa8130ca61d [id=37a2a807288046ba8cdc6ac961416990]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:59.583894 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 321) took 3156 ms. Trace:
I20251029 02:51:59.584035 1174 rpcz_store.cc:276] 1029 02:51:56.426936 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:56.429395 (+ 2459us) service_pool.cc:225] Handling call
1029 02:51:59.583624 (+3154229us) ts_tablet_manager.cc:1151] Deleting tablet dd6ad641fbda472fab07aa35c5e4523a
1029 02:51:59.583641 (+ 17us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:59.583827 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3153714}
I20251029 02:51:59.585110 741 rpcz_store.cc:276] 1029 02:51:58.451663 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:58.878728 (+427065us) service_pool.cc:225] Handling call
1029 02:51:58.905974 (+ 27246us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1029 02:51:58.920022 (+ 14048us) ts_tablet_manager.cc:682] Creating new metadata...
1029 02:51:59.317345 (+397323us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:59.557427 (+240082us) tablet_replica.cc:206] Creating consensus instance
1029 02:51:59.557554 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251029 02:51:59.587019 1174 tablet_replica.cc:333] T 7c041f4909274cd5a03b9fd78353faf9 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:51:59.587221 1174 raft_consensus.cc:2243] T 7c041f4909274cd5a03b9fd78353faf9 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:59.587400 1174 raft_consensus.cc:2272] T 7c041f4909274cd5a03b9fd78353faf9 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:59.587954 1174 ts_tablet_manager.cc:1916] T 7c041f4909274cd5a03b9fd78353faf9 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:59.593797 1172 tablet_bootstrap.cc:654] T 7309d3190cfa415281c5253a634fa7f3 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
W20251029 02:51:59.594090 711 leader_election.cc:343] T 1fc87e2d7b4240e08f2b4ac7277293ff P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
W20251029 02:51:59.594502 711 leader_election.cc:343] T 9e07291d6b5645feb4d5890f58e1ef74 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
W20251029 02:51:59.594762 711 leader_election.cc:343] T 8afc01d52d784f2fa559cab7ce07c8be P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:51:59.596114 752 tablet_service.cc:1505] Processing CreateTablet for tablet 5f52f7520ebc446f865aa7d999d9f7db (DEFAULT_TABLE table=default.Table_013611a621b2452c8ffb8d9161725af6 [id=02c02bd43c994c559655b1bee49893df]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:59.596709 743 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:44580 (request call id 457) took 1101 ms. Trace:
I20251029 02:51:59.596899 743 rpcz_store.cc:276] 1029 02:51:58.482243 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:59.272538 (+790295us) service_pool.cc:225] Handling call
1029 02:51:59.313068 (+ 40530us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1029 02:51:59.313102 (+ 34us) ts_tablet_manager.cc:682] Creating new metadata...
1029 02:51:59.554449 (+241347us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:59.583704 (+ 29255us) tablet_replica.cc:206] Creating consensus instance
1029 02:51:59.583880 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251029 02:51:59.598884 743 tablet_service.cc:1552] Processing DeleteTablet for tablet 023a4094b424464c8f49d6eb9cc15da4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:58 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:59.599346 743 tablet_service.cc:1552] Processing DeleteTablet for tablet 1878d7eef14a4578bc929da68872cee4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:58 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:59.599622 743 tablet_service.cc:1552] Processing DeleteTablet for tablet 7816556521144c928d7a02694e22e79f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:58 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:59.599862 743 tablet_service.cc:1552] Processing DeleteTablet for tablet 94bdbb9394f7490e99916bd818986208 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:58 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:59.600191 743 tablet_service.cc:1552] Processing DeleteTablet for tablet c9bd3f2deb5d4af09b7918269a98509f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:58 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:51:59.601249 743 tablet_service.cc:1505] Processing CreateTablet for tablet 845a69a8696e469a8548787a7645d41a (DEFAULT_TABLE table=default.Table_68075467476c41f2b5f28dcfa7633b10 [id=0c4e504854004a45be67fee7b396ff96]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:59.602257 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f52f7520ebc446f865aa7d999d9f7db. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:59.592098 748 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:44580 (request call id 451) took 1082 ms. Trace:
I20251029 02:51:59.604698 742 rpcz_store.cc:276] 1029 02:51:58.481753 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:58.959917 (+478164us) service_pool.cc:225] Handling call
1029 02:51:58.960307 (+ 390us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1029 02:51:58.960327 (+ 20us) ts_tablet_manager.cc:682] Creating new metadata...
1029 02:51:59.420808 (+460481us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:59.556720 (+135912us) tablet_replica.cc:206] Creating consensus instance
1029 02:51:59.556902 (+ 182us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251029 02:51:59.608198 748 rpcz_store.cc:276] 1029 02:51:58.481639 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:58.954949 (+473310us) service_pool.cc:225] Handling call
1029 02:51:58.955529 (+ 580us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1029 02:51:58.955556 (+ 27us) ts_tablet_manager.cc:682] Creating new metadata...
1029 02:51:59.306372 (+350816us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:59.560373 (+254001us) tablet_replica.cc:206] Creating consensus instance
1029 02:51:59.564249 (+ 3876us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251029 02:51:59.613281 1174 ts_tablet_manager.cc:1929] T 7c041f4909274cd5a03b9fd78353faf9 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:59.613448 1174 log.cc:1199] T 7c041f4909274cd5a03b9fd78353faf9 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/7c041f4909274cd5a03b9fd78353faf9
I20251029 02:51:59.613898 1174 ts_tablet_manager.cc:1950] T 7c041f4909274cd5a03b9fd78353faf9 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:51:59.614840 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 322) took 3181 ms. Trace:
I20251029 02:51:59.619421 748 tablet_service.cc:1505] Processing CreateTablet for tablet f2d47238761e4ed49c88abdbb677b18a (DEFAULT_TABLE table=default.Table_39a7937d49ce4ee290f72ed5295a2a3e [id=1571e82f4f9948089d81b0450dbfdee3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:59.620329 1042 tablet_service.cc:1552] Processing DeleteTablet for tablet f18b0ca5a6a44175a912642e9eea1778 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:58 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:59.622915 1042 tablet_service.cc:1505] Processing CreateTablet for tablet 5bd2163bc4be467fb695629bfbb5a6ca (DEFAULT_TABLE table=default.Table_a45edb83b6254a4bb8fb610220e5cf65 [id=19f7af2387764bb8813321ddb6443d8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:59.614995 1174 rpcz_store.cc:276] 1029 02:51:56.433652 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:56.433901 (+ 249us) service_pool.cc:225] Handling call
1029 02:51:59.586928 (+3153027us) ts_tablet_manager.cc:1151] Deleting tablet 7c041f4909274cd5a03b9fd78353faf9
1029 02:51:59.586940 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:59.593552 (+ 6612us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:59.613101 (+ 19549us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:59.614794 (+ 1693us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3152709}
I20251029 02:51:59.635550 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 7c041f4909274cd5a03b9fd78353faf9 (table default.Table_83d3fe1030e0461895fc2437732b4ff8 [id=e11618f2bd094526ad359c65bee6a9ba]) successfully deleted
I20251029 02:51:59.638007 1023 tablet_service.cc:1552] Processing DeleteTablet for tablet c8389c8bfead4ff7a23f363c9c6770bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:58 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:59.640417 1599 raft_consensus.cc:493] T 6f93812578a940a3b7378015e2a46f95 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:59.640609 1599 raft_consensus.cc:515] T 6f93812578a940a3b7378015e2a46f95 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:59.636300 742 tablet_service.cc:1505] Processing CreateTablet for tablet ead73d81d18a40a4b0aa9935f86421f8 (DEFAULT_TABLE table=default.Table_fd89990c08d142ffa08457a5802e5e69 [id=f3f342e7edc849fdbffea50a912b175f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:59.645817 1599 leader_election.cc:290] T 6f93812578a940a3b7378015e2a46f95 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:51:59.648451 1026 tablet_service.cc:1505] Processing CreateTablet for tablet 4cede3b183494845af91dca39a2096c4 (DEFAULT_TABLE table=default.Table_ad59ec000b0c4dbcb80dd9a5d9648ec6 [id=7f651f203c294c549a87a3c0dba15bb8]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 02:51:59.648849 847 leader_election.cc:343] T 6f93812578a940a3b7378015e2a46f95 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 6f93812578a940a3b7378015e2a46f95
I20251029 02:51:59.648957 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 323) took 3192 ms. Trace:
W20251029 02:51:59.649124 849 leader_election.cc:343] T 6f93812578a940a3b7378015e2a46f95 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Not found: Tablet not found: 6f93812578a940a3b7378015e2a46f95
I20251029 02:51:59.649376 849 leader_election.cc:304] T 6f93812578a940a3b7378015e2a46f95 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9
I20251029 02:51:59.649860 1030 tablet_service.cc:1552] Processing DeleteTablet for tablet eca016aed05e4ce09e1a2bf96e2286f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:58 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:59.649829 1174 rpcz_store.cc:276] 1029 02:51:56.456354 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:56.460015 (+ 3661us) service_pool.cc:225] Handling call
1029 02:51:59.635517 (+3175502us) ts_tablet_manager.cc:1151] Deleting tablet daa05a34c0684c07a35966290fd7d44c
1029 02:51:59.635535 (+ 18us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:59.648904 (+ 13369us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3175262}
I20251029 02:51:59.650305 1174 tablet_replica.cc:333] T c4884e22cf2044a4a1070c1b4b759d9e P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
W20251029 02:51:59.650378 1661 lock_manager.cc:585] Waited 1 seconds to obtain row lock on key '<redacted>' (key hash 8473487660152521317) tablet 00000000000000000000000000000000 cur holder 0x617000040c80
I20251029 02:51:59.650431 1599 raft_consensus.cc:2749] T 6f93812578a940a3b7378015e2a46f95 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:51:59.650683 1174 raft_consensus.cc:2243] T c4884e22cf2044a4a1070c1b4b759d9e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:59.651310 1174 raft_consensus.cc:2272] T c4884e22cf2044a4a1070c1b4b759d9e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:59.651888 1174 ts_tablet_manager.cc:1916] T c4884e22cf2044a4a1070c1b4b759d9e P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:59.652132 1023 tablet_service.cc:1505] Processing CreateTablet for tablet 4a55f22b0d3b4ebe98ec2e054d05b095 (DEFAULT_TABLE table=default.Table_2e41802f8663436e83915f560cf8f94b [id=4e2615cec64b4828b8311b37e4a568b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:59.661221 751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:44580 (request call id 456) took 1173 ms. Trace:
I20251029 02:51:59.661368 751 rpcz_store.cc:276] 1029 02:51:58.482164 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:59.232131 (+749967us) service_pool.cc:225] Handling call
1029 02:51:59.275484 (+ 43353us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1029 02:51:59.275573 (+ 89us) ts_tablet_manager.cc:682] Creating new metadata...
1029 02:51:59.360486 (+ 84913us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:59.636965 (+276479us) tablet_replica.cc:206] Creating consensus instance
1029 02:51:59.655928 (+ 18963us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251029 02:51:59.666985 744 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:44580 (request call id 447) took 1192 ms. Trace:
I20251029 02:51:59.670604 1030 tablet_service.cc:1505] Processing CreateTablet for tablet 6364601618314251b5601f30971ac61f (DEFAULT_TABLE table=default.Table_d497c064fe99434faf6566e6c5b6836c [id=5fa9ea29aa9c4175b68461684a57c420]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:59.677955 1174 ts_tablet_manager.cc:1929] T c4884e22cf2044a4a1070c1b4b759d9e P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:59.678153 1174 log.cc:1199] T c4884e22cf2044a4a1070c1b4b759d9e P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/c4884e22cf2044a4a1070c1b4b759d9e
I20251029 02:51:59.678597 1174 ts_tablet_manager.cc:1950] T c4884e22cf2044a4a1070c1b4b759d9e P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:51:59.679600 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 324) took 3178 ms. Trace:
I20251029 02:51:59.679729 1167 ts_tablet_manager.cc:1434] T c519832030884a049839d309cc738c28 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.195s user 0.002s sys 0.009s
I20251029 02:51:59.679663 1035 tablet_service.cc:1505] Processing CreateTablet for tablet d9fe530823864f4e8c2643bf7fe6d383 (DEFAULT_TABLE table=default.Table_1b9efccf77ba4f8c91c0136ca2433af8 [id=31b2158fa0a54f798794379f76d55a23]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:59.681483 1167 tablet_bootstrap.cc:492] T 6c22497760274380a832feff7c5a2fae P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:51:59.684181 1036 tablet_service.cc:1505] Processing CreateTablet for tablet eb78efbb100946a781056fab1d8bfaca (DEFAULT_TABLE table=default.Table_51aa8265801a4f8d9c0330cbf2bc72a4 [id=1efd18c0e7614e1da602a0d016391696]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:59.685454 1167 tablet_bootstrap.cc:654] T 6c22497760274380a832feff7c5a2fae P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:59.692867 1031 tablet_service.cc:1505] Processing CreateTablet for tablet 5f52f7520ebc446f865aa7d999d9f7db (DEFAULT_TABLE table=default.Table_013611a621b2452c8ffb8d9161725af6 [id=02c02bd43c994c559655b1bee49893df]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:59.695191 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27e26b5e812f4af0a4914bf9022d62d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:59.698967 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2d47238761e4ed49c88abdbb677b18a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:59.679731 1174 rpcz_store.cc:276] 1029 02:51:56.501485 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:56.502228 (+ 743us) service_pool.cc:225] Handling call
1029 02:51:59.650236 (+3148008us) ts_tablet_manager.cc:1151] Deleting tablet c4884e22cf2044a4a1070c1b4b759d9e
1029 02:51:59.650247 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:59.661249 (+ 11002us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:59.677793 (+ 16544us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:59.679532 (+ 1739us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":425,"tablet-delete.queue_time_us":3147754}
I20251029 02:51:59.700812 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 325) took 3190 ms. Trace:
I20251029 02:51:59.700901 1174 rpcz_store.cc:276] 1029 02:51:56.509893 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:56.509950 (+ 57us) service_pool.cc:225] Handling call
1029 02:51:59.700633 (+3190683us) ts_tablet_manager.cc:1151] Deleting tablet f187dbd8a55b45b292ea61db43caf47d
1029 02:51:59.700643 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:59.700756 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3190440}
I20251029 02:51:59.701210 749 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:44580 (request call id 453) took 1219 ms. Trace:
I20251029 02:51:59.701335 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 331) took 3010 ms. Trace:
I20251029 02:51:59.701444 1174 rpcz_store.cc:276] 1029 02:51:56.690867 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:56.691133 (+ 266us) service_pool.cc:225] Handling call
1029 02:51:59.701196 (+3010063us) ts_tablet_manager.cc:1151] Deleting tablet 1fc87e2d7b4240e08f2b4ac7277293ff
1029 02:51:59.701204 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:59.701280 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3009304}
I20251029 02:51:59.703259 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet c4884e22cf2044a4a1070c1b4b759d9e (table default.Table_da0be17b8bdc4e579b6c3faff71d9c98 [id=6e06ac70c6814e7d89a008bbaddccc7e]) successfully deleted
W20251029 02:51:59.703593 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet f187dbd8a55b45b292ea61db43caf47d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f187dbd8a55b45b292ea61db43caf47d already in progress: creating tablet
W20251029 02:51:59.703883 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 1fc87e2d7b4240e08f2b4ac7277293ff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1fc87e2d7b4240e08f2b4ac7277293ff already in progress: creating tablet
I20251029 02:51:59.709664 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 332) took 3018 ms. Trace:
I20251029 02:51:59.709796 1174 rpcz_store.cc:276] 1029 02:51:56.690978 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:56.691173 (+ 195us) service_pool.cc:225] Handling call
1029 02:51:59.709463 (+3018290us) ts_tablet_manager.cc:1151] Deleting tablet 59eee86f249e4d6a9d09d30276dcd659
1029 02:51:59.709474 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:59.709602 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3017137}
I20251029 02:51:59.710089 751 tablet_service.cc:1505] Processing CreateTablet for tablet 30282e0f1b98401bb450e19965f037a5 (DEFAULT_TABLE table=default.Table_a714017562834a6dbfaa057b9f9aae41 [id=a6e350736cb6499a8d93cee9be338f08]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:59.710351 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 333) took 3019 ms. Trace:
I20251029 02:51:59.710454 1174 rpcz_store.cc:276] 1029 02:51:56.691077 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:56.691211 (+ 134us) service_pool.cc:225] Handling call
1029 02:51:59.710190 (+3018979us) ts_tablet_manager.cc:1151] Deleting tablet 6d830d30b26b4ba799785fec8a6f0769
1029 02:51:59.710202 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:59.710298 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3017744}
I20251029 02:51:59.711158 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 330) took 3020 ms. Trace:
I20251029 02:51:59.711253 1174 rpcz_store.cc:276] 1029 02:51:56.690691 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:56.691277 (+ 586us) service_pool.cc:225] Handling call
1029 02:51:59.711006 (+3019729us) ts_tablet_manager.cc:1151] Deleting tablet 1bc45078a6a942c38ff3350c15885eff
1029 02:51:59.711018 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:59.711107 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3011979}
W20251029 02:51:59.712019 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 59eee86f249e4d6a9d09d30276dcd659 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 59eee86f249e4d6a9d09d30276dcd659 already in progress: creating tablet
W20251029 02:51:59.712297 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 6d830d30b26b4ba799785fec8a6f0769 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6d830d30b26b4ba799785fec8a6f0769 already in progress: creating tablet
W20251029 02:51:59.712538 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 1bc45078a6a942c38ff3350c15885eff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1bc45078a6a942c38ff3350c15885eff already in progress: creating tablet
W20251029 02:51:59.714813 518 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 67651306f3b44e7eb14dcee9a2007a81 consensus state has no leader
I20251029 02:51:59.715349 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 336) took 2883 ms. Trace:
W20251029 02:51:59.715461 522 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f4729ecd182042c2a10f67ca25224101 consensus state has no leader
I20251029 02:51:59.715437 1174 rpcz_store.cc:276] 1029 02:51:56.832080 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:56.836308 (+ 4228us) service_pool.cc:225] Handling call
1029 02:51:59.715156 (+2878848us) ts_tablet_manager.cc:1151] Deleting tablet 0f80b840ed954230bfbfe7e40127cd2f
1029 02:51:59.715164 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:59.715240 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2871608}
I20251029 02:51:59.716398 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 338) took 2810 ms. Trace:
I20251029 02:51:59.716496 1174 rpcz_store.cc:276] 1029 02:51:56.905519 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:56.905943 (+ 424us) service_pool.cc:225] Handling call
1029 02:51:59.716227 (+2810284us) ts_tablet_manager.cc:1151] Deleting tablet 1275e192871440408a8345fef9b56260
1029 02:51:59.716235 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:59.716345 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2809865}
I20251029 02:51:59.717846 526 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_9efddcb184d74a75bd0d1b72cc522e3f" } new_table_name: "default.Table_b9f8911906c3473da4de53896081cb2c" modify_external_catalogs: true
I20251029 02:51:59.719358 1174 tablet_replica.cc:333] T 8f7c746f8b6948b58676298c2c2ea221 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:51:59.719563 1174 raft_consensus.cc:2243] T 8f7c746f8b6948b58676298c2c2ea221 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Raft consensus shutting down.
W20251029 02:51:59.719480 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 0f80b840ed954230bfbfe7e40127cd2f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0f80b840ed954230bfbfe7e40127cd2f already in progress: creating tablet
I20251029 02:51:59.719935 1174 raft_consensus.cc:2272] T 8f7c746f8b6948b58676298c2c2ea221 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
W20251029 02:51:59.719949 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 1275e192871440408a8345fef9b56260 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1275e192871440408a8345fef9b56260 already in progress: creating tablet
I20251029 02:51:59.720536 1174 ts_tablet_manager.cc:1916] T 8f7c746f8b6948b58676298c2c2ea221 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:59.720628 524 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_b1d1e274a2ae4a639d80fb46e27bca52" } modify_external_catalogs: true
I20251029 02:51:59.720800 524 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_b1d1e274a2ae4a639d80fb46e27bca52" } modify_external_catalogs: true
I20251029 02:51:59.721345 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_0936756f462041d6a93c1401ceb56158"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:59.721891 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0936756f462041d6a93c1401ceb56158 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:59.722811 519 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_2a10c9dfc91d4974845dbcd500d1d8be" } modify_external_catalogs: true
I20251029 02:51:59.722982 519 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_2a10c9dfc91d4974845dbcd500d1d8be" } modify_external_catalogs: true
I20251029 02:51:59.723475 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_104d5aaf444b4e5e813288013737aa15"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:59.723924 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_104d5aaf444b4e5e813288013737aa15 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:59.725451 522 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_c4ce908a6d5c40e1b825533a63689d0a" } modify_external_catalogs: true
I20251029 02:51:59.725590 522 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_c4ce908a6d5c40e1b825533a63689d0a" } modify_external_catalogs: true
I20251029 02:51:59.726735 1167 tablet_bootstrap.cc:492] T 6c22497760274380a832feff7c5a2fae P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:51:59.726897 1167 ts_tablet_manager.cc:1403] T 6c22497760274380a832feff7c5a2fae P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.046s user 0.003s sys 0.000s
I20251029 02:51:59.727770 1167 raft_consensus.cc:359] T 6c22497760274380a832feff7c5a2fae P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:59.727972 1167 raft_consensus.cc:385] T 6c22497760274380a832feff7c5a2fae P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:59.728394 518 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_4e576f3e5a1a4b99af528a20793e1956" } new_table_name: "default.Table_6081606370774be1a2233b9304083139" modify_external_catalogs: true
I20251029 02:51:59.729214 744 rpcz_store.cc:276] 1029 02:51:58.464231 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:58.936823 (+472592us) service_pool.cc:225] Handling call
1029 02:51:58.937212 (+ 389us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1029 02:51:58.937233 (+ 21us) ts_tablet_manager.cc:682] Creating new metadata...
1029 02:51:59.426373 (+489140us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:59.606918 (+180545us) tablet_replica.cc:206] Creating consensus instance
1029 02:51:59.655793 (+ 48875us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251029 02:51:59.730351 523 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_4619a2287cb14edbb32bcbec581eb4d9" } modify_external_catalogs: true
I20251029 02:51:59.730499 523 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_4619a2287cb14edbb32bcbec581eb4d9" } modify_external_catalogs: true
I20251029 02:51:59.731276 525 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_bc32d7beda4f4957817b20124bf48e9c" } modify_external_catalogs: true
I20251029 02:51:59.731408 525 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_bc32d7beda4f4957817b20124bf48e9c" } modify_external_catalogs: true
I20251029 02:51:59.733743 895 tablet_service.cc:1552] Processing DeleteTablet for tablet c519832030884a049839d309cc738c28 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:52 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:51:59.734043 744 tablet_service.cc:1505] Processing CreateTablet for tablet df669cf8da7f4c98a3f16423c55c15eb (DEFAULT_TABLE table=default.Table_c0a7b43b54fb41c6ab5347fdb4ea3cf2 [id=c480bda570c44f07a7c2a2664233619c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:59.730233 749 rpcz_store.cc:276] 1029 02:51:58.481858 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:59.063944 (+582086us) service_pool.cc:225] Handling call
1029 02:51:59.222928 (+158984us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1029 02:51:59.222960 (+ 32us) ts_tablet_manager.cc:682] Creating new metadata...
1029 02:51:59.521282 (+298322us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:59.701025 (+179743us) tablet_replica.cc:206] Creating consensus instance
1029 02:51:59.701167 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251029 02:51:59.743798 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 845a69a8696e469a8548787a7645d41a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:59.745538 749 tablet_service.cc:1505] Processing CreateTablet for tablet fd1bf6b0f45349d0981f9ddd89269489 (DEFAULT_TABLE table=default.Table_5263ec7f27d74f73892b0ca252e9fa83 [id=18549ebdadaf443189d5f3eda682f3f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:59.728066 1167 raft_consensus.cc:740] T 6c22497760274380a832feff7c5a2fae P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:51:59.749424 1174 ts_tablet_manager.cc:1929] T 8f7c746f8b6948b58676298c2c2ea221 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20251029 02:51:59.749606 1174 log.cc:1199] T 8f7c746f8b6948b58676298c2c2ea221 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/8f7c746f8b6948b58676298c2c2ea221
I20251029 02:51:59.754045 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df669cf8da7f4c98a3f16423c55c15eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:59.764110 1174 ts_tablet_manager.cc:1950] T 8f7c746f8b6948b58676298c2c2ea221 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:51:59.764806 1062 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a5c3a15510cb4321bbb80d61fabd0c0a" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:51:59.765148 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 339) took 2851 ms. Trace:
I20251029 02:51:59.765327 1174 rpcz_store.cc:276] 1029 02:51:56.913357 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:56.913472 (+ 115us) service_pool.cc:225] Handling call
1029 02:51:59.719267 (+2805795us) ts_tablet_manager.cc:1151] Deleting tablet 8f7c746f8b6948b58676298c2c2ea221
1029 02:51:59.719277 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:59.724631 (+ 5354us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:59.749234 (+ 24603us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:59.765108 (+ 15874us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2805515}
I20251029 02:51:59.765934 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 8f7c746f8b6948b58676298c2c2ea221 (table default.Table_da4961f316764505bb9e0f191a62a327 [id=669ae9dd0a5749b1aa3557ab8183a057]) successfully deleted
I20251029 02:51:59.766150 1059 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1fc87e2d7b4240e08f2b4ac7277293ff" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:51:59.767601 1039 tablet_service.cc:1552] Processing DeleteTablet for tablet 023a4094b424464c8f49d6eb9cc15da4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:58 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:59.768014 1061 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0a47781d036149569e1164dd938a45e9" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:51:59.768348 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "392a36a6b59a42f78e3ac7c8fb63214b" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:51:59.768973 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 340) took 2846 ms. Trace:
I20251029 02:51:59.769095 1174 rpcz_store.cc:276] 1029 02:51:56.922127 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:56.922182 (+ 55us) service_pool.cc:225] Handling call
1029 02:51:59.767816 (+2845634us) ts_tablet_manager.cc:1151] Deleting tablet 51effc9324034ae79e2b25cb57a996c1
1029 02:51:59.767832 (+ 16us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:59.768929 (+ 1097us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2845414}
W20251029 02:51:59.769658 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 51effc9324034ae79e2b25cb57a996c1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 51effc9324034ae79e2b25cb57a996c1 already in progress: creating tablet
I20251029 02:51:59.746963 1167 consensus_queue.cc:260] T 6c22497760274380a832feff7c5a2fae P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:51:59.769812 1058 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e07291d6b5645feb4d5890f58e1ef74" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:51:59.774303 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 341) took 2847 ms. Trace:
I20251029 02:51:59.774475 1167 ts_tablet_manager.cc:1434] T 6c22497760274380a832feff7c5a2fae P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.047s user 0.002s sys 0.000s
I20251029 02:51:59.774946 1174 rpcz_store.cc:276] 1029 02:51:56.927291 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:56.927358 (+ 67us) service_pool.cc:225] Handling call
1029 02:51:59.774084 (+2846726us) ts_tablet_manager.cc:1151] Deleting tablet 71068739b4384badbf305313d902f0f5
1029 02:51:59.774099 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:59.774251 (+ 152us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2846479}
W20251029 02:51:59.775556 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 71068739b4384badbf305313d902f0f5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 71068739b4384badbf305313d902f0f5 already in progress: creating tablet
I20251029 02:51:59.775888 1167 tablet_bootstrap.cc:492] T 8d9722c764ac4b27a540595e9c6aafcd P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:51:59.776280 1174 tablet_replica.cc:333] T 9be9add0274d41b688edd6292a253a06 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:51:59.776806 1174 raft_consensus.cc:2243] T 9be9add0274d41b688edd6292a253a06 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:59.776885 1024 tablet_service.cc:1552] Processing DeleteTablet for tablet 1878d7eef14a4578bc929da68872cee4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:58 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:59.775573 1057 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8afc01d52d784f2fa559cab7ce07c8be" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:51:59.777393 1174 raft_consensus.cc:2272] T 9be9add0274d41b688edd6292a253a06 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:59.778028 1024 tablet_service.cc:1552] Processing DeleteTablet for tablet 94bdbb9394f7490e99916bd818986208 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:58 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:59.778116 1174 ts_tablet_manager.cc:1916] T 9be9add0274d41b688edd6292a253a06 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:59.785125 1024 tablet_service.cc:1552] Processing DeleteTablet for tablet c9bd3f2deb5d4af09b7918269a98509f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:58 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:59.790701 1037 tablet_service.cc:1505] Processing CreateTablet for tablet 845a69a8696e469a8548787a7645d41a (DEFAULT_TABLE table=default.Table_68075467476c41f2b5f28dcfa7633b10 [id=0c4e504854004a45be67fee7b396ff96]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:59.802220 1167 tablet_bootstrap.cc:654] T 8d9722c764ac4b27a540595e9c6aafcd P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:51:59.804131 1024 tablet_service.cc:1505] Processing CreateTablet for tablet ead73d81d18a40a4b0aa9935f86421f8 (DEFAULT_TABLE table=default.Table_fd89990c08d142ffa08457a5802e5e69 [id=f3f342e7edc849fdbffea50a912b175f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:59.830933 1039 tablet_service.cc:1552] Processing DeleteTablet for tablet 7816556521144c928d7a02694e22e79f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:58 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:59.830837 1029 tablet_service.cc:1505] Processing CreateTablet for tablet ff3e9227b9674d31b874823cf4a62d9b (DEFAULT_TABLE table=default.Table_99567b9d8c154303bae22fda47db1d0a [id=2b8bd2d2c2664edaad536b1d7706096b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:59.831470 1039 tablet_service.cc:1505] Processing CreateTablet for tablet 30282e0f1b98401bb450e19965f037a5 (DEFAULT_TABLE table=default.Table_a714017562834a6dbfaa057b9f9aae41 [id=a6e350736cb6499a8d93cee9be338f08]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:59.830855 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5bd2163bc4be467fb695629bfbb5a6ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:59.832098 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff3e9227b9674d31b874823cf4a62d9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:59.832700 1682 raft_consensus.cc:493] T c519832030884a049839d309cc738c28 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:51:59.832866 1682 raft_consensus.cc:515] T c519832030884a049839d309cc738c28 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:59.833637 1682 leader_election.cc:290] T c519832030884a049839d309cc738c28 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:51:59.835269 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30282e0f1b98401bb450e19965f037a5. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:51:59.835299 849 leader_election.cc:343] T c519832030884a049839d309cc738c28 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Not found: Tablet not found: c519832030884a049839d309cc738c28
I20251029 02:51:59.837658 1033 tablet_service.cc:1505] Processing CreateTablet for tablet df669cf8da7f4c98a3f16423c55c15eb (DEFAULT_TABLE table=default.Table_c0a7b43b54fb41c6ab5347fdb4ea3cf2 [id=c480bda570c44f07a7c2a2664233619c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:59.812506 1174 ts_tablet_manager.cc:1929] T 9be9add0274d41b688edd6292a253a06 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251029 02:51:59.838117 1174 log.cc:1199] T 9be9add0274d41b688edd6292a253a06 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/9be9add0274d41b688edd6292a253a06
I20251029 02:51:59.839184 1174 ts_tablet_manager.cc:1950] T 9be9add0274d41b688edd6292a253a06 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:51:59.839596 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 351) took 2662 ms. Trace:
I20251029 02:51:59.839711 1174 rpcz_store.cc:276] 1029 02:51:57.177422 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:57.177783 (+ 361us) service_pool.cc:225] Handling call
1029 02:51:59.776199 (+2598416us) ts_tablet_manager.cc:1151] Deleting tablet 9be9add0274d41b688edd6292a253a06
1029 02:51:59.776211 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:59.806865 (+ 30654us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:59.812336 (+ 5471us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:59.839524 (+ 27188us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2598250}
I20251029 02:51:59.840245 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 9be9add0274d41b688edd6292a253a06 (table default.Table_c7daf9eda79e470186f5774de2b824d3 [id=f45ce142f37f4febbceb4e568e35e0f2]) successfully deleted
I20251029 02:51:59.840373 1167 tablet_bootstrap.cc:492] T 8d9722c764ac4b27a540595e9c6aafcd P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:51:59.840544 1167 ts_tablet_manager.cc:1403] T 8d9722c764ac4b27a540595e9c6aafcd P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.066s user 0.000s sys 0.005s
I20251029 02:51:59.841899 1167 raft_consensus.cc:359] T 8d9722c764ac4b27a540595e9c6aafcd P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:59.842121 1167 raft_consensus.cc:385] T 8d9722c764ac4b27a540595e9c6aafcd P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:51:59.842195 1167 raft_consensus.cc:740] T 8d9722c764ac4b27a540595e9c6aafcd P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:51:59.848968 1601 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5e5222b26fe4145948620e51edf53fb [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9261eb85ea443d5a3beb36bd0c549f0. Latest consensus state: current_term: 2 leader_uuid: "e9261eb85ea443d5a3beb36bd0c549f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9672114a6a742e2864c16ff60ba246e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42343 } } peers { permanent_uuid: "e9261eb85ea443d5a3beb36bd0c549f0" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34239 } } peers { permanent_uuid: "e5e5222b26fe4145948620e51edf53fb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41233 } } }
I20251029 02:51:59.849300 1601 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5e5222b26fe4145948620e51edf53fb [sys.catalog]: This master's current role is: FOLLOWER
I20251029 02:51:59.849550 1614 raft_consensus.cc:493] T 00000000000000000000000000000000 P e5e5222b26fe4145948620e51edf53fb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e9261eb85ea443d5a3beb36bd0c549f0)
I20251029 02:51:59.849841 1614 raft_consensus.cc:515] T 00000000000000000000000000000000 P e5e5222b26fe4145948620e51edf53fb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9672114a6a742e2864c16ff60ba246e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42343 } } peers { permanent_uuid: "e9261eb85ea443d5a3beb36bd0c549f0" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34239 } } peers { permanent_uuid: "e5e5222b26fe4145948620e51edf53fb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41233 } }
I20251029 02:51:59.850878 1167 consensus_queue.cc:260] T 8d9722c764ac4b27a540595e9c6aafcd P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
W20251029 02:51:59.842254 990 consensus_peers.cc:597] T 9be9add0274d41b688edd6292a253a06 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 -> Peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Couldn't send request to peer c756528642a543eeacd93cef3e335a7d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9be9add0274d41b688edd6292a253a06. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:51:59.845821 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 845a69a8696e469a8548787a7645d41a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:59.857847 917 raft_consensus.cc:3060] T dd90fb9eb167478cbc3ec9363af86295 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:51:59.862751 1167 ts_tablet_manager.cc:1434] T 8d9722c764ac4b27a540595e9c6aafcd P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.022s user 0.001s sys 0.002s
I20251029 02:51:59.863276 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4cede3b183494845af91dca39a2096c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:59.854733 1025 tablet_service.cc:1505] Processing CreateTablet for tablet 7ff2399833e546448b85b9caeb5200fb (DEFAULT_TABLE table=default.Table_58fd9561b8e04989a95f097e29a79613 [id=9c994e4d21f042b2a5c1b86b27abfbb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:59.877238 1614 leader_election.cc:290] T 00000000000000000000000000000000 P e5e5222b26fe4145948620e51edf53fb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e9672114a6a742e2864c16ff60ba246e (127.0.104.62:42343), e9261eb85ea443d5a3beb36bd0c549f0 (127.0.104.61:34239)
W20251029 02:51:59.877487 847 leader_election.cc:343] T c519832030884a049839d309cc738c28 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: c519832030884a049839d309cc738c28
I20251029 02:51:59.877687 847 leader_election.cc:304] T c519832030884a049839d309cc738c28 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9
W20251029 02:51:59.877722 526 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet caef1c93e33a4390bcada59a41f43465 consensus state has no leader
I20251029 02:51:59.877997 1682 raft_consensus.cc:2749] T c519832030884a049839d309cc738c28 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:51:59.880291 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_f90425a19e964b75b94c68952d26ae1e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:59.880903 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f90425a19e964b75b94c68952d26ae1e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:59.883219 1175 ts_tablet_manager.cc:1929] T 1bab90fe904944af948dab3db06b7e34 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:59.887913 755 tablet_service.cc:1505] Processing CreateTablet for tablet 1fa8cc0c60cd43cdaa7a7646f70cfb97 (DEFAULT_TABLE table=default.Table_691548aa805841a281c6b0bbff6af9cc [id=2f66ded3048744238e4ab43d78520c2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:59.889092 1175 log.cc:1199] T 1bab90fe904944af948dab3db06b7e34 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/1bab90fe904944af948dab3db06b7e34
I20251029 02:51:59.890570 1175 ts_tablet_manager.cc:1950] T 1bab90fe904944af948dab3db06b7e34 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:51:59.890931 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 338) took 3057 ms. Trace:
I20251029 02:51:59.891053 1175 rpcz_store.cc:276] 1029 02:51:56.833034 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:56.833103 (+ 69us) service_pool.cc:225] Handling call
1029 02:51:58.923355 (+2090252us) ts_tablet_manager.cc:1151] Deleting tablet 1bab90fe904944af948dab3db06b7e34
1029 02:51:58.923363 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:59.372645 (+449282us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:59.755823 (+383178us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:59.890889 (+135066us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2089975}
I20251029 02:51:59.891587 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 337) took 3062 ms. Trace:
I20251029 02:51:59.891712 1175 rpcz_store.cc:276] 1029 02:51:56.829344 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:56.835885 (+ 6541us) service_pool.cc:225] Handling call
1029 02:51:59.891413 (+3055528us) ts_tablet_manager.cc:1151] Deleting tablet 0f80b840ed954230bfbfe7e40127cd2f
1029 02:51:59.891423 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:59.891552 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3054879}
I20251029 02:51:59.892064 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 341) took 2987 ms. Trace:
I20251029 02:51:59.892182 1175 rpcz_store.cc:276] 1029 02:51:56.905103 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:56.905165 (+ 62us) service_pool.cc:225] Handling call
1029 02:51:59.891938 (+2986773us) ts_tablet_manager.cc:1151] Deleting tablet 1275e192871440408a8345fef9b56260
1029 02:51:59.891946 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:59.892032 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2986578}
I20251029 02:51:59.893090 1175 tablet_replica.cc:333] T 8f7c746f8b6948b58676298c2c2ea221 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:51:59.893373 1175 raft_consensus.cc:2243] T 8f7c746f8b6948b58676298c2c2ea221 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:51:59.893617 1175 raft_consensus.cc:2272] T 8f7c746f8b6948b58676298c2c2ea221 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:51:59.894253 1175 ts_tablet_manager.cc:1916] T 8f7c746f8b6948b58676298c2c2ea221 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:51:59.897619 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 1bab90fe904944af948dab3db06b7e34 (table default.Table_c70eb9f284cd49109fd09e430a1cea6c [id=be817eb6ec33406eb316d36575c134fb]) successfully deleted
W20251029 02:51:59.897939 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 0f80b840ed954230bfbfe7e40127cd2f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0f80b840ed954230bfbfe7e40127cd2f already in progress: creating tablet
W20251029 02:51:59.898196 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 1275e192871440408a8345fef9b56260 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1275e192871440408a8345fef9b56260 already in progress: creating tablet
I20251029 02:51:59.902262 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd1bf6b0f45349d0981f9ddd89269489. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:59.903055 1038 tablet_service.cc:1505] Processing CreateTablet for tablet f2d47238761e4ed49c88abdbb677b18a (DEFAULT_TABLE table=default.Table_39a7937d49ce4ee290f72ed5295a2a3e [id=1571e82f4f9948089d81b0450dbfdee3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:59.904857 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ead73d81d18a40a4b0aa9935f86421f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:59.907541 737 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:44580 (request call id 458) took 1414 ms. Trace:
I20251029 02:51:59.911113 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1fa8cc0c60cd43cdaa7a7646f70cfb97. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:59.911378 737 rpcz_store.cc:276] 1029 02:51:58.482352 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:59.274153 (+791801us) service_pool.cc:225] Handling call
1029 02:51:59.301724 (+ 27571us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1029 02:51:59.302010 (+ 286us) ts_tablet_manager.cc:682] Creating new metadata...
1029 02:51:59.566563 (+264553us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:59.896342 (+329779us) tablet_replica.cc:206] Creating consensus instance
1029 02:51:59.896497 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251029 02:51:59.911832 737 tablet_service.cc:1505] Processing CreateTablet for tablet 2089b058d0e94d99b40ad2718518353e (DEFAULT_TABLE table=default.Table_0f0a1e0ac81a4b3492d304bef28ee22d [id=b64a13ef78ef4005a4ca28599f3db270]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:59.912202 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2089b058d0e94d99b40ad2718518353e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:59.912161 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_daad6fda420f4659bc87d5991d60cf2f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:59.912801 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_daad6fda420f4659bc87d5991d60cf2f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:51:59.914280 518 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c1222e30bc4141b3ae686d80053c4ef5 consensus state has no leader
I20251029 02:51:59.914565 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30282e0f1b98401bb450e19965f037a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:59.914781 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_aedb5a569ca74ef2a9092a847ad5a3f6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:59.915299 1627 raft_consensus.cc:493] T 75236b2fa82f4f28b82efae9a2c5160d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251029 02:51:59.915375 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aedb5a569ca74ef2a9092a847ad5a3f6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:59.915475 1627 raft_consensus.cc:515] T 75236b2fa82f4f28b82efae9a2c5160d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:51:59.916132 1627 leader_election.cc:290] T 75236b2fa82f4f28b82efae9a2c5160d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:51:59.913144 1042 tablet_service.cc:1505] Processing CreateTablet for tablet fd1bf6b0f45349d0981f9ddd89269489 (DEFAULT_TABLE table=default.Table_5263ec7f27d74f73892b0ca252e9fa83 [id=18549ebdadaf443189d5f3eda682f3f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:59.917026 520 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_695c13e96c124cf0a3f4b49014203bdb" } new_table_name: "default.Table_3975b6f24b5b4eb88958a46683a75c43" modify_external_catalogs: true
I20251029 02:51:59.918018 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ff2399833e546448b85b9caeb5200fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:59.917824 1167 tablet_bootstrap.cc:492] T aacb6f4b9284457bb6607e3a85d5c809 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:51:59.919665 746 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:44580 (request call id 455) took 1415 ms. Trace:
I20251029 02:51:59.919790 739 tablet_service.cc:1505] Processing CreateTablet for tablet 1cb0fdbd125446bf8d28080ea15e37ea (DEFAULT_TABLE table=default.Table_a111747c600544a3a67b954b834e4039 [id=29d9b8f74c26432c880e6ff411c1b7ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:59.921025 1704 catalog_manager.cc:805] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20251029 02:51:59.924216 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ead73d81d18a40a4b0aa9935f86421f8. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:51:59.924477 511 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet f4729ecd182042c2a10f67ca25224101 (table default.Table_57a0f7b13bf146d890469978e60ac4be [id=1dd2d5488bc647c8b82c53886583b28d], current schema version=3) failed: Not found: failed to reset TS proxy: No leader found: Tablet f4729ecd182042c2a10f67ca25224101 consensus state has no leader
I20251029 02:51:59.925547 1034 tablet_service.cc:1505] Processing CreateTablet for tablet 1fa8cc0c60cd43cdaa7a7646f70cfb97 (DEFAULT_TABLE table=default.Table_691548aa805841a281c6b0bbff6af9cc [id=2f66ded3048744238e4ab43d78520c2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:59.927302 1176 ts_tablet_manager.cc:1929] T 012ce243fd07466ab5301d1c9ce4b413 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:51:59.928062 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2d47238761e4ed49c88abdbb677b18a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:59.929131 1041 tablet_service.cc:1505] Processing CreateTablet for tablet 1cb0fdbd125446bf8d28080ea15e37ea (DEFAULT_TABLE table=default.Table_a111747c600544a3a67b954b834e4039 [id=29d9b8f74c26432c880e6ff411c1b7ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:59.939231 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd1bf6b0f45349d0981f9ddd89269489. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:59.945883 1029 tablet_service.cc:1505] Processing CreateTablet for tablet 2089b058d0e94d99b40ad2718518353e (DEFAULT_TABLE table=default.Table_0f0a1e0ac81a4b3492d304bef28ee22d [id=b64a13ef78ef4005a4ca28599f3db270]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:59.946050 1108 heartbeater.cc:344] Connected to a master server at 127.0.104.60:41233
I20251029 02:51:59.948992 523 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 294afb9b012c40c0b05b73795d6f0ac8
I20251029 02:51:59.949786 523 master_service.cc:954] ReplaceTablet: received request to replace tablet cc802bda2f6a4e439b605f2cafa37967 from {username='slave'} at 127.0.0.1:35314
I20251029 02:51:59.950568 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_6a3e403847304172bc58735bfe02dbd0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:59.950726 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1fa8cc0c60cd43cdaa7a7646f70cfb97. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:51:59.951210 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6a3e403847304172bc58735bfe02dbd0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:59.953299 524 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 7b0f7d6c1c9842789c6ce2cecaa3d5b0
I20251029 02:51:59.954113 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_8dc0ccf505314b8db07c2b5c2ad231c0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:51:59.954494 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2089b058d0e94d99b40ad2718518353e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:59.954497 1491 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" instance_seqno: 1761706309690026) as {username='slave'} at 127.0.104.3:43333; Asking this server to re-register.
W20251029 02:51:59.954800 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8dc0ccf505314b8db07c2b5c2ad231c0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:59.956235 522 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet e7b7de85c13b4ddc9c995ed2a2ccd921
I20251029 02:51:59.956861 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_3698a931084b485bb2e3689ded4a7fbc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:59.957348 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3698a931084b485bb2e3689ded4a7fbc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:59.958017 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1cb0fdbd125446bf8d28080ea15e37ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:59.958668 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_3c09e84f78ee464b823fe6c104c2e207"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:59.959120 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3c09e84f78ee464b823fe6c104c2e207 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:51:59.959278 1108 heartbeater.cc:461] Registering TS with master...
I20251029 02:51:59.960714 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_bdc0e5608cc44945849e3a3744ca6d9d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:51:59.961249 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bdc0e5608cc44945849e3a3744ca6d9d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:51:59.962559 520 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ce98420e61904af78d8fa03724c74489 has no consensus state
I20251029 02:51:59.962714 1491 ts_manager.cc:194] Registered new tserver with Master: 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:51:59.962886 520 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_b6110a49346640fb8926a871940e1aad" } new_table_name: "default.Table_e7bb3e262a894c2f84b6edfc3917b30d" modify_external_catalogs: true
I20251029 02:51:59.963937 465 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e5e5222b26fe4145948620e51edf53fb" candidate_term: 3 candidate_status { last_received { term: 2 index: 479 } } ignore_live_leader: false dest_uuid: "e9672114a6a742e2864c16ff60ba246e" is_pre_election: true
I20251029 02:51:59.965220 1028 tablet_service.cc:1505] Processing CreateTablet for tablet 85159079149241adae174fc0453ed0e7 (DEFAULT_TABLE table=default.Table_cb67eff3fb7c4cd9bc24d6bf31c3cd8f [id=3c7affb3187c4d1d987fdf2102c30a99]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:59.965878 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85159079149241adae174fc0453ed0e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:59.966440 536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e5e5222b26fe4145948620e51edf53fb" candidate_term: 3 candidate_status { last_received { term: 2 index: 479 } } ignore_live_leader: false dest_uuid: "e9261eb85ea443d5a3beb36bd0c549f0" is_pre_election: true
I20251029 02:51:59.967308 525 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 44c0488eba0543078ff4f0b3d69366f2
I20251029 02:51:59.968117 525 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_17185d86d94d40dc9d61f6f60d45bf51" } new_table_name: "default.Table_fa37ddda630a405d85ae341c82b4f204" modify_external_catalogs: true
I20251029 02:51:59.969578 519 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 8afc01d52d784f2fa559cab7ce07c8be
I20251029 02:51:59.971297 517 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 7bb85a12f573494cbb3118d1947eda27
I20251029 02:51:59.971814 517 catalog_manager.cc:6719] ReplaceTablet: tablet 7bb85a12f573494cbb3118d1947eda27 deleted and replaced by tablet a9f78f61ee2841dea71bbbfceacc3ea6
I20251029 02:51:59.972304 1417 leader_election.cc:304] T 00000000000000000000000000000000 P e5e5222b26fe4145948620e51edf53fb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5e5222b26fe4145948620e51edf53fb; no voters: e9261eb85ea443d5a3beb36bd0c549f0, e9672114a6a742e2864c16ff60ba246e
I20251029 02:51:59.973017 1032 tablet_service.cc:1505] Processing CreateTablet for tablet 07316861d2da4f55bd41111e400677fe (DEFAULT_TABLE table=default.Table_fc1eba7107f64286916e3dfce5f828b6 [id=7a248a5faa8346e09038e9e1c892625b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:59.973280 1614 raft_consensus.cc:2749] T 00000000000000000000000000000000 P e5e5222b26fe4145948620e51edf53fb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251029 02:51:59.976781 1039 tablet_service.cc:1505] Processing CreateTablet for tablet ea7b977224f24bf49dadfb83ffa28310 (DEFAULT_TABLE table=default.Table_da40f10638dc4f96a028f73fd7f001f1 [id=579d0cc01040411f9de03e699ff959e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:59.979003 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07316861d2da4f55bd41111e400677fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:59.981381 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea7b977224f24bf49dadfb83ffa28310. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:59.982160 1040 tablet_service.cc:1552] Processing DeleteTablet for tablet 24f07b1efac64b099b0b5ed467463749 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:59 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:51:59.985769 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet df669cf8da7f4c98a3f16423c55c15eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:59.987318 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6364601618314251b5601f30971ac61f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:59.988657 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb78efbb100946a781056fab1d8bfaca. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:59.990057 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a55f22b0d3b4ebe98ec2e054d05b095. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:51:59.997674 1040 tablet_service.cc:1505] Processing CreateTablet for tablet 85f979babbd34eaeaa89ea5906ddab90 (DEFAULT_TABLE table=default.Table_8c062218c19c46169ae3652f572ed821 [id=8a5e8af85fb74b71b4fa8568c0606319]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:51:59.999316 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85f979babbd34eaeaa89ea5906ddab90. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:00.012369 746 rpcz_store.cc:276] 1029 02:51:58.482029 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:59.222048 (+740019us) service_pool.cc:225] Handling call
1029 02:51:59.224120 (+ 2072us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1029 02:51:59.227151 (+ 3031us) ts_tablet_manager.cc:682] Creating new metadata...
1029 02:51:59.605080 (+377929us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:59.887487 (+282407us) tablet_replica.cc:206] Creating consensus instance
1029 02:51:59.897251 (+ 9764us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251029 02:52:00.013656 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f52f7520ebc446f865aa7d999d9f7db. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:00.015098 989 leader_election.cc:304] T d2990e9f626c40618c118266bc595684 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters: c756528642a543eeacd93cef3e335a7d
I20251029 02:52:00.015934 1686 raft_consensus.cc:2804] T d2990e9f626c40618c118266bc595684 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:00.016086 1686 raft_consensus.cc:493] T d2990e9f626c40618c118266bc595684 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:00.016237 1686 raft_consensus.cc:3060] T d2990e9f626c40618c118266bc595684 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
W20251029 02:52:00.017824 520 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 82289e0f823c4d43bfd8c4e878572b2b consensus state has no leader
I20251029 02:52:00.019799 1686 raft_consensus.cc:515] T d2990e9f626c40618c118266bc595684 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:00.016644 745 tablet_service.cc:1505] Processing CreateTablet for tablet 85159079149241adae174fc0453ed0e7 (DEFAULT_TABLE table=default.Table_cb67eff3fb7c4cd9bc24d6bf31c3cd8f [id=3c7affb3187c4d1d987fdf2102c30a99]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:00.026791 1665 raft_consensus.cc:493] T 2242f5b2c20346d9b753d7cd29242441 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:00.029081 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1cb0fdbd125446bf8d28080ea15e37ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:00.033896 1172 tablet_bootstrap.cc:492] T 7309d3190cfa415281c5253a634fa7f3 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
W20251029 02:52:00.034041 525 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b125cc9aaf4248b8bfa48fb152b832a6 consensus state has no leader
W20251029 02:52:00.036615 1685 log.cc:927] Time spent T 24016ceaa84147eca0b94b3ad288e772 P b495a49349324f7d81908f2da700f0d9: Append to log took a long time: real 0.081s user 0.000s sys 0.000s
I20251029 02:52:00.041188 746 tablet_service.cc:1552] Processing DeleteTablet for tablet 24f07b1efac64b099b0b5ed467463749 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:59 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:00.044533 1166 tablet_bootstrap.cc:492] T fd900eec3f0c43bf965c148ea90870a3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:00.045246 1176 log.cc:1199] T 012ce243fd07466ab5301d1c9ce4b413 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/012ce243fd07466ab5301d1c9ce4b413
I20251029 02:52:00.049031 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85159079149241adae174fc0453ed0e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:00.055140 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75236b2fa82f4f28b82efae9a2c5160d" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:00.055723 895 tablet_service.cc:1505] Processing CreateTablet for tablet a3d1c98fc694486991af6de1d9c45751 (DEFAULT_TABLE table=default.Table_556d748de1214c5592a0f74d11d7a7d9 [id=7252bdb49e2b4bfb9a031ede4d2e454d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:00.056102 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3d1c98fc694486991af6de1d9c45751. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:00.056334 747 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:44580 (request call id 472) took 1046 ms. Trace:
I20251029 02:52:00.058856 881 tablet_service.cc:1505] Processing CreateTablet for tablet e7b7de85c13b4ddc9c995ed2a2ccd921 (DEFAULT_TABLE table=default.Table_c4ce908a6d5c40e1b825533a63689d0a [id=146ba242a24c423198155dec5cd58e9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:00.059332 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7b7de85c13b4ddc9c995ed2a2ccd921. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:00.060375 753 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:44580 (request call id 459) took 1533 ms. Trace:
I20251029 02:52:00.060489 753 rpcz_store.cc:276] 1029 02:51:58.482426 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:59.299251 (+816825us) service_pool.cc:225] Handling call
1029 02:51:59.314281 (+ 15030us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1029 02:51:59.314307 (+ 26us) ts_tablet_manager.cc:682] Creating new metadata...
1029 02:51:59.718263 (+403956us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:00.015808 (+297545us) tablet_replica.cc:206] Creating consensus instance
1029 02:52:00.015994 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251029 02:52:00.064337 753 tablet_service.cc:1505] Processing CreateTablet for tablet 85f979babbd34eaeaa89ea5906ddab90 (DEFAULT_TABLE table=default.Table_8c062218c19c46169ae3652f572ed821 [id=8a5e8af85fb74b71b4fa8568c0606319]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:00.064770 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85f979babbd34eaeaa89ea5906ddab90. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:00.064904 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9fe530823864f4e8c2643bf7fe6d383. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:00.065793 746 tablet_service.cc:1505] Processing CreateTablet for tablet 07316861d2da4f55bd41111e400677fe (DEFAULT_TABLE table=default.Table_fc1eba7107f64286916e3dfce5f828b6 [id=7a248a5faa8346e09038e9e1c892625b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:00.067772 1042 tablet_service.cc:1552] Processing DeleteTablet for tablet 5842e9bba4fa4295ba21e90d645a56df with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:59 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:00.068091 1038 tablet_service.cc:1505] Processing CreateTablet for tablet 15cadd681ceb4daba178a9cc2b7aec08 (DEFAULT_TABLE table=default.Table_bc803f454de540c28f741096815c1ce2 [id=f132223faea74eb2b424cecb26440aac]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:00.068508 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15cadd681ceb4daba178a9cc2b7aec08. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:00.070201 1176 ts_tablet_manager.cc:1950] T 012ce243fd07466ab5301d1c9ce4b413 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:00.071359 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07316861d2da4f55bd41111e400677fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:00.066881 1665 raft_consensus.cc:515] T 2242f5b2c20346d9b753d7cd29242441 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:00.073341 1166 ts_tablet_manager.cc:1403] T fd900eec3f0c43bf965c148ea90870a3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.852s user 0.016s sys 0.051s
I20251029 02:52:00.074496 1042 tablet_service.cc:1552] Processing DeleteTablet for tablet 40e844fd7ad74680bcbe7bef936c5b9d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:59 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:00.074489 752 tablet_service.cc:1505] Processing CreateTablet for tablet ea7b977224f24bf49dadfb83ffa28310 (DEFAULT_TABLE table=default.Table_da40f10638dc4f96a028f73fd7f001f1 [id=579d0cc01040411f9de03e699ff959e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:00.074803 1172 ts_tablet_manager.cc:1403] T 7309d3190cfa415281c5253a634fa7f3 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.592s user 0.013s sys 0.012s
I20251029 02:52:00.074803 1042 tablet_service.cc:1552] Processing DeleteTablet for tablet 28f7ceb91a34450cad4b5b45f5285b41 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:59 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:00.075910 1042 tablet_service.cc:1552] Processing DeleteTablet for tablet 58dde9397b434b089ade9968f369006a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:59 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:00.076325 1042 tablet_service.cc:1505] Processing CreateTablet for tablet 5171c8aa52ac497d9858bb6d0ea8dfee (DEFAULT_TABLE table=default.Table_724cce6ff005499cae9f0077c069e82a [id=def36e1439084a659906905bd1eb7993]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:00.076449 747 rpcz_store.cc:276] 1029 02:51:59.010259 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:59.427209 (+416950us) service_pool.cc:225] Handling call
1029 02:51:59.427710 (+ 501us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1029 02:51:59.427735 (+ 25us) ts_tablet_manager.cc:682] Creating new metadata...
1029 02:51:59.702629 (+274894us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:00.050927 (+348298us) tablet_replica.cc:206] Creating consensus instance
1029 02:52:00.056294 (+ 5367us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251029 02:52:00.076948 747 tablet_service.cc:1505] Processing CreateTablet for tablet 15cadd681ceb4daba178a9cc2b7aec08 (DEFAULT_TABLE table=default.Table_bc803f454de540c28f741096815c1ce2 [id=f132223faea74eb2b424cecb26440aac]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:00.077531 522 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet aacb6f4b9284457bb6607e3a85d5c809 (table default.Table_f35ae25e36f64d2886752bb3b1d66710 [id=9f95160639ad40e995a6c89e297a0fad]). Expected version 1 got 0
I20251029 02:52:00.078594 891 tablet_service.cc:1505] Processing CreateTablet for tablet 8cae9d4ddb0744298b82f4d9cee061b3 (DEFAULT_TABLE table=default.Table_8ecb63fece7348379a7d341b1c4b5891 [id=9ee86407c70046c18eeea55a7b42f052]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:00.079026 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8cae9d4ddb0744298b82f4d9cee061b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:00.084370 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea7b977224f24bf49dadfb83ffa28310. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:00.085556 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 364) took 3123 ms. Trace:
I20251029 02:52:00.087394 900 tablet_service.cc:1552] Processing DeleteTablet for tablet 8f7c746f8b6948b58676298c2c2ea221 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:56 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:00.088092 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15cadd681ceb4daba178a9cc2b7aec08. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:00.088191 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 352) took 2885 ms. Trace:
I20251029 02:52:00.089628 898 tablet_service.cc:1505] Processing CreateTablet for tablet 72ac7d5f409344319fac08797e0d26bd (DEFAULT_TABLE table=default.Table_290338e0424d46cf9166ed27b1c765a1 [id=a882e9d5d98141d8a83c6382672d34ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:00.090013 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72ac7d5f409344319fac08797e0d26bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:00.092823 522 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_eef9ea2caa6e4d73a71eebd2ab5f5cde" } new_table_name: "default.Table_9bf4e25d441f4d4dbd6651386eec512e" modify_external_catalogs: true
I20251029 02:52:00.093885 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5171c8aa52ac497d9858bb6d0ea8dfee. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:00.094395 1034 tablet_service.cc:1505] Processing CreateTablet for tablet a945d64287f94ca697447e540fab39ce (DEFAULT_TABLE table=default.Table_d901c631b88743509930ea084b31e7a8 [id=74b92cc3eadb43ec96c77b178917a04d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:00.094554 1665 leader_election.cc:290] T 2242f5b2c20346d9b753d7cd29242441 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:00.094827 883 tablet_service.cc:1505] Processing CreateTablet for tablet ecf3b900206b4c86a7adbc83a769d90e (DEFAULT_TABLE table=default.Table_9cd59d4e874c4239b50b475aa8def07c [id=1d130644f20d4ea5bcbe83e01d970ce7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:00.095571 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2242f5b2c20346d9b753d7cd29242441" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:00.097291 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ecf3b900206b4c86a7adbc83a769d90e. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:00.097440 989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:38481, user_credentials={real_user=slave}} blocked reactor thread for 81702.9us
I20251029 02:52:00.096426 1174 rpcz_store.cc:276] 1029 02:51:57.177512 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:57.177975 (+ 463us) service_pool.cc:225] Handling call
1029 02:51:59.911803 (+2733828us) ts_tablet_manager.cc:1151] Deleting tablet e1fa4980f2624321b617b84c68b90113
1029 02:51:59.911816 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:00.063347 (+151531us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2733633}
I20251029 02:52:00.099277 900 tablet_service.cc:1552] Processing DeleteTablet for tablet 294afb9b012c40c0b05b73795d6f0ac8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:59 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:00.099478 884 tablet_service.cc:1552] Processing DeleteTablet for tablet 7b0f7d6c1c9842789c6ce2cecaa3d5b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:59 UTC) from {username='slave'} at 127.0.0.1:35702
W20251029 02:52:00.099643 711 leader_election.cc:343] T 2242f5b2c20346d9b753d7cd29242441 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:00.100160 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 350) took 2922 ms. Trace:
I20251029 02:52:00.100246 1174 rpcz_store.cc:276] 1029 02:51:57.177282 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:57.177695 (+ 413us) service_pool.cc:225] Handling call
1029 02:52:00.098731 (+2921036us) ts_tablet_manager.cc:1151] Deleting tablet 59d03a190b784ab18a3e63a572675891
1029 02:52:00.098742 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:00.099698 (+ 956us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2885038}
I20251029 02:52:00.100625 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 353) took 2897 ms. Trace:
I20251029 02:52:00.100706 1174 rpcz_store.cc:276] 1029 02:51:57.202918 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:57.203584 (+ 666us) service_pool.cc:225] Handling call
1029 02:52:00.100468 (+2896884us) ts_tablet_manager.cc:1151] Deleting tablet 7a1866afc7564217ad777e7ce53786c5
1029 02:52:00.100477 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:00.100574 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2885714}
I20251029 02:52:00.101074 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 354) took 2875 ms. Trace:
I20251029 02:52:00.101156 1174 rpcz_store.cc:276] 1029 02:51:57.225564 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:57.225636 (+ 72us) service_pool.cc:225] Handling call
1029 02:52:00.100927 (+2875291us) ts_tablet_manager.cc:1151] Deleting tablet 69597d48631d43b6a5b92abbf60d5ed3
1029 02:52:00.100937 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:00.101022 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2875071}
I20251029 02:52:00.101471 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 355) took 2841 ms. Trace:
I20251029 02:52:00.101585 1174 rpcz_store.cc:276] 1029 02:51:57.259828 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:57.260254 (+ 426us) service_pool.cc:225] Handling call
1029 02:52:00.101344 (+2841090us) ts_tablet_manager.cc:1151] Deleting tablet 13ee62fc2339410b9db70dfd3d3c6fa6
1029 02:52:00.101352 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:00.101424 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2781866}
I20251029 02:52:00.101918 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 372) took 2703 ms. Trace:
I20251029 02:52:00.102008 1174 rpcz_store.cc:276] 1029 02:51:57.398935 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:57.399027 (+ 92us) service_pool.cc:225] Handling call
1029 02:52:00.101783 (+2702756us) ts_tablet_manager.cc:1151] Deleting tablet 589a521f5cc24e6289c37b9cf256bce5
1029 02:52:00.101791 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:00.101868 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2696317}
I20251029 02:52:00.102358 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 373) took 2655 ms. Trace:
I20251029 02:52:00.102780 884 tablet_service.cc:1552] Processing DeleteTablet for tablet e7b7de85c13b4ddc9c995ed2a2ccd921 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:59 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:00.103345 884 tablet_service.cc:1552] Processing DeleteTablet for tablet 44c0488eba0543078ff4f0b3d69366f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:59 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:00.106403 893 tablet_service.cc:1505] Processing CreateTablet for tablet fe9e660de57a46c7ba1d82f25e12b4b0 (DEFAULT_TABLE table=default.Table_b0b0cdffdbbc4f89a6117b857e80df26 [id=6add86002712479781451e6413c94e66]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:00.107254 890 tablet_service.cc:1505] Processing CreateTablet for tablet ad2654beffef467dbfbf6472103d25f4 (DEFAULT_TABLE table=default.Table_21064497551747b08fa4839601b7c680 [id=00d800d749494f78a7c14ddc9a169e16]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 02:52:00.107969 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 59d03a190b784ab18a3e63a572675891 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 59d03a190b784ab18a3e63a572675891 already in progress: creating tablet
W20251029 02:52:00.108160 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 7a1866afc7564217ad777e7ce53786c5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7a1866afc7564217ad777e7ce53786c5 already in progress: creating tablet
W20251029 02:52:00.108359 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 69597d48631d43b6a5b92abbf60d5ed3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 69597d48631d43b6a5b92abbf60d5ed3 already in progress: creating tablet
I20251029 02:52:00.108206 1172 raft_consensus.cc:359] T 7309d3190cfa415281c5253a634fa7f3 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:00.108520 1172 raft_consensus.cc:385] T 7309d3190cfa415281c5253a634fa7f3 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251029 02:52:00.108575 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 13ee62fc2339410b9db70dfd3d3c6fa6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 13ee62fc2339410b9db70dfd3d3c6fa6 already in progress: creating tablet
I20251029 02:52:00.108608 1172 raft_consensus.cc:740] T 7309d3190cfa415281c5253a634fa7f3 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
W20251029 02:52:00.108741 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 589a521f5cc24e6289c37b9cf256bce5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 589a521f5cc24e6289c37b9cf256bce5 already in progress: creating tablet
I20251029 02:52:00.108896 1172 consensus_queue.cc:260] T 7309d3190cfa415281c5253a634fa7f3 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:00.109583 1172 ts_tablet_manager.cc:1434] T 7309d3190cfa415281c5253a634fa7f3 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.035s user 0.001s sys 0.001s
I20251029 02:52:00.109660 900 tablet_service.cc:1552] Processing DeleteTablet for tablet 7bb85a12f573494cbb3118d1947eda27 with delete_type TABLET_DATA_DELETED (replaced by tablet a9f78f61ee2841dea71bbbfceacc3ea6) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:00.109938 884 tablet_service.cc:1552] Processing DeleteTablet for tablet 8afc01d52d784f2fa559cab7ce07c8be with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:59 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:00.110546 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe9e660de57a46c7ba1d82f25e12b4b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:00.111356 879 tablet_service.cc:1505] Processing CreateTablet for tablet a47821980eb4413b8af40eb5415b5d72 (DEFAULT_TABLE table=default.Table_240f7d6a65f04099abf3e204b2f47fb6 [id=a0e3209e45574a8e8efb821e4a0f56dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:00.111738 919 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2990e9f626c40618c118266bc595684" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
I20251029 02:52:00.112227 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad2654beffef467dbfbf6472103d25f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:00.112702 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a47821980eb4413b8af40eb5415b5d72. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:00.113505 1167 tablet_bootstrap.cc:654] T aacb6f4b9284457bb6607e3a85d5c809 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:00.115001 884 tablet_service.cc:1505] Processing CreateTablet for tablet 64fbb4d0b5ab4b5bb1f21195894a66c6 (DEFAULT_TABLE table=default.Table_4559c2fec511424f8d43a3d208c5ee61 [id=e1f28b0e027b4749a63a9610c9c9ddeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:00.115352 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64fbb4d0b5ab4b5bb1f21195894a66c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:00.117653 899 tablet_service.cc:1505] Processing CreateTablet for tablet 6c10980d0c1042539ac8fb6406403ce0 (DEFAULT_TABLE table=default.Table_c32e2f0d85f9486d8957cedb7e09dda5 [id=4994066ff383406d8bae1be2e82183ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:00.118008 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c10980d0c1042539ac8fb6406403ce0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:00.118829 900 tablet_service.cc:1505] Processing CreateTablet for tablet 2e003e140a244d46b15a2c3e84f68936 (DEFAULT_TABLE table=default.Table_a8855248e99e4374aaaf0d4f66e9cad1 [id=18b70ff334f443e6a2022ad1a1c6b5de]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:00.119133 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e003e140a244d46b15a2c3e84f68936. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:00.122128 525 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_559b8d2aa7c04ba68f184405d44898d6" } new_table_name: "default.Table_a2eda0beb1fb4fd3aa2a64eded1e17ad" modify_external_catalogs: true
W20251029 02:52:00.122660 712 leader_election.cc:343] T a5c3a15510cb4321bbb80d61fabd0c0a P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:00.122820 712 leader_election.cc:304] T a5c3a15510cb4321bbb80d61fabd0c0a P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:00.123039 892 tablet_service.cc:1505] Processing CreateTablet for tablet ce98420e61904af78d8fa03724c74489 (DEFAULT_TABLE table=default.Table_3975b6f24b5b4eb88958a46683a75c43 [id=bde03d34cbf546d3ad07852b4339dd0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:00.123360 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce98420e61904af78d8fa03724c74489. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:00.123374 712 leader_election.cc:343] T 1fc87e2d7b4240e08f2b4ac7277293ff P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:00.123502 712 leader_election.cc:304] T 1fc87e2d7b4240e08f2b4ac7277293ff P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:00.124135 1665 raft_consensus.cc:2749] T a5c3a15510cb4321bbb80d61fabd0c0a P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:00.124322 1665 raft_consensus.cc:2749] T 1fc87e2d7b4240e08f2b4ac7277293ff P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:00.102434 1174 rpcz_store.cc:276] 1029 02:51:57.447011 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:57.449360 (+ 2349us) service_pool.cc:225] Handling call
1029 02:52:00.102214 (+2652854us) ts_tablet_manager.cc:1151] Deleting tablet faa835de84ff45b19525d1d603678987
1029 02:52:00.102222 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:00.102308 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2632312}
I20251029 02:52:00.125058 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 374) took 2636 ms. Trace:
I20251029 02:52:00.125151 1174 rpcz_store.cc:276] 1029 02:51:57.488754 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:57.488819 (+ 65us) service_pool.cc:225] Handling call
1029 02:52:00.124858 (+2636039us) ts_tablet_manager.cc:1151] Deleting tablet c32c28b0e74a4af3a0e69bbf24ab1a1f
1029 02:52:00.124867 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:00.125002 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2635767}
I20251029 02:52:00.125525 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 376) took 2620 ms. Trace:
I20251029 02:52:00.125603 1174 rpcz_store.cc:276] 1029 02:51:57.505225 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:57.505300 (+ 75us) service_pool.cc:225] Handling call
1029 02:52:00.125375 (+2620075us) ts_tablet_manager.cc:1151] Deleting tablet 4aa78f4ac38e4919864c924e50f1aaf4
1029 02:52:00.125384 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:00.125462 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2619590}
I20251029 02:52:00.125926 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 386) took 2354 ms. Trace:
I20251029 02:52:00.126009 1174 rpcz_store.cc:276] 1029 02:51:57.771021 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:57.771128 (+ 107us) service_pool.cc:225] Handling call
1029 02:52:00.125789 (+2354661us) ts_tablet_manager.cc:1151] Deleting tablet 2f198c3259864f7ab3625cb7bfae20b0
1029 02:52:00.125797 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:00.125877 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2351349}
I20251029 02:52:00.126322 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 392) took 2316 ms. Trace:
I20251029 02:52:00.126400 1174 rpcz_store.cc:276] 1029 02:51:57.809401 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:57.849960 (+ 40559us) service_pool.cc:225] Handling call
1029 02:52:00.126197 (+2276237us) ts_tablet_manager.cc:1151] Deleting tablet 9e07291d6b5645feb4d5890f58e1ef74
1029 02:52:00.126206 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:00.126276 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2259824}
I20251029 02:52:00.126752 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 393) took 2313 ms. Trace:
I20251029 02:52:00.126830 1174 rpcz_store.cc:276] 1029 02:51:57.813293 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:57.856158 (+ 42865us) service_pool.cc:225] Handling call
1029 02:52:00.126623 (+2270465us) ts_tablet_manager.cc:1151] Deleting tablet 23b9b2ea814645eba63e5f6b12cfb966
1029 02:52:00.126632 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:00.126704 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2252410}
I20251029 02:52:00.127089 1174 tablet_replica.cc:333] T fa5127bcfcda40a7b5ad4ef016af072a P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:00.127264 1174 raft_consensus.cc:2243] T fa5127bcfcda40a7b5ad4ef016af072a P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:00.127472 1174 raft_consensus.cc:2272] T fa5127bcfcda40a7b5ad4ef016af072a P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:00.128028 1174 ts_tablet_manager.cc:1916] T fa5127bcfcda40a7b5ad4ef016af072a P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:00.128986 750 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:44580 (request call id 473) took 1118 ms. Trace:
W20251029 02:52:00.129014 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet faa835de84ff45b19525d1d603678987 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet faa835de84ff45b19525d1d603678987 already in progress: creating tablet
I20251029 02:52:00.129112 750 rpcz_store.cc:276] 1029 02:51:59.010366 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:59.445248 (+434882us) service_pool.cc:225] Handling call
1029 02:51:59.449685 (+ 4437us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1029 02:51:59.449708 (+ 23us) ts_tablet_manager.cc:682] Creating new metadata...
1029 02:51:59.901028 (+451320us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:00.104457 (+203429us) tablet_replica.cc:206] Creating consensus instance
1029 02:52:00.128943 (+ 24486us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251029 02:52:00.129251 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet c32c28b0e74a4af3a0e69bbf24ab1a1f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c32c28b0e74a4af3a0e69bbf24ab1a1f already in progress: creating tablet
I20251029 02:52:00.129375 740 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:44580 (request call id 475) took 1099 ms. Trace:
W20251029 02:52:00.129456 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 4aa78f4ac38e4919864c924e50f1aaf4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4aa78f4ac38e4919864c924e50f1aaf4 already in progress: creating tablet
I20251029 02:52:00.129524 738 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:44580 (request call id 467) took 1119 ms. Trace:
I20251029 02:52:00.129602 738 rpcz_store.cc:276] 1029 02:51:59.009619 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:59.338032 (+328413us) service_pool.cc:225] Handling call
1029 02:51:59.445972 (+107940us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1029 02:51:59.445998 (+ 26us) ts_tablet_manager.cc:682] Creating new metadata...
1029 02:51:59.910047 (+464049us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:00.107075 (+197028us) tablet_replica.cc:206] Creating consensus instance
1029 02:52:00.129467 (+ 22392us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251029 02:52:00.129549 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_c4adcf442af54ea08664c405b8a6dfe4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:00.129681 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 2f198c3259864f7ab3625cb7bfae20b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2f198c3259864f7ab3625cb7bfae20b0 already in progress: creating tablet
W20251029 02:52:00.129864 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 9e07291d6b5645feb4d5890f58e1ef74 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9e07291d6b5645feb4d5890f58e1ef74 already in progress: creating tablet
W20251029 02:52:00.130065 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 23b9b2ea814645eba63e5f6b12cfb966 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 23b9b2ea814645eba63e5f6b12cfb966 already in progress: creating tablet
I20251029 02:52:00.130102 738 tablet_service.cc:1552] Processing DeleteTablet for tablet 40e844fd7ad74680bcbe7bef936c5b9d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:59 UTC) from {username='slave'} at 127.0.0.1:44580
W20251029 02:52:00.130131 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c4adcf442af54ea08664c405b8a6dfe4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:00.130304 738 tablet_service.cc:1552] Processing DeleteTablet for tablet 28f7ceb91a34450cad4b5b45f5285b41 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:59 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:00.130537 738 tablet_service.cc:1552] Processing DeleteTablet for tablet 58dde9397b434b089ade9968f369006a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:59 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:00.130937 738 tablet_service.cc:1505] Processing CreateTablet for tablet 5171c8aa52ac497d9858bb6d0ea8dfee (DEFAULT_TABLE table=default.Table_724cce6ff005499cae9f0077c069e82a [id=def36e1439084a659906905bd1eb7993]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:00.131340 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5171c8aa52ac497d9858bb6d0ea8dfee. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:00.129376 750 tablet_service.cc:1552] Processing DeleteTablet for tablet 5842e9bba4fa4295ba21e90d645a56df with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:59 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:00.133913 750 tablet_service.cc:1505] Processing CreateTablet for tablet a945d64287f94ca697447e540fab39ce (DEFAULT_TABLE table=default.Table_d901c631b88743509930ea084b31e7a8 [id=74b92cc3eadb43ec96c77b178917a04d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:00.134279 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a945d64287f94ca697447e540fab39ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:00.134814 1172 tablet_bootstrap.cc:492] T dbc2092b63e0417997565c8fadb53ae0 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:00.136566 1172 tablet_bootstrap.cc:654] T dbc2092b63e0417997565c8fadb53ae0 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
W20251029 02:52:00.136869 525 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f93b0c39d7174dabbe018b1546dbc198 consensus state has no leader
I20251029 02:52:00.139719 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_ff3d06085614492a97e31873f16f4fdc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:00.140117 756 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:44580 (request call id 460) took 1657 ms. Trace:
I20251029 02:52:00.140205 756 rpcz_store.cc:276] 1029 02:51:58.482540 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:59.316894 (+834354us) service_pool.cc:225] Handling call
1029 02:51:59.322531 (+ 5637us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1029 02:51:59.322559 (+ 28us) ts_tablet_manager.cc:682] Creating new metadata...
1029 02:51:59.742567 (+420008us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:00.139930 (+397363us) tablet_replica.cc:206] Creating consensus instance
1029 02:52:00.140062 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251029 02:52:00.140410 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ff3d06085614492a97e31873f16f4fdc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:00.140699 756 tablet_service.cc:1505] Processing CreateTablet for tablet b41e114ed6e94f5895bdec9bee836d2d (DEFAULT_TABLE table=default.Table_203b46ee33744bf3bfe75190ab006c13 [id=825464f47f3a45f79eb319c3ca3e54e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:00.140882 750 tablet_service.cc:1505] Processing CreateTablet for tablet a04bc4d9605d4c049a151d985035bb63 (DEFAULT_TABLE table=default.Table_62644549a49a4f039492b17deac6113d [id=bb15ae87ebcd4a39a99982967fd215b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:00.141075 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b41e114ed6e94f5895bdec9bee836d2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:00.141392 749 tablet_service.cc:1505] Processing CreateTablet for tablet 9c6f4dc4b0c74ddd80d9dc762619b1ec (DEFAULT_TABLE table=default.Table_51e08a9cce7d4d9f9b358a693206462f [id=e1f53eaba1d64ef495c67d40be796e49]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:00.141736 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c6f4dc4b0c74ddd80d9dc762619b1ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:00.129477 740 rpcz_store.cc:276] 1029 02:51:59.029572 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:59.520152 (+490580us) service_pool.cc:225] Handling call
1029 02:51:59.558645 (+ 38493us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1029 02:51:59.558679 (+ 34us) ts_tablet_manager.cc:682] Creating new metadata...
1029 02:51:59.930554 (+371875us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:00.129186 (+198632us) tablet_replica.cc:206] Creating consensus instance
1029 02:52:00.129310 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20251029 02:52:00.155086 990 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:41701, user_credentials={real_user=slave}} blocked reactor thread for 49212.6us
I20251029 02:52:00.157605 1057 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2242f5b2c20346d9b753d7cd29242441" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:00.165424 740 tablet_service.cc:1505] Processing CreateTablet for tablet c8b6afc8c77c4a7fbe1956cb30bba656 (DEFAULT_TABLE table=default.Table_db8d41696cb54486aa9fa06d4396e499 [id=f87557451c8c466c8f6ea573155aa684]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:00.167076 522 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_3b4bdfaf4ddf43cfa2d2287594c57816" } modify_external_catalogs: true
I20251029 02:52:00.167349 522 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_3b4bdfaf4ddf43cfa2d2287594c57816" } modify_external_catalogs: true
I20251029 02:52:00.169716 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a04bc4d9605d4c049a151d985035bb63. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:00.172641 885 tablet_service.cc:1505] Processing CreateTablet for tablet 8f1d5847265b4bb0b29b5e513c35b9bb (DEFAULT_TABLE table=default.Table_877abcb0d221493c8875368e1c75382b [id=6ea46a01a86140b9ae63e33b90b03dd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:00.173271 881 tablet_service.cc:1505] Processing CreateTablet for tablet 479fd93eb02540868931faee38830223 (DEFAULT_TABLE table=default.Table_d44327be26be4ff1b62a7be7dada45c0 [id=97e41fe5933b4e6e98f6692542883406]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:00.173771 895 tablet_service.cc:1505] Processing CreateTablet for tablet 45d33b70d549488c802957bb2d5b2821 (DEFAULT_TABLE table=default.Table_360a4b04c44345e0a36d04e6d90c698f [id=6044f9e5029b4985b5540d35286f8586]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:00.177599 1514 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42490 (request call id 0) took 3401 ms. Trace:
I20251029 02:52:00.181367 523 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_47c159158e26411c934a19f853eb19e9" } new_table_name: "default.Table_0ae102f09fdc41bcbead0148653df9e2" modify_external_catalogs: true
I20251029 02:52:00.182386 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_4780e4a5339245d1acdf62a82234a1d2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:00.183058 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4780e4a5339245d1acdf62a82234a1d2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:00.183375 524 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_beeb4b56920e4aff85dfbb9758fa414c" } modify_external_catalogs: true
I20251029 02:52:00.183717 520 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_cd55951068e2462ab1b5b075a5b40767" } modify_external_catalogs: true
I20251029 02:52:00.184043 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_b131ed8021574ae885778d1e8da0ad8e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:00.184520 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b131ed8021574ae885778d1e8da0ad8e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:00.185348 521 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_ad59ec000b0c4dbcb80dd9a5d9648ec6" } modify_external_catalogs: true
I20251029 02:52:00.185660 526 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_0f0a1e0ac81a4b3492d304bef28ee22d" } modify_external_catalogs: true
I20251029 02:52:00.185974 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_e1ecca1682bb467789ae7beec4a3f11d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:00.186420 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e1ecca1682bb467789ae7beec4a3f11d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:00.190553 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f1d5847265b4bb0b29b5e513c35b9bb. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:00.192044 712 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:45003, user_credentials={real_user=slave}} blocked reactor thread for 68659.9us
W20251029 02:52:00.192312 712 leader_election.cc:343] T 0a47781d036149569e1164dd938a45e9 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:00.193495 1166 raft_consensus.cc:359] T fd900eec3f0c43bf965c148ea90870a3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:00.195194 743 tablet_service.cc:1552] Processing DeleteTablet for tablet fcb3db8399aa4cf188b07dda220f7f5b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:59 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:00.195457 743 tablet_service.cc:1552] Processing DeleteTablet for tablet 845a69a8696e469a8548787a7645d41a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:59 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:00.195632 743 tablet_service.cc:1552] Processing DeleteTablet for tablet 38f83ebcb2a440bc8e6a8d36607e6666 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:59 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:00.195812 743 tablet_service.cc:1552] Processing DeleteTablet for tablet 8df97119d0654d7d92bb5bd17c5d8562 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:59 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:00.195972 743 tablet_service.cc:1552] Processing DeleteTablet for tablet f65fe9a5137b48669afef51ad7dd76ae with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:59 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:00.196341 743 tablet_service.cc:1505] Processing CreateTablet for tablet a3d1c98fc694486991af6de1d9c45751 (DEFAULT_TABLE table=default.Table_556d748de1214c5592a0f74d11d7a7d9 [id=7252bdb49e2b4bfb9a031ede4d2e454d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:00.200510 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 479fd93eb02540868931faee38830223. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:00.201246 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8b6afc8c77c4a7fbe1956cb30bba656. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:00.206398 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45d33b70d549488c802957bb2d5b2821. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:00.207962 748 tablet_service.cc:1505] Processing CreateTablet for tablet a7685b0bed174ed0b25f99ef8a46fca3 (DEFAULT_TABLE table=default.Table_beeb4b56920e4aff85dfbb9758fa414c [id=213b1abd06cf4e22b05bfb35460bdd3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:00.209488 741 tablet_service.cc:1505] Processing CreateTablet for tablet 699ee74f39b342639ab84d0d65e53a70 (DEFAULT_TABLE table=default.Table_1ff672db4f064544b523c0ae9667d967 [id=60cf9a35c3c848d8a4e7cd7a8128b12f]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 02:52:00.210188 1692 log.cc:927] Time spent T 70e8cfe5eaa9492c85874dc3214a1e0b P b495a49349324f7d81908f2da700f0d9: Append to log took a long time: real 0.158s user 0.000s sys 0.000s
I20251029 02:52:00.219492 754 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:44580 (request call id 474) took 1209 ms. Trace:
I20251029 02:52:00.220137 519 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_8462690958454b098e0531b1086e60f7" } new_table_name: "default.Table_075076404d494b29b2cb90e9f10219e5" modify_external_catalogs: true
I20251029 02:52:00.222280 754 rpcz_store.cc:276] 1029 02:51:59.010448 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:59.506925 (+496477us) service_pool.cc:225] Handling call
1029 02:51:59.508140 (+ 1215us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1029 02:51:59.508172 (+ 32us) ts_tablet_manager.cc:682] Creating new metadata...
1029 02:51:59.903376 (+395204us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:00.219277 (+315901us) tablet_replica.cc:206] Creating consensus instance
1029 02:52:00.219441 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251029 02:52:00.222875 754 tablet_service.cc:1505] Processing CreateTablet for tablet e7b7de85c13b4ddc9c995ed2a2ccd921 (DEFAULT_TABLE table=default.Table_c4ce908a6d5c40e1b825533a63689d0a [id=146ba242a24c423198155dec5cd58e9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:00.226716 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3d1c98fc694486991af6de1d9c45751. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:00.194447 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a945d64287f94ca697447e540fab39ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:00.174078 1176 rpcz_store.cc:276] 1029 02:51:56.955106 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:56.955242 (+ 136us) service_pool.cc:225] Handling call
1029 02:51:59.191290 (+2236048us) ts_tablet_manager.cc:1151] Deleting tablet 012ce243fd07466ab5301d1c9ce4b413
1029 02:51:59.191299 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:51:59.788097 (+596798us) tablet_metadata.cc:641] Metadata flushed
1029 02:51:59.901261 (+113164us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:00.078457 (+177196us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":184993,"tablet-delete.queue_time_us":2235845}
I20251029 02:52:00.227378 1166 raft_consensus.cc:385] T fd900eec3f0c43bf965c148ea90870a3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:00.227320 1027 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:37626 (request call id 490) took 1094 ms. Trace:
I20251029 02:52:00.192430 712 leader_election.cc:304] T 0a47781d036149569e1164dd938a45e9 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:00.238569 1717 raft_consensus.cc:2749] T 0a47781d036149569e1164dd938a45e9 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:00.242067 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7b7de85c13b4ddc9c995ed2a2ccd921. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:00.245748 1166 raft_consensus.cc:740] T fd900eec3f0c43bf965c148ea90870a3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:00.248795 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 012ce243fd07466ab5301d1c9ce4b413 (table default.Table_cb594ce4077d4d02aa70ef94d5f801cc [id=b792f15b5ea34e44b32b3087b983567c]) successfully deleted
W20251029 02:52:00.252905 712 leader_election.cc:343] T 392a36a6b59a42f78e3ac7c8fb63214b P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:00.255419 712 leader_election.cc:304] T 392a36a6b59a42f78e3ac7c8fb63214b P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:00.257858 1717 raft_consensus.cc:2749] T 392a36a6b59a42f78e3ac7c8fb63214b P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:00.261335 524 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_beeb4b56920e4aff85dfbb9758fa414c" } modify_external_catalogs: true
I20251029 02:52:00.265252 520 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_cd55951068e2462ab1b5b075a5b40767" } modify_external_catalogs: true
I20251029 02:52:00.266477 526 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_0f0a1e0ac81a4b3492d304bef28ee22d" } modify_external_catalogs: true
I20251029 02:52:00.267792 521 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_ad59ec000b0c4dbcb80dd9a5d9648ec6" } modify_external_catalogs: true
I20251029 02:52:00.269858 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 699ee74f39b342639ab84d0d65e53a70. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:00.274358 712 leader_election.cc:343] T 9e07291d6b5645feb4d5890f58e1ef74 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:00.274621 737 tablet_service.cc:1505] Processing CreateTablet for tablet ecf3b900206b4c86a7adbc83a769d90e (DEFAULT_TABLE table=default.Table_9cd59d4e874c4239b50b475aa8def07c [id=1d130644f20d4ea5bcbe83e01d970ce7]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 02:52:00.277626 523 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6dff036b32904ddb9598be4b79504bb6 consensus state has no leader
I20251029 02:52:00.278503 1175 ts_tablet_manager.cc:1929] T 8f7c746f8b6948b58676298c2c2ea221 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20251029 02:52:00.279561 712 leader_election.cc:304] T 9e07291d6b5645feb4d5890f58e1ef74 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:00.280432 742 tablet_service.cc:1552] Processing DeleteTablet for tablet 7b0f7d6c1c9842789c6ce2cecaa3d5b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:59 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:00.281703 749 tablet_service.cc:1505] Processing CreateTablet for tablet 8cae9d4ddb0744298b82f4d9cee061b3 (DEFAULT_TABLE table=default.Table_8ecb63fece7348379a7d341b1c4b5891 [id=9ee86407c70046c18eeea55a7b42f052]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:00.282753 756 tablet_service.cc:1505] Processing CreateTablet for tablet 72ac7d5f409344319fac08797e0d26bd (DEFAULT_TABLE table=default.Table_290338e0424d46cf9166ed27b1c765a1 [id=a882e9d5d98141d8a83c6382672d34ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:00.284068 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7685b0bed174ed0b25f99ef8a46fca3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:00.286684 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ecf3b900206b4c86a7adbc83a769d90e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:00.287386 1175 log.cc:1199] T 8f7c746f8b6948b58676298c2c2ea221 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/8f7c746f8b6948b58676298c2c2ea221
I20251029 02:52:00.287417 1717 raft_consensus.cc:2749] T 9e07291d6b5645feb4d5890f58e1ef74 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 02:52:00.287622 712 leader_election.cc:343] T 8afc01d52d784f2fa559cab7ce07c8be P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:00.287750 712 leader_election.cc:304] T 8afc01d52d784f2fa559cab7ce07c8be P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:00.287868 1175 ts_tablet_manager.cc:1950] T 8f7c746f8b6948b58676298c2c2ea221 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:00.288022 1717 raft_consensus.cc:2749] T 8afc01d52d784f2fa559cab7ce07c8be P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:00.288218 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 342) took 3376 ms. Trace:
I20251029 02:52:00.288312 1175 rpcz_store.cc:276] 1029 02:51:56.911890 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:56.912573 (+ 683us) service_pool.cc:225] Handling call
1029 02:51:59.892404 (+2979831us) ts_tablet_manager.cc:1151] Deleting tablet 8f7c746f8b6948b58676298c2c2ea221
1029 02:51:59.892413 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:00.097979 (+205566us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:00.278397 (+180418us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:00.288155 (+ 9758us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2979590}
W20251029 02:52:00.288502 712 leader_election.cc:343] T 2242f5b2c20346d9b753d7cd29242441 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:00.288741 712 leader_election.cc:304] T 2242f5b2c20346d9b753d7cd29242441 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:00.288815 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 343) took 3365 ms. Trace:
I20251029 02:52:00.288892 1175 rpcz_store.cc:276] 1029 02:51:56.923698 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:56.923753 (+ 55us) service_pool.cc:225] Handling call
1029 02:52:00.288638 (+3364885us) ts_tablet_manager.cc:1151] Deleting tablet 51effc9324034ae79e2b25cb57a996c1
1029 02:52:00.288646 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:00.288760 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3364677}
I20251029 02:52:00.289023 1717 raft_consensus.cc:2749] T 2242f5b2c20346d9b753d7cd29242441 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:00.290772 742 tablet_service.cc:1552] Processing DeleteTablet for tablet e7b7de85c13b4ddc9c995ed2a2ccd921 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:59 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:00.296334 742 tablet_service.cc:1552] Processing DeleteTablet for tablet 8afc01d52d784f2fa559cab7ce07c8be with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:59 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:00.296594 742 tablet_service.cc:1552] Processing DeleteTablet for tablet 7bb85a12f573494cbb3118d1947eda27 with delete_type TABLET_DATA_DELETED (replaced by tablet a9f78f61ee2841dea71bbbfceacc3ea6) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:00.296790 742 tablet_service.cc:1552] Processing DeleteTablet for tablet 2242f5b2c20346d9b753d7cd29242441 with delete_type TABLET_DATA_DELETED (replaced by tablet e1fa4980f2624321b617b84c68b90113) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:00.296790 1174 ts_tablet_manager.cc:1929] T fa5127bcfcda40a7b5ad4ef016af072a P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251029 02:52:00.296996 1174 log.cc:1199] T fa5127bcfcda40a7b5ad4ef016af072a P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/fa5127bcfcda40a7b5ad4ef016af072a
I20251029 02:52:00.297644 742 tablet_service.cc:1505] Processing CreateTablet for tablet fe9e660de57a46c7ba1d82f25e12b4b0 (DEFAULT_TABLE table=default.Table_b0b0cdffdbbc4f89a6117b857e80df26 [id=6add86002712479781451e6413c94e66]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:00.298090 755 tablet_service.cc:1552] Processing DeleteTablet for tablet 294afb9b012c40c0b05b73795d6f0ac8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:59 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:00.303826 1167 tablet_bootstrap.cc:492] T aacb6f4b9284457bb6607e3a85d5c809 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:00.304114 1167 ts_tablet_manager.cc:1403] T aacb6f4b9284457bb6607e3a85d5c809 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.415s user 0.002s sys 0.012s
I20251029 02:52:00.306219 744 tablet_service.cc:1552] Processing DeleteTablet for tablet 44c0488eba0543078ff4f0b3d69366f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:59 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:00.307029 755 tablet_service.cc:1505] Processing CreateTablet for tablet a47821980eb4413b8af40eb5415b5d72 (DEFAULT_TABLE table=default.Table_240f7d6a65f04099abf3e204b2f47fb6 [id=a0e3209e45574a8e8efb821e4a0f56dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:00.307396 1167 raft_consensus.cc:359] T aacb6f4b9284457bb6607e3a85d5c809 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:00.289264 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 8f7c746f8b6948b58676298c2c2ea221 (table default.Table_da4961f316764505bb9e0f191a62a327 [id=669ae9dd0a5749b1aa3557ab8183a057]) successfully deleted
W20251029 02:52:00.309594 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 51effc9324034ae79e2b25cb57a996c1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 51effc9324034ae79e2b25cb57a996c1 already in progress: creating tablet
W20251029 02:52:00.303009 519 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c45ffc304e73490387ce800b702b5881 consensus state has no leader
I20251029 02:52:00.309938 1167 raft_consensus.cc:385] T aacb6f4b9284457bb6607e3a85d5c809 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:00.310940 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_88c311b8b4534d5190945358ee18fe6a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:00.311517 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_88c311b8b4534d5190945358ee18fe6a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:00.312181 1491 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "c756528642a543eeacd93cef3e335a7d" instance_seqno: 1761706309176588) as {username='slave'} at 127.0.104.2:51269; Asking this server to re-register.
I20251029 02:52:00.313663 966 heartbeater.cc:461] Registering TS with master...
I20251029 02:52:00.315017 1682 raft_consensus.cc:493] T 8d9722c764ac4b27a540595e9c6aafcd P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:00.291293 771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2990e9f626c40618c118266bc595684" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:00.315205 1682 raft_consensus.cc:515] T 8d9722c764ac4b27a540595e9c6aafcd P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:00.315151 518 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet d2990e9f626c40618c118266bc595684 (table default.Table_bd020e7c45a54c6e92211c4261293575 [id=f733948e92d8470da9d8050bd26a06b1]). Expected version 1 got 0
I20251029 02:52:00.315552 518 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet aacb6f4b9284457bb6607e3a85d5c809 (table default.Table_f35ae25e36f64d2886752bb3b1d66710 [id=9f95160639ad40e995a6c89e297a0fad]). Expected version 1 got 0
I20251029 02:52:00.315797 518 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet 41a6139fd9b84707a8c7eebc0381e383 (table default.Table_e48aadc66a0e48658d45d0812c29cbdf [id=f45315b686594867aae0114f4ce2f26c]). Expected version 1 got 0
I20251029 02:52:00.315882 1682 leader_election.cc:290] T 8d9722c764ac4b27a540595e9c6aafcd P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:00.316071 518 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet dd90fb9eb167478cbc3ec9363af86295 (table default.Table_8b6217e43e8d467693c104ef84e78911 [id=052abf3dcc904bdcb869d5b885723dab]). Expected version 4 got 3
I20251029 02:52:00.314456 1174 ts_tablet_manager.cc:1950] T fa5127bcfcda40a7b5ad4ef016af072a P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:00.314107 1167 raft_consensus.cc:740] T aacb6f4b9284457bb6607e3a85d5c809 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:00.318161 771 raft_consensus.cc:3060] T d2990e9f626c40618c118266bc595684 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:00.263928 1027 rpcz_store.cc:276] 1029 02:51:59.106249 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:59.576678 (+470429us) service_pool.cc:225] Handling call
1029 02:51:59.582930 (+ 6252us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1029 02:51:59.582958 (+ 28us) ts_tablet_manager.cc:682] Creating new metadata...
1029 02:51:59.960729 (+377771us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:00.175731 (+215002us) tablet_replica.cc:206] Creating consensus instance
1029 02:52:00.200362 (+ 24631us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1920}
I20251029 02:52:00.309228 522 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 97b2ecae572a4b7ea1b9595074556708
W20251029 02:52:00.319334 847 leader_election.cc:343] T 8d9722c764ac4b27a540595e9c6aafcd P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 8d9722c764ac4b27a540595e9c6aafcd
I20251029 02:52:00.319772 1686 leader_election.cc:290] T d2990e9f626c40618c118266bc595684 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Requested vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:00.320384 895 tablet_service.cc:1552] Processing DeleteTablet for tablet 97b2ecae572a4b7ea1b9595074556708 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:00 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:00.320427 1166 consensus_queue.cc:260] T fd900eec3f0c43bf965c148ea90870a3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:00.320650 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_5668f431df944d4291ec4d158ff43e57"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:00.321262 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5668f431df944d4291ec4d158ff43e57 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:00.320971 1167 consensus_queue.cc:260] T aacb6f4b9284457bb6607e3a85d5c809 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:00.321290 1036 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:37626 (request call id 498) took 1212 ms. Trace:
I20251029 02:52:00.326030 1682 raft_consensus.cc:493] T 6c22497760274380a832feff7c5a2fae P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:00.326201 1682 raft_consensus.cc:515] T 6c22497760274380a832feff7c5a2fae P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:00.326890 1682 leader_election.cc:290] T 6c22497760274380a832feff7c5a2fae P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:00.327631 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 395) took 2508 ms. Trace:
I20251029 02:52:00.330509 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 344) took 3363 ms. Trace:
I20251029 02:52:00.330817 747 tablet_service.cc:1505] Processing CreateTablet for tablet 64fbb4d0b5ab4b5bb1f21195894a66c6 (DEFAULT_TABLE table=default.Table_4559c2fec511424f8d43a3d208c5ee61 [id=e1f28b0e027b4749a63a9610c9c9ddeb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:00.331581 520 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 998b9cbe76514665be76c6c387dc75b3
I20251029 02:52:00.331918 744 tablet_service.cc:1505] Processing CreateTablet for tablet ad2654beffef467dbfbf6472103d25f4 (DEFAULT_TABLE table=default.Table_21064497551747b08fa4839601b7c680 [id=00d800d749494f78a7c14ddc9a169e16]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:00.332484 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_dc5f0ee620604d15a617a3c7ed345c37"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:00.331625 1167 ts_tablet_manager.cc:1434] T aacb6f4b9284457bb6607e3a85d5c809 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.027s user 0.002s sys 0.001s
W20251029 02:52:00.333127 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dc5f0ee620604d15a617a3c7ed345c37 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:00.324434 1726 tablet.cc:1722] T 24016ceaa84147eca0b94b3ad288e772 P b495a49349324f7d81908f2da700f0d9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 02:52:00.334837 895 tablet_service.cc:1552] Processing DeleteTablet for tablet 998b9cbe76514665be76c6c387dc75b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:00 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:00.333236 1174 rpcz_store.cc:276] 1029 02:51:57.818811 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:57.866408 (+ 47597us) service_pool.cc:225] Handling call
1029 02:52:00.127012 (+2260604us) ts_tablet_manager.cc:1151] Deleting tablet fa5127bcfcda40a7b5ad4ef016af072a
1029 02:52:00.127020 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:00.222025 (+ 95005us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:00.289815 (+ 67790us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:00.327592 (+ 37777us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2233479}
I20251029 02:52:00.335443 1167 tablet_bootstrap.cc:492] T f71388036478404d878a52fc20ece216 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:00.336057 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_9dfd24b5c34c4b9f965292b798351bc1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:00.336612 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9dfd24b5c34c4b9f965292b798351bc1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:00.337841 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet fa5127bcfcda40a7b5ad4ef016af072a (table default.Table_5113d1a33e8e4bd2af5bc51e689f304f [id=ae5d56d40fa941a68f2b10d5e6e9ac83]) successfully deleted
I20251029 02:52:00.337482 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 394) took 2521 ms. Trace:
I20251029 02:52:00.338912 524 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet a7685b0bed174ed0b25f99ef8a46fca3
W20251029 02:52:00.339190 847 leader_election.cc:343] T 6c22497760274380a832feff7c5a2fae P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 6c22497760274380a832feff7c5a2fae
I20251029 02:52:00.339645 524 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_10befcd256124bc187c2f0ed7887d711" } new_table_name: "default.Table_8dacd5c558a04e47b424533b985aee9e" modify_external_catalogs: true
I20251029 02:52:00.338456 1174 rpcz_store.cc:276] 1029 02:51:57.815936 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:57.864313 (+ 48377us) service_pool.cc:225] Handling call
1029 02:52:00.336301 (+2471988us) ts_tablet_manager.cc:1151] Deleting tablet afb19ba39399410bb90b6225d7b5b5b1
1029 02:52:00.336312 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:00.337275 (+ 963us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2423137}
W20251029 02:52:00.341385 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet afb19ba39399410bb90b6225d7b5b5b1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet afb19ba39399410bb90b6225d7b5b5b1 already in progress: creating tablet
I20251029 02:52:00.341490 895 tablet_service.cc:1552] Processing DeleteTablet for tablet a7685b0bed174ed0b25f99ef8a46fca3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:00 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:00.343147 525 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_e0854491deb34525bce9198a673fb943" } new_table_name: "default.Table_a45ccb3fef5d4947ab7a8709b7c32119" modify_external_catalogs: true
I20251029 02:52:00.345155 521 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 4cede3b183494845af91dca39a2096c4
I20251029 02:52:00.349274 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_25a8f4870ca94816934d0214177fc278"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:00.349808 895 tablet_service.cc:1552] Processing DeleteTablet for tablet 4cede3b183494845af91dca39a2096c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:00 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:00.351374 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 399) took 2308 ms. Trace:
W20251029 02:52:00.352799 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_25a8f4870ca94816934d0214177fc278 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:00.336083 752 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:44580 (request call id 495) took 1004 ms. Trace:
I20251029 02:52:00.353737 752 rpcz_store.cc:276] 1029 02:51:59.326361 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:00.059787 (+733426us) service_pool.cc:225] Handling call
1029 02:52:00.076211 (+ 16424us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1029 02:52:00.076237 (+ 26us) ts_tablet_manager.cc:682] Creating new metadata...
1029 02:52:00.187454 (+111217us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:00.327509 (+140055us) tablet_replica.cc:206] Creating consensus instance
1029 02:52:00.330714 (+ 3205us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251029 02:52:00.352421 1174 rpcz_store.cc:276] 1029 02:51:58.033972 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:58.034536 (+ 564us) service_pool.cc:225] Handling call
1029 02:52:00.340699 (+2306163us) ts_tablet_manager.cc:1151] Deleting tablet 84993b2f63e7421290a161dde4bb0607
1029 02:52:00.340712 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:00.342358 (+ 1646us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2305740}
I20251029 02:52:00.354386 752 tablet_service.cc:1505] Processing CreateTablet for tablet 2e003e140a244d46b15a2c3e84f68936 (DEFAULT_TABLE table=default.Table_a8855248e99e4374aaaf0d4f66e9cad1 [id=18b70ff334f443e6a2022ad1a1c6b5de]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:00.336357 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8cae9d4ddb0744298b82f4d9cee061b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:00.338356 745 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:44580 (request call id 492) took 1012 ms. Trace:
I20251029 02:52:00.355388 745 rpcz_store.cc:276] 1029 02:51:59.326071 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:59.915281 (+589210us) service_pool.cc:225] Handling call
1029 02:52:00.026310 (+111029us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1029 02:52:00.026347 (+ 37us) ts_tablet_manager.cc:682] Creating new metadata...
1029 02:52:00.314316 (+287969us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:00.338185 (+ 23869us) tablet_replica.cc:206] Creating consensus instance
1029 02:52:00.338314 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251029 02:52:00.355729 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad2654beffef467dbfbf6472103d25f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:00.354055 521 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_5391971cdf974108a468e2d263d1cec1" } new_table_name: "default.Table_2a63af6894694f16b45f4bf48f73a68f" modify_external_catalogs: true
I20251029 02:52:00.355991 745 tablet_service.cc:1505] Processing CreateTablet for tablet 6c10980d0c1042539ac8fb6406403ce0 (DEFAULT_TABLE table=default.Table_c32e2f0d85f9486d8957cedb7e09dda5 [id=4994066ff383406d8bae1be2e82183ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:00.356362 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c10980d0c1042539ac8fb6406403ce0. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:00.357285 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 84993b2f63e7421290a161dde4bb0607 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 84993b2f63e7421290a161dde4bb0607 already in progress: creating tablet
I20251029 02:52:00.335832 1175 rpcz_store.cc:276] 1029 02:51:56.927106 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:56.927255 (+ 149us) service_pool.cc:225] Handling call
1029 02:52:00.289568 (+3362313us) ts_tablet_manager.cc:1151] Deleting tablet 71068739b4384badbf305313d902f0f5
1029 02:52:00.289579 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:00.290648 (+ 1069us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3360461}
I20251029 02:52:00.358783 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 397) took 2330 ms. Trace:
I20251029 02:52:00.358880 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 353) took 3202 ms. Trace:
I20251029 02:52:00.358996 1175 rpcz_store.cc:276] 1029 02:51:57.156412 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:57.156513 (+ 101us) service_pool.cc:225] Handling call
1029 02:52:00.358679 (+3202166us) ts_tablet_manager.cc:1151] Deleting tablet 59d03a190b784ab18a3e63a572675891
1029 02:52:00.358688 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:00.358810 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3201876}
I20251029 02:52:00.359342 1175 tablet_replica.cc:333] T 9be9add0274d41b688edd6292a253a06 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:00.359583 1175 raft_consensus.cc:2243] T 9be9add0274d41b688edd6292a253a06 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:00.359833 1175 raft_consensus.cc:2272] T 9be9add0274d41b688edd6292a253a06 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:00.360457 1175 ts_tablet_manager.cc:1916] T 9be9add0274d41b688edd6292a253a06 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:00.343783 739 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:44580 (request call id 490) took 1012 ms. Trace:
I20251029 02:52:00.362648 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e003e140a244d46b15a2c3e84f68936. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:00.362612 739 rpcz_store.cc:276] 1029 02:51:59.323348 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:59.905353 (+582005us) service_pool.cc:225] Handling call
1029 02:51:59.919971 (+ 14618us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1029 02:51:59.919995 (+ 24us) ts_tablet_manager.cc:682] Creating new metadata...
1029 02:52:00.145414 (+225419us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:00.335824 (+190410us) tablet_replica.cc:206] Creating consensus instance
1029 02:52:00.335985 (+ 161us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251029 02:52:00.362983 739 tablet_service.cc:1552] Processing DeleteTablet for tablet 7309d3190cfa415281c5253a634fa7f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:52 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:00.363408 739 tablet_service.cc:1505] Processing CreateTablet for tablet ce98420e61904af78d8fa03724c74489 (DEFAULT_TABLE table=default.Table_3975b6f24b5b4eb88958a46683a75c43 [id=bde03d34cbf546d3ad07852b4339dd0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:00.363791 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce98420e61904af78d8fa03724c74489. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:00.366566 1174 rpcz_store.cc:276] 1029 02:51:58.028631 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:58.029517 (+ 886us) service_pool.cc:225] Handling call
1029 02:52:00.357003 (+2327486us) ts_tablet_manager.cc:1151] Deleting tablet 49d9ce32e4f04f11be2ecefc5d523a8b
1029 02:52:00.357316 (+ 313us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:00.358730 (+ 1414us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2314224}
W20251029 02:52:00.367112 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 49d9ce32e4f04f11be2ecefc5d523a8b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 49d9ce32e4f04f11be2ecefc5d523a8b already in progress: creating tablet
I20251029 02:52:00.343895 746 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:44580 (request call id 494) took 1009 ms. Trace:
I20251029 02:52:00.346375 771 raft_consensus.cc:2468] T d2990e9f626c40618c118266bc595684 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 1.
I20251029 02:52:00.370944 746 rpcz_store.cc:276] 1029 02:51:59.326258 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:00.049290 (+723032us) service_pool.cc:225] Handling call
1029 02:52:00.065979 (+ 16689us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1029 02:52:00.066008 (+ 29us) ts_tablet_manager.cc:682] Creating new metadata...
1029 02:52:00.310493 (+244485us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:00.336043 (+ 25550us) tablet_replica.cc:206] Creating consensus instance
1029 02:52:00.336144 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2579}
I20251029 02:52:00.371585 746 tablet_service.cc:1505] Processing CreateTablet for tablet 479fd93eb02540868931faee38830223 (DEFAULT_TABLE table=default.Table_d44327be26be4ff1b62a7be7dada45c0 [id=97e41fe5933b4e6e98f6692542883406]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:00.371991 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 479fd93eb02540868931faee38830223. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:00.372778 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a47821980eb4413b8af40eb5415b5d72. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:00.375535 1167 tablet_bootstrap.cc:654] T f71388036478404d878a52fc20ece216 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:00.375897 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 401) took 2320 ms. Trace:
W20251029 02:52:00.377271 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 71068739b4384badbf305313d902f0f5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 71068739b4384badbf305313d902f0f5 already in progress: creating tablet
I20251029 02:52:00.376612 1174 rpcz_store.cc:276] 1029 02:51:58.055090 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:58.055149 (+ 59us) service_pool.cc:225] Handling call
1029 02:52:00.375318 (+2320169us) ts_tablet_manager.cc:1151] Deleting tablet a5c3a15510cb4321bbb80d61fabd0c0a
1029 02:52:00.375330 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:00.375775 (+ 445us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2319424}
W20251029 02:52:00.377539 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 59d03a190b784ab18a3e63a572675891 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 59d03a190b784ab18a3e63a572675891 already in progress: creating tablet
W20251029 02:52:00.378062 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet a5c3a15510cb4321bbb80d61fabd0c0a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a5c3a15510cb4321bbb80d61fabd0c0a already in progress: creating tablet
I20251029 02:52:00.378672 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64fbb4d0b5ab4b5bb1f21195894a66c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:00.379276 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_9bc6d7e931a34e00b8a1c2233fa53c3f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:00.379928 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9bc6d7e931a34e00b8a1c2233fa53c3f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:00.379845 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 403) took 2302 ms. Trace:
I20251029 02:52:00.380323 1174 rpcz_store.cc:276] 1029 02:51:58.076849 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:58.077497 (+ 648us) service_pool.cc:225] Handling call
1029 02:52:00.378342 (+2300845us) ts_tablet_manager.cc:1151] Deleting tablet bfec87f690a640c39c58c64e16cc67e7
1029 02:52:00.378499 (+ 157us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:00.379106 (+ 607us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2297155}
W20251029 02:52:00.381629 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet bfec87f690a640c39c58c64e16cc67e7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bfec87f690a640c39c58c64e16cc67e7 already in progress: creating tablet
I20251029 02:52:00.382562 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72ac7d5f409344319fac08797e0d26bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:00.382400 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 409) took 2260 ms. Trace:
I20251029 02:52:00.383196 1174 rpcz_store.cc:276] 1029 02:51:58.121269 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:58.121320 (+ 51us) service_pool.cc:225] Handling call
1029 02:52:00.381702 (+2260382us) ts_tablet_manager.cc:1151] Deleting tablet cc802bda2f6a4e439b605f2cafa37967
1029 02:52:00.381717 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:00.382211 (+ 494us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2259996}
W20251029 02:52:00.384297 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet cc802bda2f6a4e439b605f2cafa37967 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cc802bda2f6a4e439b605f2cafa37967 already in progress: creating tablet
I20251029 02:52:00.384666 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe9e660de57a46c7ba1d82f25e12b4b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:00.384989 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 425) took 2107 ms. Trace:
I20251029 02:52:00.385438 1174 rpcz_store.cc:276] 1029 02:51:58.277644 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:58.277706 (+ 62us) service_pool.cc:225] Handling call
1029 02:52:00.384336 (+2106630us) ts_tablet_manager.cc:1151] Deleting tablet 061f3d38c4604fa4b57e74ff6f6fc7da
1029 02:52:00.384449 (+ 113us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:00.384866 (+ 417us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2106392}
W20251029 02:52:00.386929 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 061f3d38c4604fa4b57e74ff6f6fc7da because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 061f3d38c4604fa4b57e74ff6f6fc7da already in progress: creating tablet
I20251029 02:52:00.387494 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 427) took 2105 ms. Trace:
I20251029 02:52:00.388288 1174 rpcz_store.cc:276] 1029 02:51:58.281908 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:58.281958 (+ 50us) service_pool.cc:225] Handling call
1029 02:52:00.386488 (+2104530us) ts_tablet_manager.cc:1151] Deleting tablet 30d03e1e42d34242bd354f9e9ec4f127
1029 02:52:00.386600 (+ 112us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:00.387306 (+ 706us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2104342}
W20251029 02:52:00.389818 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 30d03e1e42d34242bd354f9e9ec4f127 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 30d03e1e42d34242bd354f9e9ec4f127 already in progress: creating tablet
I20251029 02:52:00.390466 1166 ts_tablet_manager.cc:1434] T fd900eec3f0c43bf965c148ea90870a3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.317s user 0.003s sys 0.000s
I20251029 02:52:00.391202 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 428) took 2106 ms. Trace:
I20251029 02:52:00.392040 1174 rpcz_store.cc:276] 1029 02:51:58.284550 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:58.284599 (+ 49us) service_pool.cc:225] Handling call
1029 02:52:00.390056 (+2105457us) ts_tablet_manager.cc:1151] Deleting tablet 897a9a75d13f493e90526da4ca7f54d2
1029 02:52:00.390067 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:00.390944 (+ 877us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2103774}
W20251029 02:52:00.394292 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 897a9a75d13f493e90526da4ca7f54d2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 897a9a75d13f493e90526da4ca7f54d2 already in progress: creating tablet
I20251029 02:52:00.393108 1030 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:37626 (request call id 495) took 1148 ms. Trace:
I20251029 02:52:00.395488 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 431) took 2100 ms. Trace:
I20251029 02:52:00.396600 1174 rpcz_store.cc:276] 1029 02:51:58.294443 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:58.294499 (+ 56us) service_pool.cc:225] Handling call
1029 02:52:00.394409 (+2099910us) ts_tablet_manager.cc:1151] Deleting tablet aee42fd9ca894319b3759beb2bae342e
1029 02:52:00.394423 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:00.395219 (+ 796us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2098723}
W20251029 02:52:00.399785 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet aee42fd9ca894319b3759beb2bae342e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aee42fd9ca894319b3759beb2bae342e already in progress: creating tablet
W20251029 02:52:00.400483 849 leader_election.cc:343] T 6c22497760274380a832feff7c5a2fae P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Not found: Tablet not found: 6c22497760274380a832feff7c5a2fae
I20251029 02:52:00.400655 849 leader_election.cc:304] T 6c22497760274380a832feff7c5a2fae P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9
I20251029 02:52:00.401163 1682 raft_consensus.cc:2749] T 6c22497760274380a832feff7c5a2fae P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 02:52:00.402092 1693 log.cc:927] Time spent T 00000000000000000000000000000000 P e5e5222b26fe4145948620e51edf53fb: Append to log took a long time: real 0.199s user 0.000s sys 0.000s
I20251029 02:52:00.404925 1176 tablet_replica.cc:333] T 8d9722c764ac4b27a540595e9c6aafcd P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:00.404945 1037 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:37626 (request call id 505) took 1241 ms. Trace:
I20251029 02:52:00.398828 1027 tablet_service.cc:1505] Processing CreateTablet for tablet b41e114ed6e94f5895bdec9bee836d2d (DEFAULT_TABLE table=default.Table_203b46ee33744bf3bfe75190ab006c13 [id=825464f47f3a45f79eb319c3ca3e54e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:00.407711 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 438) took 2054 ms. Trace:
W20251029 02:52:00.409020 989 leader_election.cc:343] T 75236b2fa82f4f28b82efae9a2c5160d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 75236b2fa82f4f28b82efae9a2c5160d
I20251029 02:52:00.409660 1730 tablet.cc:1722] T 70e8cfe5eaa9492c85874dc3214a1e0b P c756528642a543eeacd93cef3e335a7d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 02:52:00.409327 1174 rpcz_store.cc:276] 1029 02:51:58.353225 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:58.407806 (+ 54581us) service_pool.cc:225] Handling call
1029 02:52:00.405709 (+1997903us) ts_tablet_manager.cc:1151] Deleting tablet 5ae5f33567324d90b7a5a6e0f1b5fb67
1029 02:52:00.406018 (+ 309us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:00.407375 (+ 1357us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1913018}
W20251029 02:52:00.412423 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 5ae5f33567324d90b7a5a6e0f1b5fb67 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5ae5f33567324d90b7a5a6e0f1b5fb67 already in progress: creating tablet
I20251029 02:52:00.413496 1732 tablet.cc:1722] T dd90fb9eb167478cbc3ec9363af86295 P c756528642a543eeacd93cef3e335a7d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251029 02:52:00.414741 1174 tablet_replica.cc:333] T d6e1170dbb594a5696a3899cde8da7c9 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:00.418128 1491 ts_manager.cc:194] Registered new tserver with Master: c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:52:00.423722 1033 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:37626 (request call id 509) took 1241 ms. Trace:
I20251029 02:52:00.424916 1032 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:37626 (request call id 516) took 1241 ms. Trace:
I20251029 02:52:00.426546 1174 raft_consensus.cc:2243] T d6e1170dbb594a5696a3899cde8da7c9 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:00.428710 1174 raft_consensus.cc:2272] T d6e1170dbb594a5696a3899cde8da7c9 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:00.431203 738 tablet_service.cc:1505] Processing CreateTablet for tablet 45d33b70d549488c802957bb2d5b2821 (DEFAULT_TABLE table=default.Table_360a4b04c44345e0a36d04e6d90c698f [id=6044f9e5029b4985b5540d35286f8586]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:00.436836 1172 tablet_bootstrap.cc:492] T dbc2092b63e0417997565c8fadb53ae0 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:00.440299 1174 ts_tablet_manager.cc:1916] T d6e1170dbb594a5696a3899cde8da7c9 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 02:52:00.442620 990 leader_election.cc:343] T 75236b2fa82f4f28b82efae9a2c5160d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:00.445017 746 tablet_service.cc:1552] Processing DeleteTablet for tablet 998b9cbe76514665be76c6c387dc75b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:00 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:00.444578 1730 tablet.cc:1717] T dd90fb9eb167478cbc3ec9363af86295 P c756528642a543eeacd93cef3e335a7d: Skipping requested alter to schema version 3, tablet already version 3
I20251029 02:52:00.450223 1167 tablet_bootstrap.cc:492] T f71388036478404d878a52fc20ece216 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:00.450636 1167 ts_tablet_manager.cc:1403] T f71388036478404d878a52fc20ece216 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.117s user 0.007s sys 0.002s
I20251029 02:52:00.450946 743 tablet_service.cc:1505] Processing CreateTablet for tablet 8f1d5847265b4bb0b29b5e513c35b9bb (DEFAULT_TABLE table=default.Table_877abcb0d221493c8875368e1c75382b [id=6ea46a01a86140b9ae63e33b90b03dd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:00.451710 1167 raft_consensus.cc:359] T f71388036478404d878a52fc20ece216 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:00.452059 1167 raft_consensus.cc:385] T f71388036478404d878a52fc20ece216 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:00.452189 1167 raft_consensus.cc:740] T f71388036478404d878a52fc20ece216 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:00.452754 1167 consensus_queue.cc:260] T f71388036478404d878a52fc20ece216 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:00.454315 753 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:44580 (request call id 496) took 1127 ms. Trace:
I20251029 02:52:00.454437 753 rpcz_store.cc:276] 1029 02:51:59.326500 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:00.060743 (+734243us) service_pool.cc:225] Handling call
1029 02:52:00.064569 (+ 3826us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1029 02:52:00.064597 (+ 28us) ts_tablet_manager.cc:682] Creating new metadata...
1029 02:52:00.324394 (+259797us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:00.454058 (+129664us) tablet_replica.cc:206] Creating consensus instance
1029 02:52:00.454242 (+ 184us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251029 02:52:00.455389 773 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.104.3:43159 (request call id 104) took 1012 ms. Trace:
I20251029 02:52:00.455583 773 rpcz_store.cc:276] 1029 02:51:59.442602 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:59.442719 (+ 117us) service_pool.cc:225] Handling call
1029 02:51:59.443814 (+ 1095us) raft_consensus.cc:1471] Updating replica for 1 ops
1029 02:51:59.445594 (+ 1780us) raft_consensus.cc:1534] Early marking committed up to index 0
1029 02:51:59.445599 (+ 5us) raft_consensus.cc:1539] Triggering prepare for 1 ops
1029 02:51:59.459238 (+ 13639us) alter_schema_op.cc:128] START. Timestamp: 1761706318457650
1029 02:51:59.922272 (+463034us) log.cc:844] Serialized 173 byte log entry
1029 02:52:00.263389 (+341117us) raft_consensus.cc:1645] Marking committed up to 3
1029 02:52:00.263401 (+ 12us) raft_consensus.cc:1697] Filling consensus response to leader.
1029 02:52:00.263545 (+ 144us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
1029 02:52:00.455254 (+191709us) raft_consensus.cc:1685] finished
1029 02:52:00.455258 (+ 4us) raft_consensus.cc:1691] UpdateReplicas() finished
1029 02:52:00.455326 (+ 68us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1029 02:51:59.697978 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
1029 02:51:59.698046 (+ 68us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
1029 02:51:59.698053 (+ 7us) alter_schema_op.cc:66] Acquired schema lock
1029 02:51:59.698058 (+ 5us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
Metrics: {"thread_start_us":340970,"threads_started":1,"wal-append.queue_time_us":466776,"child_traces":[["op",{"prepare.queue_time_us":238662,"prepare.run_cpu_time_us":95,"prepare.run_wall_time_us":95,"thread_start_us":411632,"threads_started":1}]]}
I20251029 02:52:00.458869 1167 ts_tablet_manager.cc:1434] T f71388036478404d878a52fc20ece216 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20251029 02:52:00.461872 1167 tablet_bootstrap.cc:492] T 012ce243fd07466ab5301d1c9ce4b413 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:00.463332 1172 ts_tablet_manager.cc:1403] T dbc2092b63e0417997565c8fadb53ae0 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.302s user 0.008s sys 0.003s
I20251029 02:52:00.464826 746 tablet_service.cc:1552] Processing DeleteTablet for tablet 4cede3b183494845af91dca39a2096c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:00 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:00.465008 739 tablet_service.cc:1552] Processing DeleteTablet for tablet 97b2ecae572a4b7ea1b9595074556708 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:00 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:00.466364 744 tablet_service.cc:1552] Processing DeleteTablet for tablet a7685b0bed174ed0b25f99ef8a46fca3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:00 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:00.469775 1175 ts_tablet_manager.cc:1929] T 9be9add0274d41b688edd6292a253a06 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251029 02:52:00.470024 1175 log.cc:1199] T 9be9add0274d41b688edd6292a253a06 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/9be9add0274d41b688edd6292a253a06
I20251029 02:52:00.470619 1175 ts_tablet_manager.cc:1950] T 9be9add0274d41b688edd6292a253a06 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:00.471539 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 354) took 3302 ms. Trace:
I20251029 02:52:00.471732 1175 rpcz_store.cc:276] 1029 02:51:57.168896 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:57.170869 (+ 1973us) service_pool.cc:225] Handling call
1029 02:52:00.359258 (+3188389us) ts_tablet_manager.cc:1151] Deleting tablet 9be9add0274d41b688edd6292a253a06
1029 02:52:00.359267 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:00.376635 (+ 17368us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:00.463645 (+ 87010us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:00.471497 (+ 7852us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3187456}
I20251029 02:52:00.472463 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 355) took 3301 ms. Trace:
I20251029 02:52:00.472669 1175 rpcz_store.cc:276] 1029 02:51:57.171383 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:57.171445 (+ 62us) service_pool.cc:225] Handling call
1029 02:52:00.472254 (+3300809us) ts_tablet_manager.cc:1151] Deleting tablet e1fa4980f2624321b617b84c68b90113
1029 02:52:00.472266 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:00.472419 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3300322}
I20251029 02:52:00.473456 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 356) took 3272 ms. Trace:
I20251029 02:52:00.473735 1175 rpcz_store.cc:276] 1029 02:51:57.200966 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:57.203468 (+ 2502us) service_pool.cc:225] Handling call
1029 02:52:00.473257 (+3269789us) ts_tablet_manager.cc:1151] Deleting tablet 7a1866afc7564217ad777e7ce53786c5
1029 02:52:00.473267 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:00.473401 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3269551}
I20251029 02:52:00.474902 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 9be9add0274d41b688edd6292a253a06 (table default.Table_c7daf9eda79e470186f5774de2b824d3 [id=f45ce142f37f4febbceb4e568e35e0f2]) successfully deleted
I20251029 02:52:00.477785 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45d33b70d549488c802957bb2d5b2821. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:00.480952 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 7a1866afc7564217ad777e7ce53786c5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7a1866afc7564217ad777e7ce53786c5 already in progress: creating tablet
I20251029 02:52:00.481570 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f1d5847265b4bb0b29b5e513c35b9bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:00.448935 1491 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "b495a49349324f7d81908f2da700f0d9" instance_seqno: 1761706308660093) as {username='slave'} at 127.0.104.1:40633; Asking this server to re-register.
I20251029 02:52:00.484915 1167 tablet_bootstrap.cc:654] T 012ce243fd07466ab5301d1c9ce4b413 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:00.488488 740 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:44580 (request call id 507) took 1018 ms. Trace:
I20251029 02:52:00.486419 1174 ts_tablet_manager.cc:1929] T d6e1170dbb594a5696a3899cde8da7c9 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:00.494558 822 heartbeater.cc:461] Registering TS with master...
I20251029 02:52:00.494781 1174 log.cc:1199] T d6e1170dbb594a5696a3899cde8da7c9 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/d6e1170dbb594a5696a3899cde8da7c9
I20251029 02:52:00.495373 1174 ts_tablet_manager.cc:1950] T d6e1170dbb594a5696a3899cde8da7c9 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:00.495707 740 rpcz_store.cc:276] 1029 02:51:59.459161 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:00.143928 (+684767us) service_pool.cc:225] Handling call
1029 02:52:00.165738 (+ 21810us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1029 02:52:00.165785 (+ 47us) ts_tablet_manager.cc:682] Creating new metadata...
1029 02:52:00.397968 (+232183us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:00.477486 (+ 79518us) tablet_replica.cc:206] Creating consensus instance
1029 02:52:00.477695 (+ 209us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251029 02:52:00.496433 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 455) took 1916 ms. Trace:
I20251029 02:52:00.498579 1174 rpcz_store.cc:276] 1029 02:51:58.580210 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:58.680766 (+100556us) service_pool.cc:225] Handling call
1029 02:52:00.413563 (+1732797us) ts_tablet_manager.cc:1151] Deleting tablet d6e1170dbb594a5696a3899cde8da7c9
1029 02:52:00.413878 (+ 315us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:00.465111 (+ 51233us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:00.486284 (+ 21173us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:00.496369 (+ 10085us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1732134}
I20251029 02:52:00.501459 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet d6e1170dbb594a5696a3899cde8da7c9 (table default.Table_4d2c99159e924a06858977465efa38a6 [id=ad583c111f524bb0a40dcf1eb4d5c527]) successfully deleted
I20251029 02:52:00.502105 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 456) took 1891 ms. Trace:
I20251029 02:52:00.505438 1174 rpcz_store.cc:276] 1029 02:51:58.610595 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:58.681134 (+ 70539us) service_pool.cc:225] Handling call
1029 02:52:00.501936 (+1820802us) ts_tablet_manager.cc:1151] Deleting tablet c8389c8bfead4ff7a23f363c9c6770bf
1029 02:52:00.501945 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:00.502053 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1799008}
W20251029 02:52:00.505998 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet c8389c8bfead4ff7a23f363c9c6770bf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c8389c8bfead4ff7a23f363c9c6770bf already in progress: creating tablet
I20251029 02:52:00.507565 1167 tablet_bootstrap.cc:492] T 012ce243fd07466ab5301d1c9ce4b413 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:00.507526 517 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet dd90fb9eb167478cbc3ec9363af86295 (table default.Table_8b6217e43e8d467693c104ef84e78911 [id=052abf3dcc904bdcb869d5b885723dab]). Expected version 4 got 0
I20251029 02:52:00.507784 1167 ts_tablet_manager.cc:1403] T 012ce243fd07466ab5301d1c9ce4b413 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.047s user 0.002s sys 0.002s
I20251029 02:52:00.507972 1172 raft_consensus.cc:359] T dbc2092b63e0417997565c8fadb53ae0 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:00.508198 1172 raft_consensus.cc:385] T dbc2092b63e0417997565c8fadb53ae0 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:00.508303 1172 raft_consensus.cc:740] T dbc2092b63e0417997565c8fadb53ae0 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:00.508571 1172 consensus_queue.cc:260] T dbc2092b63e0417997565c8fadb53ae0 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:00.508829 895 tablet_service.cc:1552] Processing DeleteTablet for tablet 6c22497760274380a832feff7c5a2fae with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:53 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:00.509053 1167 raft_consensus.cc:359] T 012ce243fd07466ab5301d1c9ce4b413 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:00.509289 1167 raft_consensus.cc:385] T 012ce243fd07466ab5301d1c9ce4b413 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:00.509613 895 tablet_service.cc:1552] Processing DeleteTablet for tablet 8d9722c764ac4b27a540595e9c6aafcd with delete_type TABLET_DATA_DELETED (replaced by tablet 51effc9324034ae79e2b25cb57a996c1) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:00.509443 1167 raft_consensus.cc:740] T 012ce243fd07466ab5301d1c9ce4b413 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:00.509933 1172 ts_tablet_manager.cc:1434] T dbc2092b63e0417997565c8fadb53ae0 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.046s user 0.002s sys 0.000s
I20251029 02:52:00.510110 1167 consensus_queue.cc:260] T 012ce243fd07466ab5301d1c9ce4b413 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:00.510557 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_e7b916cfb7f64f5d9e1ed5b9337f4206"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:00.511543 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e7b916cfb7f64f5d9e1ed5b9337f4206 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:00.512930 1167 ts_tablet_manager.cc:1434] T 012ce243fd07466ab5301d1c9ce4b413 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251029 02:52:00.514704 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 457) took 1870 ms. Trace:
I20251029 02:52:00.514891 1174 rpcz_store.cc:276] 1029 02:51:58.638757 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:58.702981 (+ 64224us) service_pool.cc:225] Handling call
1029 02:52:00.506336 (+1803355us) ts_tablet_manager.cc:1151] Deleting tablet 86fc22bb84794149b598b97df41eb881
1029 02:52:00.506347 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:00.506443 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1802423}
I20251029 02:52:00.515666 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 458) took 1866 ms. Trace:
I20251029 02:52:00.519670 1174 rpcz_store.cc:276] 1029 02:51:58.648820 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:58.703942 (+ 55122us) service_pool.cc:225] Handling call
1029 02:52:00.515470 (+1811528us) ts_tablet_manager.cc:1151] Deleting tablet f18b0ca5a6a44175a912642e9eea1778
1029 02:52:00.515481 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:00.515606 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1787734}
I20251029 02:52:00.521683 1491 ts_manager.cc:194] Registered new tserver with Master: b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:00.515268 1167 tablet_bootstrap.cc:492] T 14698af6348f41e384f10569392d9b23 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:00.531096 751 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:44580 (request call id 486) took 1208 ms. Trace:
I20251029 02:52:00.531278 751 rpcz_store.cc:276] 1029 02:51:59.314806 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:59.661667 (+346861us) service_pool.cc:225] Handling call
1029 02:51:59.710359 (+ 48692us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1029 02:51:59.710383 (+ 24us) ts_tablet_manager.cc:682] Creating new metadata...
1029 02:52:00.303444 (+593061us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:00.510929 (+207485us) tablet_replica.cc:206] Creating consensus instance
1029 02:52:00.523657 (+ 12728us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251029 02:52:00.532296 1172 tablet_bootstrap.cc:492] T 1f5b40c0403d4a49a4bfde222d8c947a P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:00.533289 1167 tablet_bootstrap.cc:654] T 14698af6348f41e384f10569392d9b23 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
W20251029 02:52:00.511933 525 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6142018e44af4d32818adefa2ee43d02 consensus state has no leader
I20251029 02:52:00.533874 525 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_73bc152b9f224b07ae2524e0e18f7df0" } modify_external_catalogs: true
I20251029 02:52:00.534184 525 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_73bc152b9f224b07ae2524e0e18f7df0" } modify_external_catalogs: true
W20251029 02:52:00.511097 524 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b3de6670357c4ae59eb77469b2fb434c consensus state has no leader
I20251029 02:52:00.536291 745 tablet_service.cc:1552] Processing DeleteTablet for tablet dbc2092b63e0417997565c8fadb53ae0 with delete_type TABLET_DATA_DELETED (replaced by tablet 290d9d46914b4f328c7f2d9afe901889) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:00.534381 1729 tablet.cc:1722] T 70e8cfe5eaa9492c85874dc3214a1e0b P b495a49349324f7d81908f2da700f0d9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 02:52:00.536532 524 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_5a915593503d490fa12b10fccd76ac3e" } modify_external_catalogs: true
I20251029 02:52:00.536706 524 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_5a915593503d490fa12b10fccd76ac3e" } modify_external_catalogs: true
I20251029 02:52:00.514920 520 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet dd90fb9eb167478cbc3ec9363af86295 (table default.Table_8b6217e43e8d467693c104ef84e78911 [id=052abf3dcc904bdcb869d5b885723dab]). Expected version 4 got 3
W20251029 02:52:00.517089 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 86fc22bb84794149b598b97df41eb881 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 86fc22bb84794149b598b97df41eb881 already in progress: creating tablet
W20251029 02:52:00.537819 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet f18b0ca5a6a44175a912642e9eea1778 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f18b0ca5a6a44175a912642e9eea1778 already in progress: creating tablet
I20251029 02:52:00.538174 520 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet aacb6f4b9284457bb6607e3a85d5c809 (table default.Table_f35ae25e36f64d2886752bb3b1d66710 [id=9f95160639ad40e995a6c89e297a0fad]). Expected version 1 got 0
I20251029 02:52:00.538404 520 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet 70e8cfe5eaa9492c85874dc3214a1e0b (table default.Table_9bf4e25d441f4d4dbd6651386eec512e [id=82df85af001340efb570040e047fa136]). Expected version 2 got 1
I20251029 02:52:00.541080 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_bc03c0e4711a4763a6f67e7c0687215e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:00.533710 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 460) took 1862 ms. Trace:
W20251029 02:52:00.541767 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bc03c0e4711a4763a6f67e7c0687215e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:00.540782 895 tablet_service.cc:1552] Processing DeleteTablet for tablet f71388036478404d878a52fc20ece216 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:52 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:00.542101 1174 rpcz_store.cc:276] 1029 02:51:58.671676 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:58.727759 (+ 56083us) service_pool.cc:225] Handling call
1029 02:52:00.533522 (+1805763us) ts_tablet_manager.cc:1151] Deleting tablet eca016aed05e4ce09e1a2bf96e2286f5
1029 02:52:00.533534 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:00.533653 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1805540}
W20251029 02:52:00.542646 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet eca016aed05e4ce09e1a2bf96e2286f5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eca016aed05e4ce09e1a2bf96e2286f5 already in progress: creating tablet
I20251029 02:52:00.543116 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 459) took 1892 ms. Trace:
I20251029 02:52:00.543251 1174 rpcz_store.cc:276] 1029 02:51:58.651034 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:58.727318 (+ 76284us) service_pool.cc:225] Handling call
1029 02:52:00.542942 (+1815624us) ts_tablet_manager.cc:1151] Deleting tablet 358bdfbcb4ee440c899fb6eedef68f44
1029 02:52:00.542954 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:00.543063 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1809063}
W20251029 02:52:00.543890 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 358bdfbcb4ee440c899fb6eedef68f44 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 358bdfbcb4ee440c899fb6eedef68f44 already in progress: creating tablet
I20251029 02:52:00.544349 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 471) took 1660 ms. Trace:
I20251029 02:52:00.544482 1174 rpcz_store.cc:276] 1029 02:51:58.884217 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:58.884839 (+ 622us) service_pool.cc:225] Handling call
1029 02:52:00.544162 (+1659323us) ts_tablet_manager.cc:1151] Deleting tablet 023a4094b424464c8f49d6eb9cc15da4
1029 02:52:00.544171 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:00.544297 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1579549}
W20251029 02:52:00.544939 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 023a4094b424464c8f49d6eb9cc15da4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 023a4094b424464c8f49d6eb9cc15da4 already in progress: creating tablet
I20251029 02:52:00.547209 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 472) took 1662 ms. Trace:
I20251029 02:52:00.547364 1174 rpcz_store.cc:276] 1029 02:51:58.884562 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:58.932542 (+ 47980us) service_pool.cc:225] Handling call
1029 02:52:00.545254 (+1612712us) ts_tablet_manager.cc:1151] Deleting tablet 1878d7eef14a4578bc929da68872cee4
1029 02:52:00.545265 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:00.547161 (+ 1896us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1576226}
W20251029 02:52:00.547853 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 1878d7eef14a4578bc929da68872cee4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1878d7eef14a4578bc929da68872cee4 already in progress: creating tablet
I20251029 02:52:00.525244 522 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_73e59d41837947c4a4b59f6639ad6f06" } new_table_name: "default.Table_443f33587b8f428484b99e07355d69ca" modify_external_catalogs: true
I20251029 02:52:00.548290 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 473) took 1663 ms. Trace:
I20251029 02:52:00.548503 1174 rpcz_store.cc:276] 1029 02:51:58.884842 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:58.938952 (+ 54110us) service_pool.cc:225] Handling call
1029 02:52:00.548137 (+1609185us) ts_tablet_manager.cc:1151] Deleting tablet 7816556521144c928d7a02694e22e79f
1029 02:52:00.548148 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:00.548241 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1576842}
W20251029 02:52:00.548985 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 7816556521144c928d7a02694e22e79f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7816556521144c928d7a02694e22e79f already in progress: creating tablet
W20251029 02:52:00.517066 521 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet da2a21c3236a4bd99872b89b12a36508 consensus state has no leader
I20251029 02:52:00.549906 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 475) took 1603 ms. Trace:
I20251029 02:52:00.549942 1175 tablet_replica.cc:333] T 69597d48631d43b6a5b92abbf60d5ed3 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:00.550019 1174 rpcz_store.cc:276] 1029 02:51:58.945494 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:58.950129 (+ 4635us) service_pool.cc:225] Handling call
1029 02:52:00.549241 (+1599112us) ts_tablet_manager.cc:1151] Deleting tablet c9bd3f2deb5d4af09b7918269a98509f
1029 02:52:00.549250 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:00.549332 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1577936}
W20251029 02:52:00.551028 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet c9bd3f2deb5d4af09b7918269a98509f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c9bd3f2deb5d4af09b7918269a98509f already in progress: creating tablet
I20251029 02:52:00.551470 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 474) took 1606 ms. Trace:
I20251029 02:52:00.554948 521 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_8fee569eca38413dbac56707725c6a46" } modify_external_catalogs: true
I20251029 02:52:00.551919 895 tablet_service.cc:1552] Processing DeleteTablet for tablet 012ce243fd07466ab5301d1c9ce4b413 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:55 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:00.555506 1174 rpcz_store.cc:276] 1029 02:51:58.945383 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:58.950601 (+ 5218us) service_pool.cc:225] Handling call
1029 02:52:00.551312 (+1600711us) ts_tablet_manager.cc:1151] Deleting tablet 94bdbb9394f7490e99916bd818986208
1029 02:52:00.551325 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:00.551423 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1579961}
I20251029 02:52:00.555574 521 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_8fee569eca38413dbac56707725c6a46" } modify_external_catalogs: true
W20251029 02:52:00.556025 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 94bdbb9394f7490e99916bd818986208 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 94bdbb9394f7490e99916bd818986208 already in progress: creating tablet
I20251029 02:52:00.556393 1174 tablet_replica.cc:333] T 6f93812578a940a3b7378015e2a46f95 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:00.556613 1174 raft_consensus.cc:2243] T 6f93812578a940a3b7378015e2a46f95 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:00.557127 1174 raft_consensus.cc:2272] T 6f93812578a940a3b7378015e2a46f95 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:00.558053 1174 ts_tablet_manager.cc:1916] T 6f93812578a940a3b7378015e2a46f95 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:00.560451 1041 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:37626 (request call id 512) took 1389 ms. Trace:
I20251029 02:52:00.560868 1036 rpcz_store.cc:276] 1029 02:51:59.108633 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:59.683948 (+575315us) service_pool.cc:225] Handling call
1029 02:51:59.684352 (+ 404us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1029 02:51:59.684373 (+ 21us) ts_tablet_manager.cc:682] Creating new metadata...
1029 02:52:00.032744 (+348371us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:00.173004 (+140260us) tablet_replica.cc:206] Creating consensus instance
1029 02:52:00.321174 (+148170us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251029 02:52:00.561477 1036 tablet_service.cc:1505] Processing CreateTablet for tablet a04bc4d9605d4c049a151d985035bb63 (DEFAULT_TABLE table=default.Table_62644549a49a4f039492b17deac6113d [id=bb15ae87ebcd4a39a99982967fd215b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:00.562036 1057 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8d9722c764ac4b27a540595e9c6aafcd" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:00.562173 1166 tablet_bootstrap.cc:492] T 9e07291d6b5645feb4d5890f58e1ef74 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:00.562081 1037 rpcz_store.cc:276] 1029 02:51:59.163540 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:59.779400 (+615860us) service_pool.cc:225] Handling call
1029 02:51:59.804482 (+ 25082us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1029 02:51:59.804520 (+ 38us) ts_tablet_manager.cc:682] Creating new metadata...
1029 02:51:59.862023 (+ 57503us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:00.217238 (+355215us) tablet_replica.cc:206] Creating consensus instance
1029 02:52:00.404898 (+187660us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251029 02:52:00.562311 1029 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:37626 (request call id 513) took 1234 ms. Trace:
I20251029 02:52:00.562395 1029 rpcz_store.cc:276] 1029 02:51:59.170907 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:59.932661 (+761754us) service_pool.cc:225] Handling call
1029 02:51:59.946170 (+ 13509us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1029 02:51:59.946200 (+ 30us) ts_tablet_manager.cc:682] Creating new metadata...
1029 02:52:00.043630 (+ 97430us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:00.324309 (+280679us) tablet_replica.cc:206] Creating consensus instance
1029 02:52:00.405566 (+ 81257us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251029 02:52:00.562475 1030 rpcz_store.cc:276] 1029 02:51:59.108354 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:59.650023 (+541669us) service_pool.cc:225] Handling call
1029 02:51:59.696249 (+ 46226us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1029 02:51:59.696271 (+ 22us) ts_tablet_manager.cc:682] Creating new metadata...
1029 02:52:00.028351 (+332080us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:00.193618 (+165267us) tablet_replica.cc:206] Creating consensus instance
1029 02:52:00.256994 (+ 63376us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251029 02:52:00.562675 1037 tablet_service.cc:1505] Processing CreateTablet for tablet 9c6f4dc4b0c74ddd80d9dc762619b1ec (DEFAULT_TABLE table=default.Table_51e08a9cce7d4d9f9b358a693206462f [id=e1f53eaba1d64ef495c67d40be796e49]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:00.562839 1029 tablet_service.cc:1505] Processing CreateTablet for tablet c8b6afc8c77c4a7fbe1956cb30bba656 (DEFAULT_TABLE table=default.Table_db8d41696cb54486aa9fa06d4396e499 [id=f87557451c8c466c8f6ea573155aa684]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:00.562956 1030 tablet_service.cc:1505] Processing CreateTablet for tablet a7685b0bed174ed0b25f99ef8a46fca3 (DEFAULT_TABLE table=default.Table_beeb4b56920e4aff85dfbb9758fa414c [id=213b1abd06cf4e22b05bfb35460bdd3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:00.563131 989 leader_election.cc:304] T d2990e9f626c40618c118266bc595684 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters:
W20251029 02:52:00.563378 989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:38481, user_credentials={real_user=slave}} blocked reactor thread for 153547us
I20251029 02:52:00.563434 1167 tablet_bootstrap.cc:492] T 14698af6348f41e384f10569392d9b23 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:00.563467 1722 raft_consensus.cc:2804] T d2990e9f626c40618c118266bc595684 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:00.563601 1167 ts_tablet_manager.cc:1403] T 14698af6348f41e384f10569392d9b23 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.048s user 0.002s sys 0.002s
I20251029 02:52:00.563608 1722 raft_consensus.cc:697] T d2990e9f626c40618c118266bc595684 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Becoming Leader. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Running, Role: LEADER
I20251029 02:52:00.563720 1026 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:37626 (request call id 491) took 1457 ms. Trace:
I20251029 02:52:00.563819 1026 rpcz_store.cc:276] 1029 02:51:59.106315 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:59.577523 (+471208us) service_pool.cc:225] Handling call
1029 02:51:59.670072 (+ 92549us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1029 02:51:59.670110 (+ 38us) ts_tablet_manager.cc:682] Creating new metadata...
1029 02:52:00.084394 (+414284us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:00.459983 (+375589us) tablet_replica.cc:206] Creating consensus instance
1029 02:52:00.563652 (+103669us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251029 02:52:00.563932 1722 consensus_queue.cc:237] T d2990e9f626c40618c118266bc595684 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:00.564224 1026 tablet_service.cc:1505] Processing CreateTablet for tablet 699ee74f39b342639ab84d0d65e53a70 (DEFAULT_TABLE table=default.Table_1ff672db4f064544b523c0ae9667d967 [id=60cf9a35c3c848d8a4e7cd7a8128b12f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:00.564694 1167 raft_consensus.cc:359] T 14698af6348f41e384f10569392d9b23 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:00.564936 1167 raft_consensus.cc:385] T 14698af6348f41e384f10569392d9b23 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:00.565021 1167 raft_consensus.cc:740] T 14698af6348f41e384f10569392d9b23 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:00.562114 1176 raft_consensus.cc:2243] T 8d9722c764ac4b27a540595e9c6aafcd P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:00.565212 1025 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:37626 (request call id 494) took 1456 ms. Trace:
I20251029 02:52:00.565245 1176 raft_consensus.cc:2272] T 8d9722c764ac4b27a540595e9c6aafcd P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:00.565312 1025 rpcz_store.cc:276] 1029 02:51:59.108244 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:59.646554 (+538310us) service_pool.cc:225] Handling call
1029 02:51:59.864437 (+217883us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1029 02:51:59.900318 (+ 35881us) ts_tablet_manager.cc:682] Creating new metadata...
1029 02:52:00.194130 (+293812us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:00.461787 (+267657us) tablet_replica.cc:206] Creating consensus instance
1029 02:52:00.564584 (+102797us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251029 02:52:00.565318 1167 consensus_queue.cc:260] T 14698af6348f41e384f10569392d9b23 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:00.565730 1025 tablet_service.cc:1505] Processing CreateTablet for tablet ce98420e61904af78d8fa03724c74489 (DEFAULT_TABLE table=default.Table_3975b6f24b5b4eb88958a46683a75c43 [id=bde03d34cbf546d3ad07852b4339dd0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:00.565809 1176 ts_tablet_manager.cc:1916] T 8d9722c764ac4b27a540595e9c6aafcd P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 02:52:00.566676 849 leader_election.cc:343] T 8d9722c764ac4b27a540595e9c6aafcd P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:00.566841 849 leader_election.cc:304] T 8d9722c764ac4b27a540595e9c6aafcd P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9
I20251029 02:52:00.566816 1035 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:37626 (request call id 497) took 1458 ms. Trace:
I20251029 02:52:00.567062 1035 rpcz_store.cc:276] 1029 02:51:59.108555 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:59.679045 (+570490us) service_pool.cc:225] Handling call
1029 02:51:59.680435 (+ 1390us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1029 02:51:59.680514 (+ 79us) ts_tablet_manager.cc:682] Creating new metadata...
1029 02:52:00.456096 (+775582us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:00.462541 (+ 6445us) tablet_replica.cc:206] Creating consensus instance
1029 02:52:00.566787 (+104246us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251029 02:52:00.567410 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet fcb3db8399aa4cf188b07dda220f7f5b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:59 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:00.567623 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet 845a69a8696e469a8548787a7645d41a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:59 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:00.567793 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet 38f83ebcb2a440bc8e6a8d36607e6666 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:59 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:00.567981 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet 8df97119d0654d7d92bb5bd17c5d8562 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:59 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:00.568125 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet f65fe9a5137b48669afef51ad7dd76ae with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:59 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:00.568624 1682 raft_consensus.cc:2749] T 8d9722c764ac4b27a540595e9c6aafcd P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:00.568861 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b41e114ed6e94f5895bdec9bee836d2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:00.569155 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c6f4dc4b0c74ddd80d9dc762619b1ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:00.569490 1167 ts_tablet_manager.cc:1434] T 14698af6348f41e384f10569392d9b23 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251029 02:52:00.570150 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 699ee74f39b342639ab84d0d65e53a70. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:00.571048 1166 tablet_bootstrap.cc:654] T 9e07291d6b5645feb4d5890f58e1ef74 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:00.571532 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8b6afc8c77c4a7fbe1956cb30bba656. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:00.568454 1035 tablet_service.cc:1505] Processing CreateTablet for tablet a3d1c98fc694486991af6de1d9c45751 (DEFAULT_TABLE table=default.Table_556d748de1214c5592a0f74d11d7a7d9 [id=7252bdb49e2b4bfb9a031ede4d2e454d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:00.572889 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_be523b14f1164046a23c062a7907de02"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:00.573799 1167 tablet_bootstrap.cc:492] T d2990e9f626c40618c118266bc595684 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:00.575794 1167 tablet_bootstrap.cc:654] T d2990e9f626c40618c118266bc595684 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:00.523962 523 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_3cca74f150f549b7aef4a213cafe99e5" } new_table_name: "default.Table_84e9fe6549cc42ed94c0f4c0b489f726" modify_external_catalogs: true
I20251029 02:52:00.581212 1028 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:37626 (request call id 514) took 1401 ms. Trace:
I20251029 02:52:00.581353 1028 rpcz_store.cc:276] 1029 02:51:59.170988 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:59.933311 (+762323us) service_pool.cc:225] Handling call
1029 02:51:59.965485 (+ 32174us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1029 02:51:59.965519 (+ 34us) ts_tablet_manager.cc:682] Creating new metadata...
1029 02:52:00.178022 (+212503us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:00.466194 (+288172us) tablet_replica.cc:206] Creating consensus instance
1029 02:52:00.571954 (+105760us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251029 02:52:00.583302 748 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:44580 (request call id 508) took 1101 ms. Trace:
I20251029 02:52:00.583935 1023 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:37626 (request call id 493) took 1465 ms. Trace:
I20251029 02:52:00.584096 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce98420e61904af78d8fa03724c74489. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:00.584252 1039 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:37626 (request call id 517) took 1401 ms. Trace:
I20251029 02:52:00.584343 1031 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:37626 (request call id 499) took 1463 ms. Trace:
I20251029 02:52:00.586699 1032 rpcz_store.cc:276] 1029 02:51:59.171167 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:59.972685 (+801518us) service_pool.cc:225] Handling call
1029 02:51:59.973248 (+ 563us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1029 02:51:59.973287 (+ 39us) ts_tablet_manager.cc:682] Creating new metadata...
1029 02:52:00.040664 (+ 67377us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:00.392985 (+352321us) tablet_replica.cc:206] Creating consensus instance
1029 02:52:00.413048 (+ 20063us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251029 02:52:00.587893 1040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:37626 (request call id 518) took 1404 ms. Trace:
I20251029 02:52:00.594756 1028 tablet_service.cc:1505] Processing CreateTablet for tablet e7b7de85c13b4ddc9c995ed2a2ccd921 (DEFAULT_TABLE table=default.Table_c4ce908a6d5c40e1b825533a63689d0a [id=146ba242a24c423198155dec5cd58e9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:00.595839 1174 ts_tablet_manager.cc:1929] T 6f93812578a940a3b7378015e2a46f95 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:00.596400 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3d1c98fc694486991af6de1d9c45751. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:00.596906 1032 tablet_service.cc:1505] Processing CreateTablet for tablet 8cae9d4ddb0744298b82f4d9cee061b3 (DEFAULT_TABLE table=default.Table_8ecb63fece7348379a7d341b1c4b5891 [id=9ee86407c70046c18eeea55a7b42f052]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:00.597236 1174 log.cc:1199] T 6f93812578a940a3b7378015e2a46f95 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/6f93812578a940a3b7378015e2a46f95
I20251029 02:52:00.597811 1174 ts_tablet_manager.cc:1950] T 6f93812578a940a3b7378015e2a46f95 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:00.602360 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 484) took 1588 ms. Trace:
I20251029 02:52:00.602661 1174 rpcz_store.cc:276] 1029 02:51:59.013532 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:59.013667 (+ 135us) service_pool.cc:225] Handling call
1029 02:52:00.556307 (+1542640us) ts_tablet_manager.cc:1151] Deleting tablet 6f93812578a940a3b7378015e2a46f95
1029 02:52:00.556317 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:00.591070 (+ 34753us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:00.595687 (+ 4617us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:00.602317 (+ 6630us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1542410}
I20251029 02:52:00.606735 895 tablet_service.cc:1552] Processing DeleteTablet for tablet 14698af6348f41e384f10569392d9b23 with delete_type TABLET_DATA_DELETED (replaced by tablet 7309d3190cfa415281c5253a634fa7f3) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:00.607161 750 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:44580 (request call id 505) took 1114 ms. Trace:
I20251029 02:52:00.607334 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 6f93812578a940a3b7378015e2a46f95 (table default.Table_3cca74f150f549b7aef4a213cafe99e5 [id=feb1f56ae44249c6adce65637017bb43]) successfully deleted
I20251029 02:52:00.607339 750 rpcz_store.cc:276] 1029 02:51:59.443763 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:00.140652 (+696889us) service_pool.cc:225] Handling call
1029 02:52:00.141019 (+ 367us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1029 02:52:00.141036 (+ 17us) ts_tablet_manager.cc:682] Creating new metadata...
1029 02:52:00.516027 (+374991us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:00.558553 (+ 42526us) tablet_replica.cc:206] Creating consensus instance
1029 02:52:00.558681 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251029 02:52:00.607786 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7b7de85c13b4ddc9c995ed2a2ccd921. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:00.607829 748 rpcz_store.cc:276] 1029 02:51:59.459338 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:00.153374 (+694036us) service_pool.cc:225] Handling call
1029 02:52:00.208253 (+ 54879us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1029 02:52:00.208290 (+ 37us) ts_tablet_manager.cc:682] Creating new metadata...
1029 02:52:00.465453 (+257163us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:00.560728 (+ 95275us) tablet_replica.cc:206] Creating consensus instance
1029 02:52:00.560860 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251029 02:52:00.608057 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 487) took 1444 ms. Trace:
I20251029 02:52:00.608223 1174 rpcz_store.cc:276] 1029 02:51:59.164034 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:59.164324 (+ 290us) service_pool.cc:225] Handling call
1029 02:52:00.607748 (+1443424us) ts_tablet_manager.cc:1151] Deleting tablet 24f07b1efac64b099b0b5ed467463749
1029 02:52:00.607772 (+ 24us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:00.607942 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1432461}
W20251029 02:52:00.608747 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 24f07b1efac64b099b0b5ed467463749 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 24f07b1efac64b099b0b5ed467463749 already in progress: creating tablet
I20251029 02:52:00.609170 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 493) took 1434 ms. Trace:
I20251029 02:52:00.609318 1174 rpcz_store.cc:276] 1029 02:51:59.174980 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:59.175378 (+ 398us) service_pool.cc:225] Handling call
1029 02:52:00.609046 (+1433668us) ts_tablet_manager.cc:1151] Deleting tablet 5842e9bba4fa4295ba21e90d645a56df
1029 02:52:00.609056 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:00.609140 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1432911}
W20251029 02:52:00.609843 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 5842e9bba4fa4295ba21e90d645a56df because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5842e9bba4fa4295ba21e90d645a56df already in progress: creating tablet
I20251029 02:52:00.608875 1041 rpcz_store.cc:276] 1029 02:51:59.170818 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:59.914039 (+743221us) service_pool.cc:225] Handling call
1029 02:51:59.933609 (+ 19570us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1029 02:51:59.933641 (+ 32us) ts_tablet_manager.cc:682] Creating new metadata...
1029 02:52:00.071964 (+138323us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:00.431075 (+359111us) tablet_replica.cc:206] Creating consensus instance
1029 02:52:00.560413 (+129338us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251029 02:52:00.610316 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 496) took 1434 ms. Trace:
I20251029 02:52:00.610460 1174 rpcz_store.cc:276] 1029 02:51:59.176189 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:59.176302 (+ 113us) service_pool.cc:225] Handling call
1029 02:52:00.610106 (+1433804us) ts_tablet_manager.cc:1151] Deleting tablet 58dde9397b434b089ade9968f369006a
1029 02:52:00.610116 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:00.610284 (+ 168us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1362181}
W20251029 02:52:00.611071 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 58dde9397b434b089ade9968f369006a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 58dde9397b434b089ade9968f369006a already in progress: creating tablet
W20251029 02:52:00.575644 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_be523b14f1164046a23c062a7907de02 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:00.612008 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 494) took 1436 ms. Trace:
I20251029 02:52:00.612171 1174 rpcz_store.cc:276] 1029 02:51:59.175091 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:59.175355 (+ 264us) service_pool.cc:225] Handling call
1029 02:52:00.611377 (+1436022us) ts_tablet_manager.cc:1151] Deleting tablet 40e844fd7ad74680bcbe7bef936c5b9d
1029 02:52:00.611845 (+ 468us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:00.611964 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1363373}
W20251029 02:52:00.612648 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 40e844fd7ad74680bcbe7bef936c5b9d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 40e844fd7ad74680bcbe7bef936c5b9d already in progress: creating tablet
I20251029 02:52:00.613085 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 495) took 1437 ms. Trace:
I20251029 02:52:00.613113 1041 tablet_service.cc:1505] Processing CreateTablet for tablet ecf3b900206b4c86a7adbc83a769d90e (DEFAULT_TABLE table=default.Table_9cd59d4e874c4239b50b475aa8def07c [id=1d130644f20d4ea5bcbe83e01d970ce7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:00.613238 1174 rpcz_store.cc:276] 1029 02:51:59.176081 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:59.176222 (+ 141us) service_pool.cc:225] Handling call
1029 02:52:00.612941 (+1436719us) ts_tablet_manager.cc:1151] Deleting tablet 28f7ceb91a34450cad4b5b45f5285b41
1029 02:52:00.612958 (+ 17us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:00.613037 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":896,"tablet-delete.queue_time_us":1358304}
W20251029 02:52:00.613790 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 28f7ceb91a34450cad4b5b45f5285b41 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 28f7ceb91a34450cad4b5b45f5285b41 already in progress: creating tablet
I20251029 02:52:00.614192 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 507) took 1211 ms. Trace:
I20251029 02:52:00.614360 1174 rpcz_store.cc:276] 1029 02:51:59.402834 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:59.402932 (+ 98us) service_pool.cc:225] Handling call
1029 02:52:00.614056 (+1211124us) ts_tablet_manager.cc:1151] Deleting tablet 8df97119d0654d7d92bb5bd17c5d8562
1029 02:52:00.614066 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:00.614157 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1209579}
W20251029 02:52:00.614915 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 8df97119d0654d7d92bb5bd17c5d8562 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8df97119d0654d7d92bb5bd17c5d8562 already in progress: creating tablet
I20251029 02:52:00.615337 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 505) took 1212 ms. Trace:
I20251029 02:52:00.618557 741 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:44580 (request call id 509) took 1141 ms. Trace:
I20251029 02:52:00.619450 1175 raft_consensus.cc:2243] T 69597d48631d43b6a5b92abbf60d5ed3 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:00.622143 741 rpcz_store.cc:276] 1029 02:51:59.476818 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:00.188580 (+711762us) service_pool.cc:225] Handling call
1029 02:52:00.214550 (+ 25970us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1029 02:52:00.214618 (+ 68us) ts_tablet_manager.cc:682] Creating new metadata...
1029 02:52:00.529790 (+315172us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:00.618244 (+ 88454us) tablet_replica.cc:206] Creating consensus instance
1029 02:52:00.618491 (+ 247us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251029 02:52:00.622589 1174 rpcz_store.cc:276] 1029 02:51:59.402596 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:59.402741 (+ 145us) service_pool.cc:225] Handling call
1029 02:52:00.615203 (+1212462us) ts_tablet_manager.cc:1151] Deleting tablet 845a69a8696e469a8548787a7645d41a
1029 02:52:00.615214 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:00.615305 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1210084}
W20251029 02:52:00.623180 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 845a69a8696e469a8548787a7645d41a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 845a69a8696e469a8548787a7645d41a already in progress: creating tablet
I20251029 02:52:00.623732 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 508) took 1219 ms. Trace:
I20251029 02:52:00.630079 1175 raft_consensus.cc:2272] T 69597d48631d43b6a5b92abbf60d5ed3 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:00.631836 1726 tablet.cc:1722] T 59af906061d049c99ae2153945597007 P b495a49349324f7d81908f2da700f0d9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 02:52:00.624492 1174 rpcz_store.cc:276] 1029 02:51:59.404156 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:59.405387 (+ 1231us) service_pool.cc:225] Handling call
1029 02:52:00.623540 (+1218153us) ts_tablet_manager.cc:1151] Deleting tablet f65fe9a5137b48669afef51ad7dd76ae
1029 02:52:00.623552 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:00.623697 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1217972}
W20251029 02:52:00.635308 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet f65fe9a5137b48669afef51ad7dd76ae because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f65fe9a5137b48669afef51ad7dd76ae already in progress: creating tablet
I20251029 02:52:00.635903 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 504) took 1233 ms. Trace:
I20251029 02:52:00.636085 1174 rpcz_store.cc:276] 1029 02:51:59.402409 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:59.403900 (+ 1491us) service_pool.cc:225] Handling call
1029 02:52:00.635701 (+1231801us) ts_tablet_manager.cc:1151] Deleting tablet fcb3db8399aa4cf188b07dda220f7f5b
1029 02:52:00.635718 (+ 17us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:00.635864 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1219618}
W20251029 02:52:00.637970 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet fcb3db8399aa4cf188b07dda220f7f5b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fcb3db8399aa4cf188b07dda220f7f5b already in progress: creating tablet
I20251029 02:52:00.638875 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 506) took 1236 ms. Trace:
I20251029 02:52:00.639174 1038 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:37626 (request call id 519) took 1382 ms. Trace:
I20251029 02:52:00.639374 1024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:37626 (request call id 506) took 1395 ms. Trace:
I20251029 02:52:00.640030 1174 rpcz_store.cc:276] 1029 02:51:59.402711 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:59.403594 (+ 883us) service_pool.cc:225] Handling call
1029 02:52:00.638687 (+1235093us) ts_tablet_manager.cc:1151] Deleting tablet 38f83ebcb2a440bc8e6a8d36607e6666
1029 02:52:00.638700 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:00.638841 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1210836}
W20251029 02:52:00.640595 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 38f83ebcb2a440bc8e6a8d36607e6666 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 38f83ebcb2a440bc8e6a8d36607e6666 already in progress: creating tablet
I20251029 02:52:00.640990 1174 tablet_replica.cc:333] T c519832030884a049839d309cc738c28 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:00.641253 1174 raft_consensus.cc:2243] T c519832030884a049839d309cc738c28 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:00.642329 1024 rpcz_store.cc:276] 1029 02:51:59.165511 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:59.785320 (+619809us) service_pool.cc:225] Handling call
1029 02:51:59.808035 (+ 22715us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1029 02:51:59.808070 (+ 35us) ts_tablet_manager.cc:682] Creating new metadata...
1029 02:52:00.051839 (+243769us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:00.426118 (+374279us) tablet_replica.cc:206] Creating consensus instance
1029 02:52:00.560680 (+134562us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251029 02:52:00.642607 1033 rpcz_store.cc:276] 1029 02:51:59.170421 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:59.837370 (+666949us) service_pool.cc:225] Handling call
1029 02:51:59.900457 (+ 63087us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1029 02:51:59.900516 (+ 59us) ts_tablet_manager.cc:682] Creating new metadata...
1029 02:52:00.023632 (+123116us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:00.236199 (+212567us) tablet_replica.cc:206] Creating consensus instance
1029 02:52:00.412044 (+175845us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251029 02:52:00.642705 1174 raft_consensus.cc:2272] T c519832030884a049839d309cc738c28 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:00.643428 1174 ts_tablet_manager.cc:1916] T c519832030884a049839d309cc738c28 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:00.648252 1665 raft_consensus.cc:493] T 078ace5eada044c88d2462efb2b57aa8 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:00.648466 1665 raft_consensus.cc:515] T 078ace5eada044c88d2462efb2b57aa8 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:00.650099 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "078ace5eada044c88d2462efb2b57aa8" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:00.650362 1665 leader_election.cc:290] T 078ace5eada044c88d2462efb2b57aa8 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:52:00.650942 1175 ts_tablet_manager.cc:1916] T 69597d48631d43b6a5b92abbf60d5ed3 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:00.650853 1729 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:44580 (request call id 477) took 1427 ms. Trace:
W20251029 02:52:00.656296 711 leader_election.cc:343] T 078ace5eada044c88d2462efb2b57aa8 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:00.662599 1023 rpcz_store.cc:276] 1029 02:51:59.106440 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:59.638247 (+531807us) service_pool.cc:225] Handling call
1029 02:51:59.652381 (+ 14134us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1029 02:51:59.652418 (+ 37us) ts_tablet_manager.cc:682] Creating new metadata...
1029 02:52:00.037326 (+384908us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:00.464476 (+427150us) tablet_replica.cc:206] Creating consensus instance
1029 02:52:00.571875 (+107399us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251029 02:52:00.663030 1023 tablet_service.cc:1552] Processing DeleteTablet for tablet 7b0f7d6c1c9842789c6ce2cecaa3d5b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:59 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:00.663465 1034 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:37626 (request call id 525) took 1335 ms. Trace:
I20251029 02:52:00.642175 1038 rpcz_store.cc:276] 1029 02:51:59.178956 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:00.044333 (+865377us) service_pool.cc:225] Handling call
1029 02:52:00.068278 (+ 23945us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1029 02:52:00.068305 (+ 27us) ts_tablet_manager.cc:682] Creating new metadata...
1029 02:52:00.401957 (+333652us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:00.450031 (+ 48074us) tablet_replica.cc:206] Creating consensus instance
1029 02:52:00.560754 (+110723us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251029 02:52:00.664758 1023 tablet_service.cc:1552] Processing DeleteTablet for tablet e7b7de85c13b4ddc9c995ed2a2ccd921 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:59 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:00.665027 1023 tablet_service.cc:1552] Processing DeleteTablet for tablet 8afc01d52d784f2fa559cab7ce07c8be with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:59 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:00.666996 1730 tablet.cc:1722] T 70e8cfe5eaa9492c85874dc3214a1e0b P c756528642a543eeacd93cef3e335a7d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 02:52:00.667337 1033 tablet_service.cc:1552] Processing DeleteTablet for tablet 294afb9b012c40c0b05b73795d6f0ac8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:59 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:00.668131 1031 rpcz_store.cc:276] 1029 02:51:59.108752 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:59.691495 (+582743us) service_pool.cc:225] Handling call
1029 02:51:59.694024 (+ 2529us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1029 02:51:59.694177 (+ 153us) ts_tablet_manager.cc:682] Creating new metadata...
1029 02:52:00.435125 (+740948us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:00.471207 (+ 36082us) tablet_replica.cc:206] Creating consensus instance
1029 02:52:00.572432 (+101225us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251029 02:52:00.669035 1034 rpcz_store.cc:276] 1029 02:51:59.242322 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:00.089763 (+847441us) service_pool.cc:225] Handling call
1029 02:52:00.094642 (+ 4879us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1029 02:52:00.094671 (+ 29us) ts_tablet_manager.cc:682] Creating new metadata...
1029 02:52:00.473550 (+378879us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:00.481767 (+ 8217us) tablet_replica.cc:206] Creating consensus instance
1029 02:52:00.576874 (+ 95107us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251029 02:52:00.668180 990 leader_election.cc:304] T 75236b2fa82f4f28b82efae9a2c5160d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5; no voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:00.671574 1172 tablet_bootstrap.cc:654] T 1f5b40c0403d4a49a4bfde222d8c947a P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:00.672819 1729 rpcz_store.cc:276] 1029 02:51:59.219856 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:59.585373 (+365517us) service_pool.cc:225] Handling call
1029 02:52:00.646809 (+1061436us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1029 02:51:59.698201 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
1029 02:51:59.698248 (+ 47us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
1029 02:51:59.698254 (+ 6us) alter_schema_op.cc:66] Acquired schema lock
1029 02:51:59.698257 (+ 3us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
1029 02:51:59.698300 (+ 43us) alter_schema_op.cc:128] START. Timestamp: 1761706319698233
1029 02:51:59.698303 (+ 3us) op_driver.cc:348] REPLICATION: starting
1029 02:51:59.698660 (+ 357us) log.cc:844] Serialized 173 byte log entry
1029 02:52:00.188516 (+489856us) spinlock_profiling.cc:243] Waited 152 ms on lock 0x60c000421100. stack: 00007f1174e9f9f6 00007f1174e9f7b0 00007f1173e7b3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f1188a5cd65 00007f1180615f90 00007f1174ec4380 0000000000a75106 00007f1174ee3a2a 00007f1174edf44c 00007f1174ede89e 00007f118211eb43 00007f11820c11cf 00007f1184bb20d3
1029 02:52:00.332441 (+143925us) op_driver.cc:464] REPLICATION: finished
1029 02:52:00.521168 (+188727us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
1029 02:52:00.547450 (+ 26282us) log.cc:844] Serialized 5 byte log entry
1029 02:52:00.626525 (+ 79075us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:00.646460 (+ 19935us) log.cc:844] Serialized 14 byte log entry
1029 02:52:00.646754 (+ 294us) alter_schema_op.cc:73] Released schema lock
1029 02:52:00.646761 (+ 7us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":182310,"mutex_wait_us":13192,"prepare.queue_time_us":89817,"prepare.run_cpu_time_us":5105,"prepare.run_wall_time_us":55872,"raft.queue_time_us":46,"raft.run_cpu_time_us":370,"raft.run_wall_time_us":372,"replication_time_us":634102,"spinlock_wait_cycles":348962688,"thread_start_us":517405,"threads_started":3,"wal-append.queue_time_us":176897}]]}
I20251029 02:52:00.674093 1729 tablet.cc:1722] T dd90fb9eb167478cbc3ec9363af86295 P b495a49349324f7d81908f2da700f0d9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20251029 02:52:00.675223 1031 tablet_service.cc:1505] Processing CreateTablet for tablet a47821980eb4413b8af40eb5415b5d72 (DEFAULT_TABLE table=default.Table_240f7d6a65f04099abf3e204b2f47fb6 [id=a0e3209e45574a8e8efb821e4a0f56dc]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 02:52:00.675858 990 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:41701, user_credentials={real_user=slave}} blocked reactor thread for 247915us
I20251029 02:52:00.677410 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8cae9d4ddb0744298b82f4d9cee061b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:00.681219 1174 ts_tablet_manager.cc:1929] T c519832030884a049839d309cc738c28 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:00.683275 1033 tablet_service.cc:1505] Processing CreateTablet for tablet fe9e660de57a46c7ba1d82f25e12b4b0 (DEFAULT_TABLE table=default.Table_b0b0cdffdbbc4f89a6117b857e80df26 [id=6add86002712479781451e6413c94e66]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:00.685041 1737 tablet.cc:1722] T aacb6f4b9284457bb6607e3a85d5c809 P b495a49349324f7d81908f2da700f0d9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 02:52:00.683391 1743 raft_consensus.cc:493] T 00000000000000000000000000000000 P e5e5222b26fe4145948620e51edf53fb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e9261eb85ea443d5a3beb36bd0c549f0)
I20251029 02:52:00.686692 1743 raft_consensus.cc:515] T 00000000000000000000000000000000 P e5e5222b26fe4145948620e51edf53fb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9672114a6a742e2864c16ff60ba246e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42343 } } peers { permanent_uuid: "e9261eb85ea443d5a3beb36bd0c549f0" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34239 } } peers { permanent_uuid: "e5e5222b26fe4145948620e51edf53fb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41233 } }
I20251029 02:52:00.688779 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a47821980eb4413b8af40eb5415b5d72. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:00.689612 536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e5e5222b26fe4145948620e51edf53fb" candidate_term: 3 candidate_status { last_received { term: 2 index: 479 } } ignore_live_leader: false dest_uuid: "e9261eb85ea443d5a3beb36bd0c549f0" is_pre_election: true
I20251029 02:52:00.692080 1174 log.cc:1199] T c519832030884a049839d309cc738c28 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/c519832030884a049839d309cc738c28
I20251029 02:52:00.692263 1743 leader_election.cc:290] T 00000000000000000000000000000000 P e5e5222b26fe4145948620e51edf53fb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e9672114a6a742e2864c16ff60ba246e (127.0.104.62:42343), e9261eb85ea443d5a3beb36bd0c549f0 (127.0.104.61:34239)
I20251029 02:52:00.694785 1174 ts_tablet_manager.cc:1950] T c519832030884a049839d309cc738c28 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:00.696559 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet c519832030884a049839d309cc738c28 (table default.Table_e95a0439ea534f538683392bbc8e5219 [id=8ad554168c7e4b86a373a3c4579e6211]) successfully deleted
I20251029 02:52:00.696992 1039 rpcz_store.cc:276] 1029 02:51:59.171245 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:59.975896 (+804651us) service_pool.cc:225] Handling call
1029 02:51:59.977113 (+ 1217us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1029 02:51:59.977156 (+ 43us) ts_tablet_manager.cc:682] Creating new metadata...
1029 02:52:00.416524 (+439368us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:00.468475 (+ 51951us) tablet_replica.cc:206] Creating consensus instance
1029 02:52:00.572358 (+103883us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251029 02:52:00.697104 1726 tablet.cc:1722] T 70e8cfe5eaa9492c85874dc3214a1e0b P b495a49349324f7d81908f2da700f0d9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 02:52:00.697588 1039 tablet_service.cc:1505] Processing CreateTablet for tablet 64fbb4d0b5ab4b5bb1f21195894a66c6 (DEFAULT_TABLE table=default.Table_4559c2fec511424f8d43a3d208c5ee61 [id=e1f28b0e027b4749a63a9610c9c9ddeb]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 02:52:00.698752 512 catalog_manager.cc:4952] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 8f7c746f8b6948b58676298c2c2ea221 because the tablet was not found. No further retry: Not found: Tablet not found: 8f7c746f8b6948b58676298c2c2ea221
I20251029 02:52:00.699076 1027 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:37626 (request call id 526) took 1443 ms. Trace:
W20251029 02:52:00.699605 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 294afb9b012c40c0b05b73795d6f0ac8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 294afb9b012c40c0b05b73795d6f0ac8 already in progress: creating tablet
W20251029 02:52:00.700346 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 7b0f7d6c1c9842789c6ce2cecaa3d5b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7b0f7d6c1c9842789c6ce2cecaa3d5b0 already in progress: creating tablet
I20251029 02:52:00.701243 1038 tablet_service.cc:1552] Processing DeleteTablet for tablet 44c0488eba0543078ff4f0b3d69366f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:59 UTC) from {username='slave'} at 127.0.0.1:37626
W20251029 02:52:00.701797 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet e7b7de85c13b4ddc9c995ed2a2ccd921 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e7b7de85c13b4ddc9c995ed2a2ccd921 already in progress: creating tablet
I20251029 02:52:00.701918 1024 tablet_service.cc:1505] Processing CreateTablet for tablet 72ac7d5f409344319fac08797e0d26bd (DEFAULT_TABLE table=default.Table_290338e0424d46cf9166ed27b1c765a1 [id=a882e9d5d98141d8a83c6382672d34ae]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 02:52:00.702406 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 44c0488eba0543078ff4f0b3d69366f2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 44c0488eba0543078ff4f0b3d69366f2 already in progress: creating tablet
I20251029 02:52:00.702924 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "078ace5eada044c88d2462efb2b57aa8" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:52:00.703094 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 7bb85a12f573494cbb3118d1947eda27 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7bb85a12f573494cbb3118d1947eda27 already in progress: creating tablet
W20251029 02:52:00.703740 712 leader_election.cc:343] T 078ace5eada044c88d2462efb2b57aa8 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:00.703904 712 leader_election.cc:304] T 078ace5eada044c88d2462efb2b57aa8 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:00.704376 1665 raft_consensus.cc:2749] T 078ace5eada044c88d2462efb2b57aa8 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 02:52:00.704407 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 8afc01d52d784f2fa559cab7ce07c8be because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8afc01d52d784f2fa559cab7ce07c8be already in progress: creating tablet
W20251029 02:52:00.705083 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 97b2ecae572a4b7ea1b9595074556708 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 97b2ecae572a4b7ea1b9595074556708 already in progress: creating tablet
I20251029 02:52:00.705458 1038 tablet_service.cc:1505] Processing CreateTablet for tablet 2e003e140a244d46b15a2c3e84f68936 (DEFAULT_TABLE table=default.Table_a8855248e99e4374aaaf0d4f66e9cad1 [id=18b70ff334f443e6a2022ad1a1c6b5de]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 02:52:00.705850 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 998b9cbe76514665be76c6c387dc75b3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 998b9cbe76514665be76c6c387dc75b3 already in progress: creating tablet
I20251029 02:52:00.706291 1040 rpcz_store.cc:276] 1029 02:51:59.177998 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:59.982426 (+804428us) service_pool.cc:225] Handling call
1029 02:51:59.999071 (+ 16645us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1029 02:51:59.999110 (+ 39us) ts_tablet_manager.cc:682] Creating new metadata...
1029 02:52:00.084859 (+ 85749us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:00.454293 (+369434us) tablet_replica.cc:206] Creating consensus instance
1029 02:52:00.576028 (+121735us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251029 02:52:00.706821 1034 tablet_service.cc:1505] Processing CreateTablet for tablet ad2654beffef467dbfbf6472103d25f4 (DEFAULT_TABLE table=default.Table_21064497551747b08fa4839601b7c680 [id=00d800d749494f78a7c14ddc9a169e16]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 02:52:00.706961 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet a7685b0bed174ed0b25f99ef8a46fca3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a7685b0bed174ed0b25f99ef8a46fca3 already in progress: creating tablet
I20251029 02:52:00.707386 1721 raft_consensus.cc:493] T fd900eec3f0c43bf965c148ea90870a3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:00.707531 1721 raft_consensus.cc:515] T fd900eec3f0c43bf965c148ea90870a3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:00.707504 463 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e5e5222b26fe4145948620e51edf53fb" candidate_term: 3 candidate_status { last_received { term: 2 index: 479 } } ignore_live_leader: false dest_uuid: "e9672114a6a742e2864c16ff60ba246e" is_pre_election: true
I20251029 02:52:00.708154 1721 leader_election.cc:290] T fd900eec3f0c43bf965c148ea90870a3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:00.708722 1023 tablet_service.cc:1552] Processing DeleteTablet for tablet 7bb85a12f573494cbb3118d1947eda27 with delete_type TABLET_DATA_DELETED (replaced by tablet a9f78f61ee2841dea71bbbfceacc3ea6) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:00.709164 1023 tablet_service.cc:1505] Processing CreateTablet for tablet 479fd93eb02540868931faee38830223 (DEFAULT_TABLE table=default.Table_d44327be26be4ff1b62a7be7dada45c0 [id=97e41fe5933b4e6e98f6692542883406]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:00.709200 1412 leader_election.cc:304] T 00000000000000000000000000000000 P e5e5222b26fe4145948620e51edf53fb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5e5222b26fe4145948620e51edf53fb; no voters: e9261eb85ea443d5a3beb36bd0c549f0, e9672114a6a742e2864c16ff60ba246e
W20251029 02:52:00.709272 989 leader_election.cc:343] T fd900eec3f0c43bf965c148ea90870a3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: fd900eec3f0c43bf965c148ea90870a3
W20251029 02:52:00.709563 990 leader_election.cc:343] T d2990e9f626c40618c118266bc595684 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:00.709661 1743 raft_consensus.cc:2749] T 00000000000000000000000000000000 P e5e5222b26fe4145948620e51edf53fb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20251029 02:52:00.710218 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 4cede3b183494845af91dca39a2096c4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4cede3b183494845af91dca39a2096c4 already in progress: creating tablet
I20251029 02:52:00.712790 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd900eec3f0c43bf965c148ea90870a3" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:00.714039 1730 tablet.cc:1722] T 41a6139fd9b84707a8c7eebc0381e383 P c756528642a543eeacd93cef3e335a7d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 02:52:00.714425 1174 tablet_replica.cc:333] T 6c22497760274380a832feff7c5a2fae P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:00.714854 1732 tablet.cc:1722] T dd90fb9eb167478cbc3ec9363af86295 P c756528642a543eeacd93cef3e335a7d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20251029 02:52:00.716542 917 raft_consensus.cc:3060] T aacb6f4b9284457bb6607e3a85d5c809 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:00.723234 1713 raft_consensus.cc:2749] T 75236b2fa82f4f28b82efae9a2c5160d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:00.723618 1729 tablet.cc:1722] T 41a6139fd9b84707a8c7eebc0381e383 P b495a49349324f7d81908f2da700f0d9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 02:52:00.723961 1741 tablet.cc:1722] T dd90fb9eb167478cbc3ec9363af86295 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20251029 02:52:00.724450 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ecf3b900206b4c86a7adbc83a769d90e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:00.724747 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e003e140a244d46b15a2c3e84f68936. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:00.724962 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe9e660de57a46c7ba1d82f25e12b4b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:00.714717 1174 raft_consensus.cc:2243] T 6c22497760274380a832feff7c5a2fae P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:00.725140 1174 raft_consensus.cc:2272] T 6c22497760274380a832feff7c5a2fae P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:00.725668 1174 ts_tablet_manager.cc:1916] T 6c22497760274380a832feff7c5a2fae P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:00.726245 1027 rpcz_store.cc:276] 1029 02:51:59.245307 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:00.319054 (+1073747us) service_pool.cc:225] Handling call
1029 02:52:00.405527 (+ 86473us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1029 02:52:00.411904 (+ 6377us) ts_tablet_manager.cc:682] Creating new metadata...
1029 02:52:00.672187 (+260283us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:00.687222 (+ 15035us) tablet_replica.cc:206] Creating consensus instance
1029 02:52:00.687485 (+ 263us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":896}
I20251029 02:52:00.727638 1175 ts_tablet_manager.cc:1929] T 69597d48631d43b6a5b92abbf60d5ed3 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:00.727581 1027 tablet_service.cc:1505] Processing CreateTablet for tablet 45d33b70d549488c802957bb2d5b2821 (DEFAULT_TABLE table=default.Table_360a4b04c44345e0a36d04e6d90c698f [id=6044f9e5029b4985b5540d35286f8586]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:00.728526 1736 tablet.cc:1722] T 70e8cfe5eaa9492c85874dc3214a1e0b P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 02:52:00.729300 1740 tablet.cc:1722] T aacb6f4b9284457bb6607e3a85d5c809 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 02:52:00.729664 1175 log.cc:1199] T 69597d48631d43b6a5b92abbf60d5ed3 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/69597d48631d43b6a5b92abbf60d5ed3
I20251029 02:52:00.730137 1175 ts_tablet_manager.cc:1950] T 69597d48631d43b6a5b92abbf60d5ed3 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:00.732240 1040 tablet_service.cc:1505] Processing CreateTablet for tablet 6c10980d0c1042539ac8fb6406403ce0 (DEFAULT_TABLE table=default.Table_c32e2f0d85f9486d8957cedb7e09dda5 [id=4994066ff383406d8bae1be2e82183ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:00.735810 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 357) took 3515 ms. Trace:
I20251029 02:52:00.735913 1732 tablet.cc:1722] T aacb6f4b9284457bb6607e3a85d5c809 P c756528642a543eeacd93cef3e335a7d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 02:52:00.736011 1175 rpcz_store.cc:276] 1029 02:51:57.220657 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:57.220717 (+ 60us) service_pool.cc:225] Handling call
1029 02:52:00.528258 (+3307541us) ts_tablet_manager.cc:1151] Deleting tablet 69597d48631d43b6a5b92abbf60d5ed3
1029 02:52:00.539189 (+ 10931us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:00.715222 (+176033us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:00.723489 (+ 8267us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:00.735740 (+ 12251us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3307330}
I20251029 02:52:00.736482 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 69597d48631d43b6a5b92abbf60d5ed3 (table default.Table_cec9500b890447609a5feb7346bcf7a0 [id=ffce392867b14b6ea9db44f4bdee57c3]) successfully deleted
I20251029 02:52:00.736951 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 358) took 3480 ms. Trace:
I20251029 02:52:00.737058 1175 rpcz_store.cc:276] 1029 02:51:57.256537 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:57.256839 (+ 302us) service_pool.cc:225] Handling call
1029 02:52:00.736762 (+3479923us) ts_tablet_manager.cc:1151] Deleting tablet 13ee62fc2339410b9db70dfd3d3c6fa6
1029 02:52:00.736772 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:00.736899 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3479601}
W20251029 02:52:00.737509 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 13ee62fc2339410b9db70dfd3d3c6fa6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 13ee62fc2339410b9db70dfd3d3c6fa6 already in progress: creating tablet
I20251029 02:52:00.739593 1682 raft_consensus.cc:493] T f71388036478404d878a52fc20ece216 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:00.739754 1682 raft_consensus.cc:515] T f71388036478404d878a52fc20ece216 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:00.740485 1682 leader_election.cc:290] T f71388036478404d878a52fc20ece216 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
W20251029 02:52:00.741343 847 leader_election.cc:343] T f71388036478404d878a52fc20ece216 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: f71388036478404d878a52fc20ece216
I20251029 02:52:00.741495 1029 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:37626 (request call id 529) took 1415 ms. Trace:
I20251029 02:52:00.741694 1175 tablet_replica.cc:333] T 3cded156380f45cc95893825c71ba0e9 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:00.741899 1175 raft_consensus.cc:2243] T 3cded156380f45cc95893825c71ba0e9 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:00.742089 1175 raft_consensus.cc:2272] T 3cded156380f45cc95893825c71ba0e9 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251029 02:52:00.742478 990 consensus_peers.cc:597] T fa5127bcfcda40a7b5ad4ef016af072a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 -> Peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Couldn't send request to peer c756528642a543eeacd93cef3e335a7d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fa5127bcfcda40a7b5ad4ef016af072a. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:00.744699 1175 ts_tablet_manager.cc:1916] T 3cded156380f45cc95893825c71ba0e9 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:00.744979 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64fbb4d0b5ab4b5bb1f21195894a66c6. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:00.745383 849 leader_election.cc:343] T f71388036478404d878a52fc20ece216 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Not found: Tablet not found: f71388036478404d878a52fc20ece216
I20251029 02:52:00.745575 849 leader_election.cc:304] T f71388036478404d878a52fc20ece216 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9
I20251029 02:52:00.747339 1682 raft_consensus.cc:2749] T f71388036478404d878a52fc20ece216 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:00.758986 743 tablet_service.cc:1505] Processing CreateTablet for tablet 76287edc126a403dbbec4a3b41effe84 (DEFAULT_TABLE table=default.Table_ff3d06085614492a97e31873f16f4fdc [id=e49dd0af2f6e4c91a2aa0ec24881ae48]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:00.759460 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76287edc126a403dbbec4a3b41effe84. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:00.759682 895 tablet_service.cc:1505] Processing CreateTablet for tablet 76287edc126a403dbbec4a3b41effe84 (DEFAULT_TABLE table=default.Table_ff3d06085614492a97e31873f16f4fdc [id=e49dd0af2f6e4c91a2aa0ec24881ae48]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:00.760113 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76287edc126a403dbbec4a3b41effe84. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:00.762851 742 tablet_service.cc:1505] Processing CreateTablet for tablet 19609bc7f06645dc811b4bae8ce506c8 (DEFAULT_TABLE table=default.Table_c4adcf442af54ea08664c405b8a6dfe4 [id=d061b05cb1564a4591068af6baa5a069]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:00.762869 884 tablet_service.cc:1505] Processing CreateTablet for tablet aad32e3581d94cc384c1938d881063fa (DEFAULT_TABLE table=default.Table_3698a931084b485bb2e3689ded4a7fbc [id=c6ea82a97e17409a8e44e333c29453c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:00.763221 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19609bc7f06645dc811b4bae8ce506c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:00.763247 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aad32e3581d94cc384c1938d881063fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:00.766831 881 tablet_service.cc:1505] Processing CreateTablet for tablet 19609bc7f06645dc811b4bae8ce506c8 (DEFAULT_TABLE table=default.Table_c4adcf442af54ea08664c405b8a6dfe4 [id=d061b05cb1564a4591068af6baa5a069]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:00.767231 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19609bc7f06645dc811b4bae8ce506c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:00.770927 741 tablet_service.cc:1505] Processing CreateTablet for tablet aad32e3581d94cc384c1938d881063fa (DEFAULT_TABLE table=default.Table_3698a931084b485bb2e3689ded4a7fbc [id=c6ea82a97e17409a8e44e333c29453c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:00.771400 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aad32e3581d94cc384c1938d881063fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:00.778627 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_f8c6e4111f9945dfb9e1a7dadc4f5381"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:00.779333 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f8c6e4111f9945dfb9e1a7dadc4f5381 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:00.780853 885 tablet_service.cc:1505] Processing CreateTablet for tablet 4d51120775424146b6ab3e6d2e4710bf (DEFAULT_TABLE table=default.Table_8dc0ccf505314b8db07c2b5c2ad231c0 [id=987af87f0a2b4b5294a578f7205f1eb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:00.781282 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d51120775424146b6ab3e6d2e4710bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:00.783032 520 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_c2719a3251c84739a6681f80f1d8f94d" } modify_external_catalogs: true
I20251029 02:52:00.783219 520 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_c2719a3251c84739a6681f80f1d8f94d" } modify_external_catalogs: true
I20251029 02:52:00.747910 1029 rpcz_store.cc:276] 1029 02:51:59.327511 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:00.562575 (+1235064us) service_pool.cc:225] Handling call
1029 02:52:00.562982 (+ 407us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1029 02:52:00.562999 (+ 17us) ts_tablet_manager.cc:682] Creating new metadata...
1029 02:52:00.639752 (+ 76753us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:00.731910 (+ 92158us) tablet_replica.cc:206] Creating consensus instance
1029 02:52:00.732083 (+ 173us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251029 02:52:00.783574 1037 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:37626 (request call id 528) took 1438 ms. Trace:
I20251029 02:52:00.796355 899 tablet_service.cc:1505] Processing CreateTablet for tablet 04528b4b700744aab24f6b1d5cb2787a (DEFAULT_TABLE table=default.Table_bdc0e5608cc44945849e3a3744ca6d9d [id=7aac9c86fb794cd59ab1efb5e34741a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:00.796800 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04528b4b700744aab24f6b1d5cb2787a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:00.794296 1172 tablet_bootstrap.cc:492] T 1f5b40c0403d4a49a4bfde222d8c947a P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:00.798591 1665 raft_consensus.cc:493] T 4bf06476c87a4549a73f5994314bb75f P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:00.798763 1665 raft_consensus.cc:515] T 4bf06476c87a4549a73f5994314bb75f P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:00.799547 1665 leader_election.cc:290] T 4bf06476c87a4549a73f5994314bb75f P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:52:00.800609 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4bf06476c87a4549a73f5994314bb75f" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:00.801254 711 leader_election.cc:343] T 4bf06476c87a4549a73f5994314bb75f P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:00.801672 741 tablet_service.cc:1505] Processing CreateTablet for tablet 4d51120775424146b6ab3e6d2e4710bf (DEFAULT_TABLE table=default.Table_8dc0ccf505314b8db07c2b5c2ad231c0 [id=987af87f0a2b4b5294a578f7205f1eb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:00.802111 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d51120775424146b6ab3e6d2e4710bf. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:00.807816 990 leader_election.cc:343] T fd900eec3f0c43bf965c148ea90870a3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:00.805963 1172 ts_tablet_manager.cc:1403] T 1f5b40c0403d4a49a4bfde222d8c947a P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.283s user 0.008s sys 0.004s
I20251029 02:52:00.807991 990 leader_election.cc:304] T fd900eec3f0c43bf965c148ea90870a3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5; no voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:00.808427 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 479fd93eb02540868931faee38830223. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:00.808559 1722 raft_consensus.cc:2749] T fd900eec3f0c43bf965c148ea90870a3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:00.809602 1176 ts_tablet_manager.cc:1929] T 8d9722c764ac4b27a540595e9c6aafcd P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:00.809450 1037 rpcz_store.cc:276] 1029 02:51:59.306001 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:00.562397 (+1256396us) service_pool.cc:225] Handling call
1029 02:52:00.562862 (+ 465us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1029 02:52:00.562893 (+ 31us) ts_tablet_manager.cc:682] Creating new metadata...
1029 02:52:00.609850 (+ 46957us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:00.743079 (+133229us) tablet_replica.cc:206] Creating consensus instance
1029 02:52:00.743227 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251029 02:52:00.810930 1176 log.cc:1199] T 8d9722c764ac4b27a540595e9c6aafcd P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/8d9722c764ac4b27a540595e9c6aafcd
I20251029 02:52:00.811334 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad2654beffef467dbfbf6472103d25f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:00.814658 1029 tablet_service.cc:1505] Processing CreateTablet for tablet 8f1d5847265b4bb0b29b5e513c35b9bb (DEFAULT_TABLE table=default.Table_877abcb0d221493c8875368e1c75382b [id=6ea46a01a86140b9ae63e33b90b03dd3]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 02:52:00.815057 1735 log.cc:927] Time spent T d2990e9f626c40618c118266bc595684 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Append to log took a long time: real 0.083s user 0.000s sys 0.000s
W20251029 02:52:00.815028 990 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:41701, user_credentials={real_user=slave}} blocked reactor thread for 65531.1us
I20251029 02:52:00.816125 883 tablet_service.cc:1505] Processing CreateTablet for tablet ac162bce5b6e4080839095a045ef0cc0 (DEFAULT_TABLE table=default.Table_104d5aaf444b4e5e813288013737aa15 [id=991375eeb3f14176bd84a23bd3366566]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:00.816519 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac162bce5b6e4080839095a045ef0cc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:00.817903 1166 tablet_bootstrap.cc:492] T 9e07291d6b5645feb4d5890f58e1ef74 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:00.818064 1166 ts_tablet_manager.cc:1403] T 9e07291d6b5645feb4d5890f58e1ef74 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.413s user 0.012s sys 0.003s
I20251029 02:52:00.818849 1166 raft_consensus.cc:359] T 9e07291d6b5645feb4d5890f58e1ef74 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:00.819046 1166 raft_consensus.cc:385] T 9e07291d6b5645feb4d5890f58e1ef74 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:00.819165 1166 raft_consensus.cc:740] T 9e07291d6b5645feb4d5890f58e1ef74 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:00.819370 1166 consensus_queue.cc:260] T 9e07291d6b5645feb4d5890f58e1ef74 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:00.819552 519 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet aacb6f4b9284457bb6607e3a85d5c809 (table default.Table_f35ae25e36f64d2886752bb3b1d66710 [id=9f95160639ad40e995a6c89e297a0fad]). Expected version 1 got 0
I20251029 02:52:00.819897 1166 ts_tablet_manager.cc:1434] T 9e07291d6b5645feb4d5890f58e1ef74 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 02:52:00.820984 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 45d33b70d549488c802957bb2d5b2821. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:00.821583 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c10980d0c1042539ac8fb6406403ce0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:00.822269 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72ac7d5f409344319fac08797e0d26bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:00.822826 1037 tablet_service.cc:1552] Processing DeleteTablet for tablet 75236b2fa82f4f28b82efae9a2c5160d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:52 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:00.822243 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4bf06476c87a4549a73f5994314bb75f" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:00.825080 893 tablet_service.cc:1505] Processing CreateTablet for tablet eaf4018c0d624b1cb8c6528eea7b689e (DEFAULT_TABLE table=default.Table_aedb5a569ca74ef2a9092a847ad5a3f6 [id=16ad7720626546e6a63766039d0b36c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:00.825546 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eaf4018c0d624b1cb8c6528eea7b689e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:00.830327 879 tablet_service.cc:1505] Processing CreateTablet for tablet c0936e54f1d246388a2d82dbc52aac42 (DEFAULT_TABLE table=default.Table_f90425a19e964b75b94c68952d26ae1e [id=48211a42281c40a99c8cf27174cf6679]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:00.830857 892 tablet_service.cc:1505] Processing CreateTablet for tablet a9f78f61ee2841dea71bbbfceacc3ea6 (DEFAULT_TABLE table=default.Table_0100f32eabf24d88a739a3cdac3597e1 [id=f22450cafadd404d8d224cdee646b5a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:00.831496 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0936e54f1d246388a2d82dbc52aac42. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:00.814895 1745 tablet.cc:1722] T 41a6139fd9b84707a8c7eebc0381e383 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 02:52:00.833454 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a04bc4d9605d4c049a151d985035bb63. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:00.835144 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f1d5847265b4bb0b29b5e513c35b9bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:00.835420 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9f78f61ee2841dea71bbbfceacc3ea6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:00.837244 749 tablet_service.cc:1505] Processing CreateTablet for tablet c73059366f4b45528f8b907263c4d085 (DEFAULT_TABLE table=default.Table_daad6fda420f4659bc87d5991d60cf2f [id=9cd9b155c5ad467c8da68c8c03a68206]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:00.837574 754 tablet_service.cc:1505] Processing CreateTablet for tablet a9f78f61ee2841dea71bbbfceacc3ea6 (DEFAULT_TABLE table=default.Table_0100f32eabf24d88a739a3cdac3597e1 [id=f22450cafadd404d8d224cdee646b5a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:00.837776 738 tablet_service.cc:1505] Processing CreateTablet for tablet ac162bce5b6e4080839095a045ef0cc0 (DEFAULT_TABLE table=default.Table_104d5aaf444b4e5e813288013737aa15 [id=991375eeb3f14176bd84a23bd3366566]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:00.838066 748 tablet_service.cc:1505] Processing CreateTablet for tablet eaf4018c0d624b1cb8c6528eea7b689e (DEFAULT_TABLE table=default.Table_aedb5a569ca74ef2a9092a847ad5a3f6 [id=16ad7720626546e6a63766039d0b36c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:00.838722 743 tablet_service.cc:1505] Processing CreateTablet for tablet 04528b4b700744aab24f6b1d5cb2787a (DEFAULT_TABLE table=default.Table_bdc0e5608cc44945849e3a3744ca6d9d [id=7aac9c86fb794cd59ab1efb5e34741a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:00.839075 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a7685b0bed174ed0b25f99ef8a46fca3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:00.839092 1172 raft_consensus.cc:359] T 1f5b40c0403d4a49a4bfde222d8c947a P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:00.841586 895 tablet_service.cc:1505] Processing CreateTablet for tablet 9a58ecfa646c41cdae4d75724767d0d9 (DEFAULT_TABLE table=default.Table_3c09e84f78ee464b823fe6c104c2e207 [id=4b62bbd2d37949aabd475744524eb11a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:00.842015 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a58ecfa646c41cdae4d75724767d0d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:00.844947 1037 tablet_service.cc:1552] Processing DeleteTablet for tablet 97b2ecae572a4b7ea1b9595074556708 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:00 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:00.846345 1172 raft_consensus.cc:385] T 1f5b40c0403d4a49a4bfde222d8c947a P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:00.846609 1172 raft_consensus.cc:740] T 1f5b40c0403d4a49a4bfde222d8c947a P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:00.847240 1172 consensus_queue.cc:260] T 1f5b40c0403d4a49a4bfde222d8c947a P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
W20251029 02:52:00.848980 712 leader_election.cc:343] T 4bf06476c87a4549a73f5994314bb75f P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:00.848922 742 tablet_service.cc:1505] Processing CreateTablet for tablet 9a58ecfa646c41cdae4d75724767d0d9 (DEFAULT_TABLE table=default.Table_3c09e84f78ee464b823fe6c104c2e207 [id=4b62bbd2d37949aabd475744524eb11a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:00.849109 1172 ts_tablet_manager.cc:1434] T 1f5b40c0403d4a49a4bfde222d8c947a P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.040s user 0.001s sys 0.001s
I20251029 02:52:00.849143 712 leader_election.cc:304] T 4bf06476c87a4549a73f5994314bb75f P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:00.849845 1665 raft_consensus.cc:2749] T 4bf06476c87a4549a73f5994314bb75f P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:00.850203 750 tablet_service.cc:1505] Processing CreateTablet for tablet c0936e54f1d246388a2d82dbc52aac42 (DEFAULT_TABLE table=default.Table_f90425a19e964b75b94c68952d26ae1e [id=48211a42281c40a99c8cf27174cf6679]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:00.850497 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c73059366f4b45528f8b907263c4d085. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:00.851991 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eaf4018c0d624b1cb8c6528eea7b689e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:00.852479 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9f78f61ee2841dea71bbbfceacc3ea6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:00.853034 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac162bce5b6e4080839095a045ef0cc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:00.855252 899 tablet_service.cc:1505] Processing CreateTablet for tablet c73059366f4b45528f8b907263c4d085 (DEFAULT_TABLE table=default.Table_daad6fda420f4659bc87d5991d60cf2f [id=9cd9b155c5ad467c8da68c8c03a68206]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:00.855652 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c73059366f4b45528f8b907263c4d085. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:00.859279 1175 ts_tablet_manager.cc:1929] T 3cded156380f45cc95893825c71ba0e9 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:00.859421 1175 log.cc:1199] T 3cded156380f45cc95893825c71ba0e9 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/3cded156380f45cc95893825c71ba0e9
I20251029 02:52:00.864174 1174 ts_tablet_manager.cc:1929] T 6c22497760274380a832feff7c5a2fae P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:00.865535 1172 tablet_bootstrap.cc:492] T 51effc9324034ae79e2b25cb57a996c1 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:00.866294 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a58ecfa646c41cdae4d75724767d0d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:00.867812 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0936e54f1d246388a2d82dbc52aac42. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:00.870846 522 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c9e923724ca9402682f13b252dbc37aa consensus state has no leader
I20251029 02:52:00.877597 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_dca7cb07591d4d238e29e54969bca2b8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:00.878358 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dca7cb07591d4d238e29e54969bca2b8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:00.880297 526 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 2089b058d0e94d99b40ad2718518353e
I20251029 02:52:00.881029 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_151d1210897f41ceb7e07cca8a1e478c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:00.881554 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_151d1210897f41ceb7e07cca8a1e478c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:00.882706 524 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet d31e548feae24b739087c1e211d514f9
I20251029 02:52:00.883370 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_cb5488bb7e104676aad8f3db9be87802"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:00.883886 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cb5488bb7e104676aad8f3db9be87802 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:00.884958 525 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 2b278db5b2b04344b864797055a5f665
I20251029 02:52:00.886094 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_f8f6696a8a1c4fcca702c03886a45f62"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:00.886570 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f8f6696a8a1c4fcca702c03886a45f62 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:52:00.888219 523 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ba699862e6f646c6beaa39f080d0faba consensus state has no leader
I20251029 02:52:00.888635 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_f861260a80434489a8a2a44e9f0b6998"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:00.889086 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f861260a80434489a8a2a44e9f0b6998 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:00.890177 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04528b4b700744aab24f6b1d5cb2787a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:00.892740 1166 tablet_bootstrap.cc:492] T b5d5b737c86844d58c51b2388c0b1705 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:00.896420 1174 log.cc:1199] T 6c22497760274380a832feff7c5a2fae P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/6c22497760274380a832feff7c5a2fae
I20251029 02:52:00.896836 1174 ts_tablet_manager.cc:1950] T 6c22497760274380a832feff7c5a2fae P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:00.897847 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 6c22497760274380a832feff7c5a2fae (table default.Table_0fa5f3e932c846168255e7540c35a515 [id=c2475b97464f45da8990e004660c42b3]) successfully deleted
I20251029 02:52:00.880496 1167 tablet_bootstrap.cc:492] T d2990e9f626c40618c118266bc595684 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:00.881024 885 tablet_service.cc:1552] Processing DeleteTablet for tablet 2089b058d0e94d99b40ad2718518353e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:00 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:00.882431 893 tablet_service.cc:1505] Processing CreateTablet for tablet 60a964a5181a45a186921c31cbdd3db7 (DEFAULT_TABLE table=default.Table_6a3e403847304172bc58735bfe02dbd0 [id=8eb84ca5a1ac4d65a459573a06763a69]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:00.898626 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60a964a5181a45a186921c31cbdd3db7. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:00.898945 989 consensus_peers.cc:597] T 9be9add0274d41b688edd6292a253a06 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 -> Peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Couldn't send request to peer b495a49349324f7d81908f2da700f0d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9be9add0274d41b688edd6292a253a06. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:00.883426 892 tablet_service.cc:1552] Processing DeleteTablet for tablet d31e548feae24b739087c1e211d514f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:00 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:00.902122 1176 ts_tablet_manager.cc:1950] T 8d9722c764ac4b27a540595e9c6aafcd P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:00.885635 879 tablet_service.cc:1552] Processing DeleteTablet for tablet 2b278db5b2b04344b864797055a5f665 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:00 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:00.903331 1037 tablet_service.cc:1552] Processing DeleteTablet for tablet 998b9cbe76514665be76c6c387dc75b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:00 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:00.887019 881 tablet_service.cc:1505] Processing CreateTablet for tablet 76a7e8517db140e8bdd3b67ce423ec90 (DEFAULT_TABLE table=default.Table_0936756f462041d6a93c1401ceb56158 [id=05921c5e2e534071859f555b510a97ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:00.905700 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76a7e8517db140e8bdd3b67ce423ec90. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:00.909056 1037 tablet_service.cc:1552] Processing DeleteTablet for tablet a7685b0bed174ed0b25f99ef8a46fca3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:00 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:00.909300 1037 tablet_service.cc:1552] Processing DeleteTablet for tablet 4cede3b183494845af91dca39a2096c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:00 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:00.909305 1042 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:37626 (request call id 524) took 1721 ms. Trace:
I20251029 02:52:00.912148 1042 rpcz_store.cc:276] 1029 02:51:59.187229 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:00.076045 (+888816us) service_pool.cc:225] Handling call
1029 02:52:00.076517 (+ 472us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1029 02:52:00.076540 (+ 23us) ts_tablet_manager.cc:682] Creating new metadata...
1029 02:52:00.488667 (+412127us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:00.860057 (+371390us) tablet_replica.cc:206] Creating consensus instance
1029 02:52:00.908171 (+ 48114us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251029 02:52:00.912475 1167 ts_tablet_manager.cc:1403] T d2990e9f626c40618c118266bc595684 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.339s user 0.000s sys 0.007s
I20251029 02:52:00.912684 1042 tablet_service.cc:1505] Processing CreateTablet for tablet 76287edc126a403dbbec4a3b41effe84 (DEFAULT_TABLE table=default.Table_ff3d06085614492a97e31873f16f4fdc [id=e49dd0af2f6e4c91a2aa0ec24881ae48]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:00.913421 1167 raft_consensus.cc:359] T d2990e9f626c40618c118266bc595684 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:00.913686 1167 raft_consensus.cc:385] T d2990e9f626c40618c118266bc595684 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:00.913839 1167 raft_consensus.cc:740] T d2990e9f626c40618c118266bc595684 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:00.914125 1167 consensus_queue.cc:260] T d2990e9f626c40618c118266bc595684 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:00.918004 1167 ts_tablet_manager.cc:1434] T d2990e9f626c40618c118266bc595684 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20251029 02:52:00.920996 1175 ts_tablet_manager.cc:1950] T 3cded156380f45cc95893825c71ba0e9 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:00.925756 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 361) took 3622 ms. Trace:
W20251029 02:52:00.926369 511 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet f4729ecd182042c2a10f67ca25224101 (table default.Table_57a0f7b13bf146d890469978e60ac4be [id=1dd2d5488bc647c8b82c53886583b28d], current schema version=3) failed: Not found: failed to reset TS proxy: No leader found: Tablet f4729ecd182042c2a10f67ca25224101 consensus state has no leader
I20251029 02:52:00.925081 741 tablet_service.cc:1552] Processing DeleteTablet for tablet d31e548feae24b739087c1e211d514f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:00 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:00.926600 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 2b278db5b2b04344b864797055a5f665 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:00 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:00.930218 1167 tablet_bootstrap.cc:492] T fd900eec3f0c43bf965c148ea90870a3 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:00.930277 1037 tablet_service.cc:1552] Processing DeleteTablet for tablet fd900eec3f0c43bf965c148ea90870a3 with delete_type TABLET_DATA_DELETED (replaced by tablet b2b2d64dcf964da8be4b544dbbe31977) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:00.930748 1037 tablet_service.cc:1505] Processing CreateTablet for tablet 19609bc7f06645dc811b4bae8ce506c8 (DEFAULT_TABLE table=default.Table_c4adcf442af54ea08664c405b8a6dfe4 [id=d061b05cb1564a4591068af6baa5a069]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:00.932718 521 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet d29cc2a0c8b84e3ebadc7f571d666847
I20251029 02:52:00.932721 751 tablet_service.cc:1505] Processing CreateTablet for tablet 76a7e8517db140e8bdd3b67ce423ec90 (DEFAULT_TABLE table=default.Table_0936756f462041d6a93c1401ceb56158 [id=05921c5e2e534071859f555b510a97ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:00.933260 755 tablet_service.cc:1505] Processing CreateTablet for tablet 60a964a5181a45a186921c31cbdd3db7 (DEFAULT_TABLE table=default.Table_6a3e403847304172bc58735bfe02dbd0 [id=8eb84ca5a1ac4d65a459573a06763a69]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:00.933523 879 tablet_service.cc:1552] Processing DeleteTablet for tablet d29cc2a0c8b84e3ebadc7f571d666847 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:00 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:00.935132 521 catalog_manager.cc:5649] T d2990e9f626c40618c118266bc595684 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } health_report { overall_health: UNKNOWN } } }
I20251029 02:52:00.935729 521 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet d2990e9f626c40618c118266bc595684 (table default.Table_bd020e7c45a54c6e92211c4261293575 [id=f733948e92d8470da9d8050bd26a06b1]). Expected version 1 got 0
I20251029 02:52:00.935992 1167 tablet_bootstrap.cc:654] T fd900eec3f0c43bf965c148ea90870a3 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:00.936137 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60a964a5181a45a186921c31cbdd3db7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:00.936789 520 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 44ae2b07edca44afbe1b1fb3a5b1115d
I20251029 02:52:00.937541 879 tablet_service.cc:1552] Processing DeleteTablet for tablet 44ae2b07edca44afbe1b1fb3a5b1115d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:00 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:00.937635 520 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_b89b7b9f911e4c308272ed13a311ac9b" } new_table_name: "default.Table_2846af457a4b4e27a739af50c7f8c7f7" modify_external_catalogs: true
I20251029 02:52:00.937760 756 tablet_service.cc:1552] Processing DeleteTablet for tablet 2089b058d0e94d99b40ad2718518353e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:00 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:00.938005 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76a7e8517db140e8bdd3b67ce423ec90. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:00.938495 1174 tablet_replica.cc:333] T 8d9722c764ac4b27a540595e9c6aafcd P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:00.938799 1174 raft_consensus.cc:2243] T 8d9722c764ac4b27a540595e9c6aafcd P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:00.939020 1174 raft_consensus.cc:2272] T 8d9722c764ac4b27a540595e9c6aafcd P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:00.939146 756 tablet_service.cc:1552] Processing DeleteTablet for tablet 44ae2b07edca44afbe1b1fb3a5b1115d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:00 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:00.939548 1174 ts_tablet_manager.cc:1916] T 8d9722c764ac4b27a540595e9c6aafcd P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:00.939924 1175 rpcz_store.cc:276] 1029 02:51:57.303179 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:57.303470 (+ 291us) service_pool.cc:225] Handling call
1029 02:52:00.738364 (+3434894us) ts_tablet_manager.cc:1151] Deleting tablet 3cded156380f45cc95893825c71ba0e9
1029 02:52:00.738373 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:00.751434 (+ 13061us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:00.814017 (+ 62583us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:00.925720 (+111703us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3431639}
I20251029 02:52:00.940353 1175 tablet_replica.cc:333] T faa835de84ff45b19525d1d603678987 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:00.940569 1175 raft_consensus.cc:2243] T faa835de84ff45b19525d1d603678987 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:00.940732 1175 raft_consensus.cc:2272] T faa835de84ff45b19525d1d603678987 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:00.941247 1175 ts_tablet_manager.cc:1916] T faa835de84ff45b19525d1d603678987 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:00.943079 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 3cded156380f45cc95893825c71ba0e9 (table default.Table_6254452e8df24c21ab7d8ae32a06bdfb [id=46182aaaeecc4a4d9ec60cefb55ff50d]) successfully deleted
I20251029 02:52:00.943852 517 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_ed3714840df249bfb7cd3f3d9abaa578" } new_table_name: "default.Table_e75db5cf8f5f49ea837cb6d63f4b7ebf" modify_external_catalogs: true
I20251029 02:52:00.955363 741 tablet_service.cc:1552] Processing DeleteTablet for tablet d29cc2a0c8b84e3ebadc7f571d666847 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:00 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:00.958534 1740 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:37626 (request call id 483) took 1852 ms. Trace:
I20251029 02:52:00.958673 1740 rpcz_store.cc:276] 1029 02:51:59.105803 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:59.387079 (+281276us) service_pool.cc:225] Handling call
1029 02:52:00.958463 (+1571384us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1029 02:51:59.566588 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
1029 02:51:59.570809 (+ 4221us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
1029 02:51:59.570820 (+ 11us) alter_schema_op.cc:66] Acquired schema lock
1029 02:51:59.570828 (+ 8us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
1029 02:51:59.570867 (+ 39us) alter_schema_op.cc:128] START. Timestamp: 1761706319570820
1029 02:51:59.570870 (+ 3us) op_driver.cc:348] REPLICATION: starting
1029 02:51:59.573076 (+ 2206us) log.cc:844] Serialized 173 byte log entry
1029 02:52:00.599950 (+1026874us) op_driver.cc:464] REPLICATION: finished
1029 02:52:00.705355 (+105405us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
1029 02:52:00.820292 (+114937us) log.cc:844] Serialized 5 byte log entry
1029 02:52:00.958046 (+137754us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:00.958369 (+ 323us) log.cc:844] Serialized 14 byte log entry
1029 02:52:00.958420 (+ 51us) alter_schema_op.cc:73] Released schema lock
1029 02:52:00.958424 (+ 4us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":105316,"prepare.queue_time_us":153536,"prepare.run_cpu_time_us":22610,"prepare.run_wall_time_us":77806,"raft.queue_time_us":51,"raft.run_cpu_time_us":5,"raft.run_wall_time_us":5,"replication_time_us":1029041,"thread_start_us":280624,"threads_started":3,"wal-append.queue_time_us":69671}]]}
I20251029 02:52:00.964118 1167 tablet_bootstrap.cc:492] T fd900eec3f0c43bf965c148ea90870a3 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:00.964311 1167 ts_tablet_manager.cc:1403] T fd900eec3f0c43bf965c148ea90870a3 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.034s user 0.002s sys 0.002s
I20251029 02:52:00.965186 1167 raft_consensus.cc:359] T fd900eec3f0c43bf965c148ea90870a3 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:00.965418 1167 raft_consensus.cc:385] T fd900eec3f0c43bf965c148ea90870a3 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:00.965503 1167 raft_consensus.cc:740] T fd900eec3f0c43bf965c148ea90870a3 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:00.965777 1167 consensus_queue.cc:260] T fd900eec3f0c43bf965c148ea90870a3 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:00.965998 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 363) took 4006 ms. Trace:
I20251029 02:52:00.966120 1176 rpcz_store.cc:276] 1029 02:51:56.954965 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:56.955556 (+ 591us) service_pool.cc:225] Handling call
1029 02:52:00.228293 (+3272737us) ts_tablet_manager.cc:1151] Deleting tablet 8d9722c764ac4b27a540595e9c6aafcd
1029 02:52:00.404830 (+176537us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:00.675988 (+271158us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:00.788216 (+112228us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:00.961395 (+173179us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3272529}
I20251029 02:52:00.972965 1172 tablet_bootstrap.cc:654] T 51effc9324034ae79e2b25cb57a996c1 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:00.980643 1167 ts_tablet_manager.cc:1434] T fd900eec3f0c43bf965c148ea90870a3 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.016s user 0.001s sys 0.001s
I20251029 02:52:00.981029 1167 tablet_bootstrap.cc:492] T aa776cde2f494fa5810539edf9f289a1 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:00.984822 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 8d9722c764ac4b27a540595e9c6aafcd (table default.Table_ee95bb03ab1246e8826f28083411050d [id=09fbf31759f047be924a5ef1f0030765]) successfully deleted
I20251029 02:52:00.986547 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 373) took 3802 ms. Trace:
I20251029 02:52:00.986685 1176 rpcz_store.cc:276] 1029 02:51:57.163740 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:57.163794 (+ 54us) service_pool.cc:225] Handling call
1029 02:52:00.966385 (+3802591us) ts_tablet_manager.cc:1151] Deleting tablet 59d03a190b784ab18a3e63a572675891
1029 02:52:00.966394 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:00.966503 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3802269}
I20251029 02:52:00.987591 1174 ts_tablet_manager.cc:1929] T 8d9722c764ac4b27a540595e9c6aafcd P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:00.987102 1175 ts_tablet_manager.cc:1929] T faa835de84ff45b19525d1d603678987 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:00.987747 1174 log.cc:1199] T 8d9722c764ac4b27a540595e9c6aafcd P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/8d9722c764ac4b27a540595e9c6aafcd
I20251029 02:52:00.988160 1174 ts_tablet_manager.cc:1950] T 8d9722c764ac4b27a540595e9c6aafcd P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:00.988183 1175 log.cc:1199] T faa835de84ff45b19525d1d603678987 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/faa835de84ff45b19525d1d603678987
I20251029 02:52:00.989413 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 8d9722c764ac4b27a540595e9c6aafcd (table default.Table_ee95bb03ab1246e8826f28083411050d [id=09fbf31759f047be924a5ef1f0030765]) successfully deleted
I20251029 02:52:00.990672 1032 tablet_service.cc:1505] Processing CreateTablet for tablet 04528b4b700744aab24f6b1d5cb2787a (DEFAULT_TABLE table=default.Table_bdc0e5608cc44945849e3a3744ca6d9d [id=7aac9c86fb794cd59ab1efb5e34741a5]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 02:52:00.990278 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 59d03a190b784ab18a3e63a572675891 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 59d03a190b784ab18a3e63a572675891 already in progress: creating tablet
I20251029 02:52:00.991089 1175 ts_tablet_manager.cc:1950] T faa835de84ff45b19525d1d603678987 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:00.991997 1665 raft_consensus.cc:493] T 7309d3190cfa415281c5253a634fa7f3 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:00.992156 1665 raft_consensus.cc:515] T 7309d3190cfa415281c5253a634fa7f3 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:00.992340 1167 tablet_bootstrap.cc:654] T aa776cde2f494fa5810539edf9f289a1 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:00.992892 1665 leader_election.cc:290] T 7309d3190cfa415281c5253a634fa7f3 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:00.993805 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7309d3190cfa415281c5253a634fa7f3" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:00.994649 711 leader_election.cc:343] T 7309d3190cfa415281c5253a634fa7f3 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:00.996790 1172 tablet_bootstrap.cc:492] T 51effc9324034ae79e2b25cb57a996c1 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:00.997057 1172 ts_tablet_manager.cc:1403] T 51effc9324034ae79e2b25cb57a996c1 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.138s user 0.005s sys 0.005s
I20251029 02:52:00.997218 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 377) took 2749 ms. Trace:
I20251029 02:52:00.998399 1174 tablet_replica.cc:333] T f71388036478404d878a52fc20ece216 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:00.998610 1174 raft_consensus.cc:2243] T f71388036478404d878a52fc20ece216 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:00.998066 1175 rpcz_store.cc:276] 1029 02:51:58.247034 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:58.247098 (+ 64us) service_pool.cc:225] Handling call
1029 02:52:00.940284 (+2693186us) ts_tablet_manager.cc:1151] Deleting tablet faa835de84ff45b19525d1d603678987
1029 02:52:00.940294 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:00.952826 (+ 12532us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:00.986300 (+ 33474us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:00.996963 (+ 10663us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2692834}
I20251029 02:52:00.998813 1174 raft_consensus.cc:2272] T f71388036478404d878a52fc20ece216 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:00.999368 1174 ts_tablet_manager.cc:1916] T f71388036478404d878a52fc20ece216 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:00.999538 1172 raft_consensus.cc:359] T 51effc9324034ae79e2b25cb57a996c1 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:00.999914 1172 raft_consensus.cc:385] T 51effc9324034ae79e2b25cb57a996c1 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:01.000111 1172 raft_consensus.cc:740] T 51effc9324034ae79e2b25cb57a996c1 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:01.000571 1167 tablet_bootstrap.cc:492] T aa776cde2f494fa5810539edf9f289a1 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:01.000705 1167 ts_tablet_manager.cc:1403] T aa776cde2f494fa5810539edf9f289a1 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.002s
I20251029 02:52:01.000537 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 378) took 2751 ms. Trace:
I20251029 02:52:01.000751 1172 consensus_queue.cc:260] T 51effc9324034ae79e2b25cb57a996c1 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:01.000948 1175 rpcz_store.cc:276] 1029 02:51:58.247787 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:58.247840 (+ 53us) service_pool.cc:225] Handling call
1029 02:52:00.999082 (+2751242us) ts_tablet_manager.cc:1151] Deleting tablet c32c28b0e74a4af3a0e69bbf24ab1a1f
1029 02:52:00.999093 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:00.999473 (+ 380us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2750952}
I20251029 02:52:01.001480 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet faa835de84ff45b19525d1d603678987 (table default.Table_d9e164af67334c98a5d0821f2d80a35e [id=a184f139921d4864b1627057a7359810]) successfully deleted
I20251029 02:52:01.001593 1167 raft_consensus.cc:359] T aa776cde2f494fa5810539edf9f289a1 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:01.001783 1167 raft_consensus.cc:385] T aa776cde2f494fa5810539edf9f289a1 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:01.001883 1167 raft_consensus.cc:740] T aa776cde2f494fa5810539edf9f289a1 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
W20251029 02:52:01.001905 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet c32c28b0e74a4af3a0e69bbf24ab1a1f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c32c28b0e74a4af3a0e69bbf24ab1a1f already in progress: creating tablet
I20251029 02:52:01.002139 1167 consensus_queue.cc:260] T aa776cde2f494fa5810539edf9f289a1 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:01.002755 1167 ts_tablet_manager.cc:1434] T aa776cde2f494fa5810539edf9f289a1 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 02:52:01.003059 1167 tablet_bootstrap.cc:492] T 75236b2fa82f4f28b82efae9a2c5160d P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:01.003175 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 376) took 2756 ms. Trace:
I20251029 02:52:01.003722 1172 ts_tablet_manager.cc:1434] T 51effc9324034ae79e2b25cb57a996c1 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20251029 02:52:01.003485 1175 rpcz_store.cc:276] 1029 02:51:58.246898 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:58.257448 (+ 10550us) service_pool.cc:225] Handling call
1029 02:52:01.002369 (+2744921us) ts_tablet_manager.cc:1151] Deleting tablet 589a521f5cc24e6289c37b9cf256bce5
1029 02:52:01.002380 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:01.002918 (+ 538us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2744581}
W20251029 02:52:01.004948 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 589a521f5cc24e6289c37b9cf256bce5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 589a521f5cc24e6289c37b9cf256bce5 already in progress: creating tablet
I20251029 02:52:01.005717 1172 tablet_bootstrap.cc:492] T 24cb2a71ba3841d4a5e3e5c6fe99ae2f P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:01.006218 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 380) took 2751 ms. Trace:
I20251029 02:52:01.006866 1175 rpcz_store.cc:276] 1029 02:51:58.293503 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:58.301265 (+ 7762us) service_pool.cc:225] Handling call
1029 02:52:01.005395 (+2704130us) ts_tablet_manager.cc:1151] Deleting tablet 4aa78f4ac38e4919864c924e50f1aaf4
1029 02:52:01.005858 (+ 463us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:01.006022 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":218496,"tablet-delete.queue_time_us":2703904}
W20251029 02:52:01.008126 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 4aa78f4ac38e4919864c924e50f1aaf4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4aa78f4ac38e4919864c924e50f1aaf4 already in progress: creating tablet
I20251029 02:52:01.009362 1704 catalog_manager.cc:1269] Loaded cluster ID: 96c87f4c6dea451cb609ec9ee56d85bf
I20251029 02:52:01.009343 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 391) took 2653 ms. Trace:
I20251029 02:52:01.009743 1175 rpcz_store.cc:276] 1029 02:51:58.355919 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:58.356001 (+ 82us) service_pool.cc:225] Handling call
1029 02:52:01.008792 (+2652791us) ts_tablet_manager.cc:1151] Deleting tablet 2f198c3259864f7ab3625cb7bfae20b0
1029 02:52:01.008802 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:01.009198 (+ 396us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2539893}
W20251029 02:52:01.010900 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 2f198c3259864f7ab3625cb7bfae20b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2f198c3259864f7ab3625cb7bfae20b0 already in progress: creating tablet
I20251029 02:52:01.012359 1704 catalog_manager.cc:1562] T 00000000000000000000000000000000 P e5e5222b26fe4145948620e51edf53fb: loading cluster ID for follower catalog manager: success
I20251029 02:52:01.012665 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 405) took 2636 ms. Trace:
I20251029 02:52:01.012900 1175 rpcz_store.cc:276] 1029 02:51:58.375864 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:58.468834 (+ 92970us) service_pool.cc:225] Handling call
1029 02:52:01.011338 (+2542504us) ts_tablet_manager.cc:1151] Deleting tablet 84993b2f63e7421290a161dde4bb0607
1029 02:52:01.011450 (+ 112us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:01.011746 (+ 296us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2542256}
W20251029 02:52:01.013718 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 84993b2f63e7421290a161dde4bb0607 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 84993b2f63e7421290a161dde4bb0607 already in progress: creating tablet
I20251029 02:52:01.014420 1175 tablet_replica.cc:333] T 9e07291d6b5645feb4d5890f58e1ef74 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:01.014729 1029 tablet_service.cc:1505] Processing CreateTablet for tablet 4d51120775424146b6ab3e6d2e4710bf (DEFAULT_TABLE table=default.Table_8dc0ccf505314b8db07c2b5c2ad231c0 [id=987af87f0a2b4b5294a578f7205f1eb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:01.015034 1025 tablet_service.cc:1505] Processing CreateTablet for tablet aad32e3581d94cc384c1938d881063fa (DEFAULT_TABLE table=default.Table_3698a931084b485bb2e3689ded4a7fbc [id=c6ea82a97e17409a8e44e333c29453c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:01.015281 1175 raft_consensus.cc:2243] T 9e07291d6b5645feb4d5890f58e1ef74 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:01.015785 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19609bc7f06645dc811b4bae8ce506c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:01.016700 1175 raft_consensus.cc:2272] T 9e07291d6b5645feb4d5890f58e1ef74 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:01.017971 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76287edc126a403dbbec4a3b41effe84. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:01.021329 1176 tablet_replica.cc:333] T 9be9add0274d41b688edd6292a253a06 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:01.021539 1176 raft_consensus.cc:2243] T 9be9add0274d41b688edd6292a253a06 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:01.023427 1176 raft_consensus.cc:2272] T 9be9add0274d41b688edd6292a253a06 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:01.023468 1057 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7309d3190cfa415281c5253a634fa7f3" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:01.024227 1176 ts_tablet_manager.cc:1916] T 9be9add0274d41b688edd6292a253a06 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:01.025271 1172 tablet_bootstrap.cc:654] T 24cb2a71ba3841d4a5e3e5c6fe99ae2f P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
W20251029 02:52:01.025847 712 leader_election.cc:343] T 7309d3190cfa415281c5253a634fa7f3 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:01.025976 712 leader_election.cc:304] T 7309d3190cfa415281c5253a634fa7f3 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:01.026809 1665 raft_consensus.cc:2749] T 7309d3190cfa415281c5253a634fa7f3 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:01.035003 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 04528b4b700744aab24f6b1d5cb2787a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:01.035049 1166 tablet_bootstrap.cc:654] T b5d5b737c86844d58c51b2388c0b1705 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:00.177724 1514 rpcz_store.cc:276] 1029 02:51:56.776149 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:56.776202 (+ 53us) service_pool.cc:225] Handling call
1029 02:51:57.583031 (+806829us) raft_consensus.cc:1471] Updating replica for 474 ops
1029 02:51:58.476991 (+893960us) raft_consensus.cc:1534] Early marking committed up to index 0
1029 02:51:58.477001 (+ 10us) raft_consensus.cc:1539] Triggering prepare for 474 ops
1029 02:51:58.477818 (+ 817us) write_op.cc:270] Start()
1029 02:51:58.477893 (+ 75us) write_op.cc:276] Timestamp: P: 1761706309793767 usec, L: 0
1029 02:51:58.611645 (+133752us) write_op.cc:270] Start()
1029 02:51:58.611687 (+ 42us) write_op.cc:276] Timestamp: P: 1761706309794561 usec, L: 0
1029 02:51:58.611940 (+ 253us) write_op.cc:270] Start()
1029 02:51:58.611980 (+ 40us) write_op.cc:276] Timestamp: P: 1761706309795389 usec, L: 0
1029 02:51:58.612172 (+ 192us) write_op.cc:270] Start()
1029 02:51:58.612196 (+ 24us) write_op.cc:276] Timestamp: P: 1761706309803248 usec, L: 0
1029 02:51:58.612358 (+ 162us) write_op.cc:270] Start()
1029 02:51:58.612380 (+ 22us) write_op.cc:276] Timestamp: P: 1761706309803825 usec, L: 0
1029 02:51:58.612590 (+ 210us) write_op.cc:270] Start()
1029 02:51:58.612614 (+ 24us) write_op.cc:276] Timestamp: P: 1761706309804317 usec, L: 0
1029 02:51:58.612761 (+ 147us) write_op.cc:270] Start()
1029 02:51:58.612802 (+ 41us) write_op.cc:276] Timestamp: P: 1761706309804949 usec, L: 0
1029 02:51:58.612947 (+ 145us) write_op.cc:270] Start()
1029 02:51:58.612965 (+ 18us) write_op.cc:276] Timestamp: P: 1761706309805507 usec, L: 0
1029 02:51:58.613098 (+ 133us) write_op.cc:270] Start()
1029 02:51:58.613115 (+ 17us) write_op.cc:276] Timestamp: P: 1761706309805931 usec, L: 0
1029 02:51:58.613257 (+ 142us) write_op.cc:270] Start()
1029 02:51:58.613280 (+ 23us) write_op.cc:276] Timestamp: P: 1761706309808912 usec, L: 0
1029 02:51:58.613440 (+ 160us) write_op.cc:270] Start()
1029 02:51:58.613460 (+ 20us) write_op.cc:276] Timestamp: P: 1761706309833450 usec, L: 0
1029 02:51:58.613697 (+ 237us) write_op.cc:270] Start()
1029 02:51:58.613719 (+ 22us) write_op.cc:276] Timestamp: P: 1761706309840878 usec, L: 0
1029 02:51:58.613863 (+ 144us) write_op.cc:270] Start()
1029 02:51:58.613882 (+ 19us) write_op.cc:276] Timestamp: P: 1761706309841425 usec, L: 0
1029 02:51:58.614004 (+ 122us) write_op.cc:270] Start()
1029 02:51:58.614028 (+ 24us) write_op.cc:276] Timestamp: P: 1761706309841925 usec, L: 0
1029 02:51:58.614179 (+ 151us) write_op.cc:270] Start()
1029 02:51:58.614198 (+ 19us) write_op.cc:276] Timestamp: P: 1761706309842431 usec, L: 0
1029 02:51:58.614326 (+ 128us) write_op.cc:270] Start()
1029 02:51:58.614345 (+ 19us) write_op.cc:276] Timestamp: P: 1761706309842953 usec, L: 0
1029 02:51:58.614531 (+ 186us) write_op.cc:270] Start()
1029 02:51:58.614560 (+ 29us) write_op.cc:276] Timestamp: P: 1761706309843996 usec, L: 0
1029 02:51:58.631084 (+ 16524us) write_op.cc:270] Start()
1029 02:51:58.631114 (+ 30us) write_op.cc:276] Timestamp: P: 1761706309844517 usec, L: 0
1029 02:51:58.631282 (+ 168us) write_op.cc:270] Start()
1029 02:51:58.631300 (+ 18us) write_op.cc:276] Timestamp: P: 1761706309845001 usec, L: 0
1029 02:51:58.631424 (+ 124us) write_op.cc:270] Start()
1029 02:51:58.631442 (+ 18us) write_op.cc:276] Timestamp: P: 1761706309845796 usec, L: 0
1029 02:51:58.631628 (+ 186us) write_op.cc:270] Start()
1029 02:51:58.631647 (+ 19us) write_op.cc:276] Timestamp: P: 1761706309847117 usec, L: 0
1029 02:51:58.631796 (+ 149us) write_op.cc:270] Start()
1029 02:51:58.631815 (+ 19us) write_op.cc:276] Timestamp: P: 1761706309869703 usec, L: 0
1029 02:51:58.631940 (+ 125us) write_op.cc:270] Start()
1029 02:51:58.631959 (+ 19us) write_op.cc:276] Timestamp: P: 1761706309981146 usec, L: 0
1029 02:51:58.632110 (+ 151us) write_op.cc:270] Start()
1029 02:51:58.632130 (+ 20us) write_op.cc:276] Timestamp: P: 1761706309995927 usec, L: 0
1029 02:51:58.632295 (+ 165us) write_op.cc:270] Start()
1029 02:51:58.632317 (+ 22us) write_op.cc:276] Timestamp: P: 1761706309996811 usec, L: 0
1029 02:51:58.632450 (+ 133us) write_op.cc:270] Start()
1029 02:51:58.632469 (+ 19us) write_op.cc:276] Timestamp: P: 1761706310047077 usec, L: 0
1029 02:51:58.632593 (+ 124us) write_op.cc:270] Start()
1029 02:51:58.632617 (+ 24us) write_op.cc:276] Timestamp: P: 1761706310047906 usec, L: 0
1029 02:51:58.632791 (+ 174us) write_op.cc:270] Start()
1029 02:51:58.632809 (+ 18us) write_op.cc:276] Timestamp: P: 1761706310049087 usec, L: 0
1029 02:51:58.632929 (+ 120us) write_op.cc:270] Start()
1029 02:51:58.632946 (+ 17us) write_op.cc:276] Timestamp: P: 1761706310091423 usec, L: 0
1029 02:51:58.633053 (+ 107us) write_op.cc:270] Start()
1029 02:51:58.633070 (+ 17us) write_op.cc:276] Timestamp: P: 1761706310147788 usec, L: 0
1029 02:51:58.633183 (+ 113us) write_op.cc:270] Start()
1029 02:51:58.633199 (+ 16us) write_op.cc:276] Timestamp: P: 1761706310165368 usec, L: 0
1029 02:51:58.633311 (+ 112us) write_op.cc:270] Start()
1029 02:51:58.633328 (+ 17us) write_op.cc:276] Timestamp: P: 1761706310179292 usec, L: 0
1029 02:51:58.633469 (+ 141us) write_op.cc:270] Start()
1029 02:51:58.633486 (+ 17us) write_op.cc:276] Timestamp: P: 1761706310288057 usec, L: 0
1029 02:51:58.663861 (+ 30375us) write_op.cc:270] Start()
1029 02:51:58.663908 (+ 47us) write_op.cc:276] Timestamp: P: 1761706310288796 usec, L: 0
1029 02:51:58.664141 (+ 233us) write_op.cc:270] Start()
1029 02:51:58.664165 (+ 24us) write_op.cc:276] Timestamp: P: 1761706310289408 usec, L: 0
1029 02:51:58.664313 (+ 148us) write_op.cc:270] Start()
1029 02:51:58.664337 (+ 24us) write_op.cc:276] Timestamp: P: 1761706310289999 usec, L: 0
1029 02:51:58.664473 (+ 136us) write_op.cc:270] Start()
1029 02:51:58.664508 (+ 35us) write_op.cc:276] Timestamp: P: 1761706310291873 usec, L: 0
1029 02:51:58.664663 (+ 155us) write_op.cc:270] Start()
1029 02:51:58.664684 (+ 21us) write_op.cc:276] Timestamp: P: 1761706310292548 usec, L: 0
1029 02:51:58.664813 (+ 129us) write_op.cc:270] Start()
1029 02:51:58.664830 (+ 17us) write_op.cc:276] Timestamp: P: 1761706310293880 usec, L: 0
1029 02:51:58.664979 (+ 149us) write_op.cc:270] Start()
1029 02:51:58.664996 (+ 17us) write_op.cc:276] Timestamp: P: 1761706310294787 usec, L: 0
1029 02:51:58.665124 (+ 128us) write_op.cc:270] Start()
1029 02:51:58.665141 (+ 17us) write_op.cc:276] Timestamp: P: 1761706310319120 usec, L: 0
1029 02:51:58.665251 (+ 110us) write_op.cc:270] Start()
1029 02:51:58.665269 (+ 18us) write_op.cc:276] Timestamp: P: 1761706310320023 usec, L: 0
1029 02:51:58.665403 (+ 134us) write_op.cc:270] Start()
1029 02:51:58.665423 (+ 20us) write_op.cc:276] Timestamp: P: 1761706310320805 usec, L: 0
1029 02:51:58.665560 (+ 137us) write_op.cc:270] Start()
1029 02:51:58.665579 (+ 19us) write_op.cc:276] Timestamp: P: 1761706310323512 usec, L: 0
1029 02:51:58.665709 (+ 130us) write_op.cc:270] Start()
1029 02:51:58.665730 (+ 21us) write_op.cc:276] Timestamp: P: 1761706310362791 usec, L: 0
1029 02:51:58.665907 (+ 177us) write_op.cc:270] Start()
1029 02:51:58.665937 (+ 30us) write_op.cc:276] Timestamp: P: 1761706310363767 usec, L: 0
1029 02:51:58.666058 (+ 121us) write_op.cc:270] Start()
1029 02:51:58.666080 (+ 22us) write_op.cc:276] Timestamp: P: 1761706310396894 usec, L: 0
1029 02:51:58.666376 (+ 296us) write_op.cc:270] Start()
1029 02:51:58.666399 (+ 23us) write_op.cc:276] Timestamp: P: 1761706310399847 usec, L: 0
1029 02:51:58.666555 (+ 156us) write_op.cc:270] Start()
1029 02:51:58.666577 (+ 22us) write_op.cc:276] Timestamp: P: 1761706310445237 usec, L: 0
1029 02:51:58.666724 (+ 147us) write_op.cc:270] Start()
1029 02:51:58.666750 (+ 26us) write_op.cc:276] Timestamp: P: 1761706310455559 usec, L: 0
1029 02:51:58.666996 (+ 246us) write_op.cc:270] Start()
1029 02:51:58.667019 (+ 23us) write_op.cc:276] Timestamp: P: 1761706310456435 usec, L: 0
1029 02:51:58.667156 (+ 137us) write_op.cc:270] Start()
1029 02:51:58.667174 (+ 18us) write_op.cc:276] Timestamp: P: 1761706310466396 usec, L: 0
1029 02:51:58.667297 (+ 123us) write_op.cc:270] Start()
1029 02:51:58.667313 (+ 16us) write_op.cc:276] Timestamp: P: 1761706310485522 usec, L: 0
1029 02:51:58.667424 (+ 111us) write_op.cc:270] Start()
1029 02:51:58.667441 (+ 17us) write_op.cc:276] Timestamp: P: 1761706310491078 usec, L: 0
1029 02:51:58.667561 (+ 120us) write_op.cc:270] Start()
1029 02:51:58.667578 (+ 17us) write_op.cc:276] Timestamp: P: 1761706310504846 usec, L: 0
1029 02:51:58.667709 (+ 131us) write_op.cc:270] Start()
1029 02:51:58.667728 (+ 19us) write_op.cc:276] Timestamp: P: 1761706310505556 usec, L: 0
1029 02:51:58.667850 (+ 122us) write_op.cc:270] Start()
1029 02:51:58.667869 (+ 19us) write_op.cc:276] Timestamp: P: 1761706310525272 usec, L: 0
1029 02:51:58.667984 (+ 115us) write_op.cc:270] Start()
1029 02:51:58.668012 (+ 28us) write_op.cc:276] Timestamp: P: 1761706310527647 usec, L: 0
1029 02:51:58.668123 (+ 111us) write_op.cc:270] Start()
1029 02:51:58.668140 (+ 17us) write_op.cc:276] Timestamp: P: 1761706310543293 usec, L: 0
1029 02:51:58.668263 (+ 123us) write_op.cc:270] Start()
1029 02:51:58.668299 (+ 36us) write_op.cc:276] Timestamp: P: 1761706310581481 usec, L: 0
1029 02:51:58.668449 (+ 150us) write_op.cc:270] Start()
1029 02:51:58.668467 (+ 18us) write_op.cc:276] Timestamp: P: 1761706310583407 usec, L: 0
1029 02:51:58.668598 (+ 131us) write_op.cc:270] Start()
1029 02:51:58.668617 (+ 19us) write_op.cc:276] Timestamp: P: 1761706310589820 usec, L: 0
1029 02:51:58.668751 (+ 134us) write_op.cc:270] Start()
1029 02:51:58.668771 (+ 20us) write_op.cc:276] Timestamp: P: 1761706310672757 usec, L: 0
1029 02:51:58.668936 (+ 165us) write_op.cc:270] Start()
1029 02:51:58.668955 (+ 19us) write_op.cc:276] Timestamp: P: 1761706310673508 usec, L: 0
1029 02:51:58.669097 (+ 142us) write_op.cc:270] Start()
1029 02:51:58.669116 (+ 19us) write_op.cc:276] Timestamp: P: 1761706310674082 usec, L: 0
1029 02:51:58.669251 (+ 135us) write_op.cc:270] Start()
1029 02:51:58.669270 (+ 19us) write_op.cc:276] Timestamp: P: 1761706310674674 usec, L: 0
1029 02:51:58.669388 (+ 118us) write_op.cc:270] Start()
1029 02:51:58.669410 (+ 22us) write_op.cc:276] Timestamp: P: 1761706310675193 usec, L: 0
1029 02:51:58.669522 (+ 112us) write_op.cc:270] Start()
1029 02:51:58.669539 (+ 17us) write_op.cc:276] Timestamp: P: 1761706310675701 usec, L: 0
1029 02:51:58.669657 (+ 118us) write_op.cc:270] Start()
1029 02:51:58.669673 (+ 16us) write_op.cc:276] Timestamp: P: 1761706310676171 usec, L: 0
1029 02:51:58.669782 (+ 109us) write_op.cc:270] Start()
1029 02:51:58.669803 (+ 21us) write_op.cc:276] Timestamp: P: 1761706310737856 usec, L: 0
1029 02:51:58.669923 (+ 120us) write_op.cc:270] Start()
1029 02:51:58.669940 (+ 17us) write_op.cc:276] Timestamp: P: 1761706310760978 usec, L: 0
1029 02:51:58.670056 (+ 116us) write_op.cc:270] Start()
1029 02:51:58.670071 (+ 15us) write_op.cc:276] Timestamp: P: 1761706310772427 usec, L: 0
1029 02:51:58.670177 (+ 106us) write_op.cc:270] Start()
1029 02:51:58.670191 (+ 14us) write_op.cc:276] Timestamp: P: 1761706310783821 usec, L: 0
1029 02:51:58.670311 (+ 120us) write_op.cc:270] Start()
1029 02:51:58.670326 (+ 15us) write_op.cc:276] Timestamp: P: 1761706310807159 usec, L: 0
1029 02:51:58.670433 (+ 107us) write_op.cc:270] Start()
1029 02:51:58.670448 (+ 15us) write_op.cc:276] Timestamp: P: 1761706310813170 usec, L: 0
1029 02:51:58.670580 (+ 132us) write_op.cc:270] Start()
1029 02:51:58.670604 (+ 24us) write_op.cc:276] Timestamp: P: 1761706310815061 usec, L: 0
1029 02:51:58.670763 (+ 159us) write_op.cc:270] Start()
1029 02:51:58.670786 (+ 23us) write_op.cc:276] Timestamp: P: 1761706310837272 usec, L: 0
1029 02:51:58.670927 (+ 141us) write_op.cc:270] Start()
1029 02:51:58.670945 (+ 18us) write_op.cc:276] Timestamp: P: 1761706310852964 usec, L: 0
1029 02:51:58.671081 (+ 136us) write_op.cc:270] Start()
1029 02:51:58.671139 (+ 58us) write_op.cc:276] Timestamp: P: 1761706310863180 usec, L: 0
1029 02:51:58.671325 (+ 186us) write_op.cc:270] Start()
1029 02:51:58.671343 (+ 18us) write_op.cc:276] Timestamp: P: 1761706310876402 usec, L: 0
1029 02:51:58.671497 (+ 154us) write_op.cc:270] Start()
1029 02:51:58.671520 (+ 23us) write_op.cc:276] Timestamp: P: 1761706310880933 usec, L: 0
1029 02:51:58.671661 (+ 141us) write_op.cc:270] Start()
1029 02:51:58.671698 (+ 37us) write_op.cc:276] Timestamp: P: 1761706310898113 usec, L: 0
1029 02:51:58.671867 (+ 169us) write_op.cc:270] Start()
1029 02:51:58.671890 (+ 23us) write_op.cc:276] Timestamp: P: 1761706310899414 usec, L: 0
1029 02:51:58.672026 (+ 136us) write_op.cc:270] Start()
1029 02:51:58.672050 (+ 24us) write_op.cc:276] Timestamp: P: 1761706310900356 usec, L: 0
1029 02:51:58.672202 (+ 152us) write_op.cc:270] Start()
1029 02:51:58.672222 (+ 20us) write_op.cc:276] Timestamp: P: 1761706310901462 usec, L: 0
1029 02:51:58.672364 (+ 142us) write_op.cc:270] Start()
1029 02:51:58.672388 (+ 24us) write_op.cc:276] Timestamp: P: 1761706310902358 usec, L: 0
1029 02:51:58.672525 (+ 137us) write_op.cc:270] Start()
1029 02:51:58.672545 (+ 20us) write_op.cc:276] Timestamp: P: 1761706310903195 usec, L: 0
1029 02:51:58.672687 (+ 142us) write_op.cc:270] Start()
1029 02:51:58.672711 (+ 24us) write_op.cc:276] Timestamp: P: 1761706310941194 usec, L: 0
1029 02:51:58.672870 (+ 159us) write_op.cc:270] Start()
1029 02:51:58.672891 (+ 21us) write_op.cc:276] Timestamp: P: 1761706310946718 usec, L: 0
1029 02:51:58.673054 (+ 163us) write_op.cc:270] Start()
1029 02:51:58.673074 (+ 20us) write_op.cc:276] Timestamp: P: 1761706310947642 usec, L: 0
1029 02:51:58.673259 (+ 185us) write_op.cc:270] Start()
1029 02:51:58.673283 (+ 24us) write_op.cc:276] Timestamp: P: 1761706310948329 usec, L: 0
1029 02:51:58.673431 (+ 148us) write_op.cc:270] Start()
1029 02:51:58.673456 (+ 25us) write_op.cc:276] Timestamp: P: 1761706310948876 usec, L: 0
1029 02:51:58.673616 (+ 160us) write_op.cc:270] Start()
1029 02:51:58.673641 (+ 25us) write_op.cc:276] Timestamp: P: 1761706310966184 usec, L: 0
1029 02:51:58.673799 (+ 158us) write_op.cc:270] Start()
1029 02:51:58.673822 (+ 23us) write_op.cc:276] Timestamp: P: 1761706310992619 usec, L: 0
1029 02:51:58.673977 (+ 155us) write_op.cc:270] Start()
1029 02:51:58.673997 (+ 20us) write_op.cc:276] Timestamp: P: 1761706310994383 usec, L: 0
1029 02:51:58.674188 (+ 191us) write_op.cc:270] Start()
1029 02:51:58.674219 (+ 31us) write_op.cc:276] Timestamp: P: 1761706310995383 usec, L: 0
1029 02:51:58.674372 (+ 153us) write_op.cc:270] Start()
1029 02:51:58.674391 (+ 19us) write_op.cc:276] Timestamp: P: 1761706311011711 usec, L: 0
1029 02:51:58.674531 (+ 140us) write_op.cc:270] Start()
1029 02:51:58.674552 (+ 21us) write_op.cc:276] Timestamp: P: 1761706311077266 usec, L: 0
1029 02:51:58.674699 (+ 147us) write_op.cc:270] Start()
1029 02:51:58.674717 (+ 18us) write_op.cc:276] Timestamp: P: 1761706311078012 usec, L: 0
1029 02:51:58.674844 (+ 127us) write_op.cc:270] Start()
1029 02:51:58.674860 (+ 16us) write_op.cc:276] Timestamp: P: 1761706311078692 usec, L: 0
1029 02:51:58.674992 (+ 132us) write_op.cc:270] Start()
1029 02:51:58.675011 (+ 19us) write_op.cc:276] Timestamp: P: 1761706311079239 usec, L: 0
1029 02:51:58.675134 (+ 123us) write_op.cc:270] Start()
1029 02:51:58.675151 (+ 17us) write_op.cc:276] Timestamp: P: 1761706311079761 usec, L: 0
1029 02:51:58.675258 (+ 107us) write_op.cc:270] Start()
1029 02:51:58.675276 (+ 18us) write_op.cc:276] Timestamp: P: 1761706311080346 usec, L: 0
1029 02:51:58.675402 (+ 126us) write_op.cc:270] Start()
1029 02:51:58.675419 (+ 17us) write_op.cc:276] Timestamp: P: 1761706311080870 usec, L: 0
1029 02:51:58.675530 (+ 111us) write_op.cc:270] Start()
1029 02:51:58.675547 (+ 17us) write_op.cc:276] Timestamp: P: 1761706311094084 usec, L: 0
1029 02:51:58.808856 (+133309us) write_op.cc:270] Start()
1029 02:51:58.808891 (+ 35us) write_op.cc:276] Timestamp: P: 1761706311160905 usec, L: 0
1029 02:51:58.809091 (+ 200us) write_op.cc:270] Start()
1029 02:51:58.809114 (+ 23us) write_op.cc:276] Timestamp: P: 1761706311161772 usec, L: 0
1029 02:51:58.809267 (+ 153us) write_op.cc:270] Start()
1029 02:51:58.809288 (+ 21us) write_op.cc:276] Timestamp: P: 1761706311170364 usec, L: 0
1029 02:51:58.809445 (+ 157us) write_op.cc:270] Start()
1029 02:51:58.809466 (+ 21us) write_op.cc:276] Timestamp: P: 1761706311181048 usec, L: 0
1029 02:51:58.809609 (+ 143us) write_op.cc:270] Start()
1029 02:51:58.809628 (+ 19us) write_op.cc:276] Timestamp: P: 1761706311181783 usec, L: 0
1029 02:51:58.809782 (+ 154us) write_op.cc:270] Start()
1029 02:51:58.809807 (+ 25us) write_op.cc:276] Timestamp: P: 1761706311205353 usec, L: 0
1029 02:51:58.809974 (+ 167us) write_op.cc:270] Start()
1029 02:51:58.809998 (+ 24us) write_op.cc:276] Timestamp: P: 1761706311206641 usec, L: 0
1029 02:51:58.822110 (+ 12112us) write_op.cc:270] Start()
1029 02:51:58.822147 (+ 37us) write_op.cc:276] Timestamp: P: 1761706311207506 usec, L: 0
1029 02:51:58.822350 (+ 203us) write_op.cc:270] Start()
1029 02:51:58.822373 (+ 23us) write_op.cc:276] Timestamp: P: 1761706311209559 usec, L: 0
1029 02:51:58.822552 (+ 179us) write_op.cc:270] Start()
1029 02:51:58.822577 (+ 25us) write_op.cc:276] Timestamp: P: 1761706311254167 usec, L: 0
1029 02:51:58.822745 (+ 168us) write_op.cc:270] Start()
1029 02:51:58.822769 (+ 24us) write_op.cc:276] Timestamp: P: 1761706311279069 usec, L: 0
1029 02:51:58.822918 (+ 149us) write_op.cc:270] Start()
1029 02:51:58.822941 (+ 23us) write_op.cc:276] Timestamp: P: 1761706311293907 usec, L: 0
1029 02:51:58.823093 (+ 152us) write_op.cc:270] Start()
1029 02:51:58.823116 (+ 23us) write_op.cc:276] Timestamp: P: 1761706311315201 usec, L: 0
1029 02:51:58.823249 (+ 133us) write_op.cc:270] Start()
1029 02:51:58.851684 (+ 28435us) write_op.cc:276] Timestamp: P: 1761706311338317 usec, L: 0
1029 02:51:58.851979 (+ 295us) write_op.cc:270] Start()
1029 02:51:58.852006 (+ 27us) write_op.cc:276] Timestamp: P: 1761706311381497 usec, L: 0
1029 02:51:58.852132 (+ 126us) write_op.cc:270] Start()
1029 02:51:58.852152 (+ 20us) write_op.cc:276] Timestamp: P: 1761706311395783 usec, L: 0
1029 02:51:58.852287 (+ 135us) write_op.cc:270] Start()
1029 02:51:58.852308 (+ 21us) write_op.cc:276] Timestamp: P: 1761706311423972 usec, L: 0
1029 02:51:58.852439 (+ 131us) write_op.cc:270] Start()
1029 02:51:58.852458 (+ 19us) write_op.cc:276] Timestamp: P: 1761706311426370 usec, L: 0
1029 02:51:58.852583 (+ 125us) write_op.cc:270] Start()
1029 02:51:58.852602 (+ 19us) write_op.cc:276] Timestamp: P: 1761706311444220 usec, L: 0
1029 02:51:58.852748 (+ 146us) write_op.cc:270] Start()
1029 02:51:58.852778 (+ 30us) write_op.cc:276] Timestamp: P: 1761706311456222 usec, L: 0
1029 02:51:58.852919 (+ 141us) write_op.cc:270] Start()
1029 02:51:58.852938 (+ 19us) write_op.cc:276] Timestamp: P: 1761706311461258 usec, L: 0
1029 02:51:58.853067 (+ 129us) write_op.cc:270] Start()
1029 02:51:58.853085 (+ 18us) write_op.cc:276] Timestamp: P: 1761706311480560 usec, L: 0
1029 02:51:58.853221 (+ 136us) write_op.cc:270] Start()
1029 02:51:58.853242 (+ 21us) write_op.cc:276] Timestamp: P: 1761706311482349 usec, L: 0
1029 02:51:58.853394 (+ 152us) write_op.cc:270] Start()
1029 02:51:58.853425 (+ 31us) write_op.cc:276] Timestamp: P: 1761706311483576 usec, L: 0
1029 02:51:58.853576 (+ 151us) write_op.cc:270] Start()
1029 02:51:58.853598 (+ 22us) write_op.cc:276] Timestamp: P: 1761706311526254 usec, L: 0
1029 02:51:58.853769 (+ 171us) write_op.cc:270] Start()
1029 02:51:58.853807 (+ 38us) write_op.cc:276] Timestamp: P: 1761706311535638 usec, L: 0
1029 02:51:58.853933 (+ 126us) write_op.cc:270] Start()
1029 02:51:58.853956 (+ 23us) write_op.cc:276] Timestamp: P: 1761706311545534 usec, L: 0
1029 02:51:58.854092 (+ 136us) write_op.cc:270] Start()
1029 02:51:58.854115 (+ 23us) write_op.cc:276] Timestamp: P: 1761706311548459 usec, L: 0
1029 02:51:58.854248 (+ 133us) write_op.cc:270] Start()
1029 02:51:58.854269 (+ 21us) write_op.cc:276] Timestamp: P: 1761706311554649 usec, L: 0
1029 02:51:58.854388 (+ 119us) write_op.cc:270] Start()
1029 02:51:58.854407 (+ 19us) write_op.cc:276] Timestamp: P: 1761706311555792 usec, L: 0
1029 02:51:58.854608 (+ 201us) write_op.cc:270] Start()
1029 02:51:58.854628 (+ 20us) write_op.cc:276] Timestamp: P: 1761706311570877 usec, L: 0
1029 02:51:58.854758 (+ 130us) write_op.cc:270] Start()
1029 02:51:58.854779 (+ 21us) write_op.cc:276] Timestamp: P: 1761706311573928 usec, L: 0
1029 02:51:58.854906 (+ 127us) write_op.cc:270] Start()
1029 02:51:58.854923 (+ 17us) write_op.cc:276] Timestamp: P: 1761706311575684 usec, L: 0
1029 02:51:58.855036 (+ 113us) write_op.cc:270] Start()
1029 02:51:58.855053 (+ 17us) write_op.cc:276] Timestamp: P: 1761706311576923 usec, L: 0
1029 02:51:58.855170 (+ 117us) write_op.cc:270] Start()
1029 02:51:58.855188 (+ 18us) write_op.cc:276] Timestamp: P: 1761706311577811 usec, L: 0
1029 02:51:58.855313 (+ 125us) write_op.cc:270] Start()
1029 02:51:58.855331 (+ 18us) write_op.cc:276] Timestamp: P: 1761706311586508 usec, L: 0
1029 02:51:58.855479 (+ 148us) write_op.cc:270] Start()
1029 02:51:58.855499 (+ 20us) write_op.cc:276] Timestamp: P: 1761706311588287 usec, L: 0
1029 02:51:58.855622 (+ 123us) write_op.cc:270] Start()
1029 02:51:58.855640 (+ 18us) write_op.cc:276] Timestamp: P: 1761706311590760 usec, L: 0
1029 02:51:58.855775 (+ 135us) write_op.cc:270] Start()
1029 02:51:58.855795 (+ 20us) write_op.cc:276] Timestamp: P: 1761706311606142 usec, L: 0
1029 02:51:58.855930 (+ 135us) write_op.cc:270] Start()
1029 02:51:58.855949 (+ 19us) write_op.cc:276] Timestamp: P: 1761706311620808 usec, L: 0
1029 02:51:58.856081 (+ 132us) write_op.cc:270] Start()
1029 02:51:58.856118 (+ 37us) write_op.cc:276] Timestamp: P: 1761706311647395 usec, L: 0
1029 02:51:58.856253 (+ 135us) write_op.cc:270] Start()
1029 02:51:58.856275 (+ 22us) write_op.cc:276] Timestamp: P: 1761706311654811 usec, L: 0
1029 02:51:58.856398 (+ 123us) write_op.cc:270] Start()
1029 02:51:58.856433 (+ 35us) write_op.cc:276] Timestamp: P: 1761706311655435 usec, L: 0
1029 02:51:58.856570 (+ 137us) write_op.cc:270] Start()
1029 02:51:58.856593 (+ 23us) write_op.cc:276] Timestamp: P: 1761706311658760 usec, L: 0
1029 02:51:58.856743 (+ 150us) write_op.cc:270] Start()
1029 02:51:58.856762 (+ 19us) write_op.cc:276] Timestamp: P: 1761706311666938 usec, L: 0
1029 02:51:58.856889 (+ 127us) write_op.cc:270] Start()
1029 02:51:58.856911 (+ 22us) write_op.cc:276] Timestamp: P: 1761706311698085 usec, L: 0
1029 02:51:58.857056 (+ 145us) write_op.cc:270] Start()
1029 02:51:58.857076 (+ 20us) write_op.cc:276] Timestamp: P: 1761706311704419 usec, L: 0
1029 02:51:58.857210 (+ 134us) write_op.cc:270] Start()
1029 02:51:58.857230 (+ 20us) write_op.cc:276] Timestamp: P: 1761706311705104 usec, L: 0
1029 02:51:58.857406 (+ 176us) write_op.cc:270] Start()
1029 02:51:58.857439 (+ 33us) write_op.cc:276] Timestamp: P: 1761706311705746 usec, L: 0
1029 02:51:58.857586 (+ 147us) write_op.cc:270] Start()
1029 02:51:58.857607 (+ 21us) write_op.cc:276] Timestamp: P: 1761706311753545 usec, L: 0
1029 02:51:58.857733 (+ 126us) write_op.cc:270] Start()
1029 02:51:58.857754 (+ 21us) write_op.cc:276] Timestamp: P: 1761706311769698 usec, L: 0
1029 02:51:58.857894 (+ 140us) write_op.cc:270] Start()
1029 02:51:58.857913 (+ 19us) write_op.cc:276] Timestamp: P: 1761706311782630 usec, L: 0
1029 02:51:58.858058 (+ 145us) write_op.cc:270] Start()
1029 02:51:58.858078 (+ 20us) write_op.cc:276] Timestamp: P: 1761706311791194 usec, L: 0
1029 02:51:58.858217 (+ 139us) write_op.cc:270] Start()
1029 02:51:58.858238 (+ 21us) write_op.cc:276] Timestamp: P: 1761706311791882 usec, L: 0
1029 02:51:58.858391 (+ 153us) write_op.cc:270] Start()
1029 02:51:58.858424 (+ 33us) write_op.cc:276] Timestamp: P: 1761706311792413 usec, L: 0
1029 02:51:58.858979 (+ 555us) write_op.cc:270] Start()
1029 02:51:58.859005 (+ 26us) write_op.cc:276] Timestamp: P: 1761706311795164 usec, L: 0
1029 02:51:58.859171 (+ 166us) write_op.cc:270] Start()
1029 02:51:58.859193 (+ 22us) write_op.cc:276] Timestamp: P: 1761706311811845 usec, L: 0
1029 02:51:58.859339 (+ 146us) write_op.cc:270] Start()
1029 02:51:58.859360 (+ 21us) write_op.cc:276] Timestamp: P: 1761706311813766 usec, L: 0
1029 02:51:58.859504 (+ 144us) write_op.cc:270] Start()
1029 02:51:58.859527 (+ 23us) write_op.cc:276] Timestamp: P: 1761706311815177 usec, L: 0
1029 02:51:58.859665 (+ 138us) write_op.cc:270] Start()
1029 02:51:58.859688 (+ 23us) write_op.cc:276] Timestamp: P: 1761706311816222 usec, L: 0
1029 02:51:58.859835 (+ 147us) write_op.cc:270] Start()
1029 02:51:58.859856 (+ 21us) write_op.cc:276] Timestamp: P: 1761706311826518 usec, L: 0
1029 02:51:58.860041 (+ 185us) write_op.cc:270] Start()
1029 02:51:58.860099 (+ 58us) write_op.cc:276] Timestamp: P: 1761706311833884 usec, L: 0
1029 02:51:58.860294 (+ 195us) write_op.cc:270] Start()
1029 02:51:58.860318 (+ 24us) write_op.cc:276] Timestamp: P: 1761706311844602 usec, L: 0
1029 02:51:58.860481 (+ 163us) write_op.cc:270] Start()
1029 02:51:58.860499 (+ 18us) write_op.cc:276] Timestamp: P: 1761706311860631 usec, L: 0
1029 02:51:58.860622 (+ 123us) write_op.cc:270] Start()
1029 02:51:58.860639 (+ 17us) write_op.cc:276] Timestamp: P: 1761706311872705 usec, L: 0
1029 02:51:58.860784 (+ 145us) write_op.cc:270] Start()
1029 02:51:58.860804 (+ 20us) write_op.cc:276] Timestamp: P: 1761706311920553 usec, L: 0
1029 02:51:58.860929 (+ 125us) write_op.cc:270] Start()
1029 02:51:58.860949 (+ 20us) write_op.cc:276] Timestamp: P: 1761706311945825 usec, L: 0
1029 02:51:58.861080 (+ 131us) write_op.cc:270] Start()
1029 02:51:58.861099 (+ 19us) write_op.cc:276] Timestamp: P: 1761706311963932 usec, L: 0
1029 02:51:58.861235 (+ 136us) write_op.cc:270] Start()
1029 02:51:58.861256 (+ 21us) write_op.cc:276] Timestamp: P: 1761706311972420 usec, L: 0
1029 02:51:58.861374 (+ 118us) write_op.cc:270] Start()
1029 02:51:58.861391 (+ 17us) write_op.cc:276] Timestamp: P: 1761706311983374 usec, L: 0
1029 02:51:58.861569 (+ 178us) write_op.cc:270] Start()
1029 02:51:58.861590 (+ 21us) write_op.cc:276] Timestamp: P: 1761706311992196 usec, L: 0
1029 02:51:58.861722 (+ 132us) write_op.cc:270] Start()
1029 02:51:58.861741 (+ 19us) write_op.cc:276] Timestamp: P: 1761706312027288 usec, L: 0
1029 02:51:58.861908 (+ 167us) write_op.cc:270] Start()
1029 02:51:58.861931 (+ 23us) write_op.cc:276] Timestamp: P: 1761706312037539 usec, L: 0
1029 02:51:58.862075 (+ 144us) write_op.cc:270] Start()
1029 02:51:58.862095 (+ 20us) write_op.cc:276] Timestamp: P: 1761706312046594 usec, L: 0
1029 02:51:58.862238 (+ 143us) write_op.cc:270] Start()
1029 02:51:58.862259 (+ 21us) write_op.cc:276] Timestamp: P: 1761706312047226 usec, L: 0
1029 02:51:58.862404 (+ 145us) write_op.cc:270] Start()
1029 02:51:58.862438 (+ 34us) write_op.cc:276] Timestamp: P: 1761706312053660 usec, L: 0
1029 02:51:58.987715 (+125277us) write_op.cc:270] Start()
1029 02:51:58.987750 (+ 35us) write_op.cc:276] Timestamp: P: 1761706312058840 usec, L: 0
1029 02:51:58.987954 (+ 204us) write_op.cc:270] Start()
1029 02:51:58.987977 (+ 23us) write_op.cc:276] Timestamp: P: 1761706312088462 usec, L: 0
1029 02:51:58.988111 (+ 134us) write_op.cc:270] Start()
1029 02:51:58.988130 (+ 19us) write_op.cc:276] Timestamp: P: 1761706312103457 usec, L: 0
1029 02:51:58.988254 (+ 124us) write_op.cc:270] Start()
1029 02:51:58.988273 (+ 19us) write_op.cc:276] Timestamp: P: 1761706312114189 usec, L: 0
1029 02:51:58.988466 (+ 193us) write_op.cc:270] Start()
1029 02:51:58.988487 (+ 21us) write_op.cc:276] Timestamp: P: 1761706312131224 usec, L: 0
1029 02:51:58.988628 (+ 141us) write_op.cc:270] Start()
1029 02:51:58.988652 (+ 24us) write_op.cc:276] Timestamp: P: 1761706312139993 usec, L: 0
1029 02:51:58.988796 (+ 144us) write_op.cc:270] Start()
1029 02:51:58.988818 (+ 22us) write_op.cc:276] Timestamp: P: 1761706312150731 usec, L: 0
1029 02:51:58.988957 (+ 139us) write_op.cc:270] Start()
1029 02:51:58.988978 (+ 21us) write_op.cc:276] Timestamp: P: 1761706312184830 usec, L: 0
1029 02:51:58.989137 (+ 159us) write_op.cc:270] Start()
1029 02:51:58.989158 (+ 21us) write_op.cc:276] Timestamp: P: 1761706312201425 usec, L: 0
1029 02:51:58.989295 (+ 137us) write_op.cc:270] Start()
1029 02:51:58.989318 (+ 23us) write_op.cc:276] Timestamp: P: 1761706312250910 usec, L: 0
1029 02:51:58.989492 (+ 174us) write_op.cc:270] Start()
1029 02:51:58.989514 (+ 22us) write_op.cc:276] Timestamp: P: 1761706312251664 usec, L: 0
1029 02:51:58.989657 (+ 143us) write_op.cc:270] Start()
1029 02:51:58.989685 (+ 28us) write_op.cc:276] Timestamp: P: 1761706312252271 usec, L: 0
1029 02:51:58.989806 (+ 121us) write_op.cc:270] Start()
1029 02:51:58.989824 (+ 18us) write_op.cc:276] Timestamp: P: 1761706312302892 usec, L: 0
1029 02:51:58.989961 (+ 137us) write_op.cc:270] Start()
1029 02:51:58.989979 (+ 18us) write_op.cc:276] Timestamp: P: 1761706312304332 usec, L: 0
1029 02:51:58.990091 (+ 112us) write_op.cc:270] Start()
1029 02:51:58.990108 (+ 17us) write_op.cc:276] Timestamp: P: 1761706312305027 usec, L: 0
1029 02:51:58.990238 (+ 130us) write_op.cc:270] Start()
1029 02:51:58.990258 (+ 20us) write_op.cc:276] Timestamp: P: 17617063
I20251029 02:52:01.038857 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d51120775424146b6ab3e6d2e4710bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:01.039223 1704 catalog_manager.cc:1584] T 00000000000000000000000000000000 P e5e5222b26fe4145948620e51edf53fb: acquiring CA information for follower catalog manager: success
W20251029 02:52:01.040400 1741 tablet_replica.cc:1410] Time spent flushing the Log queue took a long time: real 0.229s user 0.007s sys 0.001s
I20251029 02:52:01.043331 1745 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:37626 (request call id 484) took 1937 ms. Trace:
I20251029 02:52:01.043463 1745 rpcz_store.cc:276] 1029 02:51:59.105863 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:59.453379 (+347516us) service_pool.cc:225] Handling call
1029 02:52:01.043267 (+1589888us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1029 02:51:59.566636 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
1029 02:51:59.576670 (+ 10034us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
1029 02:51:59.576679 (+ 9us) alter_schema_op.cc:66] Acquired schema lock
1029 02:51:59.576686 (+ 7us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
1029 02:51:59.576735 (+ 49us) alter_schema_op.cc:128] START. Timestamp: 1761706319576677
1029 02:51:59.576739 (+ 4us) op_driver.cc:348] REPLICATION: starting
1029 02:51:59.583213 (+ 6474us) log.cc:844] Serialized 173 byte log entry
1029 02:52:00.688842 (+1105629us) op_driver.cc:464] REPLICATION: finished
1029 02:52:00.814830 (+125988us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
1029 02:52:00.834004 (+ 19174us) log.cc:844] Serialized 5 byte log entry
1029 02:52:01.042885 (+208881us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:01.043182 (+ 297us) log.cc:844] Serialized 14 byte log entry
1029 02:52:01.043223 (+ 41us) alter_schema_op.cc:73] Released schema lock
1029 02:52:01.043228 (+ 5us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":125897,"prepare.queue_time_us":106916,"prepare.run_cpu_time_us":137341,"prepare.run_wall_time_us":374150,"raft.queue_time_us":59,"raft.run_cpu_time_us":442,"raft.run_wall_time_us":442,"replication_time_us":1112059,"thread_start_us":592054,"threads_started":3,"wal-append.queue_time_us":344734}]]}
I20251029 02:52:01.049831 1439 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "e9261eb85ea443d5a3beb36bd0c549f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9672114a6a742e2864c16ff60ba246e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42343 } } peers { permanent_uuid: "e9261eb85ea443d5a3beb36bd0c549f0" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34239 } } peers { permanent_uuid: "e5e5222b26fe4145948620e51edf53fb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41233 } } }
I20251029 02:52:01.050858 1439 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0 [sys.catalog]: This master's current role is: LEADER
I20251029 02:52:01.055400 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aad32e3581d94cc384c1938d881063fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:01.075524 1026 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:37626 (request call id 531) took 1740 ms. Trace:
I20251029 02:52:01.075681 1026 rpcz_store.cc:276] 1029 02:51:59.334328 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:00.563992 (+1229664us) service_pool.cc:225] Handling call
1029 02:52:00.564378 (+ 386us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1029 02:52:00.564397 (+ 19us) ts_tablet_manager.cc:682] Creating new metadata...
1029 02:52:00.680682 (+116285us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:01.027070 (+346388us) tablet_replica.cc:206] Creating consensus instance
1029 02:52:01.074359 (+ 47289us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251029 02:52:01.059969 1174 ts_tablet_manager.cc:1929] T f71388036478404d878a52fc20ece216 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:01.076969 1174 log.cc:1199] T f71388036478404d878a52fc20ece216 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/f71388036478404d878a52fc20ece216
I20251029 02:52:01.077376 1174 ts_tablet_manager.cc:1950] T f71388036478404d878a52fc20ece216 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:01.078792 1174 tablet_replica.cc:333] T 012ce243fd07466ab5301d1c9ce4b413 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:01.078935 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet f71388036478404d878a52fc20ece216 (table default.Table_03f7d6f02b064939ba7464c5173e02e3 [id=9898cffc2b964059ba3163ee4d1395c2]) successfully deleted
I20251029 02:52:01.079031 1174 raft_consensus.cc:2243] T 012ce243fd07466ab5301d1c9ce4b413 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:01.079633 1174 raft_consensus.cc:2272] T 012ce243fd07466ab5301d1c9ce4b413 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:01.080228 1174 ts_tablet_manager.cc:1916] T 012ce243fd07466ab5301d1c9ce4b413 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:01.086702 1028 tablet_service.cc:1505] Processing CreateTablet for tablet 9a58ecfa646c41cdae4d75724767d0d9 (DEFAULT_TABLE table=default.Table_3c09e84f78ee464b823fe6c104c2e207 [id=4b62bbd2d37949aabd475744524eb11a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:01.040009 1167 tablet_bootstrap.cc:654] T 75236b2fa82f4f28b82efae9a2c5160d P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:01.092118 1026 tablet_service.cc:1505] Processing CreateTablet for tablet c73059366f4b45528f8b907263c4d085 (DEFAULT_TABLE table=default.Table_daad6fda420f4659bc87d5991d60cf2f [id=9cd9b155c5ad467c8da68c8c03a68206]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:01.092483 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a58ecfa646c41cdae4d75724767d0d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:01.093612 1041 tablet_service.cc:1505] Processing CreateTablet for tablet ac162bce5b6e4080839095a045ef0cc0 (DEFAULT_TABLE table=default.Table_104d5aaf444b4e5e813288013737aa15 [id=991375eeb3f14176bd84a23bd3366566]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:01.102214 1704 catalog_manager.cc:1612] T 00000000000000000000000000000000 P e5e5222b26fe4145948620e51edf53fb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251029 02:52:01.103361 1717 raft_consensus.cc:493] T dbc2092b63e0417997565c8fadb53ae0 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:01.105234 1717 raft_consensus.cc:515] T dbc2092b63e0417997565c8fadb53ae0 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:01.106694 1717 leader_election.cc:290] T dbc2092b63e0417997565c8fadb53ae0 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:52:01.108268 918 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dbc2092b63e0417997565c8fadb53ae0" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:01.109248 1175 ts_tablet_manager.cc:1916] T 9e07291d6b5645feb4d5890f58e1ef74 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 02:52:01.113575 711 leader_election.cc:343] T dbc2092b63e0417997565c8fadb53ae0 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:01.112533 1039 tablet_service.cc:1505] Processing CreateTablet for tablet eaf4018c0d624b1cb8c6528eea7b689e (DEFAULT_TABLE table=default.Table_aedb5a569ca74ef2a9092a847ad5a3f6 [id=16ad7720626546e6a63766039d0b36c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:01.120045 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac162bce5b6e4080839095a045ef0cc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:01.122014 1717 raft_consensus.cc:493] T 51effc9324034ae79e2b25cb57a996c1 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:01.122160 1717 raft_consensus.cc:515] T 51effc9324034ae79e2b25cb57a996c1 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:01.122932 1717 leader_election.cc:290] T 51effc9324034ae79e2b25cb57a996c1 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:01.123672 918 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51effc9324034ae79e2b25cb57a996c1" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:01.124197 711 leader_election.cc:343] T 51effc9324034ae79e2b25cb57a996c1 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:01.125128 1741 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:37626 (request call id 432) took 3012 ms. Trace:
I20251029 02:52:01.131810 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c73059366f4b45528f8b907263c4d085. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:01.134742 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eaf4018c0d624b1cb8c6528eea7b689e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:01.135274 1736 tablet.cc:1722] T 70e8cfe5eaa9492c85874dc3214a1e0b P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 02:52:01.136837 1167 tablet_bootstrap.cc:492] T 75236b2fa82f4f28b82efae9a2c5160d P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:01.136834 1057 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dbc2092b63e0417997565c8fadb53ae0" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:01.137015 1167 ts_tablet_manager.cc:1403] T 75236b2fa82f4f28b82efae9a2c5160d P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.134s user 0.004s sys 0.001s
I20251029 02:52:01.146503 1167 raft_consensus.cc:359] T 75236b2fa82f4f28b82efae9a2c5160d P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:01.146752 1167 raft_consensus.cc:385] T 75236b2fa82f4f28b82efae9a2c5160d P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:01.146842 1167 raft_consensus.cc:740] T 75236b2fa82f4f28b82efae9a2c5160d P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
W20251029 02:52:01.147816 712 leader_election.cc:343] T dbc2092b63e0417997565c8fadb53ae0 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:01.147958 712 leader_election.cc:304] T dbc2092b63e0417997565c8fadb53ae0 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:01.148504 1717 raft_consensus.cc:2749] T dbc2092b63e0417997565c8fadb53ae0 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:01.150238 1174 ts_tablet_manager.cc:1929] T 012ce243fd07466ab5301d1c9ce4b413 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:01.150389 1174 log.cc:1199] T 012ce243fd07466ab5301d1c9ce4b413 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/012ce243fd07466ab5301d1c9ce4b413
I20251029 02:52:01.157193 1172 tablet_bootstrap.cc:492] T 24cb2a71ba3841d4a5e3e5c6fe99ae2f P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:01.157377 1172 ts_tablet_manager.cc:1403] T 24cb2a71ba3841d4a5e3e5c6fe99ae2f P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.152s user 0.000s sys 0.005s
I20251029 02:52:01.158408 1057 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51effc9324034ae79e2b25cb57a996c1" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:01.158587 1172 raft_consensus.cc:359] T 24cb2a71ba3841d4a5e3e5c6fe99ae2f P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:01.158959 1172 raft_consensus.cc:385] T 24cb2a71ba3841d4a5e3e5c6fe99ae2f P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251029 02:52:01.159096 712 leader_election.cc:343] T 51effc9324034ae79e2b25cb57a996c1 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:01.159200 712 leader_election.cc:304] T 51effc9324034ae79e2b25cb57a996c1 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:01.159591 1172 raft_consensus.cc:740] T 24cb2a71ba3841d4a5e3e5c6fe99ae2f P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:01.160030 1172 consensus_queue.cc:260] T 24cb2a71ba3841d4a5e3e5c6fe99ae2f P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:01.160317 1717 raft_consensus.cc:2749] T 51effc9324034ae79e2b25cb57a996c1 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:01.160630 1172 ts_tablet_manager.cc:1434] T 24cb2a71ba3841d4a5e3e5c6fe99ae2f P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20251029 02:52:01.161425 1167 consensus_queue.cc:260] T 75236b2fa82f4f28b82efae9a2c5160d P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:01.160943 1172 tablet_bootstrap.cc:492] T 576c4e6e70e24b0399ac9597a30f0569 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:01.142094 1741 rpcz_store.cc:276] 1029 02:51:58.098502 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:58.098583 (+ 81us) service_pool.cc:225] Handling call
1029 02:52:01.110941 (+3012358us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1029 02:51:58.362458 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
1029 02:51:58.440915 (+ 78457us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
1029 02:51:58.440929 (+ 14us) alter_schema_op.cc:66] Acquired schema lock
1029 02:51:58.440939 (+ 10us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
1029 02:51:58.457715 (+ 16776us) alter_schema_op.cc:128] START. Timestamp: 1761706318457650
1029 02:51:58.457721 (+ 6us) op_driver.cc:348] REPLICATION: starting
1029 02:51:58.458054 (+ 333us) log.cc:844] Serialized 173 byte log entry
1029 02:52:00.566043 (+2107989us) op_driver.cc:464] REPLICATION: finished
1029 02:52:00.686056 (+120013us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
1029 02:52:00.825441 (+139385us) log.cc:844] Serialized 5 byte log entry
1029 02:52:01.102420 (+276979us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:01.110824 (+ 8404us) log.cc:844] Serialized 14 byte log entry
1029 02:52:01.110899 (+ 75us) alter_schema_op.cc:73] Released schema lock
1029 02:52:01.110904 (+ 5us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":115768,"prepare.queue_time_us":259645,"prepare.run_cpu_time_us":25496,"prepare.run_wall_time_us":403602,"raft.queue_time_us":67,"raft.run_cpu_time_us":247,"raft.run_wall_time_us":248,"replication_time_us":2108291,"thread_start_us":794317,"threads_started":4,"wal-append.queue_time_us":376742,"wal-append.run_cpu_time_us":603,"wal-append.run_wall_time_us":1001249}]]}
I20251029 02:52:01.165028 1033 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:37626 (request call id 548) took 1038 ms. Trace:
I20251029 02:52:01.165136 1033 rpcz_store.cc:276] 1029 02:52:00.121709 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:00.667520 (+545811us) service_pool.cc:225] Handling call
1029 02:52:00.686558 (+ 19038us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1029 02:52:00.686615 (+ 57us) ts_tablet_manager.cc:682] Creating new metadata...
1029 02:52:00.984042 (+297427us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:01.160031 (+175989us) tablet_replica.cc:206] Creating consensus instance
1029 02:52:01.160188 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251029 02:52:01.165639 1033 tablet_service.cc:1505] Processing CreateTablet for tablet c0936e54f1d246388a2d82dbc52aac42 (DEFAULT_TABLE table=default.Table_f90425a19e964b75b94c68952d26ae1e [id=48211a42281c40a99c8cf27174cf6679]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:01.166003 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0936e54f1d246388a2d82dbc52aac42. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:01.174276 1682 raft_consensus.cc:493] T d2990e9f626c40618c118266bc595684 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:01.174540 1682 raft_consensus.cc:515] T d2990e9f626c40618c118266bc595684 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:01.175472 1682 leader_election.cc:290] T d2990e9f626c40618c118266bc595684 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:01.176337 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2990e9f626c40618c118266bc595684" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:01.176719 776 raft_consensus.cc:2393] T d2990e9f626c40618c118266bc595684 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c756528642a543eeacd93cef3e335a7d in current term 1: Already voted for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in this term.
I20251029 02:52:01.179908 1167 ts_tablet_manager.cc:1434] T 75236b2fa82f4f28b82efae9a2c5160d P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.043s user 0.003s sys 0.000s
I20251029 02:52:01.185523 1172 tablet_bootstrap.cc:654] T 576c4e6e70e24b0399ac9597a30f0569 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:01.187350 1166 tablet_bootstrap.cc:492] T b5d5b737c86844d58c51b2388c0b1705 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:01.187899 1167 tablet_bootstrap.cc:492] T b5d5b737c86844d58c51b2388c0b1705 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:01.194739 1174 ts_tablet_manager.cc:1950] T 012ce243fd07466ab5301d1c9ce4b413 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:01.195840 1174 tablet_replica.cc:333] T 14698af6348f41e384f10569392d9b23 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:01.196092 1174 raft_consensus.cc:2243] T 14698af6348f41e384f10569392d9b23 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:01.196218 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 012ce243fd07466ab5301d1c9ce4b413 (table default.Table_cb594ce4077d4d02aa70ef94d5f801cc [id=b792f15b5ea34e44b32b3087b983567c]) successfully deleted
I20251029 02:52:01.196321 1174 raft_consensus.cc:2272] T 14698af6348f41e384f10569392d9b23 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:01.196981 1174 ts_tablet_manager.cc:1916] T 14698af6348f41e384f10569392d9b23 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:01.198650 1167 tablet_bootstrap.cc:654] T b5d5b737c86844d58c51b2388c0b1705 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:01.208035 1057 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2990e9f626c40618c118266bc595684" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:01.208830 849 leader_election.cc:304] T d2990e9f626c40618c118266bc595684 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9
I20251029 02:52:01.209120 1682 raft_consensus.cc:3060] T d2990e9f626c40618c118266bc595684 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:01.213234 1682 raft_consensus.cc:2749] T d2990e9f626c40618c118266bc595684 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:01.216651 1166 ts_tablet_manager.cc:1403] T b5d5b737c86844d58c51b2388c0b1705 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.336s user 0.003s sys 0.009s
I20251029 02:52:01.222267 519 catalog_manager.cc:6194] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: default.Table_e48aadc66a0e48658d45d0812c29cbdf [id=f45315b686594867aae0114f4ce2f26c] alter complete (version 1)
I20251029 02:52:01.222661 522 catalog_manager.cc:6194] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: default.Table_f35ae25e36f64d2886752bb3b1d66710 [id=9f95160639ad40e995a6c89e297a0fad] alter complete (version 1)
I20251029 02:52:01.224623 1038 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:37626 (request call id 552) took 1091 ms. Trace:
I20251029 02:52:01.227584 1038 rpcz_store.cc:276] 1029 02:52:00.122024 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:00.701425 (+579401us) service_pool.cc:225] Handling call
1029 02:52:00.722102 (+ 20677us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1029 02:52:00.722130 (+ 28us) ts_tablet_manager.cc:682] Creating new metadata...
1029 02:52:01.140226 (+418096us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:01.213713 (+ 73487us) tablet_replica.cc:206] Creating consensus instance
1029 02:52:01.213879 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251029 02:52:01.228120 1038 tablet_service.cc:1505] Processing CreateTablet for tablet a9f78f61ee2841dea71bbbfceacc3ea6 (DEFAULT_TABLE table=default.Table_0100f32eabf24d88a739a3cdac3597e1 [id=f22450cafadd404d8d224cdee646b5a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:01.228714 1040 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:37626 (request call id 553) took 1106 ms. Trace:
W20251029 02:52:01.233738 520 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1bdb67125493430398ff5c80b73de2ac consensus state has no leader
I20251029 02:52:01.235054 518 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_4f9515ed04dc4e9e8480232c53fbac2e" } new_table_name: "default.Table_7cfcc5277f9f4b1d9bc979e2b218efaf" modify_external_catalogs: true
W20251029 02:52:01.235566 517 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cedfb53078d94c81b5e294ce7438325e consensus state has no leader
I20251029 02:52:01.237177 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_f15c66cb663740828b22d54e002adeaa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:01.237547 523 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_da40f10638dc4f96a028f73fd7f001f1" } new_table_name: "default.Table_ab351955aaf84fb4b49b3a63325f94ac" modify_external_catalogs: true
W20251029 02:52:01.237784 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f15c66cb663740828b22d54e002adeaa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:01.239127 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_64d89ed6a9af4c4880c3203fddf36126"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:01.239709 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_64d89ed6a9af4c4880c3203fddf36126 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:01.234881 1031 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:37626 (request call id 549) took 1113 ms. Trace:
I20251029 02:52:01.243431 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_15da0c38ef4a4e30a558753430df9072"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:01.244033 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_15da0c38ef4a4e30a558753430df9072 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:01.245347 520 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_a45edb83b6254a4bb8fb610220e5cf65" } modify_external_catalogs: true
I20251029 02:52:01.245535 520 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_a45edb83b6254a4bb8fb610220e5cf65" } modify_external_catalogs: true
I20251029 02:52:01.245535 1031 rpcz_store.cc:276] 1029 02:52:00.121799 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:00.668769 (+546970us) service_pool.cc:225] Handling call
1029 02:52:00.675491 (+ 6722us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1029 02:52:00.675529 (+ 38us) ts_tablet_manager.cc:682] Creating new metadata...
1029 02:52:01.123790 (+448261us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:01.234660 (+110870us) tablet_replica.cc:206] Creating consensus instance
1029 02:52:01.234835 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251029 02:52:01.245929 1031 tablet_service.cc:1552] Processing DeleteTablet for tablet d31e548feae24b739087c1e211d514f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:00 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:01.246240 517 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_c4adcf442af54ea08664c405b8a6dfe4" } modify_external_catalogs: true
I20251029 02:52:01.246383 517 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_c4adcf442af54ea08664c405b8a6dfe4" } modify_external_catalogs: true
I20251029 02:52:01.247385 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_28e9d77a943b43588e674c3d4825c74f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:01.247915 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_28e9d77a943b43588e674c3d4825c74f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:01.248834 1031 tablet_service.cc:1505] Processing CreateTablet for tablet 60a964a5181a45a186921c31cbdd3db7 (DEFAULT_TABLE table=default.Table_6a3e403847304172bc58735bfe02dbd0 [id=8eb84ca5a1ac4d65a459573a06763a69]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:01.249109 1025 tablet_service.cc:1552] Processing DeleteTablet for tablet 2089b058d0e94d99b40ad2718518353e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:00 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:01.249372 1025 tablet_service.cc:1552] Processing DeleteTablet for tablet 2b278db5b2b04344b864797055a5f665 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:00 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:01.249778 1025 tablet_service.cc:1505] Processing CreateTablet for tablet 76a7e8517db140e8bdd3b67ce423ec90 (DEFAULT_TABLE table=default.Table_0936756f462041d6a93c1401ceb56158 [id=05921c5e2e534071859f555b510a97ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:01.250141 1022 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:37622 (request call id 95) took 1020 ms. Trace:
I20251029 02:52:01.250226 1022 rpcz_store.cc:276] 1029 02:52:00.229343 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:00.229451 (+ 108us) service_pool.cc:225] Handling call
1029 02:52:00.974894 (+745443us) spinlock_profiling.cc:243] Waited 17.4 ms on lock 0x6180000d188c. stack: 00007f34717829f6 00007f34717827b0 00007f347075e3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f347eba944e 00007f347e9a4ac1 00007f34814cc478 00007f347b8f2c36 00007f347b8f2965 00007f347872127e 00007f347872015e 00007f3478726395 00007f3478726cbf 00007f3478726ae8
1029 02:52:01.028140 (+ 53246us) spinlock_profiling.cc:243] Waited 2.84 ms on lock 0x6160000d1630. stack: 00007f34717829f6 00007f34717827b0 00007f347075e3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f347e9a4829 00007f34814cc478 00007f347b8f2c36 00007f347b8f2965 00007f347872127e 00007f347872015e 00007f3478726395 00007f3478726cbf 00007f3478726ae8 0000000000a8867c
1029 02:52:01.250082 (+221942us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":46607616}
I20251029 02:52:01.249071 1040 rpcz_store.cc:276] 1029 02:52:00.122092 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:00.706762 (+584670us) service_pool.cc:225] Handling call
1029 02:52:00.733284 (+ 26522us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1029 02:52:00.733311 (+ 27us) ts_tablet_manager.cc:682] Creating new metadata...
1029 02:52:01.052694 (+319383us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:01.228400 (+175706us) tablet_replica.cc:206] Creating consensus instance
1029 02:52:01.228661 (+ 261us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251029 02:52:01.253566 1175 ts_tablet_manager.cc:1929] T 9e07291d6b5645feb4d5890f58e1ef74 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:01.253710 1175 log.cc:1199] T 9e07291d6b5645feb4d5890f58e1ef74 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/9e07291d6b5645feb4d5890f58e1ef74
I20251029 02:52:01.258481 1166 raft_consensus.cc:359] T b5d5b737c86844d58c51b2388c0b1705 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:01.261420 1040 tablet_service.cc:1552] Processing DeleteTablet for tablet d29cc2a0c8b84e3ebadc7f571d666847 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:00 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:01.262145 1024 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:37626 (request call id 541) took 1133 ms. Trace:
I20251029 02:52:01.262266 1024 rpcz_store.cc:276] 1029 02:52:00.121174 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:00.642577 (+521403us) service_pool.cc:225] Handling call
1029 02:52:00.702148 (+ 59571us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1029 02:52:00.702171 (+ 23us) ts_tablet_manager.cc:682] Creating new metadata...
1029 02:52:01.049465 (+347294us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:01.243549 (+194084us) tablet_replica.cc:206] Creating consensus instance
1029 02:52:01.254882 (+ 11333us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251029 02:52:01.262604 1024 tablet_service.cc:1552] Processing DeleteTablet for tablet 44ae2b07edca44afbe1b1fb3a5b1115d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:00 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:01.262900 1020 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:37622 (request call id 94) took 1103 ms. Trace:
I20251029 02:52:01.263010 1020 rpcz_store.cc:276] 1029 02:52:00.159113 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:00.159180 (+ 67us) service_pool.cc:225] Handling call
1029 02:52:00.972017 (+812837us) spinlock_profiling.cc:243] Waited 1.73 ms on lock 0x60c0001bb1b8. stack: 00007f34717829f6 00007f34717827b0 00007f347075e3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f347cf3339b 00007f347cef0643 00007f347e9a5958 00007f347e9a4e1a 00007f34814cc478 00007f347b8f2c36 00007f347b8f2965 00007f347872127e 00007f347872015e 00007f3478726395
1029 02:52:01.027717 (+ 55700us) spinlock_profiling.cc:243] Waited 2.83 ms on lock 0x6160000d1630. stack: 00007f34717829f6 00007f34717827b0 00007f347075e3ad 0000000000983f04 0000000000983ecd 000000000098389a 00000000009823e9 00007f34814cc60f 00007f347b8f2c36 00007f347b8f2965 00007f347872127e 00007f347872015e 00007f3478726395 00007f3478726cbf 00007f3478726ae8 0000000000a8867c
1029 02:52:01.073943 (+ 46226us) spinlock_profiling.cc:243] Waited 1.91 ms on lock 0x6160001a3030. stack: 00007f34717829f6 00007f34717827b0 00007f347075e3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f347e9a4829 00007f34814cc478 00007f347b8f2c36 00007f347b8f2965 00007f347872127e 00007f347872015e 00007f3478726395 00007f3478726cbf 00007f3478726ae8 0000000000a8867c
1029 02:52:01.137954 (+ 64011us) spinlock_profiling.cc:243] Waited 2.22 ms on lock 0x616000281930. stack: 00007f34717829f6 00007f34717827b0 00007f347075e3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f347e9a58ec 00007f347e9a4e1a 00007f34814cc478 00007f347b8f2c36 00007f347b8f2965 00007f347872127e 00007f347872015e 00007f3478726395 00007f3478726cbf 00007f3478726ae8
1029 02:52:01.262841 (+124887us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":20003328}
I20251029 02:52:01.263446 1166 raft_consensus.cc:385] T b5d5b737c86844d58c51b2388c0b1705 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:01.263551 1166 raft_consensus.cc:740] T b5d5b737c86844d58c51b2388c0b1705 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:01.268024 1176 ts_tablet_manager.cc:1929] T 9be9add0274d41b688edd6292a253a06 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251029 02:52:01.271598 1175 ts_tablet_manager.cc:1950] T 9e07291d6b5645feb4d5890f58e1ef74 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:01.269021 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9f78f61ee2841dea71bbbfceacc3ea6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:01.272917 1027 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:37626 (request call id 556) took 1106 ms. Trace:
I20251029 02:52:01.273176 1023 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:37626 (request call id 555) took 1107 ms. Trace:
I20251029 02:52:01.272399 1035 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:37626 (request call id 537) took 1150 ms. Trace:
I20251029 02:52:01.275765 1036 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:37626 (request call id 527) took 1981 ms. Trace:
I20251029 02:52:01.276332 1166 consensus_queue.cc:260] T b5d5b737c86844d58c51b2388c0b1705 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:01.276988 519 catalog_manager.cc:6194] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: default.Table_9bf4e25d441f4d4dbd6651386eec512e [id=82df85af001340efb570040e047fa136] alter complete (version 2)
W20251029 02:52:01.278460 518 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 688af6580c4d4637a39f817b4e324618 consensus state has no leader
W20251029 02:52:01.278736 523 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ea7b977224f24bf49dadfb83ffa28310 consensus state has no leader
I20251029 02:52:01.281997 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76a7e8517db140e8bdd3b67ce423ec90. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:01.282872 518 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_3d101961029143a882eeafb119560e5d" } modify_external_catalogs: true
I20251029 02:52:01.283062 518 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_3d101961029143a882eeafb119560e5d" } modify_external_catalogs: true
I20251029 02:52:01.283650 526 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_fd89990c08d142ffa08457a5802e5e69" } new_table_name: "default.Table_af03737bb25a4837aaef8c02ede1a537" modify_external_catalogs: true
I20251029 02:52:01.282667 1035 rpcz_store.cc:276] 1029 02:52:00.120872 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:00.568206 (+447334us) service_pool.cc:225] Handling call
1029 02:52:00.572157 (+ 3951us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1029 02:52:00.572180 (+ 23us) ts_tablet_manager.cc:682] Creating new metadata...
1029 02:52:01.146352 (+574172us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:01.270968 (+124616us) tablet_replica.cc:206] Creating consensus instance
1029 02:52:01.271097 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251029 02:52:01.286686 1166 ts_tablet_manager.cc:1434] T b5d5b737c86844d58c51b2388c0b1705 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.061s user 0.002s sys 0.001s
I20251029 02:52:01.291502 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60a964a5181a45a186921c31cbdd3db7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:01.292542 1034 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:37626 (request call id 550) took 1169 ms. Trace:
I20251029 02:52:01.292641 1034 rpcz_store.cc:276] 1029 02:52:00.121877 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:00.669850 (+547973us) service_pool.cc:225] Handling call
1029 02:52:00.707086 (+ 37236us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1029 02:52:00.707132 (+ 46us) ts_tablet_manager.cc:682] Creating new metadata...
1029 02:52:01.206148 (+499016us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:01.284935 (+ 78787us) tablet_replica.cc:206] Creating consensus instance
1029 02:52:01.291127 (+ 6192us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251029 02:52:01.287034 1023 rpcz_store.cc:276] 1029 02:52:00.159413 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:00.708894 (+549481us) service_pool.cc:225] Handling call
1029 02:52:00.709357 (+ 463us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1029 02:52:00.709382 (+ 25us) ts_tablet_manager.cc:682] Creating new metadata...
1029 02:52:01.104865 (+395483us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:01.257289 (+152424us) tablet_replica.cc:206] Creating consensus instance
1029 02:52:01.266406 (+ 9117us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251029 02:52:01.296612 521 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_a8855248e99e4374aaaf0d4f66e9cad1" } modify_external_catalogs: true
I20251029 02:52:01.296775 521 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_a8855248e99e4374aaaf0d4f66e9cad1" } modify_external_catalogs: true
I20251029 02:52:01.297319 524 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_d477b4de8ad4449aa5d53db48d45273f" } modify_external_catalogs: true
I20251029 02:52:01.297478 524 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_d477b4de8ad4449aa5d53db48d45273f" } modify_external_catalogs: true
I20251029 02:52:01.298015 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_05a3479e89744e828535b90bbea0e8e0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:01.298698 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_05a3479e89744e828535b90bbea0e8e0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:01.293293 1027 rpcz_store.cc:276] 1029 02:52:00.159557 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:00.726941 (+567384us) service_pool.cc:225] Handling call
1029 02:52:00.728078 (+ 1137us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1029 02:52:00.728148 (+ 70us) ts_tablet_manager.cc:682] Creating new metadata...
1029 02:52:01.152204 (+424056us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:01.260763 (+108559us) tablet_replica.cc:206] Creating consensus instance
1029 02:52:01.266476 (+ 5713us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251029 02:52:01.309252 1172 tablet_bootstrap.cc:492] T 576c4e6e70e24b0399ac9597a30f0569 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:01.309718 1172 ts_tablet_manager.cc:1403] T 576c4e6e70e24b0399ac9597a30f0569 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.149s user 0.002s sys 0.002s
I20251029 02:52:01.304034 1176 log.cc:1199] T 9be9add0274d41b688edd6292a253a06 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/9be9add0274d41b688edd6292a253a06
I20251029 02:52:01.320474 1166 tablet_bootstrap.cc:492] T 078ace5eada044c88d2462efb2b57aa8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:01.323288 1172 raft_consensus.cc:359] T 576c4e6e70e24b0399ac9597a30f0569 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:01.323495 1172 raft_consensus.cc:385] T 576c4e6e70e24b0399ac9597a30f0569 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:01.323634 1172 raft_consensus.cc:740] T 576c4e6e70e24b0399ac9597a30f0569 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:01.323911 1172 consensus_queue.cc:260] T 576c4e6e70e24b0399ac9597a30f0569 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:01.324749 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 396) took 2960 ms. Trace:
I20251029 02:52:01.325105 1172 ts_tablet_manager.cc:1434] T 576c4e6e70e24b0399ac9597a30f0569 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.015s user 0.001s sys 0.001s
I20251029 02:52:01.325271 1175 rpcz_store.cc:276] 1029 02:51:58.359143 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:58.359203 (+ 60us) service_pool.cc:225] Handling call
1029 02:52:01.014182 (+2654979us) ts_tablet_manager.cc:1151] Deleting tablet 9e07291d6b5645feb4d5890f58e1ef74
1029 02:52:01.014193 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:01.162487 (+148294us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:01.253417 (+ 90930us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:01.319360 (+ 65943us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2534971}
I20251029 02:52:01.325923 1172 tablet_bootstrap.cc:492] T 7e6c4f30be624769961c9ec17c5d5dcb P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:01.327714 1172 tablet_bootstrap.cc:654] T 7e6c4f30be624769961c9ec17c5d5dcb P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:01.328243 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 9e07291d6b5645feb4d5890f58e1ef74 (table default.Table_0edd4b07469b4d24b302ecdb6859d1d1 [id=e71aea48b9884bacb250412624ec4615]) successfully deleted
I20251029 02:52:01.332185 1036 rpcz_store.cc:276] 1029 02:51:59.302975 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:00.561205 (+1258230us) service_pool.cc:225] Handling call
1029 02:52:00.561749 (+ 544us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1029 02:52:00.561777 (+ 28us) ts_tablet_manager.cc:682] Creating new metadata...
1029 02:52:01.196471 (+634694us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:01.275577 (+ 79106us) tablet_replica.cc:206] Creating consensus instance
1029 02:52:01.275730 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251029 02:52:01.334335 1175 tablet_replica.cc:333] T a5c3a15510cb4321bbb80d61fabd0c0a P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:01.334585 1175 raft_consensus.cc:2243] T a5c3a15510cb4321bbb80d61fabd0c0a P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:01.334767 1175 raft_consensus.cc:2272] T a5c3a15510cb4321bbb80d61fabd0c0a P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:01.335326 1175 ts_tablet_manager.cc:1916] T a5c3a15510cb4321bbb80d61fabd0c0a P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:01.346027 1167 tablet_bootstrap.cc:492] T b5d5b737c86844d58c51b2388c0b1705 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:01.346271 1166 tablet_bootstrap.cc:654] T 078ace5eada044c88d2462efb2b57aa8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:01.346298 1167 ts_tablet_manager.cc:1403] T b5d5b737c86844d58c51b2388c0b1705 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.158s user 0.011s sys 0.034s
I20251029 02:52:01.346638 1686 raft_consensus.cc:493] T 9e07291d6b5645feb4d5890f58e1ef74 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:01.347079 1686 raft_consensus.cc:515] T 9e07291d6b5645feb4d5890f58e1ef74 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:01.347476 1167 raft_consensus.cc:359] T b5d5b737c86844d58c51b2388c0b1705 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:01.347759 1167 raft_consensus.cc:385] T b5d5b737c86844d58c51b2388c0b1705 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:01.347802 1686 leader_election.cc:290] T 9e07291d6b5645feb4d5890f58e1ef74 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:52:01.347957 1167 raft_consensus.cc:740] T b5d5b737c86844d58c51b2388c0b1705 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:01.348505 1176 ts_tablet_manager.cc:1950] T 9be9add0274d41b688edd6292a253a06 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:01.348596 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e07291d6b5645feb4d5890f58e1ef74" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:01.349846 1167 consensus_queue.cc:260] T b5d5b737c86844d58c51b2388c0b1705 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
W20251029 02:52:01.350169 990 leader_election.cc:343] T 9e07291d6b5645feb4d5890f58e1ef74 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:01.350528 1167 ts_tablet_manager.cc:1434] T b5d5b737c86844d58c51b2388c0b1705 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251029 02:52:01.351044 1167 tablet_bootstrap.cc:492] T 4bf06476c87a4549a73f5994314bb75f P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
W20251029 02:52:01.352733 989 leader_election.cc:343] T 9e07291d6b5645feb4d5890f58e1ef74 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 9e07291d6b5645feb4d5890f58e1ef74
I20251029 02:52:01.352970 989 leader_election.cc:304] T 9e07291d6b5645feb4d5890f58e1ef74 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5; no voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:01.353143 1167 tablet_bootstrap.cc:654] T 4bf06476c87a4549a73f5994314bb75f P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:01.353927 1686 raft_consensus.cc:2749] T 9e07291d6b5645feb4d5890f58e1ef74 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:01.355630 1030 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:37626 (request call id 530) took 2021 ms. Trace:
I20251029 02:52:01.356415 1030 rpcz_store.cc:276] 1029 02:51:59.329361 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:00.562703 (+1233342us) service_pool.cc:225] Handling call
1029 02:52:00.563121 (+ 418us) ts_tablet_manager.cc:669] Acquired tablet manager lock
1029 02:52:00.563142 (+ 21us) ts_tablet_manager.cc:682] Creating new metadata...
1029 02:52:00.947493 (+384351us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:01.322659 (+375166us) tablet_replica.cc:206] Creating consensus instance
1029 02:52:01.350050 (+ 27391us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20251029 02:52:01.357771 1174 ts_tablet_manager.cc:1929] T 14698af6348f41e384f10569392d9b23 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:01.359233 1018 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:37622 (request call id 96) took 1093 ms. Trace:
I20251029 02:52:01.359474 1018 rpcz_store.cc:276] 1029 02:52:00.248367 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:00.248510 (+ 143us) service_pool.cc:225] Handling call
1029 02:52:01.028047 (+779537us) spinlock_profiling.cc:243] Waited 2.54 ms on lock 0x6160000d1630. stack: 00007f34717829f6 00007f34717827b0 00007f347075e3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f347e9a4829 00007f34814cc478 00007f347b8f2c36 00007f347b8f2965 00007f347872127e 00007f347872015e 00007f3478726395 00007f3478726cbf 00007f3478726ae8 0000000000a8867c
1029 02:52:01.341677 (+313630us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5841664}
I20251029 02:52:01.370822 522 catalog_manager.cc:6194] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: default.Table_8b6217e43e8d467693c104ef84e78911 [id=052abf3dcc904bdcb869d5b885723dab] alter complete (version 4)
I20251029 02:52:01.371385 525 master_service.cc:954] ReplaceTablet: received request to replace tablet 1f5b40c0403d4a49a4bfde222d8c947a from {username='slave'} at 127.0.0.1:35314
I20251029 02:52:01.371505 522 master_service.cc:954] ReplaceTablet: received request to replace tablet 24016ceaa84147eca0b94b3ad288e772 from {username='slave'} at 127.0.0.1:35314
W20251029 02:52:01.371548 526 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ead73d81d18a40a4b0aa9935f86421f8 consensus state has no leader
I20251029 02:52:01.382655 755 tablet_service.cc:1552] Processing DeleteTablet for tablet 576c4e6e70e24b0399ac9597a30f0569 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:54 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:01.382948 750 tablet_service.cc:1552] Processing DeleteTablet for tablet 1f5b40c0403d4a49a4bfde222d8c947a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:53 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:01.383069 523 master_service.cc:954] ReplaceTablet: received request to replace tablet f2d47238761e4ed49c88abdbb677b18a from {username='slave'} at 127.0.0.1:35314
I20251029 02:52:01.383160 751 tablet_service.cc:1552] Processing DeleteTablet for tablet 24cb2a71ba3841d4a5e3e5c6fe99ae2f with delete_type TABLET_DATA_DELETED (replaced by tablet f60e7c24982d40319801869ad060864d) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:01.383720 751 tablet_service.cc:1552] Processing DeleteTablet for tablet 51effc9324034ae79e2b25cb57a996c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:56 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:01.387915 1686 consensus_queue.cc:1048] T d2990e9f626c40618c118266bc595684 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:01.391132 1174 log.cc:1199] T 14698af6348f41e384f10569392d9b23 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/14698af6348f41e384f10569392d9b23
I20251029 02:52:01.391659 1174 ts_tablet_manager.cc:1950] T 14698af6348f41e384f10569392d9b23 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:01.395576 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 374) took 4227 ms. Trace:
I20251029 02:52:01.395956 1176 rpcz_store.cc:276] 1029 02:51:57.167686 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:57.169171 (+ 1485us) service_pool.cc:225] Handling call
1029 02:52:00.986970 (+3817799us) ts_tablet_manager.cc:1151] Deleting tablet 9be9add0274d41b688edd6292a253a06
1029 02:52:00.986983 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:01.109735 (+122752us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:01.267873 (+158138us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:01.395532 (+127659us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3817410}
I20251029 02:52:01.396682 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 375) took 4226 ms. Trace:
I20251029 02:52:01.396988 1176 rpcz_store.cc:276] 1029 02:51:57.169917 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:57.170119 (+ 202us) service_pool.cc:225] Handling call
1029 02:52:01.396481 (+4226362us) ts_tablet_manager.cc:1151] Deleting tablet e1fa4980f2624321b617b84c68b90113
1029 02:52:01.396494 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:01.396647 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4226138}
I20251029 02:52:01.397104 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 9be9add0274d41b688edd6292a253a06 (table default.Table_c7daf9eda79e470186f5774de2b824d3 [id=f45ce142f37f4febbceb4e568e35e0f2]) successfully deleted
I20251029 02:52:01.397428 1176 tablet_replica.cc:333] T 14698af6348f41e384f10569392d9b23 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:01.397864 1176 raft_consensus.cc:2243] T 14698af6348f41e384f10569392d9b23 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:01.398125 1176 raft_consensus.cc:2272] T 14698af6348f41e384f10569392d9b23 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:01.401423 1172 tablet_bootstrap.cc:492] T 7e6c4f30be624769961c9ec17c5d5dcb P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:01.401624 1172 ts_tablet_manager.cc:1403] T 7e6c4f30be624769961c9ec17c5d5dcb P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.076s user 0.003s sys 0.000s
I20251029 02:52:01.405939 1021 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:37622 (request call id 93) took 1242 ms. Trace:
I20251029 02:52:01.402395 1172 raft_consensus.cc:359] T 7e6c4f30be624769961c9ec17c5d5dcb P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:01.406140 1172 raft_consensus.cc:385] T 7e6c4f30be624769961c9ec17c5d5dcb P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:01.406239 1172 raft_consensus.cc:740] T 7e6c4f30be624769961c9ec17c5d5dcb P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:01.406527 1172 consensus_queue.cc:260] T 7e6c4f30be624769961c9ec17c5d5dcb P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:01.407198 1172 ts_tablet_manager.cc:1434] T 7e6c4f30be624769961c9ec17c5d5dcb P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20251029 02:52:01.407508 1172 tablet_bootstrap.cc:492] T c9dff4dff82f49e7ad49caad5e404f09 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:01.409210 1172 tablet_bootstrap.cc:654] T c9dff4dff82f49e7ad49caad5e404f09 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:01.409997 1682 raft_consensus.cc:493] T 75236b2fa82f4f28b82efae9a2c5160d P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:01.410172 1682 raft_consensus.cc:515] T 75236b2fa82f4f28b82efae9a2c5160d P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:01.410830 1682 leader_election.cc:290] T 75236b2fa82f4f28b82efae9a2c5160d P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
W20251029 02:52:01.411672 847 leader_election.cc:343] T 75236b2fa82f4f28b82efae9a2c5160d P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 75236b2fa82f4f28b82efae9a2c5160d
I20251029 02:52:01.411765 1167 tablet_bootstrap.cc:492] T 4bf06476c87a4549a73f5994314bb75f P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:01.411639 1021 rpcz_store.cc:276] 1029 02:52:00.158207 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:00.158265 (+ 58us) service_pool.cc:225] Handling call
1029 02:52:01.024471 (+866206us) spinlock_profiling.cc:243] Waited 5.49 ms on lock 0x6160000d1630. stack: 00007f34717829f6 00007f34717827b0 00007f347075e3ad 0000000000983f04 0000000000983ecd 000000000098389a 00000000009823e9 00007f34814cc60f 00007f347b8f2c36 00007f347b8f2965 00007f347872127e 00007f347872015e 00007f3478726395 00007f3478726cbf 00007f3478726ae8 0000000000a8867c
1029 02:52:01.142606 (+118135us) spinlock_profiling.cc:243] Waited 11.8 ms on lock 0x616000281930. stack: 00007f34717829f6 00007f34717827b0 00007f347075e3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f347e9a4829 00007f34814cc478 00007f347b8f2c36 00007f347b8f2965 00007f347872127e 00007f347872015e 00007f3478726395 00007f3478726cbf 00007f3478726ae8 0000000000a8867c
1029 02:52:01.261992 (+119386us) spinlock_profiling.cc:243] Waited 3.44 ms on lock 0x60c000225478. stack: 00007f34717829f6 00007f34717827b0 00007f347075e3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f347cf3339b 00007f347cef0643 00007f347e9a5958 00007f347e9a4e1a 00007f34814cc478 00007f347b8f2c36 00007f347b8f2965 00007f347872127e 00007f347872015e 00007f3478726395
1029 02:52:01.400305 (+138313us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":47738496}
I20251029 02:52:01.411991 1167 ts_tablet_manager.cc:1403] T 4bf06476c87a4549a73f5994314bb75f P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.061s user 0.011s sys 0.001s
I20251029 02:52:01.413100 1167 raft_consensus.cc:359] T 4bf06476c87a4549a73f5994314bb75f P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:01.414896 1167 raft_consensus.cc:385] T 4bf06476c87a4549a73f5994314bb75f P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:01.415052 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 14698af6348f41e384f10569392d9b23 (table default.Table_ef563eb489f947d4a4772678806a0777 [id=7de93d6d48ab4d58b5aed9319d7299a4]) successfully deleted
I20251029 02:52:01.415174 1167 raft_consensus.cc:740] T 4bf06476c87a4549a73f5994314bb75f P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:01.415699 1167 consensus_queue.cc:260] T 4bf06476c87a4549a73f5994314bb75f P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
W20251029 02:52:01.415797 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 2089b058d0e94d99b40ad2718518353e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2089b058d0e94d99b40ad2718518353e already in progress: creating tablet
W20251029 02:52:01.416358 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet d31e548feae24b739087c1e211d514f9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d31e548feae24b739087c1e211d514f9 already in progress: creating tablet
I20251029 02:52:01.416947 1167 ts_tablet_manager.cc:1434] T 4bf06476c87a4549a73f5994314bb75f P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251029 02:52:01.409539 1686 consensus_queue.cc:1048] T d2990e9f626c40618c118266bc595684 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:01.417862 1167 tablet_bootstrap.cc:492] T 0a47781d036149569e1164dd938a45e9 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
W20251029 02:52:01.419001 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 2b278db5b2b04344b864797055a5f665 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2b278db5b2b04344b864797055a5f665 already in progress: creating tablet
I20251029 02:52:01.410459 1019 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:37622 (request call id 92) took 1244 ms. Trace:
I20251029 02:52:01.419836 1019 rpcz_store.cc:276] 1029 02:52:00.158094 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:00.159310 (+ 1216us) service_pool.cc:225] Handling call
1029 02:52:00.427284 (+267974us) spinlock_profiling.cc:243] Waited 8.82 ms on lock 0x616000250130. stack: 00007f34717829f6 00007f34717827b0 00007f347075e3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f347e9a4829 00007f34814cc478 00007f347b8f2c36 00007f347b8f2965 00007f347872127e 00007f347872015e 00007f3478726395 00007f3478726cbf 00007f3478726ae8 0000000000a8867c
1029 02:52:01.027945 (+600661us) spinlock_profiling.cc:243] Waited 1.87 ms on lock 0x6160000d1630. stack: 00007f34717829f6 00007f34717827b0 00007f347075e3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f347e9a4829 00007f34814cc478 00007f347b8f2c36 00007f347b8f2965 00007f347872127e 00007f347872015e 00007f3478726395 00007f3478726cbf 00007f3478726ae8 0000000000a8867c
1029 02:52:01.102976 (+ 75031us) spinlock_profiling.cc:243] Waited 5.12 ms on lock 0x6160002a1430. stack: 00007f34717829f6 00007f34717827b0 00007f347075e3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f347e9a4829 00007f34814cc478 00007f347b8f2c36 00007f347b8f2965 00007f347872127e 00007f347872015e 00007f3478726395 00007f3478726cbf 00007f3478726ae8 0000000000a8867c
1029 02:52:01.132765 (+ 29789us) spinlock_profiling.cc:243] Waited 9.72 ms on lock 0x616000281930. stack: 00007f34717829f6 00007f34717827b0 00007f347075e3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f347e9a4829 00007f34814cc478 00007f347b8f2c36 00007f347b8f2965 00007f347872127e 00007f347872015e 00007f3478726395 00007f3478726cbf 00007f3478726ae8 0000000000a8867c
1029 02:52:01.205574 (+ 72809us) spinlock_profiling.cc:243] Waited 13.1 ms on lock 0x616000194330. stack: 00007f34717829f6 00007f34717827b0 00007f347075e3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f347e9a4829 00007f34814cc478 00007f347b8f2c36 00007f347b8f2965 00007f347872127e 00007f347872015e 00007f3478726395 00007f3478726cbf 00007f3478726ae8 0000000000a8867c
1029 02:52:01.402212 (+196638us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":88944128}
I20251029 02:52:01.406188 1176 ts_tablet_manager.cc:1916] T 14698af6348f41e384f10569392d9b23 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:01.411103 1057 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75236b2fa82f4f28b82efae9a2c5160d" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:01.421211 1057 raft_consensus.cc:2468] T 75236b2fa82f4f28b82efae9a2c5160d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:01.428293 849 leader_election.cc:304] T 75236b2fa82f4f28b82efae9a2c5160d P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters: b495a49349324f7d81908f2da700f0d9
I20251029 02:52:01.428854 1682 raft_consensus.cc:2804] T 75236b2fa82f4f28b82efae9a2c5160d P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:01.428966 1682 raft_consensus.cc:493] T 75236b2fa82f4f28b82efae9a2c5160d P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:01.429067 1682 raft_consensus.cc:3060] T 75236b2fa82f4f28b82efae9a2c5160d P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:01.429129 522 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 24016ceaa84147eca0b94b3ad288e772
I20251029 02:52:01.429869 522 catalog_manager.cc:6719] ReplaceTablet: tablet 24016ceaa84147eca0b94b3ad288e772 deleted and replaced by tablet 91e31f0b06924740bdf7525cae79d8ef
I20251029 02:52:01.430306 751 tablet_service.cc:1552] Processing DeleteTablet for tablet 24016ceaa84147eca0b94b3ad288e772 with delete_type TABLET_DATA_DELETED (replaced by tablet 91e31f0b06924740bdf7525cae79d8ef) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:01.430236 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_244839a45b634a3b86f305b2c1e337c6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:01.430833 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_244839a45b634a3b86f305b2c1e337c6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:01.432427 893 tablet_service.cc:1552] Processing DeleteTablet for tablet 24016ceaa84147eca0b94b3ad288e772 with delete_type TABLET_DATA_DELETED (replaced by tablet 91e31f0b06924740bdf7525cae79d8ef) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:01.433702 1682 raft_consensus.cc:515] T 75236b2fa82f4f28b82efae9a2c5160d P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:01.435555 1176 ts_tablet_manager.cc:1929] T 14698af6348f41e384f10569392d9b23 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:01.436293 1176 log.cc:1199] T 14698af6348f41e384f10569392d9b23 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/14698af6348f41e384f10569392d9b23
I20251029 02:52:01.436937 1176 ts_tablet_manager.cc:1950] T 14698af6348f41e384f10569392d9b23 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:01.438273 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 376) took 4234 ms. Trace:
I20251029 02:52:01.438596 524 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 1df0d944717d487a98ea83a8ec04bd46
I20251029 02:52:01.439563 521 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 2e003e140a244d46b15a2c3e84f68936
I20251029 02:52:01.439873 1665 raft_consensus.cc:493] T 1f5b40c0403d4a49a4bfde222d8c947a P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:01.440011 751 tablet_service.cc:1552] Processing DeleteTablet for tablet 1df0d944717d487a98ea83a8ec04bd46 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:01 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:01.440047 1665 raft_consensus.cc:515] T 1f5b40c0403d4a49a4bfde222d8c947a P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:01.440683 1665 leader_election.cc:290] T 1f5b40c0403d4a49a4bfde222d8c947a P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:52:01.438802 1176 rpcz_store.cc:276] 1029 02:51:57.204039 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:57.204089 (+ 50us) service_pool.cc:225] Handling call
1029 02:52:01.397359 (+4193270us) ts_tablet_manager.cc:1151] Deleting tablet 14698af6348f41e384f10569392d9b23
1029 02:52:01.397369 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:01.425479 (+ 28110us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:01.435414 (+ 9935us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:01.438222 (+ 2808us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4193018}
I20251029 02:52:01.441664 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 377) took 4237 ms. Trace:
I20251029 02:52:01.442134 1176 rpcz_store.cc:276] 1029 02:51:57.204284 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:57.204351 (+ 67us) service_pool.cc:225] Handling call
1029 02:52:01.441482 (+4237131us) ts_tablet_manager.cc:1151] Deleting tablet 7a1866afc7564217ad777e7ce53786c5
1029 02:52:01.441493 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:01.441611 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4236906}
I20251029 02:52:01.443511 751 tablet_service.cc:1552] Processing DeleteTablet for tablet 2e003e140a244d46b15a2c3e84f68936 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:01 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:01.444792 520 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 5bd2163bc4be467fb695629bfbb5a6ca
I20251029 02:52:01.445881 517 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 19609bc7f06645dc811b4bae8ce506c8
I20251029 02:52:01.446065 751 tablet_service.cc:1552] Processing DeleteTablet for tablet 5bd2163bc4be467fb695629bfbb5a6ca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:01 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:01.447000 751 tablet_service.cc:1552] Processing DeleteTablet for tablet 19609bc7f06645dc811b4bae8ce506c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:01 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:01.447639 519 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet d2990e9f626c40618c118266bc595684 (table default.Table_bd020e7c45a54c6e92211c4261293575 [id=f733948e92d8470da9d8050bd26a06b1]). Expected version 1 got 0
I20251029 02:52:01.448664 526 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet d2990e9f626c40618c118266bc595684 (table default.Table_bd020e7c45a54c6e92211c4261293575 [id=f733948e92d8470da9d8050bd26a06b1]). Expected version 1 got 0
I20251029 02:52:01.451041 750 tablet_service.cc:1505] Processing CreateTablet for tablet 80f54d8a486a4196aca3c28c84c0872f (DEFAULT_TABLE table=default.Table_25a8f4870ca94816934d0214177fc278 [id=b8eeea99f4df4aedbd187b8805525905]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:01.451553 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80f54d8a486a4196aca3c28c84c0872f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:01.451742 918 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1f5b40c0403d4a49a4bfde222d8c947a" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:01.452320 1174 tablet_replica.cc:333] T 24016ceaa84147eca0b94b3ad288e772 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:01.452553 1174 raft_consensus.cc:2243] T 24016ceaa84147eca0b94b3ad288e772 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:01.452742 1174 raft_consensus.cc:2272] T 24016ceaa84147eca0b94b3ad288e772 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:01.443079 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 378) took 4221 ms. Trace:
I20251029 02:52:01.453312 1174 ts_tablet_manager.cc:1916] T 24016ceaa84147eca0b94b3ad288e772 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:01.453222 1176 rpcz_store.cc:276] 1029 02:51:57.221640 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:57.221715 (+ 75us) service_pool.cc:225] Handling call
1029 02:52:01.442930 (+4221215us) ts_tablet_manager.cc:1151] Deleting tablet 69597d48631d43b6a5b92abbf60d5ed3
1029 02:52:01.442939 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:01.443031 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4220991}
I20251029 02:52:01.453691 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 379) took 4192 ms. Trace:
I20251029 02:52:01.453787 1176 rpcz_store.cc:276] 1029 02:51:57.260882 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:57.295337 (+ 34455us) service_pool.cc:225] Handling call
1029 02:52:01.453504 (+4158167us) ts_tablet_manager.cc:1151] Deleting tablet 13ee62fc2339410b9db70dfd3d3c6fa6
1029 02:52:01.453515 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:01.453629 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4157747}
I20251029 02:52:01.454152 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 398) took 4022 ms. Trace:
I20251029 02:52:01.454229 1176 rpcz_store.cc:276] 1029 02:51:57.432139 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:57.432196 (+ 57us) service_pool.cc:225] Handling call
1029 02:52:01.454015 (+4021819us) ts_tablet_manager.cc:1151] Deleting tablet faa835de84ff45b19525d1d603678987
1029 02:52:01.454024 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:01.454104 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4021557}
I20251029 02:52:01.454576 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 397) took 4061 ms. Trace:
I20251029 02:52:01.454669 1176 rpcz_store.cc:276] 1029 02:51:57.392781 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:57.392839 (+ 58us) service_pool.cc:225] Handling call
1029 02:52:01.454416 (+4061577us) ts_tablet_manager.cc:1151] Deleting tablet 589a521f5cc24e6289c37b9cf256bce5
1029 02:52:01.454423 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:01.454503 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4002216}
I20251029 02:52:01.455636 751 tablet_service.cc:1505] Processing CreateTablet for tablet 09e3ce70eb9a4a6c9035dabbf7434636 (DEFAULT_TABLE table=default.Table_be523b14f1164046a23c062a7907de02 [id=00565347222c4715b4e8471055916a52]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:01.456025 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09e3ce70eb9a4a6c9035dabbf7434636. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:01.456619 1176 tablet_replica.cc:333] T aa776cde2f494fa5810539edf9f289a1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:01.456820 1176 raft_consensus.cc:2243] T aa776cde2f494fa5810539edf9f289a1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:01.457010 1176 raft_consensus.cc:2272] T aa776cde2f494fa5810539edf9f289a1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:01.457557 1176 ts_tablet_manager.cc:1916] T aa776cde2f494fa5810539edf9f289a1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 02:52:01.460239 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet d29cc2a0c8b84e3ebadc7f571d666847 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d29cc2a0c8b84e3ebadc7f571d666847 already in progress: creating tablet
W20251029 02:52:01.460517 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 44ae2b07edca44afbe1b1fb3a5b1115d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 44ae2b07edca44afbe1b1fb3a5b1115d already in progress: creating tablet
I20251029 02:52:01.463951 895 tablet_service.cc:1552] Processing DeleteTablet for tablet 2e003e140a244d46b15a2c3e84f68936 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:01 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:01.464246 881 tablet_service.cc:1552] Processing DeleteTablet for tablet 5bd2163bc4be467fb695629bfbb5a6ca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:01 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:01.464453 879 tablet_service.cc:1552] Processing DeleteTablet for tablet 19609bc7f06645dc811b4bae8ce506c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:01 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:01.464891 892 tablet_service.cc:1505] Processing CreateTablet for tablet 09e3ce70eb9a4a6c9035dabbf7434636 (DEFAULT_TABLE table=default.Table_be523b14f1164046a23c062a7907de02 [id=00565347222c4715b4e8471055916a52]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:01.465235 893 tablet_service.cc:1552] Processing DeleteTablet for tablet 1df0d944717d487a98ea83a8ec04bd46 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:01 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:01.465370 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09e3ce70eb9a4a6c9035dabbf7434636. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:01.465701 881 tablet_service.cc:1505] Processing CreateTablet for tablet 0ffd22d8610b4329a0a4f753d9e60149 (DEFAULT_TABLE table=default.Table_9bc6d7e931a34e00b8a1c2233fa53c3f [id=c8105da2dd8540aa977bf54b09d874ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:01.466300 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ffd22d8610b4329a0a4f753d9e60149. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:01.466871 895 tablet_service.cc:1505] Processing CreateTablet for tablet 80f54d8a486a4196aca3c28c84c0872f (DEFAULT_TABLE table=default.Table_25a8f4870ca94816934d0214177fc278 [id=b8eeea99f4df4aedbd187b8805525905]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:01.467228 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80f54d8a486a4196aca3c28c84c0872f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:01.470824 899 tablet_service.cc:1505] Processing CreateTablet for tablet e2c8983cded84a1c8b4205c48a4eac33 (DEFAULT_TABLE table=default.Table_4780e4a5339245d1acdf62a82234a1d2 [id=22d68257e1714ab8a91dd28c562c4cc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:01.471184 1717 raft_consensus.cc:493] T 24cb2a71ba3841d4a5e3e5c6fe99ae2f P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:01.471312 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2c8983cded84a1c8b4205c48a4eac33. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:01.471442 1717 raft_consensus.cc:515] T 24cb2a71ba3841d4a5e3e5c6fe99ae2f P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:01.472127 1717 leader_election.cc:290] T 24cb2a71ba3841d4a5e3e5c6fe99ae2f P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:52:01.474821 884 tablet_service.cc:1505] Processing CreateTablet for tablet 050b33f64bcf42c5a5e80600520e3727 (DEFAULT_TABLE table=default.Table_bc03c0e4711a4763a6f67e7c0687215e [id=8ed7b2d71aaf4c8c9ec60648319faff2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:01.475219 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 050b33f64bcf42c5a5e80600520e3727. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:01.476163 519 master_service.cc:954] ReplaceTablet: received request to replace tablet f18b0ca5a6a44175a912642e9eea1778 from {username='slave'} at 127.0.0.1:35314
I20251029 02:52:01.476564 1167 tablet_bootstrap.cc:654] T 0a47781d036149569e1164dd938a45e9 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:01.478821 883 tablet_service.cc:1505] Processing CreateTablet for tablet a27496be3431446da9cb83452a3893e3 (DEFAULT_TABLE table=default.Table_9dfd24b5c34c4b9f965292b798351bc1 [id=5bce7a821707460aaec9997adf37a16d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:01.478909 1172 tablet_bootstrap.cc:492] T c9dff4dff82f49e7ad49caad5e404f09 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:01.479238 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a27496be3431446da9cb83452a3893e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:01.482309 918 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24cb2a71ba3841d4a5e3e5c6fe99ae2f" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:01.483480 900 tablet_service.cc:1505] Processing CreateTablet for tablet 8c972c88f64e486bb2b75306549bb23f (DEFAULT_TABLE table=default.Table_88c311b8b4534d5190945358ee18fe6a [id=c4e7234e2c3f43e6ab5f972e2cbd7785]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:01.484030 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c972c88f64e486bb2b75306549bb23f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:01.479065 1172 ts_tablet_manager.cc:1403] T c9dff4dff82f49e7ad49caad5e404f09 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.072s user 0.002s sys 0.001s
I20251029 02:52:01.485836 1172 raft_consensus.cc:359] T c9dff4dff82f49e7ad49caad5e404f09 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:01.486027 1172 raft_consensus.cc:385] T c9dff4dff82f49e7ad49caad5e404f09 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:01.486115 1172 raft_consensus.cc:740] T c9dff4dff82f49e7ad49caad5e404f09 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:01.486327 1172 consensus_queue.cc:260] T c9dff4dff82f49e7ad49caad5e404f09 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:01.488214 885 tablet_service.cc:1505] Processing CreateTablet for tablet c21ec56a12be48c4966762868e2c2aed (DEFAULT_TABLE table=default.Table_5668f431df944d4291ec4d158ff43e57 [id=749261ef7d7c45b7a91fb5db5908baa3]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 02:52:01.479544 711 leader_election.cc:343] T 1f5b40c0403d4a49a4bfde222d8c947a P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:01.488649 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c21ec56a12be48c4966762868e2c2aed. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:01.479905 755 tablet_service.cc:1505] Processing CreateTablet for tablet c21ec56a12be48c4966762868e2c2aed (DEFAULT_TABLE table=default.Table_5668f431df944d4291ec4d158ff43e57 [id=749261ef7d7c45b7a91fb5db5908baa3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:01.489070 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c21ec56a12be48c4966762868e2c2aed. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:01.489030 711 leader_election.cc:343] T 24cb2a71ba3841d4a5e3e5c6fe99ae2f P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:01.489434 745 tablet_service.cc:1505] Processing CreateTablet for tablet 8c972c88f64e486bb2b75306549bb23f (DEFAULT_TABLE table=default.Table_88c311b8b4534d5190945358ee18fe6a [id=c4e7234e2c3f43e6ab5f972e2cbd7785]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:01.489841 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c972c88f64e486bb2b75306549bb23f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:01.491281 879 tablet_service.cc:1505] Processing CreateTablet for tablet 839f3f679c344fc5a3746da6df2cfe93 (DEFAULT_TABLE table=default.Table_e1ecca1682bb467789ae7beec4a3f11d [id=b4be764ec2f64bdfa8e3abb9b92a6e93]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:01.491745 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 839f3f679c344fc5a3746da6df2cfe93. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:01.480211 742 tablet_service.cc:1505] Processing CreateTablet for tablet 0ffd22d8610b4329a0a4f753d9e60149 (DEFAULT_TABLE table=default.Table_9bc6d7e931a34e00b8a1c2233fa53c3f [id=c8105da2dd8540aa977bf54b09d874ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:01.493662 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ffd22d8610b4329a0a4f753d9e60149. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:01.480434 748 tablet_service.cc:1505] Processing CreateTablet for tablet 839f3f679c344fc5a3746da6df2cfe93 (DEFAULT_TABLE table=default.Table_e1ecca1682bb467789ae7beec4a3f11d [id=b4be764ec2f64bdfa8e3abb9b92a6e93]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:01.499138 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 839f3f679c344fc5a3746da6df2cfe93. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:01.500205 893 tablet_service.cc:1552] Processing DeleteTablet for tablet fd900eec3f0c43bf965c148ea90870a3 with delete_type TABLET_DATA_DELETED (replaced by tablet b2b2d64dcf964da8be4b544dbbe31977) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:01.500600 893 tablet_service.cc:1552] Processing DeleteTablet for tablet aa776cde2f494fa5810539edf9f289a1 with delete_type TABLET_DATA_DELETED (replaced by tablet c9bd3f2deb5d4af09b7918269a98509f) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:01.500936 893 tablet_service.cc:1552] Processing DeleteTablet for tablet b5d5b737c86844d58c51b2388c0b1705 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:54 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:01.501296 893 tablet_service.cc:1552] Processing DeleteTablet for tablet 75236b2fa82f4f28b82efae9a2c5160d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:52 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:01.501837 893 tablet_service.cc:1505] Processing CreateTablet for tablet c4b1ef53dbaa47c39361cc4ef3be71d8 (DEFAULT_TABLE table=default.Table_dc5f0ee620604d15a617a3c7ed345c37 [id=5de39f63a7a74332ba335249210b0344]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:01.502279 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4b1ef53dbaa47c39361cc4ef3be71d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:01.502806 738 tablet_service.cc:1505] Processing CreateTablet for tablet 65e761f9f9fb490586d6415a5a8565dd (DEFAULT_TABLE table=default.Table_b131ed8021574ae885778d1e8da0ad8e [id=dd34a215f0cb45b7929a1b86576027f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:01.503166 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65e761f9f9fb490586d6415a5a8565dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:01.506809 749 tablet_service.cc:1505] Processing CreateTablet for tablet 058d582d8b7f459bbacff370b26699a3 (DEFAULT_TABLE table=default.Table_e7b916cfb7f64f5d9e1ed5b9337f4206 [id=48b975892e094be093171bb527a8bce0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:01.507189 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 058d582d8b7f459bbacff370b26699a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:01.480636 741 tablet_service.cc:1505] Processing CreateTablet for tablet e2c8983cded84a1c8b4205c48a4eac33 (DEFAULT_TABLE table=default.Table_4780e4a5339245d1acdf62a82234a1d2 [id=22d68257e1714ab8a91dd28c562c4cc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:01.511085 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2c8983cded84a1c8b4205c48a4eac33. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:01.481141 756 tablet_service.cc:1505] Processing CreateTablet for tablet a27496be3431446da9cb83452a3893e3 (DEFAULT_TABLE table=default.Table_9dfd24b5c34c4b9f965292b798351bc1 [id=5bce7a821707460aaec9997adf37a16d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:01.515673 1682 leader_election.cc:290] T 75236b2fa82f4f28b82efae9a2c5160d P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:01.515774 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a27496be3431446da9cb83452a3893e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:01.515780 1057 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "75236b2fa82f4f28b82efae9a2c5160d" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:52:01.516007 1057 raft_consensus.cc:3060] T 75236b2fa82f4f28b82efae9a2c5160d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
W20251029 02:52:01.516161 847 leader_election.cc:343] T 75236b2fa82f4f28b82efae9a2c5160d P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 75236b2fa82f4f28b82efae9a2c5160d
I20251029 02:52:01.480885 754 tablet_service.cc:1505] Processing CreateTablet for tablet 050b33f64bcf42c5a5e80600520e3727 (DEFAULT_TABLE table=default.Table_bc03c0e4711a4763a6f67e7c0687215e [id=8ed7b2d71aaf4c8c9ec60648319faff2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:01.517452 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 050b33f64bcf42c5a5e80600520e3727. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:01.520488 890 tablet_service.cc:1505] Processing CreateTablet for tablet 65e761f9f9fb490586d6415a5a8565dd (DEFAULT_TABLE table=default.Table_b131ed8021574ae885778d1e8da0ad8e [id=dd34a215f0cb45b7929a1b86576027f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:01.520696 1057 raft_consensus.cc:2468] T 75236b2fa82f4f28b82efae9a2c5160d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:01.520993 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65e761f9f9fb490586d6415a5a8565dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:01.521351 849 leader_election.cc:304] T 75236b2fa82f4f28b82efae9a2c5160d P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters: b495a49349324f7d81908f2da700f0d9
I20251029 02:52:01.523206 891 tablet_service.cc:1505] Processing CreateTablet for tablet 058d582d8b7f459bbacff370b26699a3 (DEFAULT_TABLE table=default.Table_e7b916cfb7f64f5d9e1ed5b9337f4206 [id=48b975892e094be093171bb527a8bce0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:01.523590 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 058d582d8b7f459bbacff370b26699a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:01.535548 1167 tablet_bootstrap.cc:492] T 0a47781d036149569e1164dd938a45e9 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:01.535730 1167 ts_tablet_manager.cc:1403] T 0a47781d036149569e1164dd938a45e9 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.118s user 0.008s sys 0.008s
I20251029 02:52:01.536564 1167 raft_consensus.cc:359] T 0a47781d036149569e1164dd938a45e9 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:01.536767 1167 raft_consensus.cc:385] T 0a47781d036149569e1164dd938a45e9 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:01.536864 1167 raft_consensus.cc:740] T 0a47781d036149569e1164dd938a45e9 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:01.537271 1167 consensus_queue.cc:260] T 0a47781d036149569e1164dd938a45e9 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:01.539194 1682 raft_consensus.cc:2804] T 75236b2fa82f4f28b82efae9a2c5160d P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:01.543823 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 14698af6348f41e384f10569392d9b23 (table default.Table_ef563eb489f947d4a4772678806a0777 [id=7de93d6d48ab4d58b5aed9319d7299a4]) successfully deleted
W20251029 02:52:01.544098 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 7a1866afc7564217ad777e7ce53786c5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7a1866afc7564217ad777e7ce53786c5 already in progress: creating tablet
I20251029 02:52:01.550073 1176 ts_tablet_manager.cc:1929] T aa776cde2f494fa5810539edf9f289a1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:01.550307 1176 log.cc:1199] T aa776cde2f494fa5810539edf9f289a1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/aa776cde2f494fa5810539edf9f289a1
I20251029 02:52:01.550320 1166 tablet_bootstrap.cc:492] T 078ace5eada044c88d2462efb2b57aa8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:01.550523 1166 ts_tablet_manager.cc:1403] T 078ace5eada044c88d2462efb2b57aa8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.257s user 0.010s sys 0.006s
I20251029 02:52:01.550606 745 tablet_service.cc:1505] Processing CreateTablet for tablet c4b1ef53dbaa47c39361cc4ef3be71d8 (DEFAULT_TABLE table=default.Table_dc5f0ee620604d15a617a3c7ed345c37 [id=5de39f63a7a74332ba335249210b0344]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:01.550793 1176 ts_tablet_manager.cc:1950] T aa776cde2f494fa5810539edf9f289a1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:01.551046 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4b1ef53dbaa47c39361cc4ef3be71d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:01.551553 1166 raft_consensus.cc:359] T 078ace5eada044c88d2462efb2b57aa8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:01.551654 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 400) took 4057 ms. Trace:
I20251029 02:52:01.551733 1166 raft_consensus.cc:385] T 078ace5eada044c88d2462efb2b57aa8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:01.551826 1166 raft_consensus.cc:740] T 078ace5eada044c88d2462efb2b57aa8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:01.552026 1166 consensus_queue.cc:260] T 078ace5eada044c88d2462efb2b57aa8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:01.552554 1166 ts_tablet_manager.cc:1434] T 078ace5eada044c88d2462efb2b57aa8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 02:52:01.552807 1166 tablet_bootstrap.cc:492] T 4bf06476c87a4549a73f5994314bb75f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:01.554253 1057 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1f5b40c0403d4a49a4bfde222d8c947a" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:52:01.554818 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 69597d48631d43b6a5b92abbf60d5ed3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 69597d48631d43b6a5b92abbf60d5ed3 already in progress: creating tablet
W20251029 02:52:01.555042 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 13ee62fc2339410b9db70dfd3d3c6fa6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 13ee62fc2339410b9db70dfd3d3c6fa6 already in progress: creating tablet
I20251029 02:52:01.555076 1057 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24cb2a71ba3841d4a5e3e5c6fe99ae2f" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:52:01.555238 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet faa835de84ff45b19525d1d603678987 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet faa835de84ff45b19525d1d603678987 already in progress: creating tablet
W20251029 02:52:01.555401 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 589a521f5cc24e6289c37b9cf256bce5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 589a521f5cc24e6289c37b9cf256bce5 already in progress: creating tablet
I20251029 02:52:01.556109 1166 tablet_bootstrap.cc:654] T 4bf06476c87a4549a73f5994314bb75f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:01.556401 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet 24016ceaa84147eca0b94b3ad288e772 with delete_type TABLET_DATA_DELETED (replaced by tablet 91e31f0b06924740bdf7525cae79d8ef) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:01.557934 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet 1df0d944717d487a98ea83a8ec04bd46 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:01 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:01.551784 1176 rpcz_store.cc:276] 1029 02:51:57.494336 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:57.494563 (+ 227us) service_pool.cc:225] Handling call
1029 02:52:01.456545 (+3961982us) ts_tablet_manager.cc:1151] Deleting tablet aa776cde2f494fa5810539edf9f289a1
1029 02:52:01.456553 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:01.546549 (+ 89996us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:01.549932 (+ 3383us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:01.551615 (+ 1683us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3961243}
I20251029 02:52:01.558465 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet 2e003e140a244d46b15a2c3e84f68936 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:01 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:01.558604 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 401) took 4053 ms. Trace:
I20251029 02:52:01.558724 1176 rpcz_store.cc:276] 1029 02:51:57.505267 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:57.530070 (+ 24803us) service_pool.cc:225] Handling call
1029 02:52:01.558424 (+4028354us) ts_tablet_manager.cc:1151] Deleting tablet 4aa78f4ac38e4919864c924e50f1aaf4
1029 02:52:01.558434 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:01.558568 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4027884}
I20251029 02:52:01.559100 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 399) took 4067 ms. Trace:
I20251029 02:52:01.559211 1176 rpcz_store.cc:276] 1029 02:51:57.492120 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:57.530617 (+ 38497us) service_pool.cc:225] Handling call
1029 02:52:01.558954 (+4028337us) ts_tablet_manager.cc:1151] Deleting tablet c32c28b0e74a4af3a0e69bbf24ab1a1f
1029 02:52:01.558961 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:01.559068 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4027945}
I20251029 02:52:01.559469 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet 5bd2163bc4be467fb695629bfbb5a6ca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:01 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:01.559568 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 419) took 3640 ms. Trace:
I20251029 02:52:01.559674 1176 rpcz_store.cc:276] 1029 02:51:57.919497 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:57.919544 (+ 47us) service_pool.cc:225] Handling call
1029 02:52:01.559431 (+3639887us) ts_tablet_manager.cc:1151] Deleting tablet 23b9b2ea814645eba63e5f6b12cfb966
1029 02:52:01.559439 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:01.559528 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3639648}
I20251029 02:52:01.559964 1176 tablet_replica.cc:333] T 9e07291d6b5645feb4d5890f58e1ef74 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:01.560026 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet 19609bc7f06645dc811b4bae8ce506c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:01 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:01.560184 1176 raft_consensus.cc:2243] T 9e07291d6b5645feb4d5890f58e1ef74 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:01.560364 1176 raft_consensus.cc:2272] T 9e07291d6b5645feb4d5890f58e1ef74 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:01.560926 1176 ts_tablet_manager.cc:1916] T 9e07291d6b5645feb4d5890f58e1ef74 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:01.561972 1029 tablet_service.cc:1505] Processing CreateTablet for tablet 09e3ce70eb9a4a6c9035dabbf7434636 (DEFAULT_TABLE table=default.Table_be523b14f1164046a23c062a7907de02 [id=00565347222c4715b4e8471055916a52]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:01.562422 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09e3ce70eb9a4a6c9035dabbf7434636. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:01.568018 1166 tablet_bootstrap.cc:492] T 4bf06476c87a4549a73f5994314bb75f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:01.568218 1166 ts_tablet_manager.cc:1403] T 4bf06476c87a4549a73f5994314bb75f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.000s
I20251029 02:52:01.569139 1166 raft_consensus.cc:359] T 4bf06476c87a4549a73f5994314bb75f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:01.569828 1166 raft_consensus.cc:385] T 4bf06476c87a4549a73f5994314bb75f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:01.569926 1166 raft_consensus.cc:740] T 4bf06476c87a4549a73f5994314bb75f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:01.570305 1166 consensus_queue.cc:260] T 4bf06476c87a4549a73f5994314bb75f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:01.479791 1175 ts_tablet_manager.cc:1929] T a5c3a15510cb4321bbb80d61fabd0c0a P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:01.570851 1175 log.cc:1199] T a5c3a15510cb4321bbb80d61fabd0c0a P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/a5c3a15510cb4321bbb80d61fabd0c0a
I20251029 02:52:01.571314 1175 ts_tablet_manager.cc:1950] T a5c3a15510cb4321bbb80d61fabd0c0a P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:01.571534 1039 tablet_service.cc:1552] Processing DeleteTablet for tablet b5d5b737c86844d58c51b2388c0b1705 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:54 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:01.572055 1025 tablet_service.cc:1505] Processing CreateTablet for tablet 80f54d8a486a4196aca3c28c84c0872f (DEFAULT_TABLE table=default.Table_25a8f4870ca94816934d0214177fc278 [id=b8eeea99f4df4aedbd187b8805525905]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:01.572464 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 80f54d8a486a4196aca3c28c84c0872f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:01.572683 1032 tablet_service.cc:1505] Processing CreateTablet for tablet 8c972c88f64e486bb2b75306549bb23f (DEFAULT_TABLE table=default.Table_88c311b8b4534d5190945358ee18fe6a [id=c4e7234e2c3f43e6ab5f972e2cbd7785]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:01.573066 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c972c88f64e486bb2b75306549bb23f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:01.574779 1038 tablet_service.cc:1505] Processing CreateTablet for tablet 0ffd22d8610b4329a0a4f753d9e60149 (DEFAULT_TABLE table=default.Table_9bc6d7e931a34e00b8a1c2233fa53c3f [id=c8105da2dd8540aa977bf54b09d874ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:01.575100 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0ffd22d8610b4329a0a4f753d9e60149. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:01.578039 1031 tablet_service.cc:1505] Processing CreateTablet for tablet 839f3f679c344fc5a3746da6df2cfe93 (DEFAULT_TABLE table=default.Table_e1ecca1682bb467789ae7beec4a3f11d [id=b4be764ec2f64bdfa8e3abb9b92a6e93]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:01.578497 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 839f3f679c344fc5a3746da6df2cfe93. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:01.582402 1682 raft_consensus.cc:697] T 75236b2fa82f4f28b82efae9a2c5160d P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:52:01.582693 1041 tablet_service.cc:1505] Processing CreateTablet for tablet e2c8983cded84a1c8b4205c48a4eac33 (DEFAULT_TABLE table=default.Table_4780e4a5339245d1acdf62a82234a1d2 [id=22d68257e1714ab8a91dd28c562c4cc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:01.582767 1682 consensus_queue.cc:237] T 75236b2fa82f4f28b82efae9a2c5160d P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:01.583128 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2c8983cded84a1c8b4205c48a4eac33. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:01.585649 1665 raft_consensus.cc:493] T 576c4e6e70e24b0399ac9597a30f0569 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:01.585969 1665 raft_consensus.cc:515] T 576c4e6e70e24b0399ac9597a30f0569 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:01.586558 1029 tablet_service.cc:1505] Processing CreateTablet for tablet 65e761f9f9fb490586d6415a5a8565dd (DEFAULT_TABLE table=default.Table_b131ed8021574ae885778d1e8da0ad8e [id=dd34a215f0cb45b7929a1b86576027f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:01.586658 1665 leader_election.cc:290] T 576c4e6e70e24b0399ac9597a30f0569 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:52:01.586604 1028 tablet_service.cc:1505] Processing CreateTablet for tablet 050b33f64bcf42c5a5e80600520e3727 (DEFAULT_TABLE table=default.Table_bc03c0e4711a4763a6f67e7c0687215e [id=8ed7b2d71aaf4c8c9ec60648319faff2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:01.587034 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65e761f9f9fb490586d6415a5a8565dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:01.587096 1037 tablet_service.cc:1505] Processing CreateTablet for tablet a27496be3431446da9cb83452a3893e3 (DEFAULT_TABLE table=default.Table_9dfd24b5c34c4b9f965292b798351bc1 [id=5bce7a821707460aaec9997adf37a16d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:01.587468 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a27496be3431446da9cb83452a3893e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:01.589830 918 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "576c4e6e70e24b0399ac9597a30f0569" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:01.590445 711 leader_election.cc:343] T 576c4e6e70e24b0399ac9597a30f0569 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:01.591102 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 407) took 3214 ms. Trace:
I20251029 02:52:01.591202 1175 rpcz_store.cc:276] 1029 02:51:58.376096 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:58.479223 (+103127us) service_pool.cc:225] Handling call
1029 02:52:01.325687 (+2846464us) ts_tablet_manager.cc:1151] Deleting tablet a5c3a15510cb4321bbb80d61fabd0c0a
1029 02:52:01.334267 (+ 8580us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:01.339072 (+ 4805us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:01.479666 (+140594us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:01.591044 (+111378us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":18838,"tablet-delete.queue_time_us":2846177}
I20251029 02:52:01.591408 1032 tablet_service.cc:1505] Processing CreateTablet for tablet c4b1ef53dbaa47c39361cc4ef3be71d8 (DEFAULT_TABLE table=default.Table_dc5f0ee620604d15a617a3c7ed345c37 [id=5de39f63a7a74332ba335249210b0344]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:01.591643 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet a5c3a15510cb4321bbb80d61fabd0c0a (table default.Table_8cf076425649436289d327edc92d4642 [id=c124cfa700724f6db0f1a12fe495c2af]) successfully deleted
I20251029 02:52:01.591811 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4b1ef53dbaa47c39361cc4ef3be71d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:01.592095 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 397) took 3232 ms. Trace:
I20251029 02:52:01.592195 1175 rpcz_store.cc:276] 1029 02:51:58.359577 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:58.359648 (+ 71us) service_pool.cc:225] Handling call
1029 02:52:01.591936 (+3232288us) ts_tablet_manager.cc:1151] Deleting tablet 23b9b2ea814645eba63e5f6b12cfb966
1029 02:52:01.591947 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:01.592044 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3111730}
I20251029 02:52:01.592303 1039 tablet_service.cc:1505] Processing CreateTablet for tablet c21ec56a12be48c4966762868e2c2aed (DEFAULT_TABLE table=default.Table_5668f431df944d4291ec4d158ff43e57 [id=749261ef7d7c45b7a91fb5db5908baa3]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 02:52:01.592651 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 23b9b2ea814645eba63e5f6b12cfb966 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 23b9b2ea814645eba63e5f6b12cfb966 already in progress: creating tablet
I20251029 02:52:01.592726 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c21ec56a12be48c4966762868e2c2aed. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:01.593034 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 409) took 3216 ms. Trace:
I20251029 02:52:01.593163 1175 rpcz_store.cc:276] 1029 02:51:58.376324 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:58.480245 (+103921us) service_pool.cc:225] Handling call
1029 02:52:01.592894 (+3112649us) ts_tablet_manager.cc:1151] Deleting tablet bfec87f690a640c39c58c64e16cc67e7
1029 02:52:01.592904 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:01.592986 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3112422}
W20251029 02:52:01.593552 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet bfec87f690a640c39c58c64e16cc67e7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bfec87f690a640c39c58c64e16cc67e7 already in progress: creating tablet
I20251029 02:52:01.593847 1175 tablet_replica.cc:333] T 6feec2c8e030447fae9c292a626dcd1d P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:01.594051 1175 raft_consensus.cc:2243] T 6feec2c8e030447fae9c292a626dcd1d P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:01.594231 1175 raft_consensus.cc:2272] T 6feec2c8e030447fae9c292a626dcd1d P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:01.594841 1175 ts_tablet_manager.cc:1916] T 6feec2c8e030447fae9c292a626dcd1d P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:01.595220 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 050b33f64bcf42c5a5e80600520e3727. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:01.598766 1167 ts_tablet_manager.cc:1434] T 0a47781d036149569e1164dd938a45e9 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.063s user 0.012s sys 0.004s
I20251029 02:52:01.599146 1026 tablet_service.cc:1505] Processing CreateTablet for tablet 058d582d8b7f459bbacff370b26699a3 (DEFAULT_TABLE table=default.Table_e7b916cfb7f64f5d9e1ed5b9337f4206 [id=48b975892e094be093171bb527a8bce0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:01.599261 1167 tablet_bootstrap.cc:492] T 9964ad774890422f8e14f6b99f28cb55 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:01.599190 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_5ed09727711e4eb6a60deb327d188aea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:01.599567 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 058d582d8b7f459bbacff370b26699a3. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:01.599862 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5ed09727711e4eb6a60deb327d188aea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:01.600960 519 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet d2990e9f626c40618c118266bc595684 (table default.Table_bd020e7c45a54c6e92211c4261293575 [id=f733948e92d8470da9d8050bd26a06b1]). Expected version 1 got 0
I20251029 02:52:01.601135 1717 raft_consensus.cc:493] T 7e6c4f30be624769961c9ec17c5d5dcb P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:01.601284 1717 raft_consensus.cc:515] T 7e6c4f30be624769961c9ec17c5d5dcb P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:01.601409 525 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_88289be99f39482d837c84fecf31efb1" } new_table_name: "default.Table_9d8abd5ee0364a16bd9c5156a01dd6da" modify_external_catalogs: true
I20251029 02:52:01.601979 1717 leader_election.cc:290] T 7e6c4f30be624769961c9ec17c5d5dcb P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:52:01.602146 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_a37ba50ba83a43668c7fe1af7cca7f0d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:01.602800 526 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_f35ae25e36f64d2886752bb3b1d66710" } new_table_name: "default.Table_691b1d73c4f44f098dccf26f6a51a4d0" modify_external_catalogs: true
I20251029 02:52:01.602847 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_909da9f5c1374cc2b5c318ffc1c3ac19"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:01.603364 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_909da9f5c1374cc2b5c318ffc1c3ac19 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:01.605605 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_839c3ffd7768463da17f35d024d016a0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:01.605835 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a37ba50ba83a43668c7fe1af7cca7f0d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:52:01.606268 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_839c3ffd7768463da17f35d024d016a0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:52:01.606879 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 4aa78f4ac38e4919864c924e50f1aaf4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4aa78f4ac38e4919864c924e50f1aaf4 already in progress: creating tablet
I20251029 02:52:01.607873 1172 ts_tablet_manager.cc:1434] T c9dff4dff82f49e7ad49caad5e404f09 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.123s user 0.001s sys 0.001s
I20251029 02:52:01.608371 918 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7e6c4f30be624769961c9ec17c5d5dcb" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:01.608911 712 leader_election.cc:343] T 1f5b40c0403d4a49a4bfde222d8c947a P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:01.609057 712 leader_election.cc:304] T 1f5b40c0403d4a49a4bfde222d8c947a P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
W20251029 02:52:01.609496 712 leader_election.cc:343] T 24cb2a71ba3841d4a5e3e5c6fe99ae2f P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:01.609619 712 leader_election.cc:304] T 24cb2a71ba3841d4a5e3e5c6fe99ae2f P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:01.610831 1665 raft_consensus.cc:2749] T 1f5b40c0403d4a49a4bfde222d8c947a P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:01.610764 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_d72ae8772b6044458311fc523dca2c71"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:01.611020 1665 raft_consensus.cc:2749] T 24cb2a71ba3841d4a5e3e5c6fe99ae2f P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 02:52:01.611078 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet c32c28b0e74a4af3a0e69bbf24ab1a1f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c32c28b0e74a4af3a0e69bbf24ab1a1f already in progress: creating tablet
W20251029 02:52:01.611428 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d72ae8772b6044458311fc523dca2c71 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:01.612498 1172 tablet_bootstrap.cc:492] T 42cfdf88472b44979d86836561005ed1 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:01.613516 522 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet d2990e9f626c40618c118266bc595684 (table default.Table_bd020e7c45a54c6e92211c4261293575 [id=f733948e92d8470da9d8050bd26a06b1]). Expected version 1 got 0
I20251029 02:52:01.616693 1166 ts_tablet_manager.cc:1434] T 4bf06476c87a4549a73f5994314bb75f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.048s user 0.003s sys 0.000s
I20251029 02:52:01.616765 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_225f7c2cca034ef79a9956e3d1888c5d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:01.617298 1166 tablet_bootstrap.cc:492] T 9964ad774890422f8e14f6b99f28cb55 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
W20251029 02:52:01.617334 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_225f7c2cca034ef79a9956e3d1888c5d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:52:01.618957 711 leader_election.cc:343] T 7e6c4f30be624769961c9ec17c5d5dcb P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:01.619323 1166 tablet_bootstrap.cc:654] T 9964ad774890422f8e14f6b99f28cb55 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
W20251029 02:52:01.619592 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 23b9b2ea814645eba63e5f6b12cfb966 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 23b9b2ea814645eba63e5f6b12cfb966 already in progress: creating tablet
I20251029 02:52:01.620882 1172 tablet_bootstrap.cc:654] T 42cfdf88472b44979d86836561005ed1 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:01.624365 1167 tablet_bootstrap.cc:654] T 9964ad774890422f8e14f6b99f28cb55 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:01.624919 518 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 7d1408dc544944669de519a1553d9bb7
I20251029 02:52:01.625331 523 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet f2d47238761e4ed49c88abdbb677b18a
I20251029 02:52:01.625804 523 catalog_manager.cc:6719] ReplaceTablet: tablet f2d47238761e4ed49c88abdbb677b18a deleted and replaced by tablet b4621bf9e6324eaeaa64a6ca51cd5ca9
I20251029 02:52:01.626165 523 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_3c09e84f78ee464b823fe6c104c2e207" } new_table_name: "default.Table_75b922d164054630b5de9dbe39ec275b" modify_external_catalogs: true
I20251029 02:52:01.627489 518 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_8c062218c19c46169ae3652f572ed821" } new_table_name: "default.Table_9c0d8c29e3fe473690991c14455981ba" modify_external_catalogs: true
W20251029 02:52:01.630937 525 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c0672a948e5e41c5bbfe7e8199025986 consensus state has no leader
I20251029 02:52:01.631361 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_ba15e59ee62346c7bd87a0bf7b87c9fc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:01.631950 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ba15e59ee62346c7bd87a0bf7b87c9fc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:01.632174 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_ac4056c73c0f41cda4580d2736f4c6a5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:01.632640 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ac4056c73c0f41cda4580d2736f4c6a5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:01.635743 520 master_service.cc:954] ReplaceTablet: received request to replace tablet 7d1408dc544944669de519a1553d9bb7 from {username='slave'} at 127.0.0.1:35314
I20251029 02:52:01.635938 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_9171c03558ca4705b7f71acb9f542107"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:01.636344 520 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_5668f431df944d4291ec4d158ff43e57" } new_table_name: "default.Table_ab6134b9c1e64a52a4d125acf5127d1b" modify_external_catalogs: true
W20251029 02:52:01.636493 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9171c03558ca4705b7f71acb9f542107 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:01.640003 1773 raft_consensus.cc:493] T 4bf06476c87a4549a73f5994314bb75f P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:01.640242 1773 raft_consensus.cc:515] T 4bf06476c87a4549a73f5994314bb75f P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:01.641108 1773 leader_election.cc:290] T 4bf06476c87a4549a73f5994314bb75f P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:01.626343 749 tablet_service.cc:1552] Processing DeleteTablet for tablet f2d47238761e4ed49c88abdbb677b18a with delete_type TABLET_DATA_DELETED (replaced by tablet b4621bf9e6324eaeaa64a6ca51cd5ca9) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:01.641583 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4bf06476c87a4549a73f5994314bb75f" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:01.643285 776 raft_consensus.cc:2468] T 4bf06476c87a4549a73f5994314bb75f P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:01.644272 847 leader_election.cc:304] T 4bf06476c87a4549a73f5994314bb75f P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:01.646180 1773 raft_consensus.cc:2804] T 4bf06476c87a4549a73f5994314bb75f P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:01.646358 1773 raft_consensus.cc:493] T 4bf06476c87a4549a73f5994314bb75f P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:01.646534 1773 raft_consensus.cc:3060] T 4bf06476c87a4549a73f5994314bb75f P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:01.646881 891 tablet_service.cc:1552] Processing DeleteTablet for tablet f2d47238761e4ed49c88abdbb677b18a with delete_type TABLET_DATA_DELETED (replaced by tablet b4621bf9e6324eaeaa64a6ca51cd5ca9) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:01.647197 751 tablet_service.cc:1552] Processing DeleteTablet for tablet 7d1408dc544944669de519a1553d9bb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:01 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:01.647485 893 tablet_service.cc:1552] Processing DeleteTablet for tablet 7d1408dc544944669de519a1553d9bb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:01 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:01.649468 1167 tablet_bootstrap.cc:492] T 9964ad774890422f8e14f6b99f28cb55 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:01.649768 1167 ts_tablet_manager.cc:1403] T 9964ad774890422f8e14f6b99f28cb55 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.051s user 0.003s sys 0.004s
I20251029 02:52:01.650750 1167 raft_consensus.cc:359] T 9964ad774890422f8e14f6b99f28cb55 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:01.651062 1167 raft_consensus.cc:385] T 9964ad774890422f8e14f6b99f28cb55 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:01.651177 1167 raft_consensus.cc:740] T 9964ad774890422f8e14f6b99f28cb55 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:01.652951 1167 consensus_queue.cc:260] T 9964ad774890422f8e14f6b99f28cb55 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:01.653674 1167 ts_tablet_manager.cc:1434] T 9964ad774890422f8e14f6b99f28cb55 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251029 02:52:01.654039 1167 tablet_bootstrap.cc:492] T 078ace5eada044c88d2462efb2b57aa8 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:01.656013 1167 tablet_bootstrap.cc:654] T 078ace5eada044c88d2462efb2b57aa8 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
W20251029 02:52:01.665707 523 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9a58ecfa646c41cdae4d75724767d0d9 consensus state has no leader
W20251029 02:52:01.666764 518 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 85f979babbd34eaeaa89ea5906ddab90 consensus state has no leader
I20251029 02:52:01.669152 1682 raft_consensus.cc:493] T aa776cde2f494fa5810539edf9f289a1 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:01.669433 1682 raft_consensus.cc:515] T aa776cde2f494fa5810539edf9f289a1 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
W20251029 02:52:01.669796 520 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c21ec56a12be48c4966762868e2c2aed consensus state has no leader
W20251029 02:52:01.671031 847 leader_election.cc:343] T aa776cde2f494fa5810539edf9f289a1 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: aa776cde2f494fa5810539edf9f289a1
I20251029 02:52:01.671357 1682 leader_election.cc:290] T aa776cde2f494fa5810539edf9f289a1 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
W20251029 02:52:01.672228 849 leader_election.cc:343] T aa776cde2f494fa5810539edf9f289a1 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Not found: Tablet not found: aa776cde2f494fa5810539edf9f289a1
I20251029 02:52:01.672376 849 leader_election.cc:304] T aa776cde2f494fa5810539edf9f289a1 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9
I20251029 02:52:01.672982 1682 raft_consensus.cc:2749] T aa776cde2f494fa5810539edf9f289a1 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:01.674805 1773 raft_consensus.cc:515] T 4bf06476c87a4549a73f5994314bb75f P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:01.676467 1773 leader_election.cc:290] T 4bf06476c87a4549a73f5994314bb75f P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:01.677016 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4bf06476c87a4549a73f5994314bb75f" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:01.677269 776 raft_consensus.cc:3060] T 4bf06476c87a4549a73f5994314bb75f P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:01.642472 1057 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4bf06476c87a4549a73f5994314bb75f" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:01.679324 1057 raft_consensus.cc:2468] T 4bf06476c87a4549a73f5994314bb75f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:01.674896 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "576c4e6e70e24b0399ac9597a30f0569" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:01.675509 1058 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7e6c4f30be624769961c9ec17c5d5dcb" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:01.675875 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4bf06476c87a4549a73f5994314bb75f" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:52:01.679989 1060 raft_consensus.cc:3060] T 4bf06476c87a4549a73f5994314bb75f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:01.682142 776 raft_consensus.cc:2468] T 4bf06476c87a4549a73f5994314bb75f P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:01.682706 847 leader_election.cc:304] T 4bf06476c87a4549a73f5994314bb75f P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:01.682881 1060 raft_consensus.cc:2468] T 4bf06476c87a4549a73f5994314bb75f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:01.684154 526 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_a45ccb3fef5d4947ab7a8709b7c32119" } modify_external_catalogs: true
I20251029 02:52:01.684263 520 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_8dacd5c558a04e47b424533b985aee9e" } modify_external_catalogs: true
I20251029 02:52:01.684360 526 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_a45ccb3fef5d4947ab7a8709b7c32119" } modify_external_catalogs: true
I20251029 02:52:01.684412 520 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_8dacd5c558a04e47b424533b985aee9e" } modify_external_catalogs: true
I20251029 02:52:01.684325 518 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_6a78c7ef966c490aaa5e73f56d6d194e" } modify_external_catalogs: true
I20251029 02:52:01.684571 518 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_6a78c7ef966c490aaa5e73f56d6d194e" } modify_external_catalogs: true
I20251029 02:52:01.684825 519 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_15cde6ff49784e5082de7519ce61c81d" } modify_external_catalogs: true
I20251029 02:52:01.685002 519 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_15cde6ff49784e5082de7519ce61c81d" } modify_external_catalogs: true
I20251029 02:52:01.686137 1041 tablet_service.cc:1552] Processing DeleteTablet for tablet 7d1408dc544944669de519a1553d9bb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:01 UTC) from {username='slave'} at 127.0.0.1:37626
W20251029 02:52:01.686815 712 leader_election.cc:343] T 576c4e6e70e24b0399ac9597a30f0569 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:01.686951 712 leader_election.cc:304] T 576c4e6e70e24b0399ac9597a30f0569 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:01.687485 1717 raft_consensus.cc:2749] T 576c4e6e70e24b0399ac9597a30f0569 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:01.686353 1026 tablet_service.cc:1552] Processing DeleteTablet for tablet f2d47238761e4ed49c88abdbb677b18a with delete_type TABLET_DATA_DELETED (replaced by tablet b4621bf9e6324eaeaa64a6ca51cd5ca9) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:01.695324 1172 tablet_bootstrap.cc:492] T 42cfdf88472b44979d86836561005ed1 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:01.695538 1172 ts_tablet_manager.cc:1403] T 42cfdf88472b44979d86836561005ed1 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.084s user 0.001s sys 0.003s
I20251029 02:52:01.690644 1176 ts_tablet_manager.cc:1929] T 9e07291d6b5645feb4d5890f58e1ef74 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:01.691833 1736 tablet.cc:1722] T aacb6f4b9284457bb6607e3a85d5c809 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 02:52:01.703248 1773 raft_consensus.cc:2804] T 4bf06476c87a4549a73f5994314bb75f P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:01.703838 1773 raft_consensus.cc:697] T 4bf06476c87a4549a73f5994314bb75f P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:52:01.704140 1773 consensus_queue.cc:237] T 4bf06476c87a4549a73f5994314bb75f P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:01.706763 1176 log.cc:1199] T 9e07291d6b5645feb4d5890f58e1ef74 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/9e07291d6b5645feb4d5890f58e1ef74
I20251029 02:52:01.706894 1782 tablet.cc:1722] T aacb6f4b9284457bb6607e3a85d5c809 P c756528642a543eeacd93cef3e335a7d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 02:52:01.708612 1176 ts_tablet_manager.cc:1950] T 9e07291d6b5645feb4d5890f58e1ef74 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
W20251029 02:52:01.709115 712 leader_election.cc:343] T 7e6c4f30be624769961c9ec17c5d5dcb P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:01.709265 712 leader_election.cc:304] T 7e6c4f30be624769961c9ec17c5d5dcb P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:01.709645 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 418) took 3790 ms. Trace:
I20251029 02:52:01.709937 1665 raft_consensus.cc:2749] T 7e6c4f30be624769961c9ec17c5d5dcb P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:01.710110 1176 rpcz_store.cc:276] 1029 02:51:57.919396 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:57.922230 (+ 2834us) service_pool.cc:225] Handling call
1029 02:52:01.559900 (+3637670us) ts_tablet_manager.cc:1151] Deleting tablet 9e07291d6b5645feb4d5890f58e1ef74
1029 02:52:01.559912 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:01.615428 (+ 55516us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:01.690500 (+ 75072us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:01.709582 (+ 19082us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3637431}
I20251029 02:52:01.710402 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 7e6c4f30be624769961c9ec17c5d5dcb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:55 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:01.714293 1174 ts_tablet_manager.cc:1929] T 24016ceaa84147eca0b94b3ad288e772 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251029 02:52:01.715039 1172 raft_consensus.cc:359] T 42cfdf88472b44979d86836561005ed1 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:01.715385 1172 raft_consensus.cc:385] T 42cfdf88472b44979d86836561005ed1 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:01.715602 1172 raft_consensus.cc:740] T 42cfdf88472b44979d86836561005ed1 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:01.715646 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 411) took 3805 ms. Trace:
I20251029 02:52:01.715978 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 9e07291d6b5645feb4d5890f58e1ef74 (table default.Table_0edd4b07469b4d24b302ecdb6859d1d1 [id=e71aea48b9884bacb250412624ec4615]) successfully deleted
I20251029 02:52:01.717259 1176 rpcz_store.cc:276] 1029 02:51:57.909740 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:57.909796 (+ 56us) service_pool.cc:225] Handling call
1029 02:52:01.715470 (+3805674us) ts_tablet_manager.cc:1151] Deleting tablet 2f198c3259864f7ab3625cb7bfae20b0
1029 02:52:01.715480 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:01.715598 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3790575}
I20251029 02:52:01.717730 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 421) took 3780 ms. Trace:
W20251029 02:52:01.718641 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 2f198c3259864f7ab3625cb7bfae20b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2f198c3259864f7ab3625cb7bfae20b0 already in progress: creating tablet
I20251029 02:52:01.720327 1176 rpcz_store.cc:276] 1029 02:51:57.937455 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:57.937537 (+ 82us) service_pool.cc:225] Handling call
1029 02:52:01.717595 (+3780058us) ts_tablet_manager.cc:1151] Deleting tablet afb19ba39399410bb90b6225d7b5b5b1
1029 02:52:01.717604 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:01.717687 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3766481}
I20251029 02:52:01.720744 1176 tablet_replica.cc:333] T fa5127bcfcda40a7b5ad4ef016af072a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
W20251029 02:52:01.721026 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet afb19ba39399410bb90b6225d7b5b5b1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet afb19ba39399410bb90b6225d7b5b5b1 already in progress: creating tablet
I20251029 02:52:01.721503 1176 raft_consensus.cc:2243] T fa5127bcfcda40a7b5ad4ef016af072a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:01.723513 1176 raft_consensus.cc:2272] T fa5127bcfcda40a7b5ad4ef016af072a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:01.724282 1176 ts_tablet_manager.cc:1916] T fa5127bcfcda40a7b5ad4ef016af072a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:01.725626 1172 consensus_queue.cc:260] T 42cfdf88472b44979d86836561005ed1 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:01.726233 1172 ts_tablet_manager.cc:1434] T 42cfdf88472b44979d86836561005ed1 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.031s user 0.000s sys 0.002s
I20251029 02:52:01.729180 1175 ts_tablet_manager.cc:1929] T 6feec2c8e030447fae9c292a626dcd1d P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:01.732659 517 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet d2990e9f626c40618c118266bc595684 (table default.Table_bd020e7c45a54c6e92211c4261293575 [id=f733948e92d8470da9d8050bd26a06b1]). Expected version 1 got 0
I20251029 02:52:01.732970 517 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet 4bf06476c87a4549a73f5994314bb75f (table default.Table_bafe26b1fca14cc9b908984b7d8a2442 [id=046010a03d774e91b0ad10df97848c56]). Expected version 1 got 0
I20251029 02:52:01.734248 525 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet 4bf06476c87a4549a73f5994314bb75f (table default.Table_bafe26b1fca14cc9b908984b7d8a2442 [id=046010a03d774e91b0ad10df97848c56]). Expected version 1 got 0
I20251029 02:52:01.738854 525 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet 42cfdf88472b44979d86836561005ed1 (table default.Table_3b26d01c7bce4d678c1b65f1753e3662 [id=5e0c72957b9a491d960b33d3fffd8f70]). Expected version 1 got 0
I20251029 02:52:01.739741 526 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 6142018e44af4d32818adefa2ee43d02
I20251029 02:52:01.745878 520 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet b3de6670357c4ae59eb77469b2fb434c
I20251029 02:52:01.746029 519 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 217144957930495aa3e9dfcfde268e71
I20251029 02:52:01.746418 518 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet d041e87ceb11441983d0e38b7929c3a4
I20251029 02:52:01.753330 518 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet 42cfdf88472b44979d86836561005ed1 (table default.Table_3b26d01c7bce4d678c1b65f1753e3662 [id=5e0c72957b9a491d960b33d3fffd8f70]). Expected version 1 got 0
I20251029 02:52:01.740593 756 tablet_service.cc:1505] Processing CreateTablet for tablet 91e31f0b06924740bdf7525cae79d8ef (DEFAULT_TABLE table=default.Table_f06003da6e324504ade193bbbb11d36a [id=bd279b3ce623477ca20e8d35e3b7c0df]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:01.755249 1032 tablet_service.cc:1505] Processing CreateTablet for tablet 98bbaec01186435b8bcc7257a1390185 (DEFAULT_TABLE table=default.Table_15da0c38ef4a4e30a558753430df9072 [id=659a7343994a4172807db56112054d48]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:01.740770 749 tablet_service.cc:1505] Processing CreateTablet for tablet 67c3b526a5cd4da7800b7b72b0ebe29d (DEFAULT_TABLE table=default.Table_151d1210897f41ceb7e07cca8a1e478c [id=104a556fb8a941468f41c1bc2cdfcb7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:01.741221 754 tablet_service.cc:1552] Processing DeleteTablet for tablet 6142018e44af4d32818adefa2ee43d02 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:01 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:01.741391 751 tablet_service.cc:1505] Processing CreateTablet for tablet 2c28131d740c411dbb07898d222630ff (DEFAULT_TABLE table=default.Table_28e9d77a943b43588e674c3d4825c74f [id=56b577e6e2af4bb2895fb4e8c1573adc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:01.742074 745 tablet_service.cc:1505] Processing CreateTablet for tablet 98bbaec01186435b8bcc7257a1390185 (DEFAULT_TABLE table=default.Table_15da0c38ef4a4e30a558753430df9072 [id=659a7343994a4172807db56112054d48]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:01.743469 1172 tablet_bootstrap.cc:492] T b2b2d64dcf964da8be4b544dbbe31977 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:01.755956 1026 tablet_service.cc:1505] Processing CreateTablet for tablet 2c28131d740c411dbb07898d222630ff (DEFAULT_TABLE table=default.Table_28e9d77a943b43588e674c3d4825c74f [id=56b577e6e2af4bb2895fb4e8c1573adc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:01.747663 742 tablet_service.cc:1552] Processing DeleteTablet for tablet b3de6670357c4ae59eb77469b2fb434c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:01 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:01.747872 755 tablet_service.cc:1552] Processing DeleteTablet for tablet 217144957930495aa3e9dfcfde268e71 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:01 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:01.749788 748 tablet_service.cc:1552] Processing DeleteTablet for tablet d041e87ceb11441983d0e38b7929c3a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:01 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:01.756824 1041 tablet_service.cc:1505] Processing CreateTablet for tablet 67c3b526a5cd4da7800b7b72b0ebe29d (DEFAULT_TABLE table=default.Table_151d1210897f41ceb7e07cca8a1e478c [id=104a556fb8a941468f41c1bc2cdfcb7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:01.755412 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91e31f0b06924740bdf7525cae79d8ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:01.757673 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c28131d740c411dbb07898d222630ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:01.757622 1037 tablet_service.cc:1552] Processing DeleteTablet for tablet 6142018e44af4d32818adefa2ee43d02 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:01 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:01.758528 1037 tablet_service.cc:1505] Processing CreateTablet for tablet 91e31f0b06924740bdf7525cae79d8ef (DEFAULT_TABLE table=default.Table_f06003da6e324504ade193bbbb11d36a [id=bd279b3ce623477ca20e8d35e3b7c0df]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:01.759594 1038 tablet_service.cc:1505] Processing CreateTablet for tablet 0415b52fadf948238c4e83dc09000f96 (DEFAULT_TABLE table=default.Table_cb5488bb7e104676aad8f3db9be87802 [id=3bb4d1d8901e49a1888345fbdc1343aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:01.760751 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet 217144957930495aa3e9dfcfde268e71 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:01 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:01.761956 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0415b52fadf948238c4e83dc09000f96. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:01.762451 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67c3b526a5cd4da7800b7b72b0ebe29d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:01.762830 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67c3b526a5cd4da7800b7b72b0ebe29d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:01.764345 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c28131d740c411dbb07898d222630ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:01.765651 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91e31f0b06924740bdf7525cae79d8ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:01.766018 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98bbaec01186435b8bcc7257a1390185. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:01.754298 1784 tablet.cc:1722] T aacb6f4b9284457bb6607e3a85d5c809 P b495a49349324f7d81908f2da700f0d9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 02:52:01.773319 1172 tablet_bootstrap.cc:654] T b2b2d64dcf964da8be4b544dbbe31977 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:01.779181 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet b3de6670357c4ae59eb77469b2fb434c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:01 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:01.755465 1175 log.cc:1199] T 6feec2c8e030447fae9c292a626dcd1d P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/6feec2c8e030447fae9c292a626dcd1d
I20251029 02:52:01.784343 1682 raft_consensus.cc:493] T 9964ad774890422f8e14f6b99f28cb55 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:01.784583 1174 log.cc:1199] T 24016ceaa84147eca0b94b3ad288e772 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/24016ceaa84147eca0b94b3ad288e772
I20251029 02:52:01.784533 893 tablet_service.cc:1505] Processing CreateTablet for tablet 98bbaec01186435b8bcc7257a1390185 (DEFAULT_TABLE table=default.Table_15da0c38ef4a4e30a558753430df9072 [id=659a7343994a4172807db56112054d48]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:01.784998 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98bbaec01186435b8bcc7257a1390185. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:01.785104 1174 ts_tablet_manager.cc:1950] T 24016ceaa84147eca0b94b3ad288e772 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:01.786777 1174 tablet_replica.cc:333] T fd900eec3f0c43bf965c148ea90870a3 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:01.784533 1682 raft_consensus.cc:515] T 9964ad774890422f8e14f6b99f28cb55 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:01.787734 1174 raft_consensus.cc:2243] T fd900eec3f0c43bf965c148ea90870a3 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:01.789340 891 tablet_service.cc:1505] Processing CreateTablet for tablet 2c28131d740c411dbb07898d222630ff (DEFAULT_TABLE table=default.Table_28e9d77a943b43588e674c3d4825c74f [id=56b577e6e2af4bb2895fb4e8c1573adc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:01.790175 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9964ad774890422f8e14f6b99f28cb55" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:01.794291 518 catalog_manager.cc:6194] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: default.Table_691b1d73c4f44f098dccf26f6a51a4d0 [id=9f95160639ad40e995a6c89e297a0fad] alter complete (version 2)
I20251029 02:52:01.798238 1682 leader_election.cc:290] T 9964ad774890422f8e14f6b99f28cb55 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:01.798552 890 tablet_service.cc:1505] Processing CreateTablet for tablet 91e31f0b06924740bdf7525cae79d8ef (DEFAULT_TABLE table=default.Table_f06003da6e324504ade193bbbb11d36a [id=bd279b3ce623477ca20e8d35e3b7c0df]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:01.799965 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c28131d740c411dbb07898d222630ff. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:01.800005 849 leader_election.cc:343] T 9964ad774890422f8e14f6b99f28cb55 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:01.802609 881 tablet_service.cc:1505] Processing CreateTablet for tablet 67c3b526a5cd4da7800b7b72b0ebe29d (DEFAULT_TABLE table=default.Table_151d1210897f41ceb7e07cca8a1e478c [id=104a556fb8a941468f41c1bc2cdfcb7c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:01.803355 1167 tablet_bootstrap.cc:492] T 078ace5eada044c88d2462efb2b57aa8 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:01.803593 1174 raft_consensus.cc:2272] T fd900eec3f0c43bf965c148ea90870a3 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:01.803689 895 tablet_service.cc:1552] Processing DeleteTablet for tablet 6142018e44af4d32818adefa2ee43d02 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:01 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:01.803872 1167 ts_tablet_manager.cc:1403] T 078ace5eada044c88d2462efb2b57aa8 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.150s user 0.021s sys 0.002s
I20251029 02:52:01.802696 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98bbaec01186435b8bcc7257a1390185. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:01.805588 1682 raft_consensus.cc:493] T b5d5b737c86844d58c51b2388c0b1705 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:01.806746 1174 ts_tablet_manager.cc:1916] T fd900eec3f0c43bf965c148ea90870a3 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:01.806784 879 tablet_service.cc:1505] Processing CreateTablet for tablet 0415b52fadf948238c4e83dc09000f96 (DEFAULT_TABLE table=default.Table_cb5488bb7e104676aad8f3db9be87802 [id=3bb4d1d8901e49a1888345fbdc1343aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:01.807258 895 tablet_service.cc:1552] Processing DeleteTablet for tablet 217144957930495aa3e9dfcfde268e71 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:01 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:01.807531 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91e31f0b06924740bdf7525cae79d8ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:01.807797 885 tablet_service.cc:1552] Processing DeleteTablet for tablet b3de6670357c4ae59eb77469b2fb434c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:01 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:01.808074 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0415b52fadf948238c4e83dc09000f96. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:01.808948 1167 raft_consensus.cc:359] T 078ace5eada044c88d2462efb2b57aa8 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:01.809151 1167 raft_consensus.cc:385] T 078ace5eada044c88d2462efb2b57aa8 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:01.809881 1167 raft_consensus.cc:740] T 078ace5eada044c88d2462efb2b57aa8 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:01.810144 1167 consensus_queue.cc:260] T 078ace5eada044c88d2462efb2b57aa8 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:01.810819 892 tablet_service.cc:1505] Processing CreateTablet for tablet 87b52cb65bbc4982959b0d1e40d8b74d (DEFAULT_TABLE table=default.Table_f15c66cb663740828b22d54e002adeaa [id=b9e1da40360644e9b2f5b6ea70b8bea3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:01.812013 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet d041e87ceb11441983d0e38b7929c3a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:01 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:01.813109 885 tablet_service.cc:1552] Processing DeleteTablet for tablet d041e87ceb11441983d0e38b7929c3a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:01 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:01.814070 895 tablet_service.cc:1505] Processing CreateTablet for tablet 8f100a12d78145c2b8586a0713fc70c4 (DEFAULT_TABLE table=default.Table_05a3479e89744e828535b90bbea0e8e0 [id=872b2ef8da55474da6755e2a20d70dd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:01.814836 885 tablet_service.cc:1505] Processing CreateTablet for tablet 19e15a6df77f4e0c9b40ae4f49fdd06f (DEFAULT_TABLE table=default.Table_f8f6696a8a1c4fcca702c03886a45f62 [id=e9403bb98f0146c1a106a2e15f8a5095]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:01.815459 1167 ts_tablet_manager.cc:1434] T 078ace5eada044c88d2462efb2b57aa8 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20251029 02:52:01.815857 1167 tablet_bootstrap.cc:492] T 9e07291d6b5645feb4d5890f58e1ef74 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:01.816200 1682 raft_consensus.cc:515] T b5d5b737c86844d58c51b2388c0b1705 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:01.816969 1682 leader_election.cc:290] T b5d5b737c86844d58c51b2388c0b1705 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:01.817312 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b5d5b737c86844d58c51b2388c0b1705" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:01.817945 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 67c3b526a5cd4da7800b7b72b0ebe29d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:01.817979 900 tablet_service.cc:1505] Processing CreateTablet for tablet ae5a6caeb17f4851b27a3c6a3c0db83c (DEFAULT_TABLE table=default.Table_f8c6e4111f9945dfb9e1a7dadc4f5381 [id=8cad99658c0d4409807be58732b1598a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:01.818226 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_f17714316e664840899f4d9827cd25fb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:01.818542 1060 raft_consensus.cc:2468] T b5d5b737c86844d58c51b2388c0b1705 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:01.818758 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87b52cb65bbc4982959b0d1e40d8b74d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:01.819207 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19e15a6df77f4e0c9b40ae4f49fdd06f. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:01.819207 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f17714316e664840899f4d9827cd25fb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:01.820154 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f100a12d78145c2b8586a0713fc70c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:01.821077 849 leader_election.cc:304] T b5d5b737c86844d58c51b2388c0b1705 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:01.821362 1682 raft_consensus.cc:2804] T b5d5b737c86844d58c51b2388c0b1705 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:01.821465 1682 raft_consensus.cc:493] T b5d5b737c86844d58c51b2388c0b1705 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:01.821576 1682 raft_consensus.cc:3060] T b5d5b737c86844d58c51b2388c0b1705 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:01.827528 1774 raft_consensus.cc:493] T 078ace5eada044c88d2462efb2b57aa8 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:01.827709 1774 raft_consensus.cc:515] T 078ace5eada044c88d2462efb2b57aa8 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:01.828516 1774 leader_election.cc:290] T 078ace5eada044c88d2462efb2b57aa8 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:01.829211 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "078ace5eada044c88d2462efb2b57aa8" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:01.829344 1028 tablet_service.cc:1505] Processing CreateTablet for tablet e40d4910365f45e9a45d9b3bc5037c95 (DEFAULT_TABLE table=default.Table_dca7cb07591d4d238e29e54969bca2b8 [id=643b2db711b948ccb90aac7094a84dda]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:01.830350 1060 raft_consensus.cc:2468] T 078ace5eada044c88d2462efb2b57aa8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:01.830960 849 leader_election.cc:304] T 078ace5eada044c88d2462efb2b57aa8 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:01.831334 1774 raft_consensus.cc:2804] T 078ace5eada044c88d2462efb2b57aa8 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:01.831447 1774 raft_consensus.cc:493] T 078ace5eada044c88d2462efb2b57aa8 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:01.831529 1774 raft_consensus.cc:3060] T 078ace5eada044c88d2462efb2b57aa8 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:01.834899 1037 tablet_service.cc:1505] Processing CreateTablet for tablet 8f100a12d78145c2b8586a0713fc70c4 (DEFAULT_TABLE table=default.Table_05a3479e89744e828535b90bbea0e8e0 [id=872b2ef8da55474da6755e2a20d70dd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:01.835879 519 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_44b56ca77fed49489c18e54dffb4b108" } modify_external_catalogs: true
I20251029 02:52:01.835899 520 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_f861260a80434489a8a2a44e9f0b6998" } new_table_name: "default.Table_13dfef4db9f244d8b98b5ff7a9c7463d" modify_external_catalogs: true
I20251029 02:52:01.836092 519 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_44b56ca77fed49489c18e54dffb4b108" } modify_external_catalogs: true
I20251029 02:52:01.837960 526 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_75b922d164054630b5de9dbe39ec275b" } new_table_name: "default.Table_2a348c61abab41ca8c6ce84f4888252b" modify_external_catalogs: true
I20251029 02:52:01.838969 1774 raft_consensus.cc:515] T 078ace5eada044c88d2462efb2b57aa8 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:01.839195 1025 tablet_service.cc:1505] Processing CreateTablet for tablet 143f4a42001b43e683bbc6226c0550a9 (DEFAULT_TABLE table=default.Table_f861260a80434489a8a2a44e9f0b6998 [id=c0cf872878f740e9866f406d2d6705f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:01.840752 1774 leader_election.cc:290] T 078ace5eada044c88d2462efb2b57aa8 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:01.841856 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 24016ceaa84147eca0b94b3ad288e772 (table default.Table_f06003da6e324504ade193bbbb11d36a [id=bd279b3ce623477ca20e8d35e3b7c0df]) successfully deleted
W20251029 02:52:01.842136 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 2e003e140a244d46b15a2c3e84f68936 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2e003e140a244d46b15a2c3e84f68936 already in progress: creating tablet
W20251029 02:52:01.842357 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 5bd2163bc4be467fb695629bfbb5a6ca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5bd2163bc4be467fb695629bfbb5a6ca already in progress: creating tablet
W20251029 02:52:01.842625 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 19609bc7f06645dc811b4bae8ce506c8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 19609bc7f06645dc811b4bae8ce506c8 already in progress: creating tablet
W20251029 02:52:01.842819 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 1df0d944717d487a98ea83a8ec04bd46 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1df0d944717d487a98ea83a8ec04bd46 already in progress: creating tablet
I20251029 02:52:01.842865 883 tablet_service.cc:1505] Processing CreateTablet for tablet e40d4910365f45e9a45d9b3bc5037c95 (DEFAULT_TABLE table=default.Table_dca7cb07591d4d238e29e54969bca2b8 [id=643b2db711b948ccb90aac7094a84dda]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:01.843334 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e40d4910365f45e9a45d9b3bc5037c95. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:01.845525 520 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 143f4a42001b43e683bbc6226c0550a9 consensus state has no leader
I20251029 02:52:01.845486 899 tablet_service.cc:1505] Processing CreateTablet for tablet 143f4a42001b43e683bbc6226c0550a9 (DEFAULT_TABLE table=default.Table_f861260a80434489a8a2a44e9f0b6998 [id=c0cf872878f740e9866f406d2d6705f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:01.845883 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 143f4a42001b43e683bbc6226c0550a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:01.846467 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae5a6caeb17f4851b27a3c6a3c0db83c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:01.847600 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_be5e479a6ff642dd8b01f195cfa37d8f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:01.847805 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_3297bb18a32a4454a6458e189b9985d1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:01.848181 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_be5e479a6ff642dd8b01f195cfa37d8f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:52:01.848341 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3297bb18a32a4454a6458e189b9985d1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:01.849756 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_df78a989ef8e4c48b1716f5698c6ecf9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:01.850256 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_df78a989ef8e4c48b1716f5698c6ecf9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:01.850467 884 tablet_service.cc:1505] Processing CreateTablet for tablet fe24183d24e34579b96e6fb6bc0a914e (DEFAULT_TABLE table=default.Table_64d89ed6a9af4c4880c3203fddf36126 [id=e712c4e544f044e9ae7758ebfdefdf56]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:01.849762 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_f28c52f38da44fa6b73e9bca9b29908e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:01.851226 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f28c52f38da44fa6b73e9bca9b29908e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:01.853400 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e40d4910365f45e9a45d9b3bc5037c95. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:01.853525 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_022e72d689654b18adde8de9ec2c065a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:01.853992 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_022e72d689654b18adde8de9ec2c065a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:01.854074 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "078ace5eada044c88d2462efb2b57aa8" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:52:01.855123 1060 raft_consensus.cc:3060] T 078ace5eada044c88d2462efb2b57aa8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:01.855254 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 143f4a42001b43e683bbc6226c0550a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:01.855526 1029 tablet_service.cc:1505] Processing CreateTablet for tablet fe24183d24e34579b96e6fb6bc0a914e (DEFAULT_TABLE table=default.Table_64d89ed6a9af4c4880c3203fddf36126 [id=e712c4e544f044e9ae7758ebfdefdf56]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:01.855728 745 tablet_service.cc:1505] Processing CreateTablet for tablet 0415b52fadf948238c4e83dc09000f96 (DEFAULT_TABLE table=default.Table_cb5488bb7e104676aad8f3db9be87802 [id=3bb4d1d8901e49a1888345fbdc1343aa]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 02:52:01.856413 526 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9a58ecfa646c41cdae4d75724767d0d9 consensus state has no leader
I20251029 02:52:01.856527 1041 tablet_service.cc:1505] Processing CreateTablet for tablet 87b52cb65bbc4982959b0d1e40d8b74d (DEFAULT_TABLE table=default.Table_f15c66cb663740828b22d54e002adeaa [id=b9e1da40360644e9b2f5b6ea70b8bea3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:01.856117 1038 tablet_service.cc:1505] Processing CreateTablet for tablet 19e15a6df77f4e0c9b40ae4f49fdd06f (DEFAULT_TABLE table=default.Table_f8f6696a8a1c4fcca702c03886a45f62 [id=e9403bb98f0146c1a106a2e15f8a5095]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:01.857561 519 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet c64d6761ef924ccaa9c2579f8ba47e33
I20251029 02:52:01.860077 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe24183d24e34579b96e6fb6bc0a914e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:01.864936 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f100a12d78145c2b8586a0713fc70c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:01.870545 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_7d8a27a8c05d4e9686a1fd616c5697bd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:01.871191 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7d8a27a8c05d4e9686a1fd616c5697bd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:01.871943 522 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_1ff672db4f064544b523c0ae9667d967" } new_table_name: "default.Table_743b2905866a4205b15f193d6d404b17" modify_external_catalogs: true
I20251029 02:52:01.870611 526 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_f06003da6e324504ade193bbbb11d36a" } new_table_name: "default.Table_2662f20eb8d843b8bf6c841e49863c85" modify_external_catalogs: true
I20251029 02:52:01.874554 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_ace94370997c4944968f3e7633615ae2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:01.875078 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ace94370997c4944968f3e7633615ae2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:01.876310 525 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_943114eb062441f6bdbf5cdd16c62101" } new_table_name: "default.Table_ce73377094b942cc87faac169c0fbe4a" modify_external_catalogs: true
I20251029 02:52:01.879182 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_ad5cf71ba5be4c8a92c3783a900e0054"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:01.879664 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ad5cf71ba5be4c8a92c3783a900e0054 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:52:01.884511 526 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 91e31f0b06924740bdf7525cae79d8ef consensus state has no leader
W20251029 02:52:01.886106 522 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 699ee74f39b342639ab84d0d65e53a70 consensus state has no leader
I20251029 02:52:01.888103 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_ae1c37b89ad243f0a389a54fa71b3ae0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:01.888777 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ae1c37b89ad243f0a389a54fa71b3ae0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:01.888906 522 master_service.cc:954] ReplaceTablet: received request to replace tablet 8f1d5847265b4bb0b29b5e513c35b9bb from {username='slave'} at 127.0.0.1:35314
I20251029 02:52:01.892549 1026 tablet_service.cc:1505] Processing CreateTablet for tablet ae5a6caeb17f4851b27a3c6a3c0db83c (DEFAULT_TABLE table=default.Table_f8c6e4111f9945dfb9e1a7dadc4f5381 [id=8cad99658c0d4409807be58732b1598a]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 02:52:01.893167 525 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 02e4588b03144ea2850e13a1f8f29f06 consensus state has no leader
I20251029 02:52:01.895331 1175 ts_tablet_manager.cc:1950] T 6feec2c8e030447fae9c292a626dcd1d P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:01.895627 756 tablet_service.cc:1505] Processing CreateTablet for tablet 8f100a12d78145c2b8586a0713fc70c4 (DEFAULT_TABLE table=default.Table_05a3479e89744e828535b90bbea0e8e0 [id=872b2ef8da55474da6755e2a20d70dd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:01.895752 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87b52cb65bbc4982959b0d1e40d8b74d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:01.896159 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe24183d24e34579b96e6fb6bc0a914e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:01.896255 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 399) took 3521 ms. Trace:
I20251029 02:52:01.896355 1175 rpcz_store.cc:276] 1029 02:51:58.374764 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:58.374817 (+ 53us) service_pool.cc:225] Handling call
1029 02:52:01.593768 (+3218951us) ts_tablet_manager.cc:1151] Deleting tablet 6feec2c8e030447fae9c292a626dcd1d
1029 02:52:01.593777 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:01.598239 (+ 4462us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:01.729029 (+130790us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:01.896204 (+167175us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3110810}
I20251029 02:52:01.896688 1175 tablet_replica.cc:333] T fa5127bcfcda40a7b5ad4ef016af072a P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:01.896906 1175 raft_consensus.cc:2243] T fa5127bcfcda40a7b5ad4ef016af072a P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:01.897100 1175 raft_consensus.cc:2272] T fa5127bcfcda40a7b5ad4ef016af072a P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:01.897698 1175 ts_tablet_manager.cc:1916] T fa5127bcfcda40a7b5ad4ef016af072a P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:01.898661 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19e15a6df77f4e0c9b40ae4f49fdd06f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:01.900679 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0415b52fadf948238c4e83dc09000f96. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:01.902417 749 tablet_service.cc:1505] Processing CreateTablet for tablet 87b52cb65bbc4982959b0d1e40d8b74d (DEFAULT_TABLE table=default.Table_f15c66cb663740828b22d54e002adeaa [id=b9e1da40360644e9b2f5b6ea70b8bea3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:01.902855 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87b52cb65bbc4982959b0d1e40d8b74d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:01.902897 771 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "078ace5eada044c88d2462efb2b57aa8" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:01.903086 771 raft_consensus.cc:3060] T 078ace5eada044c88d2462efb2b57aa8 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:01.906114 771 raft_consensus.cc:2468] T 078ace5eada044c88d2462efb2b57aa8 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:01.918776 1665 raft_consensus.cc:493] T c9dff4dff82f49e7ad49caad5e404f09 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:01.926405 755 tablet_service.cc:1505] Processing CreateTablet for tablet e40d4910365f45e9a45d9b3bc5037c95 (DEFAULT_TABLE table=default.Table_dca7cb07591d4d238e29e54969bca2b8 [id=643b2db711b948ccb90aac7094a84dda]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:01.929118 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 6feec2c8e030447fae9c292a626dcd1d (table default.Table_cb3d27722cc74fa18ddc8740590639a9 [id=37499c7664774cb9bd0a80c7c5bef158]) successfully deleted
I20251029 02:52:01.930071 1665 raft_consensus.cc:515] T c9dff4dff82f49e7ad49caad5e404f09 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:01.930840 1665 leader_election.cc:290] T c9dff4dff82f49e7ad49caad5e404f09 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:52:01.931484 1060 raft_consensus.cc:2468] T 078ace5eada044c88d2462efb2b57aa8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:01.932150 893 tablet_service.cc:1552] Processing DeleteTablet for tablet c64d6761ef924ccaa9c2579f8ba47e33 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:01 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:01.932231 849 leader_election.cc:304] T 078ace5eada044c88d2462efb2b57aa8 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:01.932933 918 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9dff4dff82f49e7ad49caad5e404f09" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:01.933320 847 leader_election.cc:343] T 9964ad774890422f8e14f6b99f28cb55 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 9964ad774890422f8e14f6b99f28cb55
I20251029 02:52:01.935416 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e40d4910365f45e9a45d9b3bc5037c95. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:01.935380 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "078ace5eada044c88d2462efb2b57aa8" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:01.935598 774 raft_consensus.cc:2376] T 078ace5eada044c88d2462efb2b57aa8 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1. Re-sending same reply.
I20251029 02:52:01.936190 751 tablet_service.cc:1505] Processing CreateTablet for tablet 19e15a6df77f4e0c9b40ae4f49fdd06f (DEFAULT_TABLE table=default.Table_f8f6696a8a1c4fcca702c03886a45f62 [id=e9403bb98f0146c1a106a2e15f8a5095]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:01.936605 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19e15a6df77f4e0c9b40ae4f49fdd06f. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:01.937017 513 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 699ee74f39b342639ab84d0d65e53a70 (table default.Table_743b2905866a4205b15f193d6d404b17 [id=60cf9a35c3c848d8a4e7cd7a8128b12f], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 699ee74f39b342639ab84d0d65e53a70 consensus state has no leader
I20251029 02:52:01.937958 748 tablet_service.cc:1505] Processing CreateTablet for tablet ae5a6caeb17f4851b27a3c6a3c0db83c (DEFAULT_TABLE table=default.Table_f8c6e4111f9945dfb9e1a7dadc4f5381 [id=8cad99658c0d4409807be58732b1598a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:01.938501 754 tablet_service.cc:1552] Processing DeleteTablet for tablet c64d6761ef924ccaa9c2579f8ba47e33 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:01 UTC) from {username='slave'} at 127.0.0.1:44580
W20251029 02:52:01.942637 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.3:45003, user_credentials={real_user=slave}} blocked reactor thread for 54193.8us
I20251029 02:52:01.942962 742 tablet_service.cc:1505] Processing CreateTablet for tablet 143f4a42001b43e683bbc6226c0550a9 (DEFAULT_TABLE table=default.Table_f861260a80434489a8a2a44e9f0b6998 [id=c0cf872878f740e9866f406d2d6705f4]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 02:52:01.943548 711 leader_election.cc:343] T c9dff4dff82f49e7ad49caad5e404f09 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:01.945381 1039 tablet_service.cc:1552] Processing DeleteTablet for tablet c64d6761ef924ccaa9c2579f8ba47e33 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:01 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:01.945658 1176 ts_tablet_manager.cc:1929] T fa5127bcfcda40a7b5ad4ef016af072a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251029 02:52:01.945794 1176 log.cc:1199] T fa5127bcfcda40a7b5ad4ef016af072a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/fa5127bcfcda40a7b5ad4ef016af072a
I20251029 02:52:01.946244 1176 ts_tablet_manager.cc:1950] T fa5127bcfcda40a7b5ad4ef016af072a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:01.946542 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae5a6caeb17f4851b27a3c6a3c0db83c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:01.946615 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 422) took 4000 ms. Trace:
I20251029 02:52:01.946717 1176 rpcz_store.cc:276] 1029 02:51:57.945952 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:57.946093 (+ 141us) service_pool.cc:225] Handling call
1029 02:52:01.720656 (+3774563us) ts_tablet_manager.cc:1151] Deleting tablet fa5127bcfcda40a7b5ad4ef016af072a
1029 02:52:01.720666 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:01.824840 (+104174us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:01.918835 (+ 93995us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:01.946580 (+ 27745us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3733718}
I20251029 02:52:01.947379 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 427) took 3899 ms. Trace:
I20251029 02:52:01.947552 1176 rpcz_store.cc:276] 1029 02:51:58.048130 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:58.050052 (+ 1922us) service_pool.cc:225] Handling call
1029 02:52:01.947237 (+3897185us) ts_tablet_manager.cc:1151] Deleting tablet 84993b2f63e7421290a161dde4bb0607
1029 02:52:01.947248 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:01.947347 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3886450}
I20251029 02:52:01.947649 1682 raft_consensus.cc:515] T b5d5b737c86844d58c51b2388c0b1705 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:01.948410 1682 leader_election.cc:290] T b5d5b737c86844d58c51b2388c0b1705 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:01.948781 1058 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b5d5b737c86844d58c51b2388c0b1705" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:52:01.948959 1058 raft_consensus.cc:3060] T b5d5b737c86844d58c51b2388c0b1705 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:01.950420 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 143f4a42001b43e683bbc6226c0550a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:01.950945 519 master_service.cc:954] ReplaceTablet: received request to replace tablet 5ef87f7a3dff4f00a0e7273eaec98dff from {username='slave'} at 127.0.0.1:35314
I20251029 02:52:01.951818 1058 raft_consensus.cc:2468] T b5d5b737c86844d58c51b2388c0b1705 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:01.952888 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f100a12d78145c2b8586a0713fc70c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:01.948160 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 425) took 3911 ms. Trace:
I20251029 02:52:01.954907 1176 rpcz_store.cc:276] 1029 02:51:58.036848 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:58.037446 (+ 598us) service_pool.cc:225] Handling call
1029 02:52:01.948047 (+3910601us) ts_tablet_manager.cc:1151] Deleting tablet 49d9ce32e4f04f11be2ecefc5d523a8b
1029 02:52:01.948055 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:01.948131 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3886211}
I20251029 02:52:01.955106 749 tablet_service.cc:1505] Processing CreateTablet for tablet fe24183d24e34579b96e6fb6bc0a914e (DEFAULT_TABLE table=default.Table_64d89ed6a9af4c4880c3203fddf36126 [id=e712c4e544f044e9ae7758ebfdefdf56]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:01.955410 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 429) took 3894 ms. Trace:
I20251029 02:52:01.955478 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe24183d24e34579b96e6fb6bc0a914e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:01.955651 1176 rpcz_store.cc:276] 1029 02:51:58.061208 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:58.061898 (+ 690us) service_pool.cc:225] Handling call
1029 02:52:01.955283 (+3893385us) ts_tablet_manager.cc:1151] Deleting tablet a5c3a15510cb4321bbb80d61fabd0c0a
1029 02:52:01.955295 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:01.955382 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3872480}
I20251029 02:52:01.956103 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 430) took 3876 ms. Trace:
I20251029 02:52:01.956202 1176 rpcz_store.cc:276] 1029 02:51:58.080085 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:58.081157 (+ 1072us) service_pool.cc:225] Handling call
1029 02:52:01.955985 (+3874828us) ts_tablet_manager.cc:1151] Deleting tablet bfec87f690a640c39c58c64e16cc67e7
1029 02:52:01.955994 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:01.956076 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3860933}
I20251029 02:52:01.956487 1773 raft_consensus.cc:2804] T 078ace5eada044c88d2462efb2b57aa8 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:01.956593 847 leader_election.cc:304] T 9964ad774890422f8e14f6b99f28cb55 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9
I20251029 02:52:01.956790 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9dff4dff82f49e7ad49caad5e404f09" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:01.957239 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 438) took 3790 ms. Trace:
I20251029 02:52:01.957347 1176 rpcz_store.cc:276] 1029 02:51:58.166321 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:58.171911 (+ 5590us) service_pool.cc:225] Handling call
1029 02:52:01.957096 (+3785185us) ts_tablet_manager.cc:1151] Deleting tablet cc802bda2f6a4e439b605f2cafa37967
1029 02:52:01.957104 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:01.957198 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3720313}
W20251029 02:52:01.957607 712 leader_election.cc:343] T c9dff4dff82f49e7ad49caad5e404f09 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:01.957684 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 456) took 3645 ms. Trace:
I20251029 02:52:01.957742 712 leader_election.cc:304] T c9dff4dff82f49e7ad49caad5e404f09 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:01.957787 1176 rpcz_store.cc:276] 1029 02:51:58.312234 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:58.823144 (+510910us) service_pool.cc:225] Handling call
1029 02:52:01.957568 (+3134424us) ts_tablet_manager.cc:1151] Deleting tablet 897a9a75d13f493e90526da4ca7f54d2
1029 02:52:01.957579 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:01.957656 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3134188}
I20251029 02:52:01.958149 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 452) took 3646 ms. Trace:
I20251029 02:52:01.958256 1176 rpcz_store.cc:276] 1029 02:51:58.311851 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:58.813890 (+502039us) service_pool.cc:225] Handling call
1029 02:52:01.957997 (+3144107us) ts_tablet_manager.cc:1151] Deleting tablet 061f3d38c4604fa4b57e74ff6f6fc7da
1029 02:52:01.958006 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:01.958112 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3129079}
I20251029 02:52:01.958407 1665 raft_consensus.cc:2749] T c9dff4dff82f49e7ad49caad5e404f09 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:01.960598 1682 raft_consensus.cc:2749] T 9964ad774890422f8e14f6b99f28cb55 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 02:52:01.960659 847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:38481, user_credentials={real_user=slave}} blocked reactor thread for 54170.8us
I20251029 02:52:01.959026 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae5a6caeb17f4851b27a3c6a3c0db83c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:01.961339 1167 tablet_bootstrap.cc:654] T 9e07291d6b5645feb4d5890f58e1ef74 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:01.962435 849 leader_election.cc:304] T b5d5b737c86844d58c51b2388c0b1705 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:01.962809 1682 raft_consensus.cc:2804] T b5d5b737c86844d58c51b2388c0b1705 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:01.963042 1682 raft_consensus.cc:697] T b5d5b737c86844d58c51b2388c0b1705 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:52:01.963328 1682 consensus_queue.cc:237] T b5d5b737c86844d58c51b2388c0b1705 P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:01.963853 1773 raft_consensus.cc:697] T 078ace5eada044c88d2462efb2b57aa8 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:52:01.964123 1773 consensus_queue.cc:237] T 078ace5eada044c88d2462efb2b57aa8 P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
W20251029 02:52:01.964452 847 leader_election.cc:343] T b5d5b737c86844d58c51b2388c0b1705 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: b5d5b737c86844d58c51b2388c0b1705
W20251029 02:52:01.964936 847 leader_election.cc:343] T b5d5b737c86844d58c51b2388c0b1705 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: b5d5b737c86844d58c51b2388c0b1705
I20251029 02:52:01.969159 525 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_d9fe74a49f4843b089aae7fca9e4a826" } modify_external_catalogs: true
I20251029 02:52:01.970213 524 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_5069071caedd4971a57a438fba4fac4f" } modify_external_catalogs: true
I20251029 02:52:01.970722 520 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_15da0c38ef4a4e30a558753430df9072" } modify_external_catalogs: true
I20251029 02:52:01.971367 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet fa5127bcfcda40a7b5ad4ef016af072a (table default.Table_5113d1a33e8e4bd2af5bc51e689f304f [id=ae5d56d40fa941a68f2b10d5e6e9ac83]) successfully deleted
I20251029 02:52:01.972286 517 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_5876d3fefb8144e2a1a76782912ab03b" } modify_external_catalogs: true
I20251029 02:52:01.975509 1175 ts_tablet_manager.cc:1929] T fa5127bcfcda40a7b5ad4ef016af072a P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251029 02:52:01.976105 1175 log.cc:1199] T fa5127bcfcda40a7b5ad4ef016af072a P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/fa5127bcfcda40a7b5ad4ef016af072a
I20251029 02:52:01.978365 1175 ts_tablet_manager.cc:1950] T fa5127bcfcda40a7b5ad4ef016af072a P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:01.983623 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 459) took 3157 ms. Trace:
I20251029 02:52:01.983769 1176 rpcz_store.cc:276] 1029 02:51:58.825812 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:58.826326 (+ 514us) service_pool.cc:225] Handling call
1029 02:52:01.983413 (+3157087us) ts_tablet_manager.cc:1151] Deleting tablet aee42fd9ca894319b3759beb2bae342e
1029 02:52:01.983426 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:01.983572 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3148615}
I20251029 02:52:01.985481 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 455) took 3673 ms. Trace:
I20251029 02:52:01.985960 1176 rpcz_store.cc:276] 1029 02:51:58.312133 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:58.824021 (+511888us) service_pool.cc:225] Handling call
1029 02:52:01.985311 (+3161290us) ts_tablet_manager.cc:1151] Deleting tablet 30d03e1e42d34242bd354f9e9ec4f127
1029 02:52:01.985323 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:01.985445 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3149548}
I20251029 02:52:01.986869 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 466) took 2908 ms. Trace:
I20251029 02:52:01.987336 1176 rpcz_store.cc:276] 1029 02:51:59.078907 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:59.079699 (+ 792us) service_pool.cc:225] Handling call
1029 02:52:01.986740 (+2907041us) ts_tablet_manager.cc:1151] Deleting tablet 5ae5f33567324d90b7a5a6e0f1b5fb67
1029 02:52:01.986755 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:01.986840 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2875102}
I20251029 02:52:01.988309 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 486) took 2882 ms. Trace:
I20251029 02:52:01.988765 1176 rpcz_store.cc:276] 1029 02:51:59.105989 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:59.559042 (+453053us) service_pool.cc:225] Handling call
1029 02:52:01.988179 (+2429137us) ts_tablet_manager.cc:1151] Deleting tablet 86fc22bb84794149b598b97df41eb881
1029 02:52:01.988192 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:01.988279 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2428623}
I20251029 02:52:01.989585 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 488) took 2883 ms. Trace:
I20251029 02:52:01.990021 1176 rpcz_store.cc:276] 1029 02:51:59.106125 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:59.567003 (+460878us) service_pool.cc:225] Handling call
1029 02:52:01.989442 (+2422439us) ts_tablet_manager.cc:1151] Deleting tablet 358bdfbcb4ee440c899fb6eedef68f44
1029 02:52:01.989455 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:01.989540 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2412189}
I20251029 02:52:01.990767 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 487) took 2884 ms. Trace:
I20251029 02:52:01.991987 525 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_d9fe74a49f4843b089aae7fca9e4a826" } modify_external_catalogs: true
I20251029 02:52:01.993551 524 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_5069071caedd4971a57a438fba4fac4f" } modify_external_catalogs: true
I20251029 02:52:01.993284 526 catalog_manager.cc:5649] T 4bf06476c87a4549a73f5994314bb75f P c756528642a543eeacd93cef3e335a7d reported cstate change: term changed from 0 to 1, leader changed from <none> to c756528642a543eeacd93cef3e335a7d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "c756528642a543eeacd93cef3e335a7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } health_report { overall_health: UNKNOWN } } }
I20251029 02:52:01.993954 526 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet 4bf06476c87a4549a73f5994314bb75f (table default.Table_bafe26b1fca14cc9b908984b7d8a2442 [id=046010a03d774e91b0ad10df97848c56]). Expected version 1 got 0
I20251029 02:52:01.994243 526 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet d2990e9f626c40618c118266bc595684 (table default.Table_bd020e7c45a54c6e92211c4261293575 [id=f733948e92d8470da9d8050bd26a06b1]). Expected version 1 got 0
I20251029 02:52:01.994686 520 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_15da0c38ef4a4e30a558753430df9072" } modify_external_catalogs: true
I20251029 02:52:01.996008 522 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 8f1d5847265b4bb0b29b5e513c35b9bb
I20251029 02:52:01.997004 522 catalog_manager.cc:6719] ReplaceTablet: tablet 8f1d5847265b4bb0b29b5e513c35b9bb deleted and replaced by tablet 72e91c64a9b54cbda52ca9dfcee2ce22
I20251029 02:52:01.997339 899 tablet_service.cc:1552] Processing DeleteTablet for tablet 8f1d5847265b4bb0b29b5e513c35b9bb with delete_type TABLET_DATA_DELETED (replaced by tablet 72e91c64a9b54cbda52ca9dfcee2ce22) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:01.997412 756 tablet_service.cc:1552] Processing DeleteTablet for tablet 8f1d5847265b4bb0b29b5e513c35b9bb with delete_type TABLET_DATA_DELETED (replaced by tablet 72e91c64a9b54cbda52ca9dfcee2ce22) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:01.999373 517 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_5876d3fefb8144e2a1a76782912ab03b" } modify_external_catalogs: true
I20251029 02:52:02.002939 519 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 5ef87f7a3dff4f00a0e7273eaec98dff
I20251029 02:52:02.003445 519 catalog_manager.cc:6719] ReplaceTablet: tablet 5ef87f7a3dff4f00a0e7273eaec98dff deleted and replaced by tablet 00bc50e2c9df4f2285d91ef2ef6452af
I20251029 02:52:02.003958 756 tablet_service.cc:1552] Processing DeleteTablet for tablet 5ef87f7a3dff4f00a0e7273eaec98dff with delete_type TABLET_DATA_DELETED (replaced by tablet 00bc50e2c9df4f2285d91ef2ef6452af) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:02.004814 883 tablet_service.cc:1552] Processing DeleteTablet for tablet 5ef87f7a3dff4f00a0e7273eaec98dff with delete_type TABLET_DATA_DELETED (replaced by tablet 00bc50e2c9df4f2285d91ef2ef6452af) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:02.009505 1174 ts_tablet_manager.cc:1929] T fd900eec3f0c43bf965c148ea90870a3 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:02.009717 1174 log.cc:1199] T fd900eec3f0c43bf965c148ea90870a3 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/fd900eec3f0c43bf965c148ea90870a3
I20251029 02:52:02.009865 525 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet e65eaaae43ac435ebd665546de9d68dd
I20251029 02:52:02.010684 756 tablet_service.cc:1552] Processing DeleteTablet for tablet e65eaaae43ac435ebd665546de9d68dd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:01 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:02.010927 1174 ts_tablet_manager.cc:1950] T fd900eec3f0c43bf965c148ea90870a3 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:02.011749 1174 tablet_replica.cc:333] T aa776cde2f494fa5810539edf9f289a1 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:02.011979 1174 raft_consensus.cc:2243] T aa776cde2f494fa5810539edf9f289a1 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:02.012084 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet fd900eec3f0c43bf965c148ea90870a3 (table default.Table_e0e6e47b18d749e8ae80f822af1e23f0 [id=125ae5c7b5074378bcecb546e70f0c13]) successfully deleted
I20251029 02:52:02.012144 1174 raft_consensus.cc:2272] T aa776cde2f494fa5810539edf9f289a1 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:02.012709 1174 ts_tablet_manager.cc:1916] T aa776cde2f494fa5810539edf9f289a1 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:02.014979 524 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet e3000183230e4350b47f5085952f2db6
I20251029 02:52:02.015764 756 tablet_service.cc:1552] Processing DeleteTablet for tablet e3000183230e4350b47f5085952f2db6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:01 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:02.016660 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 400) took 3641 ms. Trace:
I20251029 02:52:02.016821 1175 rpcz_store.cc:276] 1029 02:51:58.374864 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:58.374914 (+ 50us) service_pool.cc:225] Handling call
1029 02:52:01.896622 (+3521708us) ts_tablet_manager.cc:1151] Deleting tablet fa5127bcfcda40a7b5ad4ef016af072a
1029 02:52:01.896634 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:01.963409 (+ 66775us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:01.975094 (+ 11685us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:02.016594 (+ 41500us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3411313}
I20251029 02:52:02.017128 883 tablet_service.cc:1552] Processing DeleteTablet for tablet e3000183230e4350b47f5085952f2db6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:01 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:02.017287 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet fa5127bcfcda40a7b5ad4ef016af072a (table default.Table_5113d1a33e8e4bd2af5bc51e689f304f [id=ae5d56d40fa941a68f2b10d5e6e9ac83]) successfully deleted
I20251029 02:52:02.017462 1167 tablet_bootstrap.cc:492] T 9e07291d6b5645feb4d5890f58e1ef74 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:02.017630 1167 ts_tablet_manager.cc:1403] T 9e07291d6b5645feb4d5890f58e1ef74 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.202s user 0.008s sys 0.001s
I20251029 02:52:02.017864 885 tablet_service.cc:1552] Processing DeleteTablet for tablet e65eaaae43ac435ebd665546de9d68dd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:01 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:02.018392 1167 raft_consensus.cc:359] T 9e07291d6b5645feb4d5890f58e1ef74 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:02.019152 1167 raft_consensus.cc:385] T 9e07291d6b5645feb4d5890f58e1ef74 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:02.019232 1167 raft_consensus.cc:740] T 9e07291d6b5645feb4d5890f58e1ef74 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:02.019487 1167 consensus_queue.cc:260] T 9e07291d6b5645feb4d5890f58e1ef74 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:02.020026 1167 ts_tablet_manager.cc:1434] T 9e07291d6b5645feb4d5890f58e1ef74 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 02:52:02.020334 1167 tablet_bootstrap.cc:492] T 44784a3a760946e0be24743bc2fc8138 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
W20251029 02:52:02.021275 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 84993b2f63e7421290a161dde4bb0607 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 84993b2f63e7421290a161dde4bb0607 already in progress: creating tablet
W20251029 02:52:02.021423 513 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.3:45003, user_credentials={real_user=slave}} blocked reactor thread for 49451.7us
W20251029 02:52:02.021656 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 49d9ce32e4f04f11be2ecefc5d523a8b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 49d9ce32e4f04f11be2ecefc5d523a8b already in progress: creating tablet
W20251029 02:52:02.021829 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet a5c3a15510cb4321bbb80d61fabd0c0a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a5c3a15510cb4321bbb80d61fabd0c0a already in progress: creating tablet
W20251029 02:52:02.021987 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet bfec87f690a640c39c58c64e16cc67e7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bfec87f690a640c39c58c64e16cc67e7 already in progress: creating tablet
I20251029 02:52:02.022024 1167 tablet_bootstrap.cc:654] T 44784a3a760946e0be24743bc2fc8138 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
W20251029 02:52:02.022161 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet cc802bda2f6a4e439b605f2cafa37967 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cc802bda2f6a4e439b605f2cafa37967 already in progress: creating tablet
W20251029 02:52:02.022323 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 897a9a75d13f493e90526da4ca7f54d2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 897a9a75d13f493e90526da4ca7f54d2 already in progress: creating tablet
I20251029 02:52:02.023139 1175 tablet_replica.cc:333] T 392a36a6b59a42f78e3ac7c8fb63214b P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:02.023343 1175 raft_consensus.cc:2243] T 392a36a6b59a42f78e3ac7c8fb63214b P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:02.023546 1175 raft_consensus.cc:2272] T 392a36a6b59a42f78e3ac7c8fb63214b P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:02.024129 1175 ts_tablet_manager.cc:1916] T 392a36a6b59a42f78e3ac7c8fb63214b P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 02:52:02.025167 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 061f3d38c4604fa4b57e74ff6f6fc7da because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 061f3d38c4604fa4b57e74ff6f6fc7da already in progress: creating tablet
W20251029 02:52:02.025377 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet aee42fd9ca894319b3759beb2bae342e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aee42fd9ca894319b3759beb2bae342e already in progress: creating tablet
W20251029 02:52:02.025532 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 30d03e1e42d34242bd354f9e9ec4f127 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 30d03e1e42d34242bd354f9e9ec4f127 already in progress: creating tablet
W20251029 02:52:02.025700 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 5ae5f33567324d90b7a5a6e0f1b5fb67 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5ae5f33567324d90b7a5a6e0f1b5fb67 already in progress: creating tablet
W20251029 02:52:02.026676 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 86fc22bb84794149b598b97df41eb881 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 86fc22bb84794149b598b97df41eb881 already in progress: creating tablet
W20251029 02:52:02.026906 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 358bdfbcb4ee440c899fb6eedef68f44 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 358bdfbcb4ee440c899fb6eedef68f44 already in progress: creating tablet
I20251029 02:52:02.034761 1041 tablet_service.cc:1552] Processing DeleteTablet for tablet 5ef87f7a3dff4f00a0e7273eaec98dff with delete_type TABLET_DATA_DELETED (replaced by tablet 00bc50e2c9df4f2285d91ef2ef6452af) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:02.035117 1037 tablet_service.cc:1552] Processing DeleteTablet for tablet e65eaaae43ac435ebd665546de9d68dd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:01 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:01.991140 1176 rpcz_store.cc:276] 1029 02:51:59.106048 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:59.559568 (+453520us) service_pool.cc:225] Handling call
1029 02:52:01.990641 (+2431073us) ts_tablet_manager.cc:1151] Deleting tablet f18b0ca5a6a44175a912642e9eea1778
1029 02:52:01.990650 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:01.990737 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2370057}
I20251029 02:52:02.043154 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 485) took 2937 ms. Trace:
I20251029 02:52:02.043277 1176 rpcz_store.cc:276] 1029 02:51:59.105925 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:59.558471 (+452546us) service_pool.cc:225] Handling call
1029 02:52:02.042970 (+2484499us) ts_tablet_manager.cc:1151] Deleting tablet c8389c8bfead4ff7a23f363c9c6770bf
1029 02:52:02.042984 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.043105 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2404777}
I20251029 02:52:02.043637 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 489) took 2937 ms. Trace:
I20251029 02:52:02.043779 1176 rpcz_store.cc:276] 1029 02:51:59.106182 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:59.575701 (+469519us) service_pool.cc:225] Handling call
1029 02:52:02.043497 (+2467796us) ts_tablet_manager.cc:1151] Deleting tablet eca016aed05e4ce09e1a2bf96e2286f5
1029 02:52:02.043505 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.043592 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2393498}
I20251029 02:52:02.044103 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 500) took 2935 ms. Trace:
I20251029 02:52:02.044137 1175 ts_tablet_manager.cc:1929] T 392a36a6b59a42f78e3ac7c8fb63214b P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:02.044273 1175 log.cc:1199] T 392a36a6b59a42f78e3ac7c8fb63214b P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/392a36a6b59a42f78e3ac7c8fb63214b
I20251029 02:52:02.044219 1176 rpcz_store.cc:276] 1029 02:51:59.108833 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:59.702330 (+593497us) service_pool.cc:225] Handling call
1029 02:52:02.043980 (+2341650us) ts_tablet_manager.cc:1151] Deleting tablet 023a4094b424464c8f49d6eb9cc15da4
1029 02:52:02.043988 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.044061 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2276202}
I20251029 02:52:02.044539 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 501) took 2881 ms. Trace:
I20251029 02:52:02.044705 1175 ts_tablet_manager.cc:1950] T 392a36a6b59a42f78e3ac7c8fb63214b P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:02.044651 1176 rpcz_store.cc:276] 1029 02:51:59.163116 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:59.767089 (+603973us) service_pool.cc:225] Handling call
1029 02:52:02.044405 (+2277316us) ts_tablet_manager.cc:1151] Deleting tablet 1878d7eef14a4578bc929da68872cee4
1029 02:52:02.044413 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.044505 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2267384}
I20251029 02:52:02.044973 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 503) took 2881 ms. Trace:
I20251029 02:52:02.045076 1176 rpcz_store.cc:276] 1029 02:51:59.163334 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:59.777047 (+613713us) service_pool.cc:225] Handling call
1029 02:52:02.044852 (+2267805us) ts_tablet_manager.cc:1151] Deleting tablet 94bdbb9394f7490e99916bd818986208
1029 02:52:02.044860 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.044937 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2266614}
I20251029 02:52:02.045413 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 504) took 2881 ms. Trace:
I20251029 02:52:02.045518 1176 rpcz_store.cc:276] 1029 02:51:59.163427 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:59.778264 (+614837us) service_pool.cc:225] Handling call
1029 02:52:02.045257 (+2266993us) ts_tablet_manager.cc:1151] Deleting tablet c9bd3f2deb5d4af09b7918269a98509f
1029 02:52:02.045264 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.045379 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2259962}
I20251029 02:52:02.045673 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 401) took 3670 ms. Trace:
I20251029 02:52:02.045850 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 502) took 2882 ms. Trace:
I20251029 02:52:02.045799 1175 rpcz_store.cc:276] 1029 02:51:58.375218 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:58.375266 (+ 48us) service_pool.cc:225] Handling call
1029 02:52:02.023023 (+3647757us) ts_tablet_manager.cc:1151] Deleting tablet 392a36a6b59a42f78e3ac7c8fb63214b
1029 02:52:02.023036 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.033443 (+ 10407us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:02.043967 (+ 10524us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:02.045618 (+ 1651us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3533426}
I20251029 02:52:02.045953 1176 rpcz_store.cc:276] 1029 02:51:59.163243 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:59.767809 (+604566us) service_pool.cc:225] Handling call
1029 02:52:02.045723 (+2277914us) ts_tablet_manager.cc:1151] Deleting tablet 7816556521144c928d7a02694e22e79f
1029 02:52:02.045730 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.045817 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2214610}
I20251029 02:52:02.046262 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 515) took 2875 ms. Trace:
I20251029 02:52:02.046262 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 403) took 3670 ms. Trace:
I20251029 02:52:02.046365 1176 rpcz_store.cc:276] 1029 02:51:59.171061 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:59.968648 (+797587us) service_pool.cc:225] Handling call
1029 02:52:02.046144 (+2077496us) ts_tablet_manager.cc:1151] Deleting tablet 24f07b1efac64b099b0b5ed467463749
1029 02:52:02.046151 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.046223 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2063772}
I20251029 02:52:02.046396 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 392a36a6b59a42f78e3ac7c8fb63214b (table default.Table_e523efa604564452a1f56dada6e21daf [id=2d40c37ccbd541b9a1ed21c7f6bfb6c2]) successfully deleted
I20251029 02:52:02.046905 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet 8f1d5847265b4bb0b29b5e513c35b9bb with delete_type TABLET_DATA_DELETED (replaced by tablet 72e91c64a9b54cbda52ca9dfcee2ce22) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:02.048084 1172 tablet_bootstrap.cc:492] T b2b2d64dcf964da8be4b544dbbe31977 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:02.048225 1172 ts_tablet_manager.cc:1403] T b2b2d64dcf964da8be4b544dbbe31977 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.306s user 0.006s sys 0.006s
I20251029 02:52:02.048328 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet e3000183230e4350b47f5085952f2db6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:01 UTC) from {username='slave'} at 127.0.0.1:37626
W20251029 02:52:02.049500 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet f18b0ca5a6a44175a912642e9eea1778 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f18b0ca5a6a44175a912642e9eea1778 already in progress: creating tablet
I20251029 02:52:02.050549 885 tablet_service.cc:1552] Processing DeleteTablet for tablet 9964ad774890422f8e14f6b99f28cb55 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:53 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:02.054328 1172 raft_consensus.cc:359] T b2b2d64dcf964da8be4b544dbbe31977 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:02.054579 1172 raft_consensus.cc:385] T b2b2d64dcf964da8be4b544dbbe31977 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:02.054680 1172 raft_consensus.cc:740] T b2b2d64dcf964da8be4b544dbbe31977 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:02.055181 520 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 98bbaec01186435b8bcc7257a1390185
I20251029 02:52:02.055445 517 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet c7b78fd458e4414b9fc3aa064a80b206
I20251029 02:52:02.056326 883 tablet_service.cc:1552] Processing DeleteTablet for tablet c7b78fd458e4414b9fc3aa064a80b206 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:01 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:02.056454 756 tablet_service.cc:1552] Processing DeleteTablet for tablet 98bbaec01186435b8bcc7257a1390185 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:01 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:02.056653 885 tablet_service.cc:1552] Processing DeleteTablet for tablet 98bbaec01186435b8bcc7257a1390185 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:01 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:02.056699 749 tablet_service.cc:1552] Processing DeleteTablet for tablet c7b78fd458e4414b9fc3aa064a80b206 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:01 UTC) from {username='slave'} at 127.0.0.1:44580
W20251029 02:52:02.057394 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet c8389c8bfead4ff7a23f363c9c6770bf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c8389c8bfead4ff7a23f363c9c6770bf already in progress: creating tablet
W20251029 02:52:02.058981 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet eca016aed05e4ce09e1a2bf96e2286f5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eca016aed05e4ce09e1a2bf96e2286f5 already in progress: creating tablet
W20251029 02:52:02.059978 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 023a4094b424464c8f49d6eb9cc15da4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 023a4094b424464c8f49d6eb9cc15da4 already in progress: creating tablet
W20251029 02:52:02.064503 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 1878d7eef14a4578bc929da68872cee4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1878d7eef14a4578bc929da68872cee4 already in progress: creating tablet
I20251029 02:52:02.054929 1172 consensus_queue.cc:260] T b2b2d64dcf964da8be4b544dbbe31977 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
W20251029 02:52:02.066737 990 consensus_peers.cc:597] T 24016ceaa84147eca0b94b3ad288e772 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 -> Peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Couldn't send request to peer c756528642a543eeacd93cef3e335a7d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 24016ceaa84147eca0b94b3ad288e772. This is attempt 1: this message will repeat every 5th retry.
W20251029 02:52:02.064841 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 94bdbb9394f7490e99916bd818986208 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 94bdbb9394f7490e99916bd818986208 already in progress: creating tablet
W20251029 02:52:02.067648 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet c9bd3f2deb5d4af09b7918269a98509f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c9bd3f2deb5d4af09b7918269a98509f already in progress: creating tablet
W20251029 02:52:02.067942 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 7816556521144c928d7a02694e22e79f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7816556521144c928d7a02694e22e79f already in progress: creating tablet
I20251029 02:52:02.068640 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_9bed90ff07fb4ea28bc7a94b6ebfc125"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:02.069375 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9bed90ff07fb4ea28bc7a94b6ebfc125 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:02.069689 1172 ts_tablet_manager.cc:1434] T b2b2d64dcf964da8be4b544dbbe31977 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.021s user 0.002s sys 0.000s
I20251029 02:52:02.070919 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 520) took 2891 ms. Trace:
I20251029 02:52:02.071094 1176 rpcz_store.cc:276] 1029 02:51:59.179955 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:00.067647 (+887692us) service_pool.cc:225] Handling call
1029 02:52:02.070714 (+2003067us) ts_tablet_manager.cc:1151] Deleting tablet 5842e9bba4fa4295ba21e90d645a56df
1029 02:52:02.070731 (+ 17us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.070870 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2002800}
I20251029 02:52:02.071592 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 521) took 2890 ms. Trace:
I20251029 02:52:02.071712 1176 rpcz_store.cc:276] 1029 02:51:59.181786 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:00.067924 (+886138us) service_pool.cc:225] Handling call
1029 02:52:02.071432 (+2003508us) ts_tablet_manager.cc:1151] Deleting tablet 40e844fd7ad74680bcbe7bef936c5b9d
1029 02:52:02.071444 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.071544 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1996760}
I20251029 02:52:02.072597 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 522) took 2889 ms. Trace:
I20251029 02:52:02.072713 1176 rpcz_store.cc:276] 1029 02:51:59.183492 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:00.074703 (+891211us) service_pool.cc:225] Handling call
1029 02:52:02.072448 (+1997745us) ts_tablet_manager.cc:1151] Deleting tablet 28f7ceb91a34450cad4b5b45f5285b41
1029 02:52:02.072457 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.072555 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1997547}
I20251029 02:52:02.073040 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 523) took 2888 ms. Trace:
I20251029 02:52:02.073149 1176 rpcz_store.cc:276] 1029 02:51:59.185137 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:00.074918 (+889781us) service_pool.cc:225] Handling call
1029 02:52:02.072927 (+1998009us) ts_tablet_manager.cc:1151] Deleting tablet 58dde9397b434b089ade9968f369006a
1029 02:52:02.072935 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.073011 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1996906}
I20251029 02:52:02.073246 1172 tablet_bootstrap.cc:492] T 8df97119d0654d7d92bb5bd17c5d8562 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:02.073485 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 532) took 1953 ms. Trace:
I20251029 02:52:02.073596 1176 rpcz_store.cc:276] 1029 02:52:00.120376 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:00.567340 (+446964us) service_pool.cc:225] Handling call
1029 02:52:02.073381 (+1506041us) ts_tablet_manager.cc:1151] Deleting tablet fcb3db8399aa4cf188b07dda220f7f5b
1029 02:52:02.073389 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.073459 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1505863}
I20251029 02:52:02.074796 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 533) took 1953 ms. Trace:
I20251029 02:52:02.075074 1176 rpcz_store.cc:276] 1029 02:52:00.120465 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:00.567553 (+447088us) service_pool.cc:225] Handling call
1029 02:52:02.073808 (+1506255us) ts_tablet_manager.cc:1151] Deleting tablet 845a69a8696e469a8548787a7645d41a
1029 02:52:02.073817 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.073888 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1506097}
I20251029 02:52:02.075470 1175 rpcz_store.cc:276] 1029 02:51:58.375665 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:58.388627 (+ 12962us) service_pool.cc:225] Handling call
1029 02:52:02.046100 (+3657473us) ts_tablet_manager.cc:1151] Deleting tablet 49d9ce32e4f04f11be2ecefc5d523a8b
1029 02:52:02.046113 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.046228 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3425981}
W20251029 02:52:02.075944 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 49d9ce32e4f04f11be2ecefc5d523a8b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 49d9ce32e4f04f11be2ecefc5d523a8b already in progress: creating tablet
I20251029 02:52:02.076746 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 398) took 3716 ms. Trace:
I20251029 02:52:02.076887 1175 rpcz_store.cc:276] 1029 02:51:58.374569 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:58.429692 (+ 55123us) service_pool.cc:225] Handling call
1029 02:52:02.076598 (+3646906us) ts_tablet_manager.cc:1151] Deleting tablet afb19ba39399410bb90b6225d7b5b5b1
1029 02:52:02.076608 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.076708 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3456256}
W20251029 02:52:02.077432 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet afb19ba39399410bb90b6225d7b5b5b1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet afb19ba39399410bb90b6225d7b5b5b1 already in progress: creating tablet
I20251029 02:52:02.077850 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 415) took 3701 ms. Trace:
I20251029 02:52:02.077986 1175 rpcz_store.cc:276] 1029 02:51:58.376888 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:58.590181 (+213293us) service_pool.cc:225] Handling call
1029 02:52:02.077705 (+3487524us) ts_tablet_manager.cc:1151] Deleting tablet cc802bda2f6a4e439b605f2cafa37967
1029 02:52:02.077715 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.077819 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3442832}
W20251029 02:52:02.078400 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet cc802bda2f6a4e439b605f2cafa37967 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cc802bda2f6a4e439b605f2cafa37967 already in progress: creating tablet
I20251029 02:52:02.078811 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 432) took 3700 ms. Trace:
I20251029 02:52:02.078948 1175 rpcz_store.cc:276] 1029 02:51:58.378344 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:58.773286 (+394942us) service_pool.cc:225] Handling call
1029 02:52:02.078684 (+3305398us) ts_tablet_manager.cc:1151] Deleting tablet 30d03e1e42d34242bd354f9e9ec4f127
1029 02:52:02.078694 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.078779 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3257344}
W20251029 02:52:02.079382 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 30d03e1e42d34242bd354f9e9ec4f127 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 30d03e1e42d34242bd354f9e9ec4f127 already in progress: creating tablet
I20251029 02:52:02.079749 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 437) took 3635 ms. Trace:
I20251029 02:52:02.079895 1175 rpcz_store.cc:276] 1029 02:51:58.444709 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:58.811861 (+367152us) service_pool.cc:225] Handling call
1029 02:52:02.079634 (+3267773us) ts_tablet_manager.cc:1151] Deleting tablet aee42fd9ca894319b3759beb2bae342e
1029 02:52:02.079643 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.079719 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3237253}
W20251029 02:52:02.080294 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet aee42fd9ca894319b3759beb2bae342e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aee42fd9ca894319b3759beb2bae342e already in progress: creating tablet
I20251029 02:52:02.080654 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 433) took 3702 ms. Trace:
I20251029 02:52:02.086637 1175 rpcz_store.cc:276] 1029 02:51:58.378438 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:58.781016 (+402578us) service_pool.cc:225] Handling call
1029 02:52:02.080527 (+3299511us) ts_tablet_manager.cc:1151] Deleting tablet 897a9a75d13f493e90526da4ca7f54d2
1029 02:52:02.080535 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.080624 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3201837}
I20251029 02:52:02.087209 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 534) took 1966 ms. Trace:
W20251029 02:52:02.087265 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 897a9a75d13f493e90526da4ca7f54d2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 897a9a75d13f493e90526da4ca7f54d2 already in progress: creating tablet
I20251029 02:52:02.087966 1176 rpcz_store.cc:276] 1029 02:52:00.120543 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:00.567731 (+447188us) service_pool.cc:225] Handling call
1029 02:52:02.087049 (+1519318us) ts_tablet_manager.cc:1151] Deleting tablet 38f83ebcb2a440bc8e6a8d36607e6666
1029 02:52:02.087063 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.087174 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1519168}
I20251029 02:52:02.088068 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 430) took 3709 ms. Trace:
I20251029 02:52:02.088189 1175 rpcz_store.cc:276] 1029 02:51:58.378154 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:58.751936 (+373782us) service_pool.cc:225] Handling call
1029 02:52:02.087542 (+3335606us) ts_tablet_manager.cc:1151] Deleting tablet 061f3d38c4604fa4b57e74ff6f6fc7da
1029 02:52:02.087566 (+ 24us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.088031 (+ 465us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3171293}
W20251029 02:52:02.089329 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 061f3d38c4604fa4b57e74ff6f6fc7da because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 061f3d38c4604fa4b57e74ff6f6fc7da already in progress: creating tablet
I20251029 02:52:02.089021 1172 tablet_bootstrap.cc:654] T 8df97119d0654d7d92bb5bd17c5d8562 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:02.089934 517 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_0ecb5a1d51bb40d79afa038e4f2aa4d2" } new_table_name: "default.Table_8b0c5038160542c49a0bac581c1189ca" modify_external_catalogs: true
I20251029 02:52:02.090569 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 444) took 3633 ms. Trace:
I20251029 02:52:02.090710 1175 rpcz_store.cc:276] 1029 02:51:58.457305 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:58.904760 (+447455us) service_pool.cc:225] Handling call
1029 02:52:02.090414 (+3185654us) ts_tablet_manager.cc:1151] Deleting tablet 5ae5f33567324d90b7a5a6e0f1b5fb67
1029 02:52:02.090424 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.090529 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3171535}
I20251029 02:52:02.088419 1174 ts_tablet_manager.cc:1929] T aa776cde2f494fa5810539edf9f289a1 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20251029 02:52:02.091176 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 5ae5f33567324d90b7a5a6e0f1b5fb67 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5ae5f33567324d90b7a5a6e0f1b5fb67 already in progress: creating tablet
I20251029 02:52:02.091401 1174 log.cc:1199] T aa776cde2f494fa5810539edf9f289a1 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/aa776cde2f494fa5810539edf9f289a1
I20251029 02:52:02.091735 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 462) took 3082 ms. Trace:
I20251029 02:52:02.091850 1175 rpcz_store.cc:276] 1029 02:51:59.009052 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:59.326327 (+317275us) service_pool.cc:225] Handling call
1029 02:52:02.091609 (+2765282us) ts_tablet_manager.cc:1151] Deleting tablet c8389c8bfead4ff7a23f363c9c6770bf
1029 02:52:02.091617 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.091706 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2764932}
W20251029 02:52:02.092218 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet c8389c8bfead4ff7a23f363c9c6770bf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c8389c8bfead4ff7a23f363c9c6770bf already in progress: creating tablet
I20251029 02:52:02.092500 1174 ts_tablet_manager.cc:1950] T aa776cde2f494fa5810539edf9f289a1 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:02.092654 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 464) took 3083 ms. Trace:
I20251029 02:52:02.092811 1175 rpcz_store.cc:276] 1029 02:51:59.009304 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:59.337528 (+328224us) service_pool.cc:225] Handling call
1029 02:52:02.092485 (+2754957us) ts_tablet_manager.cc:1151] Deleting tablet f18b0ca5a6a44175a912642e9eea1778
1029 02:52:02.092493 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.092623 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2754750}
W20251029 02:52:02.093338 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet f18b0ca5a6a44175a912642e9eea1778 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f18b0ca5a6a44175a912642e9eea1778 already in progress: creating tablet
I20251029 02:52:02.093770 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 466) took 3084 ms. Trace:
I20251029 02:52:02.093917 1175 rpcz_store.cc:276] 1029 02:51:59.009510 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:59.337865 (+328355us) service_pool.cc:225] Handling call
1029 02:52:02.093648 (+2755783us) ts_tablet_manager.cc:1151] Deleting tablet eca016aed05e4ce09e1a2bf96e2286f5
1029 02:52:02.093657 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.093736 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2755625}
W20251029 02:52:02.094434 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet eca016aed05e4ce09e1a2bf96e2286f5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eca016aed05e4ce09e1a2bf96e2286f5 already in progress: creating tablet
I20251029 02:52:02.094859 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 463) took 3085 ms. Trace:
I20251029 02:52:02.096004 1175 rpcz_store.cc:276] 1029 02:51:59.009169 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:59.326705 (+317536us) service_pool.cc:225] Handling call
1029 02:52:02.094732 (+2768027us) ts_tablet_manager.cc:1151] Deleting tablet 86fc22bb84794149b598b97df41eb881
1029 02:52:02.094742 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.094830 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2690750}
W20251029 02:52:02.096482 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 86fc22bb84794149b598b97df41eb881 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 86fc22bb84794149b598b97df41eb881 already in progress: creating tablet
I20251029 02:52:02.096627 774 raft_consensus.cc:1275] T 4bf06476c87a4549a73f5994314bb75f P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Refusing update from remote peer c756528642a543eeacd93cef3e335a7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251029 02:52:02.096854 1175 tablet_replica.cc:333] T 1fc87e2d7b4240e08f2b4ac7277293ff P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:02.097071 1175 raft_consensus.cc:2243] T 1fc87e2d7b4240e08f2b4ac7277293ff P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:02.097236 1175 raft_consensus.cc:2272] T 1fc87e2d7b4240e08f2b4ac7277293ff P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:02.097195 1682 consensus_queue.cc:1048] T 4bf06476c87a4549a73f5994314bb75f P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:02.096916 1060 raft_consensus.cc:1275] T 4bf06476c87a4549a73f5994314bb75f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Refusing update from remote peer c756528642a543eeacd93cef3e335a7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251029 02:52:02.100771 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 535) took 1980 ms. Trace:
I20251029 02:52:02.100939 1176 rpcz_store.cc:276] 1029 02:52:00.120670 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:00.567896 (+447226us) service_pool.cc:225] Handling call
1029 02:52:02.100605 (+1532709us) ts_tablet_manager.cc:1151] Deleting tablet 8df97119d0654d7d92bb5bd17c5d8562
1029 02:52:02.100617 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.100716 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1532543}
I20251029 02:52:02.101207 1175 ts_tablet_manager.cc:1916] T 1fc87e2d7b4240e08f2b4ac7277293ff P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:02.101501 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 536) took 1980 ms. Trace:
I20251029 02:52:02.102190 1176 rpcz_store.cc:276] 1029 02:52:00.120789 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:00.568069 (+447280us) service_pool.cc:225] Handling call
1029 02:52:02.101383 (+1533314us) ts_tablet_manager.cc:1151] Deleting tablet f65fe9a5137b48669afef51ad7dd76ae
1029 02:52:02.101392 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.101472 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1533182}
I20251029 02:52:02.102743 1682 consensus_queue.cc:1048] T 4bf06476c87a4549a73f5994314bb75f P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:02.102988 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 543) took 1981 ms. Trace:
W20251029 02:52:02.105576 517 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d7abaf84ba5d445aa17433b3be891548 consensus state has no leader
I20251029 02:52:02.107576 1172 tablet_bootstrap.cc:492] T 8df97119d0654d7d92bb5bd17c5d8562 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:02.107800 1172 ts_tablet_manager.cc:1403] T 8df97119d0654d7d92bb5bd17c5d8562 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.000s
I20251029 02:52:02.108527 520 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_31aa9d49c83b4ab29823acea7480bc96" } modify_external_catalogs: true
I20251029 02:52:02.108672 517 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_18daac68b3444f3aaf2f2e45618ef8f4" } new_table_name: "default.Table_0a039bfb2e1f458f89ffc3ee93300b8f" modify_external_catalogs: true
I20251029 02:52:02.108686 1172 raft_consensus.cc:359] T 8df97119d0654d7d92bb5bd17c5d8562 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:02.108860 1172 raft_consensus.cc:385] T 8df97119d0654d7d92bb5bd17c5d8562 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:02.108806 520 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_31aa9d49c83b4ab29823acea7480bc96" } modify_external_catalogs: true
I20251029 02:52:02.108937 1172 raft_consensus.cc:740] T 8df97119d0654d7d92bb5bd17c5d8562 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:02.109221 1172 consensus_queue.cc:260] T 8df97119d0654d7d92bb5bd17c5d8562 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:02.085914 1166 tablet_bootstrap.cc:492] T 9964ad774890422f8e14f6b99f28cb55 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:02.109607 1166 ts_tablet_manager.cc:1403] T 9964ad774890422f8e14f6b99f28cb55 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.492s user 0.007s sys 0.014s
I20251029 02:52:02.109545 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_7c3fa8fa8b3e4f3c91739b490c0e807e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:02.109822 1172 ts_tablet_manager.cc:1434] T 8df97119d0654d7d92bb5bd17c5d8562 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20251029 02:52:02.110093 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7c3fa8fa8b3e4f3c91739b490c0e807e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:02.110121 1172 tablet_bootstrap.cc:492] T f81aecb20d6e498bb9b498698886d65e P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:02.111812 1172 tablet_bootstrap.cc:654] T f81aecb20d6e498bb9b498698886d65e P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:02.115062 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_2499efd3a69d41bd8d6ddcb0987c98b4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:02.116134 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2499efd3a69d41bd8d6ddcb0987c98b4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:02.119051 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_37dad1121b2144ee9eeb3ab7c38a0609"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:02.119391 1174 tablet_replica.cc:333] T b5d5b737c86844d58c51b2388c0b1705 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:02.119576 1174 raft_consensus.cc:2243] T b5d5b737c86844d58c51b2388c0b1705 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:02.119900 1174 pending_rounds.cc:70] T b5d5b737c86844d58c51b2388c0b1705 P c756528642a543eeacd93cef3e335a7d: Trying to abort 1 pending ops.
I20251029 02:52:02.120004 1174 pending_rounds.cc:77] T b5d5b737c86844d58c51b2388c0b1705 P c756528642a543eeacd93cef3e335a7d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215949094780276736 op_type: NO_OP noop_request { }
I20251029 02:52:02.120172 1174 raft_consensus.cc:2889] T b5d5b737c86844d58c51b2388c0b1705 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 02:52:02.120308 1174 raft_consensus.cc:2272] T b5d5b737c86844d58c51b2388c0b1705 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:02.121177 1174 ts_tablet_manager.cc:1916] T b5d5b737c86844d58c51b2388c0b1705 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:02.110420 1166 raft_consensus.cc:359] T 9964ad774890422f8e14f6b99f28cb55 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:02.121789 1166 raft_consensus.cc:385] T 9964ad774890422f8e14f6b99f28cb55 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:02.121922 1166 raft_consensus.cc:740] T 9964ad774890422f8e14f6b99f28cb55 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:02.125787 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_94ed5141b07a4cd7b94c14174c30ed99"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:02.126112 519 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_691b1d73c4f44f098dccf26f6a51a4d0" } new_table_name: "default.Table_bb4222ae83ec442f8f62fbb51e9ab8eb" modify_external_catalogs: true
I20251029 02:52:02.126335 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet aa776cde2f494fa5810539edf9f289a1 (table default.Table_47a1a59db0444fa49f7274509385ff0c [id=283c09ba60114ba9b165e2ff144a6c35]) successfully deleted
W20251029 02:52:02.126763 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_94ed5141b07a4cd7b94c14174c30ed99 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:52:02.126834 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_37dad1121b2144ee9eeb3ab7c38a0609 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:02.128028 1172 tablet_bootstrap.cc:492] T f81aecb20d6e498bb9b498698886d65e P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:02.128191 1172 ts_tablet_manager.cc:1403] T f81aecb20d6e498bb9b498698886d65e P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.000s
I20251029 02:52:02.128630 1166 consensus_queue.cc:260] T 9964ad774890422f8e14f6b99f28cb55 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:02.129057 1172 raft_consensus.cc:359] T f81aecb20d6e498bb9b498698886d65e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:02.129236 1172 raft_consensus.cc:385] T f81aecb20d6e498bb9b498698886d65e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:02.129339 1172 raft_consensus.cc:740] T f81aecb20d6e498bb9b498698886d65e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:02.129869 1166 ts_tablet_manager.cc:1434] T 9964ad774890422f8e14f6b99f28cb55 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.020s user 0.002s sys 0.000s
I20251029 02:52:02.129647 1172 consensus_queue.cc:260] T f81aecb20d6e498bb9b498698886d65e P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:02.130802 1166 tablet_bootstrap.cc:492] T 69597d48631d43b6a5b92abbf60d5ed3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:02.130898 1172 ts_tablet_manager.cc:1434] T f81aecb20d6e498bb9b498698886d65e P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20251029 02:52:02.131228 1172 tablet_bootstrap.cc:492] T c9e923724ca9402682f13b252dbc37aa P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
W20251029 02:52:02.135816 517 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet adf6e736cb9c4920807ee58682e134ef consensus state has no leader
I20251029 02:52:02.137349 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_6f4e7e4f715747b492f4d5b8e3ab8c5a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:02.138072 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6f4e7e4f715747b492f4d5b8e3ab8c5a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:02.139644 1175 ts_tablet_manager.cc:1929] T 1fc87e2d7b4240e08f2b4ac7277293ff P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:02.140370 1175 log.cc:1199] T 1fc87e2d7b4240e08f2b4ac7277293ff P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/1fc87e2d7b4240e08f2b4ac7277293ff
I20251029 02:52:02.140575 1166 tablet_bootstrap.cc:654] T 69597d48631d43b6a5b92abbf60d5ed3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:02.141450 1175 ts_tablet_manager.cc:1950] T 1fc87e2d7b4240e08f2b4ac7277293ff P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:02.103080 1176 rpcz_store.cc:276] 1029 02:52:00.121309 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:00.662946 (+541637us) service_pool.cc:225] Handling call
1029 02:52:02.102844 (+1439898us) ts_tablet_manager.cc:1151] Deleting tablet 7b0f7d6c1c9842789c6ce2cecaa3d5b0
1029 02:52:02.102853 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.102960 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1439673}
I20251029 02:52:02.142441 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 544) took 2021 ms. Trace:
I20251029 02:52:02.142652 1176 rpcz_store.cc:276] 1029 02:52:00.121377 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:00.663199 (+541822us) service_pool.cc:225] Handling call
1029 02:52:02.142284 (+1479085us) ts_tablet_manager.cc:1151] Deleting tablet e7b7de85c13b4ddc9c995ed2a2ccd921
1029 02:52:02.142295 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.142405 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1477374}
I20251029 02:52:02.142939 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_35d17629e08e4b41bf6621b43343ed6b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:02.143230 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 546) took 2021 ms. Trace:
I20251029 02:52:02.143332 1176 rpcz_store.cc:276] 1029 02:52:00.121533 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:00.664934 (+543401us) service_pool.cc:225] Handling call
1029 02:52:02.143105 (+1478171us) ts_tablet_manager.cc:1151] Deleting tablet 8afc01d52d784f2fa559cab7ce07c8be
1029 02:52:02.143114 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.143202 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1477961}
W20251029 02:52:02.143464 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_35d17629e08e4b41bf6621b43343ed6b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:02.143445 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 461) took 3134 ms. Trace:
I20251029 02:52:02.143539 1175 rpcz_store.cc:276] 1029 02:51:59.008931 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:59.325462 (+316531us) service_pool.cc:225] Handling call
1029 02:52:02.096788 (+2771326us) ts_tablet_manager.cc:1151] Deleting tablet 1fc87e2d7b4240e08f2b4ac7277293ff
1029 02:52:02.096797 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.119844 (+ 23047us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:02.139481 (+ 19637us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:02.143383 (+ 3902us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2691464}
I20251029 02:52:02.143772 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 542) took 2022 ms. Trace:
I20251029 02:52:02.143869 1176 rpcz_store.cc:276] 1029 02:52:00.121250 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:00.642818 (+521568us) service_pool.cc:225] Handling call
1029 02:52:02.143657 (+1500839us) ts_tablet_manager.cc:1151] Deleting tablet 294afb9b012c40c0b05b73795d6f0ac8
1029 02:52:02.143666 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.143745 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1476167}
I20251029 02:52:02.144012 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 1fc87e2d7b4240e08f2b4ac7277293ff (table default.Table_9306136142dd41cca83965ab5a92c5a0 [id=130ea33221324d7388a014abd549b2ff]) successfully deleted
I20251029 02:52:02.144284 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 545) took 2022 ms. Trace:
I20251029 02:52:02.144394 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskYVokEG/build/asan/bin/kudu pid 424
I20251029 02:52:02.144388 1176 rpcz_store.cc:276] 1029 02:52:00.121451 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:00.664340 (+542889us) service_pool.cc:225] Handling call
1029 02:52:02.144176 (+1479836us) ts_tablet_manager.cc:1151] Deleting tablet 44c0488eba0543078ff4f0b3d69366f2
1029 02:52:02.144185 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.144256 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1442778}
I20251029 02:52:02.144819 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 547) took 2023 ms. Trace:
I20251029 02:52:02.144915 1176 rpcz_store.cc:276] 1029 02:52:00.121631 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:00.665160 (+543529us) service_pool.cc:225] Handling call
1029 02:52:02.144707 (+1479547us) ts_tablet_manager.cc:1151] Deleting tablet 7bb85a12f573494cbb3118d1947eda27
1029 02:52:02.144715 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.144790 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1435846}
I20251029 02:52:02.145292 1176 tablet_replica.cc:333] T 75236b2fa82f4f28b82efae9a2c5160d P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:02.145483 1176 raft_consensus.cc:2243] T 75236b2fa82f4f28b82efae9a2c5160d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:02.145574 485 generic_service.cc:196] Checking for leaks (request via RPC)
I20251029 02:52:02.145713 1176 raft_consensus.cc:2272] T 75236b2fa82f4f28b82efae9a2c5160d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:02.146322 1176 ts_tablet_manager.cc:1916] T 75236b2fa82f4f28b82efae9a2c5160d P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:02.148267 522 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_bd020e7c45a54c6e92211c4261293575" } new_table_name: "default.Table_da047830460443a29c5269c98fc859e9" modify_external_catalogs: true
I20251029 02:52:02.149118 1782 tablet.cc:1722] T 4bf06476c87a4549a73f5994314bb75f P c756528642a543eeacd93cef3e335a7d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 02:52:02.150171 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_bc99b82ab1b84af899404ca0f6ec25ad"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:02.150730 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bc99b82ab1b84af899404ca0f6ec25ad in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:02.153164 521 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_203b46ee33744bf3bfe75190ab006c13" } new_table_name: "default.Table_18fc160d4f4a48c792cfda3695674974" modify_external_catalogs: true
I20251029 02:52:02.161935 885 tablet_service.cc:1505] Processing CreateTablet for tablet 150eee92de394d8cb2c0b417884b8e6e (DEFAULT_TABLE table=default.Table_ad5cf71ba5be4c8a92c3783a900e0054 [id=442aabf74ce4422db409dfc194001a0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:02.162222 749 tablet_service.cc:1505] Processing CreateTablet for tablet 150eee92de394d8cb2c0b417884b8e6e (DEFAULT_TABLE table=default.Table_ad5cf71ba5be4c8a92c3783a900e0054 [id=442aabf74ce4422db409dfc194001a0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:02.162437 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 150eee92de394d8cb2c0b417884b8e6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:02.163025 518 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet 4bf06476c87a4549a73f5994314bb75f (table default.Table_bafe26b1fca14cc9b908984b7d8a2442 [id=046010a03d774e91b0ad10df97848c56]). Expected version 1 got 0
I20251029 02:52:02.164160 518 master_service.cc:954] ReplaceTablet: received request to replace tablet 5bd2163bc4be467fb695629bfbb5a6ca from {username='slave'} at 127.0.0.1:35314
I20251029 02:52:02.153128 1736 tablet.cc:1722] T 4bf06476c87a4549a73f5994314bb75f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 02:52:02.163820 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet 98bbaec01186435b8bcc7257a1390185 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:01 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:02.165004 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_f6666e856c244ff0b9b28f1ffd730bf9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:02.165244 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet c7b78fd458e4414b9fc3aa064a80b206 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:01 UTC) from {username='slave'} at 127.0.0.1:37626
W20251029 02:52:02.165784 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f6666e856c244ff0b9b28f1ffd730bf9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:02.167929 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 150eee92de394d8cb2c0b417884b8e6e. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:02.168406 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 24f07b1efac64b099b0b5ed467463749 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 24f07b1efac64b099b0b5ed467463749 already in progress: creating tablet
W20251029 02:52:02.168699 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 5842e9bba4fa4295ba21e90d645a56df because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5842e9bba4fa4295ba21e90d645a56df already in progress: creating tablet
W20251029 02:52:02.168948 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 40e844fd7ad74680bcbe7bef936c5b9d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 40e844fd7ad74680bcbe7bef936c5b9d already in progress: creating tablet
W20251029 02:52:02.169126 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 28f7ceb91a34450cad4b5b45f5285b41 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 28f7ceb91a34450cad4b5b45f5285b41 already in progress: creating tablet
W20251029 02:52:02.169282 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 58dde9397b434b089ade9968f369006a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 58dde9397b434b089ade9968f369006a already in progress: creating tablet
W20251029 02:52:02.169474 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet fcb3db8399aa4cf188b07dda220f7f5b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fcb3db8399aa4cf188b07dda220f7f5b already in progress: creating tablet
W20251029 02:52:02.169656 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 845a69a8696e469a8548787a7645d41a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 845a69a8696e469a8548787a7645d41a already in progress: creating tablet
W20251029 02:52:02.169859 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 38f83ebcb2a440bc8e6a8d36607e6666 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 38f83ebcb2a440bc8e6a8d36607e6666 already in progress: creating tablet
W20251029 02:52:02.170037 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 8df97119d0654d7d92bb5bd17c5d8562 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8df97119d0654d7d92bb5bd17c5d8562 already in progress: creating tablet
I20251029 02:52:02.170071 883 tablet_service.cc:1505] Processing CreateTablet for tablet 6ecc552c1ef643f8bb84622d9d440ba2 (DEFAULT_TABLE table=default.Table_ae1c37b89ad243f0a389a54fa71b3ae0 [id=947d1437043d4ac08cf809711bca8b33]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 02:52:02.170231 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet f65fe9a5137b48669afef51ad7dd76ae because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f65fe9a5137b48669afef51ad7dd76ae already in progress: creating tablet
I20251029 02:52:02.170444 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ecc552c1ef643f8bb84622d9d440ba2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:02.171346 756 tablet_service.cc:1505] Processing CreateTablet for tablet 6ecc552c1ef643f8bb84622d9d440ba2 (DEFAULT_TABLE table=default.Table_ae1c37b89ad243f0a389a54fa71b3ae0 [id=947d1437043d4ac08cf809711bca8b33]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:02.173591 1784 tablet.cc:1722] T 4bf06476c87a4549a73f5994314bb75f P b495a49349324f7d81908f2da700f0d9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 02:52:02.176465 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ecc552c1ef643f8bb84622d9d440ba2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:02.178124 1682 raft_consensus.cc:493] T 0a47781d036149569e1164dd938a45e9 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:02.178265 1682 raft_consensus.cc:515] T 0a47781d036149569e1164dd938a45e9 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:02.179229 1682 leader_election.cc:290] T 0a47781d036149569e1164dd938a45e9 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:02.179674 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0a47781d036149569e1164dd938a45e9" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:02.179924 774 raft_consensus.cc:2468] T 0a47781d036149569e1164dd938a45e9 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:02.180347 847 leader_election.cc:304] T 0a47781d036149569e1164dd938a45e9 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:02.180676 1773 raft_consensus.cc:2804] T 0a47781d036149569e1164dd938a45e9 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
W20251029 02:52:02.180732 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 7b0f7d6c1c9842789c6ce2cecaa3d5b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7b0f7d6c1c9842789c6ce2cecaa3d5b0 already in progress: creating tablet
I20251029 02:52:02.180907 1773 raft_consensus.cc:493] T 0a47781d036149569e1164dd938a45e9 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:02.181108 1773 raft_consensus.cc:3060] T 0a47781d036149569e1164dd938a45e9 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
W20251029 02:52:02.181121 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet e7b7de85c13b4ddc9c995ed2a2ccd921 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e7b7de85c13b4ddc9c995ed2a2ccd921 already in progress: creating tablet
W20251029 02:52:02.181332 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 8afc01d52d784f2fa559cab7ce07c8be because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8afc01d52d784f2fa559cab7ce07c8be already in progress: creating tablet
W20251029 02:52:02.181584 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 294afb9b012c40c0b05b73795d6f0ac8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 294afb9b012c40c0b05b73795d6f0ac8 already in progress: creating tablet
W20251029 02:52:02.181829 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 44c0488eba0543078ff4f0b3d69366f2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 44c0488eba0543078ff4f0b3d69366f2 already in progress: creating tablet
W20251029 02:52:02.182034 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 7bb85a12f573494cbb3118d1947eda27 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7bb85a12f573494cbb3118d1947eda27 already in progress: creating tablet
I20251029 02:52:02.183283 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0a47781d036149569e1164dd938a45e9" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:02.183779 1773 raft_consensus.cc:515] T 0a47781d036149569e1164dd938a45e9 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:02.184443 1773 leader_election.cc:290] T 0a47781d036149569e1164dd938a45e9 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
W20251029 02:52:02.184477 849 leader_election.cc:343] T 0a47781d036149569e1164dd938a45e9 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:02.184885 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0a47781d036149569e1164dd938a45e9" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:02.185092 774 raft_consensus.cc:3060] T 0a47781d036149569e1164dd938a45e9 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:02.185334 1029 tablet_service.cc:1505] Processing CreateTablet for tablet 150eee92de394d8cb2c0b417884b8e6e (DEFAULT_TABLE table=default.Table_ad5cf71ba5be4c8a92c3783a900e0054 [id=442aabf74ce4422db409dfc194001a0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:02.187741 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 150eee92de394d8cb2c0b417884b8e6e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:02.189181 774 raft_consensus.cc:2468] T 0a47781d036149569e1164dd938a45e9 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:02.189630 847 leader_election.cc:304] T 0a47781d036149569e1164dd938a45e9 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:02.190652 1773 raft_consensus.cc:2804] T 0a47781d036149569e1164dd938a45e9 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:02.190905 1773 raft_consensus.cc:697] T 0a47781d036149569e1164dd938a45e9 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:52:02.191215 1773 consensus_queue.cc:237] T 0a47781d036149569e1164dd938a45e9 P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:02.191713 1026 tablet_service.cc:1505] Processing CreateTablet for tablet 6ecc552c1ef643f8bb84622d9d440ba2 (DEFAULT_TABLE table=default.Table_ae1c37b89ad243f0a389a54fa71b3ae0 [id=947d1437043d4ac08cf809711bca8b33]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:02.192945 1167 tablet_bootstrap.cc:492] T 44784a3a760946e0be24743bc2fc8138 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:02.193073 1167 ts_tablet_manager.cc:1403] T 44784a3a760946e0be24743bc2fc8138 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.173s user 0.003s sys 0.006s
I20251029 02:52:02.193744 1167 raft_consensus.cc:359] T 44784a3a760946e0be24743bc2fc8138 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:02.193922 1167 raft_consensus.cc:385] T 44784a3a760946e0be24743bc2fc8138 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:02.193995 1167 raft_consensus.cc:740] T 44784a3a760946e0be24743bc2fc8138 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:02.194164 1167 consensus_queue.cc:260] T 44784a3a760946e0be24743bc2fc8138 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:02.195057 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 465) took 3178 ms. Trace:
I20251029 02:52:02.195356 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0a47781d036149569e1164dd938a45e9" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
W20251029 02:52:02.195940 849 leader_election.cc:343] T 0a47781d036149569e1164dd938a45e9 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:02.196259 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ecc552c1ef643f8bb84622d9d440ba2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:02.196291 1037 tablet_service.cc:1505] Processing CreateTablet for tablet de7fac8751a04c19b68a52db7f167214 (DEFAULT_TABLE table=default.Table_ace94370997c4944968f3e7633615ae2 [id=dcdf045a4fb8448f9a34bae2a5a0add1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:02.196583 885 tablet_service.cc:1505] Processing CreateTablet for tablet de7fac8751a04c19b68a52db7f167214 (DEFAULT_TABLE table=default.Table_ace94370997c4944968f3e7633615ae2 [id=dcdf045a4fb8448f9a34bae2a5a0add1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:02.196971 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de7fac8751a04c19b68a52db7f167214. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:02.196985 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de7fac8751a04c19b68a52db7f167214. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:02.196938 1175 rpcz_store.cc:276] 1029 02:51:59.009398 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:59.337752 (+328354us) service_pool.cc:225] Handling call
1029 02:52:02.153208 (+2815456us) ts_tablet_manager.cc:1151] Deleting tablet 358bdfbcb4ee440c899fb6eedef68f44
1029 02:52:02.153223 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.187591 (+ 34368us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2726022}
W20251029 02:52:02.197433 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 358bdfbcb4ee440c899fb6eedef68f44 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 358bdfbcb4ee440c899fb6eedef68f44 already in progress: creating tablet
I20251029 02:52:02.195236 1172 tablet_bootstrap.cc:654] T c9e923724ca9402682f13b252dbc37aa P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:02.196786 756 tablet_service.cc:1505] Processing CreateTablet for tablet de7fac8751a04c19b68a52db7f167214 (DEFAULT_TABLE table=default.Table_ace94370997c4944968f3e7633615ae2 [id=dcdf045a4fb8448f9a34bae2a5a0add1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:02.198984 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de7fac8751a04c19b68a52db7f167214. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:02.199201 1166 tablet_bootstrap.cc:492] T 69597d48631d43b6a5b92abbf60d5ed3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:02.200256 1166 ts_tablet_manager.cc:1403] T 69597d48631d43b6a5b92abbf60d5ed3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.069s user 0.003s sys 0.001s
I20251029 02:52:02.201581 1166 raft_consensus.cc:359] T 69597d48631d43b6a5b92abbf60d5ed3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:02.201859 1166 raft_consensus.cc:385] T 69597d48631d43b6a5b92abbf60d5ed3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:02.201943 1166 raft_consensus.cc:740] T 69597d48631d43b6a5b92abbf60d5ed3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:02.202353 1166 consensus_queue.cc:260] T 69597d48631d43b6a5b92abbf60d5ed3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:02.202927 1166 ts_tablet_manager.cc:1434] T 69597d48631d43b6a5b92abbf60d5ed3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20251029 02:52:02.203178 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 478) took 2980 ms. Trace:
I20251029 02:52:02.203509 1166 tablet_bootstrap.cc:492] T 65fdec7cf5d143208d78d4c3806b1352 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:02.203634 1175 rpcz_store.cc:276] 1029 02:51:59.222788 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:59.597525 (+374737us) service_pool.cc:225] Handling call
1029 02:52:02.201873 (+2604348us) ts_tablet_manager.cc:1151] Deleting tablet 023a4094b424464c8f49d6eb9cc15da4
1029 02:52:02.201883 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.203120 (+ 1237us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2602623}
W20251029 02:52:02.205147 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 023a4094b424464c8f49d6eb9cc15da4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 023a4094b424464c8f49d6eb9cc15da4 already in progress: creating tablet
I20251029 02:52:02.206578 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 479) took 2964 ms. Trace:
I20251029 02:52:02.207628 1175 rpcz_store.cc:276] 1029 02:51:59.241839 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:59.599283 (+357444us) service_pool.cc:225] Handling call
1029 02:52:02.206378 (+2607095us) ts_tablet_manager.cc:1151] Deleting tablet 1878d7eef14a4578bc929da68872cee4
1029 02:52:02.206391 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.206497 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2606856}
W20251029 02:52:02.209893 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 1878d7eef14a4578bc929da68872cee4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1878d7eef14a4578bc929da68872cee4 already in progress: creating tablet
I20251029 02:52:02.204324 1172 tablet_bootstrap.cc:492] T c9e923724ca9402682f13b252dbc37aa P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:02.210314 1172 ts_tablet_manager.cc:1403] T c9e923724ca9402682f13b252dbc37aa P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.079s user 0.001s sys 0.003s
I20251029 02:52:02.210711 1041 tablet_service.cc:1505] Processing CreateTablet for tablet 81ee839d811a4f46b93a57ede397bd93 (DEFAULT_TABLE table=default.Table_7d8a27a8c05d4e9686a1fd616c5697bd [id=12dedbeba5d141bfa19cd5a7ba3b20b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:02.211148 1682 raft_consensus.cc:493] T 9e07291d6b5645feb4d5890f58e1ef74 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:02.211421 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81ee839d811a4f46b93a57ede397bd93. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:02.211321 1172 raft_consensus.cc:359] T c9e923724ca9402682f13b252dbc37aa P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:02.211383 1682 raft_consensus.cc:515] T 9e07291d6b5645feb4d5890f58e1ef74 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:02.212150 1172 raft_consensus.cc:385] T c9e923724ca9402682f13b252dbc37aa P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:02.212083 1682 leader_election.cc:290] T 9e07291d6b5645feb4d5890f58e1ef74 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:02.212278 1172 raft_consensus.cc:740] T c9e923724ca9402682f13b252dbc37aa P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:02.212528 1172 consensus_queue.cc:260] T c9e923724ca9402682f13b252dbc37aa P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:02.209381 749 tablet_service.cc:1505] Processing CreateTablet for tablet 81ee839d811a4f46b93a57ede397bd93 (DEFAULT_TABLE table=default.Table_7d8a27a8c05d4e9686a1fd616c5697bd [id=12dedbeba5d141bfa19cd5a7ba3b20b7]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 02:52:02.213071 847 leader_election.cc:343] T 9e07291d6b5645feb4d5890f58e1ef74 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 9e07291d6b5645feb4d5890f58e1ef74
I20251029 02:52:02.213766 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81ee839d811a4f46b93a57ede397bd93. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:02.214159 849 leader_election.cc:343] T 9e07291d6b5645feb4d5890f58e1ef74 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Not found: Tablet not found: 9e07291d6b5645feb4d5890f58e1ef74
I20251029 02:52:02.214316 849 leader_election.cc:304] T 9e07291d6b5645feb4d5890f58e1ef74 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9
I20251029 02:52:02.214557 1682 raft_consensus.cc:2749] T 9e07291d6b5645feb4d5890f58e1ef74 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:02.215245 883 tablet_service.cc:1505] Processing CreateTablet for tablet 81ee839d811a4f46b93a57ede397bd93 (DEFAULT_TABLE table=default.Table_7d8a27a8c05d4e9686a1fd616c5697bd [id=12dedbeba5d141bfa19cd5a7ba3b20b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:02.215762 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81ee839d811a4f46b93a57ede397bd93. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:02.220127 1176 ts_tablet_manager.cc:1929] T 75236b2fa82f4f28b82efae9a2c5160d P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:02.220541 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 480) took 2973 ms. Trace:
I20251029 02:52:02.220739 1175 rpcz_store.cc:276] 1029 02:51:59.246723 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:59.599544 (+352821us) service_pool.cc:225] Handling call
1029 02:52:02.220346 (+2620802us) ts_tablet_manager.cc:1151] Deleting tablet 7816556521144c928d7a02694e22e79f
1029 02:52:02.220359 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.220481 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2620537}
I20251029 02:52:02.221097 1176 log.cc:1199] T 75236b2fa82f4f28b82efae9a2c5160d P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/75236b2fa82f4f28b82efae9a2c5160d
W20251029 02:52:02.221406 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 7816556521144c928d7a02694e22e79f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7816556521144c928d7a02694e22e79f already in progress: creating tablet
I20251029 02:52:02.221530 1176 ts_tablet_manager.cc:1950] T 75236b2fa82f4f28b82efae9a2c5160d P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:02.222031 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 481) took 2972 ms. Trace:
I20251029 02:52:02.222172 1167 ts_tablet_manager.cc:1434] T 44784a3a760946e0be24743bc2fc8138 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.029s user 0.001s sys 0.001s
I20251029 02:52:02.222167 1175 rpcz_store.cc:276] 1029 02:51:59.249359 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:59.599814 (+350455us) service_pool.cc:225] Handling call
1029 02:52:02.221887 (+2622073us) ts_tablet_manager.cc:1151] Deleting tablet 94bdbb9394f7490e99916bd818986208
1029 02:52:02.221895 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.221980 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2621809}
I20251029 02:52:02.222486 1167 tablet_bootstrap.cc:492] T fd0f89180aba43eabb20f926abccbaa6 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:02.224504 1167 tablet_bootstrap.cc:654] T fd0f89180aba43eabb20f926abccbaa6 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:02.225975 1038 tablet_service.cc:1505] Processing CreateTablet for tablet 0e9a9dd2f6e4424cb1500682d5a49cc4 (DEFAULT_TABLE table=default.Table_022e72d689654b18adde8de9ec2c065a [id=f6cf2b15e3cf4c35b1c5a0cee8942baa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:02.226673 881 tablet_service.cc:1505] Processing CreateTablet for tablet 0e9a9dd2f6e4424cb1500682d5a49cc4 (DEFAULT_TABLE table=default.Table_022e72d689654b18adde8de9ec2c065a [id=f6cf2b15e3cf4c35b1c5a0cee8942baa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:02.227067 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e9a9dd2f6e4424cb1500682d5a49cc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:02.227231 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e9a9dd2f6e4424cb1500682d5a49cc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:02.226377 742 tablet_service.cc:1505] Processing CreateTablet for tablet 0e9a9dd2f6e4424cb1500682d5a49cc4 (DEFAULT_TABLE table=default.Table_022e72d689654b18adde8de9ec2c065a [id=f6cf2b15e3cf4c35b1c5a0cee8942baa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:02.227746 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e9a9dd2f6e4424cb1500682d5a49cc4. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:02.235474 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 94bdbb9394f7490e99916bd818986208 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 94bdbb9394f7490e99916bd818986208 already in progress: creating tablet
I20251029 02:52:02.236106 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 482) took 2975 ms. Trace:
I20251029 02:52:02.237731 1175 rpcz_store.cc:276] 1029 02:51:59.260716 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:51:59.600101 (+339385us) service_pool.cc:225] Handling call
1029 02:52:02.235900 (+2635799us) ts_tablet_manager.cc:1151] Deleting tablet c9bd3f2deb5d4af09b7918269a98509f
1029 02:52:02.235912 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.236056 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2635550}
W20251029 02:52:02.239307 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet c9bd3f2deb5d4af09b7918269a98509f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c9bd3f2deb5d4af09b7918269a98509f already in progress: creating tablet
I20251029 02:52:02.240720 1039 tablet_service.cc:1505] Processing CreateTablet for tablet 9778f924d11a48b194ccf26124bd5509 (DEFAULT_TABLE table=default.Table_df78a989ef8e4c48b1716f5698c6ecf9 [id=5bd6c6f6062a42438261b3d26ea313e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:02.241462 881 tablet_service.cc:1505] Processing CreateTablet for tablet 9778f924d11a48b194ccf26124bd5509 (DEFAULT_TABLE table=default.Table_df78a989ef8e4c48b1716f5698c6ecf9 [id=5bd6c6f6062a42438261b3d26ea313e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:02.241863 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9778f924d11a48b194ccf26124bd5509. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:02.241925 742 tablet_service.cc:1505] Processing CreateTablet for tablet 9778f924d11a48b194ccf26124bd5509 (DEFAULT_TABLE table=default.Table_df78a989ef8e4c48b1716f5698c6ecf9 [id=5bd6c6f6062a42438261b3d26ea313e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:02.242594 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9778f924d11a48b194ccf26124bd5509. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:02.243758 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9778f924d11a48b194ccf26124bd5509. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:02.247193 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 493) took 2913 ms. Trace:
I20251029 02:52:02.247481 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 558) took 1813 ms. Trace:
I20251029 02:52:02.249389 1172 ts_tablet_manager.cc:1434] T c9e923724ca9402682f13b252dbc37aa P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.030s user 0.003s sys 0.000s
I20251029 02:52:02.254503 1172 tablet_bootstrap.cc:492] T cedfb53078d94c81b5e294ce7438325e P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:02.255657 1167 tablet_bootstrap.cc:492] T fd0f89180aba43eabb20f926abccbaa6 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:02.255520 1175 rpcz_store.cc:276] 1029 02:51:59.326174 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:00.016889 (+690715us) service_pool.cc:225] Handling call
1029 02:52:02.239683 (+2222794us) ts_tablet_manager.cc:1151] Deleting tablet 24f07b1efac64b099b0b5ed467463749
1029 02:52:02.239695 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.239815 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2190429}
I20251029 02:52:02.255846 1167 ts_tablet_manager.cc:1403] T fd0f89180aba43eabb20f926abccbaa6 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.033s user 0.003s sys 0.000s
I20251029 02:52:02.256035 1176 rpcz_store.cc:276] 1029 02:52:00.408426 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:00.811882 (+403456us) service_pool.cc:225] Handling call
1029 02:52:02.145231 (+1333349us) ts_tablet_manager.cc:1151] Deleting tablet 75236b2fa82f4f28b82efae9a2c5160d
1029 02:52:02.145240 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.191939 (+ 46699us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:02.220035 (+ 28096us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:02.222253 (+ 2218us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1322268}
W20251029 02:52:02.256155 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 24f07b1efac64b099b0b5ed467463749 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 24f07b1efac64b099b0b5ed467463749 already in progress: creating tablet
I20251029 02:52:02.256619 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 75236b2fa82f4f28b82efae9a2c5160d (table default.Table_0b84a2f666074fd18085ff4ce249a2fe [id=715a1df4e483412da9a63eca19b8f02d]) successfully deleted
I20251029 02:52:02.256623 1167 raft_consensus.cc:359] T fd0f89180aba43eabb20f926abccbaa6 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:02.256786 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 559) took 1848 ms. Trace:
I20251029 02:52:02.256846 1167 raft_consensus.cc:385] T fd0f89180aba43eabb20f926abccbaa6 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:02.256927 1167 raft_consensus.cc:740] T fd0f89180aba43eabb20f926abccbaa6 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:02.256948 1176 rpcz_store.cc:276] 1029 02:52:00.408596 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:00.822987 (+414391us) service_pool.cc:225] Handling call
1029 02:52:02.256452 (+1433465us) ts_tablet_manager.cc:1151] Deleting tablet 97b2ecae572a4b7ea1b9595074556708
1029 02:52:02.256462 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.256596 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1411337}
I20251029 02:52:02.257180 1167 consensus_queue.cc:260] T fd0f89180aba43eabb20f926abccbaa6 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
W20251029 02:52:02.257556 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 97b2ecae572a4b7ea1b9595074556708 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 97b2ecae572a4b7ea1b9595074556708 already in progress: creating tablet
I20251029 02:52:02.257674 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 499) took 2839 ms. Trace:
I20251029 02:52:02.259446 1038 tablet_service.cc:1505] Processing CreateTablet for tablet 4e33a29c6ef34104b73a4f1329a8387a (DEFAULT_TABLE table=default.Table_f28c52f38da44fa6b73e9bca9b29908e [id=8cec20445cb14908a738e7acb166a183]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:02.263001 1175 rpcz_store.cc:276] 1029 02:51:59.418356 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:00.130008 (+711652us) service_pool.cc:225] Handling call
1029 02:52:02.256672 (+2126664us) ts_tablet_manager.cc:1151] Deleting tablet 40e844fd7ad74680bcbe7bef936c5b9d
1029 02:52:02.256683 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.257616 (+ 933us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2126454}
W20251029 02:52:02.264088 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 40e844fd7ad74680bcbe7bef936c5b9d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 40e844fd7ad74680bcbe7bef936c5b9d already in progress: creating tablet
I20251029 02:52:02.264674 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 500) took 2834 ms. Trace:
I20251029 02:52:02.266067 881 tablet_service.cc:1505] Processing CreateTablet for tablet 4e33a29c6ef34104b73a4f1329a8387a (DEFAULT_TABLE table=default.Table_f28c52f38da44fa6b73e9bca9b29908e [id=8cec20445cb14908a738e7acb166a183]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:02.266534 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e33a29c6ef34104b73a4f1329a8387a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:02.268280 1175 rpcz_store.cc:276] 1029 02:51:59.430689 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:00.130237 (+699548us) service_pool.cc:225] Handling call
1029 02:52:02.264482 (+2134245us) ts_tablet_manager.cc:1151] Deleting tablet 28f7ceb91a34450cad4b5b45f5285b41
1029 02:52:02.264496 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.264619 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2134080}
W20251029 02:52:02.268774 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 28f7ceb91a34450cad4b5b45f5285b41 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 28f7ceb91a34450cad4b5b45f5285b41 already in progress: creating tablet
I20251029 02:52:02.266155 742 tablet_service.cc:1505] Processing CreateTablet for tablet 4e33a29c6ef34104b73a4f1329a8387a (DEFAULT_TABLE table=default.Table_f28c52f38da44fa6b73e9bca9b29908e [id=8cec20445cb14908a738e7acb166a183]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:02.268936 1166 tablet_bootstrap.cc:654] T 65fdec7cf5d143208d78d4c3806b1352 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:02.269235 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e33a29c6ef34104b73a4f1329a8387a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:02.272712 1167 ts_tablet_manager.cc:1434] T fd0f89180aba43eabb20f926abccbaa6 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
I20251029 02:52:02.273164 1167 tablet_bootstrap.cc:492] T 69597d48631d43b6a5b92abbf60d5ed3 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:02.274703 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 501) took 2843 ms. Trace:
I20251029 02:52:02.275478 1167 tablet_bootstrap.cc:654] T 69597d48631d43b6a5b92abbf60d5ed3 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:02.277681 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 560) took 1849 ms. Trace:
I20251029 02:52:02.277784 881 tablet_service.cc:1505] Processing CreateTablet for tablet 7dbc9c8b110a48f0a801ad468738c3a3 (DEFAULT_TABLE table=default.Table_3297bb18a32a4454a6458e189b9985d1 [id=b7ab42abbcef4d68bd5288794cd41ae5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:02.277897 1176 rpcz_store.cc:276] 1029 02:52:00.408708 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:00.845145 (+436437us) service_pool.cc:225] Handling call
1029 02:52:02.257868 (+1412723us) ts_tablet_manager.cc:1151] Deleting tablet 998b9cbe76514665be76c6c387dc75b3
1029 02:52:02.257878 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.257958 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1354399}
I20251029 02:52:02.278225 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7dbc9c8b110a48f0a801ad468738c3a3. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:02.278725 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 998b9cbe76514665be76c6c387dc75b3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 998b9cbe76514665be76c6c387dc75b3 already in progress: creating tablet
I20251029 02:52:02.278715 1175 rpcz_store.cc:276] 1029 02:51:59.430832 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:00.130420 (+699588us) service_pool.cc:225] Handling call
1029 02:52:02.273827 (+2143407us) ts_tablet_manager.cc:1151] Deleting tablet 58dde9397b434b089ade9968f369006a
1029 02:52:02.273834 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.274529 (+ 695us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2143147}
W20251029 02:52:02.279167 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 58dde9397b434b089ade9968f369006a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 58dde9397b434b089ade9968f369006a already in progress: creating tablet
I20251029 02:52:02.276158 1172 tablet_bootstrap.cc:654] T cedfb53078d94c81b5e294ce7438325e P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:02.277932 755 tablet_service.cc:1505] Processing CreateTablet for tablet 7dbc9c8b110a48f0a801ad468738c3a3 (DEFAULT_TABLE table=default.Table_3297bb18a32a4454a6458e189b9985d1 [id=b7ab42abbcef4d68bd5288794cd41ae5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:02.279632 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7dbc9c8b110a48f0a801ad468738c3a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:02.279786 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 561) took 1870 ms. Trace:
I20251029 02:52:02.279942 1176 rpcz_store.cc:276] 1029 02:52:00.408799 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:00.903500 (+494701us) service_pool.cc:225] Handling call
1029 02:52:02.279342 (+1375842us) ts_tablet_manager.cc:1151] Deleting tablet a7685b0bed174ed0b25f99ef8a46fca3
1029 02:52:02.279353 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.279541 (+ 188us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1370115}
W20251029 02:52:02.280608 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet a7685b0bed174ed0b25f99ef8a46fca3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a7685b0bed174ed0b25f99ef8a46fca3 already in progress: creating tablet
I20251029 02:52:02.281880 1026 tablet_service.cc:1505] Processing CreateTablet for tablet 99f4666db2804794b8db4bf109df4289 (DEFAULT_TABLE table=default.Table_9171c03558ca4705b7f71acb9f542107 [id=7c52adb35a6f4bae9cd51984758bce74]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:02.282205 1037 tablet_service.cc:1505] Processing CreateTablet for tablet 7dbc9c8b110a48f0a801ad468738c3a3 (DEFAULT_TABLE table=default.Table_3297bb18a32a4454a6458e189b9985d1 [id=b7ab42abbcef4d68bd5288794cd41ae5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:02.282783 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 562) took 1872 ms. Trace:
I20251029 02:52:02.282931 1176 rpcz_store.cc:276] 1029 02:52:00.408885 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:00.909223 (+500338us) service_pool.cc:225] Handling call
1029 02:52:02.280880 (+1371657us) ts_tablet_manager.cc:1151] Deleting tablet 4cede3b183494845af91dca39a2096c4
1029 02:52:02.280916 (+ 36us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.281057 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1371457}
W20251029 02:52:02.283532 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 4cede3b183494845af91dca39a2096c4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4cede3b183494845af91dca39a2096c4 already in progress: creating tablet
I20251029 02:52:02.284766 1176 tablet_replica.cc:333] T fd900eec3f0c43bf965c148ea90870a3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:02.285126 1176 raft_consensus.cc:2243] T fd900eec3f0c43bf965c148ea90870a3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:02.286409 1176 raft_consensus.cc:2272] T fd900eec3f0c43bf965c148ea90870a3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:02.287428 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e33a29c6ef34104b73a4f1329a8387a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:02.287916 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 498) took 2882 ms. Trace:
I20251029 02:52:02.288056 1175 rpcz_store.cc:276] 1029 02:51:59.405258 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:00.129292 (+724034us) service_pool.cc:225] Handling call
1029 02:52:02.287755 (+2158463us) ts_tablet_manager.cc:1151] Deleting tablet 5842e9bba4fa4295ba21e90d645a56df
1029 02:52:02.287763 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.287866 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2154127}
W20251029 02:52:02.288578 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 5842e9bba4fa4295ba21e90d645a56df because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5842e9bba4fa4295ba21e90d645a56df already in progress: creating tablet
I20251029 02:52:02.289036 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 510) took 2806 ms. Trace:
I20251029 02:52:02.289163 1175 rpcz_store.cc:276] 1029 02:51:59.482179 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:00.195101 (+712922us) service_pool.cc:225] Handling call
1029 02:52:02.288882 (+2093781us) ts_tablet_manager.cc:1151] Deleting tablet fcb3db8399aa4cf188b07dda220f7f5b
1029 02:52:02.288891 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.288986 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2093520}
W20251029 02:52:02.289633 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet fcb3db8399aa4cf188b07dda220f7f5b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fcb3db8399aa4cf188b07dda220f7f5b already in progress: creating tablet
I20251029 02:52:02.290047 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 511) took 2796 ms. Trace:
I20251029 02:52:02.290177 1175 rpcz_store.cc:276] 1029 02:51:59.493796 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:00.195383 (+701587us) service_pool.cc:225] Handling call
1029 02:52:02.289913 (+2094530us) ts_tablet_manager.cc:1151] Deleting tablet 845a69a8696e469a8548787a7645d41a
1029 02:52:02.289921 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.289998 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2094363}
W20251029 02:52:02.290660 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 845a69a8696e469a8548787a7645d41a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 845a69a8696e469a8548787a7645d41a already in progress: creating tablet
I20251029 02:52:02.291231 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 512) took 2794 ms. Trace:
I20251029 02:52:02.291357 1175 rpcz_store.cc:276] 1029 02:51:59.496545 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:00.195571 (+699026us) service_pool.cc:225] Handling call
1029 02:52:02.290940 (+2095369us) ts_tablet_manager.cc:1151] Deleting tablet 38f83ebcb2a440bc8e6a8d36607e6666
1029 02:52:02.290951 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.291180 (+ 229us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2095216}
W20251029 02:52:02.291819 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 38f83ebcb2a440bc8e6a8d36607e6666 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 38f83ebcb2a440bc8e6a8d36607e6666 already in progress: creating tablet
I20251029 02:52:02.292153 1665 raft_consensus.cc:493] T b2b2d64dcf964da8be4b544dbbe31977 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:02.292376 1665 raft_consensus.cc:515] T b2b2d64dcf964da8be4b544dbbe31977 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:02.293062 1665 leader_election.cc:290] T b2b2d64dcf964da8be4b544dbbe31977 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:02.293676 1175 tablet_replica.cc:333] T 8df97119d0654d7d92bb5bd17c5d8562 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:02.293910 1175 raft_consensus.cc:2243] T 8df97119d0654d7d92bb5bd17c5d8562 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:02.294148 1175 raft_consensus.cc:2272] T 8df97119d0654d7d92bb5bd17c5d8562 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:02.294931 1175 ts_tablet_manager.cc:1916] T 8df97119d0654d7d92bb5bd17c5d8562 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:02.294945 918 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b2b2d64dcf964da8be4b544dbbe31977" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:02.295431 711 leader_election.cc:343] T b2b2d64dcf964da8be4b544dbbe31977 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:02.296698 881 tablet_service.cc:1505] Processing CreateTablet for tablet 99f4666db2804794b8db4bf109df4289 (DEFAULT_TABLE table=default.Table_9171c03558ca4705b7f71acb9f542107 [id=7c52adb35a6f4bae9cd51984758bce74]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:02.296936 756 tablet_service.cc:1505] Processing CreateTablet for tablet 99f4666db2804794b8db4bf109df4289 (DEFAULT_TABLE table=default.Table_9171c03558ca4705b7f71acb9f542107 [id=7c52adb35a6f4bae9cd51984758bce74]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:02.297158 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99f4666db2804794b8db4bf109df4289. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:02.297454 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99f4666db2804794b8db4bf109df4289. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:02.300186 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b2b2d64dcf964da8be4b544dbbe31977" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:52:02.300814 712 leader_election.cc:343] T b2b2d64dcf964da8be4b544dbbe31977 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:02.300933 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99f4666db2804794b8db4bf109df4289. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:02.300947 712 leader_election.cc:304] T b2b2d64dcf964da8be4b544dbbe31977 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:02.301383 1665 raft_consensus.cc:2749] T b2b2d64dcf964da8be4b544dbbe31977 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:02.302326 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7dbc9c8b110a48f0a801ad468738c3a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:02.304405 883 tablet_service.cc:1505] Processing CreateTablet for tablet 9b7770704e674f68960be3ee57bd3c9d (DEFAULT_TABLE table=default.Table_225f7c2cca034ef79a9956e3d1888c5d [id=2b5f62944d654c47a70bcbb29539cece]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:02.304657 755 tablet_service.cc:1505] Processing CreateTablet for tablet 9b7770704e674f68960be3ee57bd3c9d (DEFAULT_TABLE table=default.Table_225f7c2cca034ef79a9956e3d1888c5d [id=2b5f62944d654c47a70bcbb29539cece]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:02.304939 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b7770704e674f68960be3ee57bd3c9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:02.305049 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b7770704e674f68960be3ee57bd3c9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:02.309284 1665 raft_consensus.cc:493] T 42cfdf88472b44979d86836561005ed1 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:02.309453 1665 raft_consensus.cc:515] T 42cfdf88472b44979d86836561005ed1 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:02.310083 1665 leader_election.cc:290] T 42cfdf88472b44979d86836561005ed1 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:52:02.311003 918 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42cfdf88472b44979d86836561005ed1" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:02.311462 711 leader_election.cc:343] T 42cfdf88472b44979d86836561005ed1 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:02.313174 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42cfdf88472b44979d86836561005ed1" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:52:02.313686 712 leader_election.cc:343] T 42cfdf88472b44979d86836561005ed1 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:02.313793 712 leader_election.cc:304] T 42cfdf88472b44979d86836561005ed1 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:02.313798 1029 tablet_service.cc:1505] Processing CreateTablet for tablet 9b7770704e674f68960be3ee57bd3c9d (DEFAULT_TABLE table=default.Table_225f7c2cca034ef79a9956e3d1888c5d [id=2b5f62944d654c47a70bcbb29539cece]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:02.314292 1665 raft_consensus.cc:2749] T 42cfdf88472b44979d86836561005ed1 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:02.315557 885 tablet_service.cc:1505] Processing CreateTablet for tablet 81f936792aae4b699b3122457d7b631d (DEFAULT_TABLE table=default.Table_ac4056c73c0f41cda4580d2736f4c6a5 [id=db1f60d5ef4144dd967b5a86fb40d5f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:02.315587 1039 tablet_service.cc:1505] Processing CreateTablet for tablet 81f936792aae4b699b3122457d7b631d (DEFAULT_TABLE table=default.Table_ac4056c73c0f41cda4580d2736f4c6a5 [id=db1f60d5ef4144dd967b5a86fb40d5f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:02.315778 749 tablet_service.cc:1505] Processing CreateTablet for tablet 81f936792aae4b699b3122457d7b631d (DEFAULT_TABLE table=default.Table_ac4056c73c0f41cda4580d2736f4c6a5 [id=db1f60d5ef4144dd967b5a86fb40d5f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:02.315896 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b7770704e674f68960be3ee57bd3c9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:02.316037 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81f936792aae4b699b3122457d7b631d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:02.316252 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81f936792aae4b699b3122457d7b631d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:02.318158 1174 ts_tablet_manager.cc:1929] T b5d5b737c86844d58c51b2388c0b1705 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:02.320595 1174 log.cc:1199] T b5d5b737c86844d58c51b2388c0b1705 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/b5d5b737c86844d58c51b2388c0b1705
I20251029 02:52:02.324080 1174 ts_tablet_manager.cc:1950] T b5d5b737c86844d58c51b2388c0b1705 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:02.328900 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81f936792aae4b699b3122457d7b631d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:02.328889 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet b5d5b737c86844d58c51b2388c0b1705 (table default.Table_8b39acd343e94119804478e17baaa785 [id=0b43261c80a5422092e96d77c0d2ee9e]) successfully deleted
I20251029 02:52:02.328933 748 tablet_service.cc:1505] Processing CreateTablet for tablet 5e122815a3c5498faf9ae7ffec62c426 (DEFAULT_TABLE table=default.Table_d72ae8772b6044458311fc523dca2c71 [id=412ae118d6c34e389747208f01a85a4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:02.329214 885 tablet_service.cc:1505] Processing CreateTablet for tablet 5e122815a3c5498faf9ae7ffec62c426 (DEFAULT_TABLE table=default.Table_d72ae8772b6044458311fc523dca2c71 [id=412ae118d6c34e389747208f01a85a4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:02.329312 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e122815a3c5498faf9ae7ffec62c426. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:02.329627 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e122815a3c5498faf9ae7ffec62c426. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:02.330025 1176 ts_tablet_manager.cc:1916] T fd900eec3f0c43bf965c148ea90870a3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:02.337459 1167 tablet_bootstrap.cc:492] T 69597d48631d43b6a5b92abbf60d5ed3 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:02.337678 1167 ts_tablet_manager.cc:1403] T 69597d48631d43b6a5b92abbf60d5ed3 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.065s user 0.004s sys 0.000s
I20251029 02:52:02.338739 1167 raft_consensus.cc:359] T 69597d48631d43b6a5b92abbf60d5ed3 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:02.339057 1167 raft_consensus.cc:385] T 69597d48631d43b6a5b92abbf60d5ed3 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:02.339195 1167 raft_consensus.cc:740] T 69597d48631d43b6a5b92abbf60d5ed3 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:02.339393 1172 tablet_bootstrap.cc:492] T cedfb53078d94c81b5e294ce7438325e P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:02.339568 1172 ts_tablet_manager.cc:1403] T cedfb53078d94c81b5e294ce7438325e P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.090s user 0.004s sys 0.000s
I20251029 02:52:02.339543 1167 consensus_queue.cc:260] T 69597d48631d43b6a5b92abbf60d5ed3 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:02.340346 1167 ts_tablet_manager.cc:1434] T 69597d48631d43b6a5b92abbf60d5ed3 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 02:52:02.340353 1172 raft_consensus.cc:359] T cedfb53078d94c81b5e294ce7438325e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:02.340827 1167 tablet_bootstrap.cc:492] T 65fdec7cf5d143208d78d4c3806b1352 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:02.341073 1172 raft_consensus.cc:385] T cedfb53078d94c81b5e294ce7438325e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:02.341190 1172 raft_consensus.cc:740] T cedfb53078d94c81b5e294ce7438325e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:02.341405 1172 consensus_queue.cc:260] T cedfb53078d94c81b5e294ce7438325e P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:02.342026 1172 ts_tablet_manager.cc:1434] T cedfb53078d94c81b5e294ce7438325e P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20251029 02:52:02.342303 1172 tablet_bootstrap.cc:492] T e1fa4980f2624321b617b84c68b90113 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:02.343518 1174 tablet_replica.cc:333] T 75236b2fa82f4f28b82efae9a2c5160d P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:02.344002 1172 tablet_bootstrap.cc:654] T e1fa4980f2624321b617b84c68b90113 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:02.344247 1041 tablet_service.cc:1505] Processing CreateTablet for tablet 5e122815a3c5498faf9ae7ffec62c426 (DEFAULT_TABLE table=default.Table_d72ae8772b6044458311fc523dca2c71 [id=412ae118d6c34e389747208f01a85a4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:02.344710 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e122815a3c5498faf9ae7ffec62c426. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:02.349948 1025 tablet_service.cc:1505] Processing CreateTablet for tablet 48aa20640dfe43288326ed6d6b9bc1cf (DEFAULT_TABLE table=default.Table_a37ba50ba83a43668c7fe1af7cca7f0d [id=6de9a1f24f954613ac4414c3e7848c9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:02.350320 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48aa20640dfe43288326ed6d6b9bc1cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:02.361441 1025 tablet_service.cc:1505] Processing CreateTablet for tablet 62e4ffea74994cb8b0474448b39fd9aa (DEFAULT_TABLE table=default.Table_909da9f5c1374cc2b5c318ffc1c3ac19 [id=a32325cb0f73491b9b6784b043dcc4ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:02.361435 1028 tablet_service.cc:1505] Processing CreateTablet for tablet bc16262db49e4fc69fe1a4f95dbde2cb (DEFAULT_TABLE table=default.Table_be5e479a6ff642dd8b01f195cfa37d8f [id=75ef6bd8e2c146c7b5996cd9ba6e2f18]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:02.361902 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62e4ffea74994cb8b0474448b39fd9aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:02.362134 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc16262db49e4fc69fe1a4f95dbde2cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:02.367044 755 tablet_service.cc:1505] Processing CreateTablet for tablet 62e4ffea74994cb8b0474448b39fd9aa (DEFAULT_TABLE table=default.Table_909da9f5c1374cc2b5c318ffc1c3ac19 [id=a32325cb0f73491b9b6784b043dcc4ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:02.368113 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62e4ffea74994cb8b0474448b39fd9aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:02.373859 742 tablet_service.cc:1505] Processing CreateTablet for tablet 48aa20640dfe43288326ed6d6b9bc1cf (DEFAULT_TABLE table=default.Table_a37ba50ba83a43668c7fe1af7cca7f0d [id=6de9a1f24f954613ac4414c3e7848c9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:02.374313 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48aa20640dfe43288326ed6d6b9bc1cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:02.377804 749 tablet_service.cc:1505] Processing CreateTablet for tablet bc16262db49e4fc69fe1a4f95dbde2cb (DEFAULT_TABLE table=default.Table_be5e479a6ff642dd8b01f195cfa37d8f [id=75ef6bd8e2c146c7b5996cd9ba6e2f18]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:02.379226 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc16262db49e4fc69fe1a4f95dbde2cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:02.380352 1028 tablet_service.cc:1505] Processing CreateTablet for tablet cdb558f14ab34d5ea4b1e9280db3c358 (DEFAULT_TABLE table=default.Table_f17714316e664840899f4d9827cd25fb [id=aea283176c6f49c1b94bc313494d486d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:02.380735 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cdb558f14ab34d5ea4b1e9280db3c358. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:02.382942 756 tablet_service.cc:1505] Processing CreateTablet for tablet cdb558f14ab34d5ea4b1e9280db3c358 (DEFAULT_TABLE table=default.Table_f17714316e664840899f4d9827cd25fb [id=aea283176c6f49c1b94bc313494d486d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:02.386332 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cdb558f14ab34d5ea4b1e9280db3c358. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:02.389123 1032 tablet_service.cc:1505] Processing CreateTablet for tablet 23856678b7aa4f9ebf65fb4564a4bf91 (DEFAULT_TABLE table=default.Table_5ed09727711e4eb6a60deb327d188aea [id=d256ddd14762490f8fbfe401d9802d4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:02.389508 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23856678b7aa4f9ebf65fb4564a4bf91. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:02.393260 1172 tablet_bootstrap.cc:492] T e1fa4980f2624321b617b84c68b90113 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:02.393487 1172 ts_tablet_manager.cc:1403] T e1fa4980f2624321b617b84c68b90113 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.051s user 0.003s sys 0.000s
I20251029 02:52:02.394330 1172 raft_consensus.cc:359] T e1fa4980f2624321b617b84c68b90113 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:02.394606 1172 raft_consensus.cc:385] T e1fa4980f2624321b617b84c68b90113 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:02.394718 1172 raft_consensus.cc:740] T e1fa4980f2624321b617b84c68b90113 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:02.395025 1172 consensus_queue.cc:260] T e1fa4980f2624321b617b84c68b90113 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:02.395686 1172 ts_tablet_manager.cc:1434] T e1fa4980f2624321b617b84c68b90113 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 02:52:02.396106 1172 tablet_bootstrap.cc:492] T 74873a05ef634867a30aa63b31fcfcc7 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:02.397696 1172 tablet_bootstrap.cc:654] T 74873a05ef634867a30aa63b31fcfcc7 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:02.385774 1175 ts_tablet_manager.cc:1929] T 8df97119d0654d7d92bb5bd17c5d8562 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:02.398343 748 tablet_service.cc:1505] Processing CreateTablet for tablet 23856678b7aa4f9ebf65fb4564a4bf91 (DEFAULT_TABLE table=default.Table_5ed09727711e4eb6a60deb327d188aea [id=d256ddd14762490f8fbfe401d9802d4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:02.398746 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23856678b7aa4f9ebf65fb4564a4bf91. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:02.398342 1175 log.cc:1199] T 8df97119d0654d7d92bb5bd17c5d8562 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/8df97119d0654d7d92bb5bd17c5d8562
I20251029 02:52:02.405583 1175 ts_tablet_manager.cc:1950] T 8df97119d0654d7d92bb5bd17c5d8562 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:02.406407 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 513) took 2902 ms. Trace:
I20251029 02:52:02.406558 1175 rpcz_store.cc:276] 1029 02:51:59.504411 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:00.195736 (+691325us) service_pool.cc:225] Handling call
1029 02:52:02.292073 (+2096337us) ts_tablet_manager.cc:1151] Deleting tablet 8df97119d0654d7d92bb5bd17c5d8562
1029 02:52:02.292082 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.378264 (+ 86182us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:02.385643 (+ 7379us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:02.406370 (+ 20727us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2096168}
I20251029 02:52:02.406983 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 514) took 2897 ms. Trace:
I20251029 02:52:02.407079 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 8df97119d0654d7d92bb5bd17c5d8562 (table default.Table_99592a0735024ba7a0d8d6145b61f2e4 [id=ced78d857c314dc7802545a7909bdf46]) successfully deleted
I20251029 02:52:02.407087 1175 rpcz_store.cc:276] 1029 02:51:59.509357 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:00.195929 (+686572us) service_pool.cc:225] Handling call
1029 02:52:02.406837 (+2210908us) ts_tablet_manager.cc:1151] Deleting tablet f65fe9a5137b48669afef51ad7dd76ae
1029 02:52:02.406847 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.406952 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2210779}
I20251029 02:52:02.407461 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 521) took 2380 ms. Trace:
W20251029 02:52:02.407591 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet f65fe9a5137b48669afef51ad7dd76ae because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f65fe9a5137b48669afef51ad7dd76ae already in progress: creating tablet
I20251029 02:52:02.407670 1032 tablet_service.cc:1505] Processing CreateTablet for tablet dd16d1c715604774baba030853f6697c (DEFAULT_TABLE table=default.Table_ba15e59ee62346c7bd87a0bf7b87c9fc [id=b1a856c68c9d45f49cd19c3c246852ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:02.409651 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd16d1c715604774baba030853f6697c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:02.348605 1773 raft_consensus.cc:493] T 44784a3a760946e0be24743bc2fc8138 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:02.407574 1175 rpcz_store.cc:276] 1029 02:52:00.026764 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:00.280344 (+253580us) service_pool.cc:225] Handling call
1029 02:52:02.407322 (+2126978us) ts_tablet_manager.cc:1151] Deleting tablet 7b0f7d6c1c9842789c6ce2cecaa3d5b0
1029 02:52:02.407330 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.407416 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2126719}
I20251029 02:52:02.359357 881 tablet_service.cc:1505] Processing CreateTablet for tablet 48aa20640dfe43288326ed6d6b9bc1cf (DEFAULT_TABLE table=default.Table_a37ba50ba83a43668c7fe1af7cca7f0d [id=6de9a1f24f954613ac4414c3e7848c9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:02.416172 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 522) took 2386 ms. Trace:
W20251029 02:52:02.416915 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 7b0f7d6c1c9842789c6ce2cecaa3d5b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7b0f7d6c1c9842789c6ce2cecaa3d5b0 already in progress: creating tablet
I20251029 02:52:02.360679 883 tablet_service.cc:1505] Processing CreateTablet for tablet 62e4ffea74994cb8b0474448b39fd9aa (DEFAULT_TABLE table=default.Table_909da9f5c1374cc2b5c318ffc1c3ac19 [id=a32325cb0f73491b9b6784b043dcc4ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:02.374662 1167 tablet_bootstrap.cc:654] T 65fdec7cf5d143208d78d4c3806b1352 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:02.377655 899 tablet_service.cc:1505] Processing CreateTablet for tablet bc16262db49e4fc69fe1a4f95dbde2cb (DEFAULT_TABLE table=default.Table_be5e479a6ff642dd8b01f195cfa37d8f [id=75ef6bd8e2c146c7b5996cd9ba6e2f18]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:02.381880 895 tablet_service.cc:1505] Processing CreateTablet for tablet cdb558f14ab34d5ea4b1e9280db3c358 (DEFAULT_TABLE table=default.Table_f17714316e664840899f4d9827cd25fb [id=aea283176c6f49c1b94bc313494d486d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:02.394733 1682 raft_consensus.cc:493] T 69597d48631d43b6a5b92abbf60d5ed3 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:02.398123 891 tablet_service.cc:1505] Processing CreateTablet for tablet 23856678b7aa4f9ebf65fb4564a4bf91 (DEFAULT_TABLE table=default.Table_5ed09727711e4eb6a60deb327d188aea [id=d256ddd14762490f8fbfe401d9802d4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:02.407205 884 tablet_service.cc:1505] Processing CreateTablet for tablet dd16d1c715604774baba030853f6697c (DEFAULT_TABLE table=default.Table_ba15e59ee62346c7bd87a0bf7b87c9fc [id=b1a856c68c9d45f49cd19c3c246852ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:02.413395 1174 raft_consensus.cc:2243] T 75236b2fa82f4f28b82efae9a2c5160d P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:02.418844 1174 pending_rounds.cc:70] T 75236b2fa82f4f28b82efae9a2c5160d P c756528642a543eeacd93cef3e335a7d: Trying to abort 1 pending ops.
I20251029 02:52:02.418905 1174 pending_rounds.cc:77] T 75236b2fa82f4f28b82efae9a2c5160d P c756528642a543eeacd93cef3e335a7d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215949093206843392 op_type: NO_OP noop_request { }
I20251029 02:52:02.414623 1773 raft_consensus.cc:515] T 44784a3a760946e0be24743bc2fc8138 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:02.419039 1174 raft_consensus.cc:2889] T 75236b2fa82f4f28b82efae9a2c5160d P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 02:52:02.419135 1174 raft_consensus.cc:2272] T 75236b2fa82f4f28b82efae9a2c5160d P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:02.419703 1773 leader_election.cc:290] T 44784a3a760946e0be24743bc2fc8138 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:02.420096 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "44784a3a760946e0be24743bc2fc8138" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:52:02.420667 849 leader_election.cc:343] T 44784a3a760946e0be24743bc2fc8138 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
W20251029 02:52:02.420905 847 leader_election.cc:343] T 44784a3a760946e0be24743bc2fc8138 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 44784a3a760946e0be24743bc2fc8138
I20251029 02:52:02.421046 847 leader_election.cc:304] T 44784a3a760946e0be24743bc2fc8138 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9
I20251029 02:52:02.421205 890 tablet_service.cc:1505] Processing CreateTablet for tablet 43a76b984da3414da701221f87349337 (DEFAULT_TABLE table=default.Table_839c3ffd7768463da17f35d024d016a0 [id=67503f4e27bd4cdfbb9b0b95706f3c4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:02.416487 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48aa20640dfe43288326ed6d6b9bc1cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:02.421921 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cdb558f14ab34d5ea4b1e9280db3c358. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:02.418267 1682 raft_consensus.cc:515] T 69597d48631d43b6a5b92abbf60d5ed3 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:02.422962 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "69597d48631d43b6a5b92abbf60d5ed3" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:02.423101 892 tablet_service.cc:1505] Processing CreateTablet for tablet 4e82c574dfa14ffbabf162a17355f841 (DEFAULT_TABLE table=default.Table_244839a45b634a3b86f305b2c1e337c6 [id=f34eb59cf65c45938cf002078006ace2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:02.423168 1060 raft_consensus.cc:2468] T 69597d48631d43b6a5b92abbf60d5ed3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:02.423300 751 tablet_service.cc:1505] Processing CreateTablet for tablet b4621bf9e6324eaeaa64a6ca51cd5ca9 (DEFAULT_TABLE table=default.Table_39a7937d49ce4ee290f72ed5295a2a3e [id=1571e82f4f9948089d81b0450dbfdee3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:02.423523 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e82c574dfa14ffbabf162a17355f841. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:02.423609 849 leader_election.cc:304] T 69597d48631d43b6a5b92abbf60d5ed3 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:02.423672 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4621bf9e6324eaeaa64a6ca51cd5ca9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:02.426640 1774 raft_consensus.cc:2749] T 44784a3a760946e0be24743bc2fc8138 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:02.426714 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc16262db49e4fc69fe1a4f95dbde2cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:02.426846 1774 raft_consensus.cc:2804] T 69597d48631d43b6a5b92abbf60d5ed3 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:02.426945 1774 raft_consensus.cc:493] T 69597d48631d43b6a5b92abbf60d5ed3 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:02.427037 1774 raft_consensus.cc:3060] T 69597d48631d43b6a5b92abbf60d5ed3 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:02.429152 1166 tablet_bootstrap.cc:492] T 65fdec7cf5d143208d78d4c3806b1352 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:02.429287 1166 ts_tablet_manager.cc:1403] T 65fdec7cf5d143208d78d4c3806b1352 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.226s user 0.005s sys 0.002s
I20251029 02:52:02.429965 1774 raft_consensus.cc:515] T 69597d48631d43b6a5b92abbf60d5ed3 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:02.430174 1166 raft_consensus.cc:359] T 65fdec7cf5d143208d78d4c3806b1352 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:02.430321 1166 raft_consensus.cc:385] T 65fdec7cf5d143208d78d4c3806b1352 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:02.430398 1166 raft_consensus.cc:740] T 65fdec7cf5d143208d78d4c3806b1352 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:02.430734 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd16d1c715604774baba030853f6697c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:02.430963 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62e4ffea74994cb8b0474448b39fd9aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:02.431058 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "69597d48631d43b6a5b92abbf60d5ed3" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:52:02.431231 1060 raft_consensus.cc:3060] T 69597d48631d43b6a5b92abbf60d5ed3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:02.432374 742 tablet_service.cc:1505] Processing CreateTablet for tablet dd16d1c715604774baba030853f6697c (DEFAULT_TABLE table=default.Table_ba15e59ee62346c7bd87a0bf7b87c9fc [id=b1a856c68c9d45f49cd19c3c246852ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:02.432786 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd16d1c715604774baba030853f6697c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:02.434186 1060 raft_consensus.cc:2468] T 69597d48631d43b6a5b92abbf60d5ed3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:02.434680 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23856678b7aa4f9ebf65fb4564a4bf91. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:02.434633 849 leader_election.cc:304] T 69597d48631d43b6a5b92abbf60d5ed3 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:02.434926 1682 leader_election.cc:290] T 69597d48631d43b6a5b92abbf60d5ed3 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:02.435079 1682 raft_consensus.cc:2804] T 69597d48631d43b6a5b92abbf60d5ed3 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:02.435060 1166 consensus_queue.cc:260] T 65fdec7cf5d143208d78d4c3806b1352 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:02.435204 1682 raft_consensus.cc:697] T 69597d48631d43b6a5b92abbf60d5ed3 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:52:02.435480 1682 consensus_queue.cc:237] T 69597d48631d43b6a5b92abbf60d5ed3 P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:02.435770 1166 ts_tablet_manager.cc:1434] T 65fdec7cf5d143208d78d4c3806b1352 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.006s user 0.001s sys 0.002s
I20251029 02:52:02.436065 1166 tablet_bootstrap.cc:492] T fd0f89180aba43eabb20f926abccbaa6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:02.436673 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43a76b984da3414da701221f87349337. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:02.438419 1166 tablet_bootstrap.cc:654] T fd0f89180aba43eabb20f926abccbaa6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
W20251029 02:52:02.439646 847 leader_election.cc:343] T 69597d48631d43b6a5b92abbf60d5ed3 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 69597d48631d43b6a5b92abbf60d5ed3
I20251029 02:52:02.441217 748 tablet_service.cc:1505] Processing CreateTablet for tablet 4e82c574dfa14ffbabf162a17355f841 (DEFAULT_TABLE table=default.Table_244839a45b634a3b86f305b2c1e337c6 [id=f34eb59cf65c45938cf002078006ace2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:02.441684 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e82c574dfa14ffbabf162a17355f841. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:02.442864 1041 tablet_service.cc:1505] Processing CreateTablet for tablet 4e82c574dfa14ffbabf162a17355f841 (DEFAULT_TABLE table=default.Table_244839a45b634a3b86f305b2c1e337c6 [id=f34eb59cf65c45938cf002078006ace2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:02.443722 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e82c574dfa14ffbabf162a17355f841. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:02.446864 1032 tablet_service.cc:1505] Processing CreateTablet for tablet b4621bf9e6324eaeaa64a6ca51cd5ca9 (DEFAULT_TABLE table=default.Table_39a7937d49ce4ee290f72ed5295a2a3e [id=1571e82f4f9948089d81b0450dbfdee3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:02.447470 879 tablet_service.cc:1505] Processing CreateTablet for tablet b4621bf9e6324eaeaa64a6ca51cd5ca9 (DEFAULT_TABLE table=default.Table_39a7937d49ce4ee290f72ed5295a2a3e [id=1571e82f4f9948089d81b0450dbfdee3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:02.449342 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4621bf9e6324eaeaa64a6ca51cd5ca9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:02.451131 756 tablet_service.cc:1505] Processing CreateTablet for tablet 43a76b984da3414da701221f87349337 (DEFAULT_TABLE table=default.Table_839c3ffd7768463da17f35d024d016a0 [id=67503f4e27bd4cdfbb9b0b95706f3c4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:02.452000 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b4621bf9e6324eaeaa64a6ca51cd5ca9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:02.453091 1176 ts_tablet_manager.cc:1929] T fd900eec3f0c43bf965c148ea90870a3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:02.454734 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43a76b984da3414da701221f87349337. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:02.457739 1172 tablet_bootstrap.cc:492] T 74873a05ef634867a30aa63b31fcfcc7 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:02.457875 1172 ts_tablet_manager.cc:1403] T 74873a05ef634867a30aa63b31fcfcc7 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.062s user 0.003s sys 0.000s
I20251029 02:52:02.458635 1172 raft_consensus.cc:359] T 74873a05ef634867a30aa63b31fcfcc7 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:02.458823 1172 raft_consensus.cc:385] T 74873a05ef634867a30aa63b31fcfcc7 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:02.458885 1172 raft_consensus.cc:740] T 74873a05ef634867a30aa63b31fcfcc7 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:02.459090 1172 consensus_queue.cc:260] T 74873a05ef634867a30aa63b31fcfcc7 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:02.459589 1172 ts_tablet_manager.cc:1434] T 74873a05ef634867a30aa63b31fcfcc7 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251029 02:52:02.459841 1172 tablet_bootstrap.cc:492] T bfec87f690a640c39c58c64e16cc67e7 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:02.461342 1172 tablet_bootstrap.cc:654] T bfec87f690a640c39c58c64e16cc67e7 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
W20251029 02:52:02.462659 847 leader_election.cc:343] T 69597d48631d43b6a5b92abbf60d5ed3 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 69597d48631d43b6a5b92abbf60d5ed3
I20251029 02:52:02.470113 1774 leader_election.cc:290] T 69597d48631d43b6a5b92abbf60d5ed3 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:02.489634 1176 log.cc:1199] T fd900eec3f0c43bf965c148ea90870a3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/fd900eec3f0c43bf965c148ea90870a3
I20251029 02:52:02.490238 1176 ts_tablet_manager.cc:1950] T fd900eec3f0c43bf965c148ea90870a3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:02.490681 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 563) took 1892 ms. Trace:
I20251029 02:52:02.491003 1176 rpcz_store.cc:276] 1029 02:52:00.598350 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:00.909436 (+311086us) service_pool.cc:225] Handling call
1029 02:52:02.283826 (+1374390us) ts_tablet_manager.cc:1151] Deleting tablet fd900eec3f0c43bf965c148ea90870a3
1029 02:52:02.283861 (+ 35us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.405339 (+121478us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:02.452927 (+ 47588us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:02.490647 (+ 37720us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1353407}
I20251029 02:52:02.491690 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet fd900eec3f0c43bf965c148ea90870a3 (table default.Table_e0e6e47b18d749e8ae80f822af1e23f0 [id=125ae5c7b5074378bcecb546e70f0c13]) successfully deleted
I20251029 02:52:02.492205 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 577) took 1520 ms. Trace:
I20251029 02:52:02.492584 1514 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:42490 (request call id 2) took 1443 ms. Trace:
I20251029 02:52:02.493748 1174 ts_tablet_manager.cc:1916] T 75236b2fa82f4f28b82efae9a2c5160d P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:02.493892 1176 rpcz_store.cc:276] 1029 02:52:00.972151 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:01.245840 (+273689us) service_pool.cc:225] Handling call
1029 02:52:02.492018 (+1246178us) ts_tablet_manager.cc:1151] Deleting tablet d31e548feae24b739087c1e211d514f9
1029 02:52:02.492029 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.492173 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1245961}
W20251029 02:52:02.510658 1485 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.1:40633 (request call id 23) took 1330 ms (client timeout 1000 ms). Trace:
I20251029 02:52:02.510999 1778 raft_consensus.cc:493] T 00000000000000000000000000000000 P e5e5222b26fe4145948620e51edf53fb [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e9261eb85ea443d5a3beb36bd0c549f0)
W20251029 02:52:02.514245 1491 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.2:51269 (request call id 38) took 1346 ms (client timeout 1000 ms). Trace:
I20251029 02:52:02.515100 1172 tablet_bootstrap.cc:492] T bfec87f690a640c39c58c64e16cc67e7 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:02.515245 1172 ts_tablet_manager.cc:1403] T bfec87f690a640c39c58c64e16cc67e7 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.055s user 0.011s sys 0.001s
I20251029 02:52:02.515975 1172 raft_consensus.cc:359] T bfec87f690a640c39c58c64e16cc67e7 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:02.516139 1172 raft_consensus.cc:385] T bfec87f690a640c39c58c64e16cc67e7 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251029 02:52:02.513289 1485 rpcz_store.cc:269] 1029 02:52:01.163121 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:01.163184 (+ 63us) service_pool.cc:225] Handling call
1029 02:52:02.492816 (+1329632us) spinlock_profiling.cc:243] Waited 1.33 s on lock 0x616000040c54. stack: 00007fc743bbc9f6 00007fc743bbc7b0 00007fc742b983ad 0000000000983f04 0000000000983ecd 00007fc752ec36bd 00007fc752ec37b3 00007fc74fa8a441 00007fc7577449dd 00007fc7578cbaa8 00007fc74e3deca6 00007fc74e3de9d5 00007fc74ab5b27e 00007fc74ab5a15e 00007fc74ab60395 00007fc74ab60cbf
1029 02:52:02.493573 (+ 757us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3055747968}
I20251029 02:52:02.516204 1172 raft_consensus.cc:740] T bfec87f690a640c39c58c64e16cc67e7 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
W20251029 02:52:02.516546 1489 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.3:43333 (request call id 23) took 1383 ms (client timeout 1000 ms). Trace:
I20251029 02:52:02.516544 1172 consensus_queue.cc:260] T bfec87f690a640c39c58c64e16cc67e7 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
W20251029 02:52:02.516665 1489 rpcz_store.cc:269] 1029 02:52:01.132169 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:01.132234 (+ 65us) service_pool.cc:225] Handling call
1029 02:52:02.512893 (+1380659us) spinlock_profiling.cc:243] Waited 1.36 s on lock 0x616000040c54. stack: 00007fc743bbc9f6 00007fc743bbc7b0 00007fc742b983ad 0000000000983f04 0000000000983ecd 00007fc752ec36bd 00007fc752ec37b3 00007fc74fa8a441 00007fc7577449dd 00007fc7578cbaa8 00007fc74e3deca6 00007fc74e3de9d5 00007fc74ab5b27e 00007fc74ab5a15e 00007fc74ab60395 00007fc74ab60cbf
1029 02:52:02.516059 (+ 3166us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3130235136}
I20251029 02:52:02.517141 1172 ts_tablet_manager.cc:1434] T bfec87f690a640c39c58c64e16cc67e7 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
W20251029 02:52:02.515733 1491 rpcz_store.cc:269] 1029 02:52:01.167069 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:01.167131 (+ 62us) service_pool.cc:225] Handling call
1029 02:52:02.511996 (+1344865us) spinlock_profiling.cc:243] Waited 1.33 s on lock 0x616000040c54. stack: 00007fc743bbc9f6 00007fc743bbc7b0 00007fc742b983ad 0000000000983f04 0000000000983ecd 00007fc752ec36bd 00007fc752ec37b3 00007fc74fa8a441 00007fc7577449dd 00007fc7578cbaa8 00007fc74e3deca6 00007fc74e3de9d5 00007fc74ab5b27e 00007fc74ab5a15e 00007fc74ab60395 00007fc74ab60cbf
1029 02:52:02.513868 (+ 1872us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3050166016}
I20251029 02:52:02.517432 1172 tablet_bootstrap.cc:492] T da2a21c3236a4bd99872b89b12a36508 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:02.512638 1778 raft_consensus.cc:515] T 00000000000000000000000000000000 P e5e5222b26fe4145948620e51edf53fb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9672114a6a742e2864c16ff60ba246e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42343 } } peers { permanent_uuid: "e9261eb85ea443d5a3beb36bd0c549f0" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34239 } } peers { permanent_uuid: "e5e5222b26fe4145948620e51edf53fb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41233 } }
I20251029 02:52:02.499356 1037 tablet_service.cc:1505] Processing CreateTablet for tablet 43a76b984da3414da701221f87349337 (DEFAULT_TABLE table=default.Table_839c3ffd7768463da17f35d024d016a0 [id=67503f4e27bd4cdfbb9b0b95706f3c4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:02.519186 1166 tablet_bootstrap.cc:492] T fd0f89180aba43eabb20f926abccbaa6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:02.521157 1166 ts_tablet_manager.cc:1403] T fd0f89180aba43eabb20f926abccbaa6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.085s user 0.003s sys 0.001s
I20251029 02:52:02.521287 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43a76b984da3414da701221f87349337. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:02.522138 1166 raft_consensus.cc:359] T fd0f89180aba43eabb20f926abccbaa6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:02.522432 1166 raft_consensus.cc:385] T fd0f89180aba43eabb20f926abccbaa6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:02.522595 1166 raft_consensus.cc:740] T fd0f89180aba43eabb20f926abccbaa6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:02.522948 1166 consensus_queue.cc:260] T fd0f89180aba43eabb20f926abccbaa6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:02.523630 1166 ts_tablet_manager.cc:1434] T fd0f89180aba43eabb20f926abccbaa6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 02:52:02.523994 1166 tablet_bootstrap.cc:492] T 44784a3a760946e0be24743bc2fc8138 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:02.524058 1172 tablet_bootstrap.cc:654] T da2a21c3236a4bd99872b89b12a36508 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:02.526198 1166 tablet_bootstrap.cc:654] T 44784a3a760946e0be24743bc2fc8138 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:02.527529 1167 tablet_bootstrap.cc:492] T 65fdec7cf5d143208d78d4c3806b1352 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
W20251029 02:52:02.527489 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet d31e548feae24b739087c1e211d514f9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d31e548feae24b739087c1e211d514f9 already in progress: creating tablet
I20251029 02:52:02.527943 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 576) took 1560 ms. Trace:
I20251029 02:52:02.528131 1176 rpcz_store.cc:276] 1029 02:52:00.967183 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:01.241970 (+274787us) service_pool.cc:225] Handling call
1029 02:52:02.527793 (+1285823us) ts_tablet_manager.cc:1151] Deleting tablet 2089b058d0e94d99b40ad2718518353e
1029 02:52:02.527802 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.527910 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1278536}
W20251029 02:52:02.528709 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 2089b058d0e94d99b40ad2718518353e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2089b058d0e94d99b40ad2718518353e already in progress: creating tablet
I20251029 02:52:02.529119 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 579) took 1544 ms. Trace:
I20251029 02:52:02.529261 1176 rpcz_store.cc:276] 1029 02:52:00.984658 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:01.249281 (+264623us) service_pool.cc:225] Handling call
1029 02:52:02.528985 (+1279704us) ts_tablet_manager.cc:1151] Deleting tablet 2b278db5b2b04344b864797055a5f665
1029 02:52:02.528994 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.529087 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1279516}
W20251029 02:52:02.530282 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 2b278db5b2b04344b864797055a5f665 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2b278db5b2b04344b864797055a5f665 already in progress: creating tablet
I20251029 02:52:02.531016 1174 ts_tablet_manager.cc:1929] T 75236b2fa82f4f28b82efae9a2c5160d P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:02.531167 1167 ts_tablet_manager.cc:1403] T 65fdec7cf5d143208d78d4c3806b1352 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.190s user 0.006s sys 0.014s
I20251029 02:52:02.532063 1167 raft_consensus.cc:359] T 65fdec7cf5d143208d78d4c3806b1352 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:02.534011 1175 rpcz_store.cc:276] 1029 02:52:00.029871 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:00.280620 (+250749us) service_pool.cc:225] Handling call
1029 02:52:02.416010 (+2135390us) ts_tablet_manager.cc:1151] Deleting tablet e7b7de85c13b4ddc9c995ed2a2ccd921
1029 02:52:02.416018 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.416140 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2119787}
W20251029 02:52:02.534735 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet e7b7de85c13b4ddc9c995ed2a2ccd921 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e7b7de85c13b4ddc9c995ed2a2ccd921 already in progress: creating tablet
I20251029 02:52:02.535434 1175 tablet_replica.cc:333] T 8afc01d52d784f2fa559cab7ce07c8be P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:02.535674 1175 raft_consensus.cc:2243] T 8afc01d52d784f2fa559cab7ce07c8be P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:02.535908 1175 raft_consensus.cc:2272] T 8afc01d52d784f2fa559cab7ce07c8be P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:02.536270 1172 tablet_bootstrap.cc:492] T da2a21c3236a4bd99872b89b12a36508 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:02.536401 1172 ts_tablet_manager.cc:1403] T da2a21c3236a4bd99872b89b12a36508 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.004s
I20251029 02:52:02.536566 1175 ts_tablet_manager.cc:1916] T 8afc01d52d784f2fa559cab7ce07c8be P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:02.537125 1172 raft_consensus.cc:359] T da2a21c3236a4bd99872b89b12a36508 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:02.537279 1172 raft_consensus.cc:385] T da2a21c3236a4bd99872b89b12a36508 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:02.537330 1172 raft_consensus.cc:740] T da2a21c3236a4bd99872b89b12a36508 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:02.537505 1172 consensus_queue.cc:260] T da2a21c3236a4bd99872b89b12a36508 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:02.538079 1172 ts_tablet_manager.cc:1434] T da2a21c3236a4bd99872b89b12a36508 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251029 02:52:02.538333 1172 tablet_bootstrap.cc:492] T 6bd3fac2502e479aa9f4ceaabbb39504 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:02.538961 536 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e5e5222b26fe4145948620e51edf53fb" candidate_term: 3 candidate_status { last_received { term: 2 index: 786 } } ignore_live_leader: false dest_uuid: "e9261eb85ea443d5a3beb36bd0c549f0" is_pre_election: true
I20251029 02:52:02.531219 1174 log.cc:1199] T 75236b2fa82f4f28b82efae9a2c5160d P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/75236b2fa82f4f28b82efae9a2c5160d
I20251029 02:52:02.539558 1174 ts_tablet_manager.cc:1950] T 75236b2fa82f4f28b82efae9a2c5160d P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:02.539882 1172 tablet_bootstrap.cc:654] T 6bd3fac2502e479aa9f4ceaabbb39504 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:02.540441 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 576) took 1039 ms. Trace:
I20251029 02:52:02.540553 1174 rpcz_store.cc:276] 1029 02:52:01.501179 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:01.501221 (+ 42us) service_pool.cc:225] Handling call
1029 02:52:02.343434 (+842213us) ts_tablet_manager.cc:1151] Deleting tablet 75236b2fa82f4f28b82efae9a2c5160d
1029 02:52:02.343444 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.509491 (+166047us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:02.530870 (+ 21379us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:02.540385 (+ 9515us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":842036}
I20251029 02:52:02.541160 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 75236b2fa82f4f28b82efae9a2c5160d (table default.Table_0b84a2f666074fd18085ff4ce249a2fe [id=715a1df4e483412da9a63eca19b8f02d]) successfully deleted
W20251029 02:52:02.541913 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet f2d47238761e4ed49c88abdbb677b18a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f2d47238761e4ed49c88abdbb677b18a already in progress: creating tablet
W20251029 02:52:02.542563 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 7d1408dc544944669de519a1553d9bb7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7d1408dc544944669de519a1553d9bb7 already in progress: creating tablet
W20251029 02:52:02.543216 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 6142018e44af4d32818adefa2ee43d02 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6142018e44af4d32818adefa2ee43d02 already in progress: creating tablet
W20251029 02:52:02.543784 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 217144957930495aa3e9dfcfde268e71 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 217144957930495aa3e9dfcfde268e71 already in progress: creating tablet
W20251029 02:52:02.544409 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet b3de6670357c4ae59eb77469b2fb434c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b3de6670357c4ae59eb77469b2fb434c already in progress: creating tablet
W20251029 02:52:02.545001 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet d041e87ceb11441983d0e38b7929c3a4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d041e87ceb11441983d0e38b7929c3a4 already in progress: creating tablet
W20251029 02:52:02.545578 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet c64d6761ef924ccaa9c2579f8ba47e33 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c64d6761ef924ccaa9c2579f8ba47e33 already in progress: creating tablet
W20251029 02:52:02.546177 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 8f1d5847265b4bb0b29b5e513c35b9bb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8f1d5847265b4bb0b29b5e513c35b9bb already in progress: creating tablet
I20251029 02:52:02.546746 1175 ts_tablet_manager.cc:1929] T 8afc01d52d784f2fa559cab7ce07c8be P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20251029 02:52:02.546789 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 5ef87f7a3dff4f00a0e7273eaec98dff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5ef87f7a3dff4f00a0e7273eaec98dff already in progress: creating tablet
W20251029 02:52:02.547314 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet e3000183230e4350b47f5085952f2db6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e3000183230e4350b47f5085952f2db6 already in progress: creating tablet
W20251029 02:52:02.547875 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet e65eaaae43ac435ebd665546de9d68dd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e65eaaae43ac435ebd665546de9d68dd already in progress: creating tablet
I20251029 02:52:02.548208 1174 tablet_replica.cc:333] T 9964ad774890422f8e14f6b99f28cb55 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:02.548434 1174 raft_consensus.cc:2243] T 9964ad774890422f8e14f6b99f28cb55 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:02.548712 1174 raft_consensus.cc:2272] T 9964ad774890422f8e14f6b99f28cb55 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:02.549310 1174 ts_tablet_manager.cc:1916] T 9964ad774890422f8e14f6b99f28cb55 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:02.551173 1167 raft_consensus.cc:385] T 65fdec7cf5d143208d78d4c3806b1352 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:02.551343 1175 log.cc:1199] T 8afc01d52d784f2fa559cab7ce07c8be P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/8afc01d52d784f2fa559cab7ce07c8be
I20251029 02:52:02.551694 1175 ts_tablet_manager.cc:1950] T 8afc01d52d784f2fa559cab7ce07c8be P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:02.551901 1172 tablet_bootstrap.cc:492] T 6bd3fac2502e479aa9f4ceaabbb39504 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:02.552032 1172 ts_tablet_manager.cc:1403] T 6bd3fac2502e479aa9f4ceaabbb39504 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20251029 02:52:02.552524 1665 raft_consensus.cc:493] T e1fa4980f2624321b617b84c68b90113 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:02.552708 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 524) took 2521 ms. Trace:
I20251029 02:52:02.552661 1665 raft_consensus.cc:515] T e1fa4980f2624321b617b84c68b90113 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:02.552768 1175 rpcz_store.cc:276] 1029 02:52:00.031216 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:00.296246 (+265030us) service_pool.cc:225] Handling call
1029 02:52:02.535351 (+2239105us) ts_tablet_manager.cc:1151] Deleting tablet 8afc01d52d784f2fa559cab7ce07c8be
1029 02:52:02.535364 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.543630 (+ 8266us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:02.546657 (+ 3027us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:02.552691 (+ 6034us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2238846}
I20251029 02:52:02.553221 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 8afc01d52d784f2fa559cab7ce07c8be (table default.Table_2a10c9dfc91d4974845dbcd500d1d8be [id=ed7295f3f21848b79af75785d3983b67]) successfully deleted
I20251029 02:52:02.553256 1665 leader_election.cc:290] T e1fa4980f2624321b617b84c68b90113 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:02.554050 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1fa4980f2624321b617b84c68b90113" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:02.554100 918 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1fa4980f2624321b617b84c68b90113" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:02.554322 1167 raft_consensus.cc:740] T 65fdec7cf5d143208d78d4c3806b1352 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:02.554291 1172 raft_consensus.cc:359] T 6bd3fac2502e479aa9f4ceaabbb39504 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:02.554417 1172 raft_consensus.cc:385] T 6bd3fac2502e479aa9f4ceaabbb39504 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:02.554488 1172 raft_consensus.cc:740] T 6bd3fac2502e479aa9f4ceaabbb39504 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
W20251029 02:52:02.554659 711 leader_election.cc:343] T e1fa4980f2624321b617b84c68b90113 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:02.554720 1172 consensus_queue.cc:260] T 6bd3fac2502e479aa9f4ceaabbb39504 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:02.555128 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 525) took 2523 ms. Trace:
I20251029 02:52:02.555282 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 582) took 1570 ms. Trace:
I20251029 02:52:02.555404 1176 rpcz_store.cc:276] 1029 02:52:00.984939 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:01.262479 (+277540us) service_pool.cc:225] Handling call
1029 02:52:02.555134 (+1292655us) ts_tablet_manager.cc:1151] Deleting tablet 44ae2b07edca44afbe1b1fb3a5b1115d
1029 02:52:02.555143 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.555248 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1292408}
W20251029 02:52:02.555109 712 leader_election.cc:343] T e1fa4980f2624321b617b84c68b90113 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:02.555460 1172 ts_tablet_manager.cc:1434] T 6bd3fac2502e479aa9f4ceaabbb39504 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251029 02:52:02.555697 712 leader_election.cc:304] T e1fa4980f2624321b617b84c68b90113 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
W20251029 02:52:02.555846 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 44ae2b07edca44afbe1b1fb3a5b1115d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 44ae2b07edca44afbe1b1fb3a5b1115d already in progress: creating tablet
I20251029 02:52:02.555665 1175 rpcz_store.cc:276] 1029 02:52:00.031302 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:00.296515 (+265213us) service_pool.cc:225] Handling call
1029 02:52:02.554998 (+2258483us) ts_tablet_manager.cc:1151] Deleting tablet 7bb85a12f573494cbb3118d1947eda27
1029 02:52:02.555009 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.555113 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2257586}
I20251029 02:52:02.556238 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 581) took 1571 ms. Trace:
I20251029 02:52:02.556210 1167 consensus_queue.cc:260] T 65fdec7cf5d143208d78d4c3806b1352 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:02.556355 1176 rpcz_store.cc:276] 1029 02:52:00.984855 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:01.250940 (+266085us) service_pool.cc:225] Handling call
1029 02:52:02.556097 (+1305157us) ts_tablet_manager.cc:1151] Deleting tablet d29cc2a0c8b84e3ebadc7f571d666847
1029 02:52:02.556106 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.556198 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1290651}
W20251029 02:52:02.556778 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet d29cc2a0c8b84e3ebadc7f571d666847 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d29cc2a0c8b84e3ebadc7f571d666847 already in progress: creating tablet
I20251029 02:52:02.557080 1176 tablet_replica.cc:333] T 24016ceaa84147eca0b94b3ad288e772 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:02.557329 1176 raft_consensus.cc:2243] T 24016ceaa84147eca0b94b3ad288e772 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:02.557716 1176 raft_consensus.cc:2272] T 24016ceaa84147eca0b94b3ad288e772 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:02.558341 1176 ts_tablet_manager.cc:1916] T 24016ceaa84147eca0b94b3ad288e772 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:02.556090 1665 raft_consensus.cc:2749] T e1fa4980f2624321b617b84c68b90113 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:02.558550 1167 ts_tablet_manager.cc:1434] T 65fdec7cf5d143208d78d4c3806b1352 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.027s user 0.002s sys 0.000s
I20251029 02:52:02.556357 1172 tablet_bootstrap.cc:492] T f187dbd8a55b45b292ea61db43caf47d P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:02.558925 1167 tablet_bootstrap.cc:492] T 5695b5a7b1854c3983e2d4ab684898b0 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:02.560380 1172 tablet_bootstrap.cc:654] T f187dbd8a55b45b292ea61db43caf47d P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
W20251029 02:52:02.561452 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 7bb85a12f573494cbb3118d1947eda27 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7bb85a12f573494cbb3118d1947eda27 already in progress: creating tablet
I20251029 02:52:02.561759 1175 tablet_replica.cc:333] T 2242f5b2c20346d9b753d7cd29242441 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:02.561951 1175 raft_consensus.cc:2243] T 2242f5b2c20346d9b753d7cd29242441 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:02.562106 1175 raft_consensus.cc:2272] T 2242f5b2c20346d9b753d7cd29242441 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:02.562736 1175 ts_tablet_manager.cc:1916] T 2242f5b2c20346d9b753d7cd29242441 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:02.563576 1167 tablet_bootstrap.cc:654] T 5695b5a7b1854c3983e2d4ab684898b0 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:02.539561 1778 leader_election.cc:290] T 00000000000000000000000000000000 P e5e5222b26fe4145948620e51edf53fb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e9672114a6a742e2864c16ff60ba246e (127.0.104.62:42343), e9261eb85ea443d5a3beb36bd0c549f0 (127.0.104.61:34239)
I20251029 02:52:02.567376 1174 ts_tablet_manager.cc:1929] T 9964ad774890422f8e14f6b99f28cb55 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:02.579713 1172 tablet_bootstrap.cc:492] T f187dbd8a55b45b292ea61db43caf47d P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:02.579869 1172 ts_tablet_manager.cc:1403] T f187dbd8a55b45b292ea61db43caf47d P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.001s
I20251029 02:52:02.580613 1172 raft_consensus.cc:359] T f187dbd8a55b45b292ea61db43caf47d P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:02.580777 1172 raft_consensus.cc:385] T f187dbd8a55b45b292ea61db43caf47d P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:02.580837 1172 raft_consensus.cc:740] T f187dbd8a55b45b292ea61db43caf47d P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:02.581073 1172 consensus_queue.cc:260] T f187dbd8a55b45b292ea61db43caf47d P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:02.581650 1172 ts_tablet_manager.cc:1434] T f187dbd8a55b45b292ea61db43caf47d P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 02:52:02.582412 1166 tablet_bootstrap.cc:492] T 44784a3a760946e0be24743bc2fc8138 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:02.582753 1172 tablet_bootstrap.cc:492] T f246f6c202cf44d48c9f5e4f335202c5 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:02.583775 1166 ts_tablet_manager.cc:1403] T 44784a3a760946e0be24743bc2fc8138 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.060s user 0.004s sys 0.000s
I20251029 02:52:02.584662 1166 raft_consensus.cc:359] T 44784a3a760946e0be24743bc2fc8138 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:02.584990 1166 raft_consensus.cc:385] T 44784a3a760946e0be24743bc2fc8138 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:02.585152 1166 raft_consensus.cc:740] T 44784a3a760946e0be24743bc2fc8138 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:02.585196 1172 tablet_bootstrap.cc:654] T f246f6c202cf44d48c9f5e4f335202c5 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:02.585489 1166 consensus_queue.cc:260] T 44784a3a760946e0be24743bc2fc8138 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:02.586226 1166 ts_tablet_manager.cc:1434] T 44784a3a760946e0be24743bc2fc8138 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 02:52:02.587776 1175 ts_tablet_manager.cc:1929] T 2242f5b2c20346d9b753d7cd29242441 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:02.588153 1166 tablet_bootstrap.cc:492] T 0a47781d036149569e1164dd938a45e9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:02.587894 1175 log.cc:1199] T 2242f5b2c20346d9b753d7cd29242441 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/2242f5b2c20346d9b753d7cd29242441
I20251029 02:52:02.589360 1175 ts_tablet_manager.cc:1950] T 2242f5b2c20346d9b753d7cd29242441 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:02.590070 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 526) took 2500 ms. Trace:
I20251029 02:52:02.590160 1175 rpcz_store.cc:276] 1029 02:52:00.089220 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:00.296710 (+207490us) service_pool.cc:225] Handling call
1029 02:52:02.561714 (+2265004us) ts_tablet_manager.cc:1151] Deleting tablet 2242f5b2c20346d9b753d7cd29242441
1029 02:52:02.561724 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.565873 (+ 4149us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:02.587672 (+ 21799us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:02.590051 (+ 2379us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":265088,"tablet-delete.queue_time_us":2264818}
I20251029 02:52:02.590626 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 520) took 2567 ms. Trace:
I20251029 02:52:02.590713 1175 rpcz_store.cc:276] 1029 02:52:00.023453 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:00.274113 (+250660us) service_pool.cc:225] Handling call
1029 02:52:02.590441 (+2316328us) ts_tablet_manager.cc:1151] Deleting tablet 294afb9b012c40c0b05b73795d6f0ac8
1029 02:52:02.590451 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.590601 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2292227}
I20251029 02:52:02.590804 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 2242f5b2c20346d9b753d7cd29242441 (table default.Table_8a5ae35851664e0f85324ec1ca672031 [id=a0e161a5d9d94a3b936669e6dfe6fb4f]) successfully deleted
I20251029 02:52:02.591033 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 523) took 2559 ms. Trace:
I20251029 02:52:02.591099 1175 rpcz_store.cc:276] 1029 02:52:00.031102 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:00.286893 (+255791us) service_pool.cc:225] Handling call
1029 02:52:02.590938 (+2304045us) ts_tablet_manager.cc:1151] Deleting tablet 44c0488eba0543078ff4f0b3d69366f2
1029 02:52:02.590946 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.591017 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2284573}
I20251029 02:52:02.591326 1175 tablet_replica.cc:333] T 7309d3190cfa415281c5253a634fa7f3 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:02.591493 1175 raft_consensus.cc:2243] T 7309d3190cfa415281c5253a634fa7f3 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:02.591641 1175 raft_consensus.cc:2272] T 7309d3190cfa415281c5253a634fa7f3 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251029 02:52:02.591640 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 294afb9b012c40c0b05b73795d6f0ac8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 294afb9b012c40c0b05b73795d6f0ac8 already in progress: creating tablet
W20251029 02:52:02.591893 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 44c0488eba0543078ff4f0b3d69366f2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 44c0488eba0543078ff4f0b3d69366f2 already in progress: creating tablet
I20251029 02:52:02.592154 1175 ts_tablet_manager.cc:1916] T 7309d3190cfa415281c5253a634fa7f3 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:02.594161 1166 tablet_bootstrap.cc:654] T 0a47781d036149569e1164dd938a45e9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:02.602792 1176 ts_tablet_manager.cc:1929] T 24016ceaa84147eca0b94b3ad288e772 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251029 02:52:02.603330 1176 log.cc:1199] T 24016ceaa84147eca0b94b3ad288e772 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/24016ceaa84147eca0b94b3ad288e772
I20251029 02:52:02.603855 1176 ts_tablet_manager.cc:1950] T 24016ceaa84147eca0b94b3ad288e772 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:02.607402 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 583) took 1051 ms. Trace:
I20251029 02:52:02.607859 1176 rpcz_store.cc:276] 1029 02:52:01.555616 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:01.556289 (+ 673us) service_pool.cc:225] Handling call
1029 02:52:02.557020 (+1000731us) ts_tablet_manager.cc:1151] Deleting tablet 24016ceaa84147eca0b94b3ad288e772
1029 02:52:02.557029 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.561684 (+ 4655us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:02.602635 (+ 40951us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:02.607329 (+ 4694us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1000497}
I20251029 02:52:02.608795 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 24016ceaa84147eca0b94b3ad288e772 (table default.Table_2662f20eb8d843b8bf6c841e49863c85 [id=bd279b3ce623477ca20e8d35e3b7c0df]) successfully deleted
I20251029 02:52:02.608999 1172 tablet_bootstrap.cc:492] T f246f6c202cf44d48c9f5e4f335202c5 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:02.609154 1172 ts_tablet_manager.cc:1403] T f246f6c202cf44d48c9f5e4f335202c5 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.003s
I20251029 02:52:02.609338 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 584) took 1051 ms. Trace:
I20251029 02:52:02.609495 1176 rpcz_store.cc:276] 1029 02:52:01.557642 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:01.557832 (+ 190us) service_pool.cc:225] Handling call
1029 02:52:02.609148 (+1051316us) ts_tablet_manager.cc:1151] Deleting tablet 1df0d944717d487a98ea83a8ec04bd46
1029 02:52:02.609162 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.609283 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1051082}
I20251029 02:52:02.609926 1172 raft_consensus.cc:359] T f246f6c202cf44d48c9f5e4f335202c5 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:02.610095 1172 raft_consensus.cc:385] T f246f6c202cf44d48c9f5e4f335202c5 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:02.610141 1172 raft_consensus.cc:740] T f246f6c202cf44d48c9f5e4f335202c5 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:02.610360 1172 consensus_queue.cc:260] T f246f6c202cf44d48c9f5e4f335202c5 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
W20251029 02:52:02.610906 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 1df0d944717d487a98ea83a8ec04bd46 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1df0d944717d487a98ea83a8ec04bd46 already in progress: creating tablet
I20251029 02:52:02.611387 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 585) took 1053 ms. Trace:
I20251029 02:52:02.611632 1176 rpcz_store.cc:276] 1029 02:52:01.558209 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:01.558380 (+ 171us) service_pool.cc:225] Handling call
1029 02:52:02.611220 (+1052840us) ts_tablet_manager.cc:1151] Deleting tablet 2e003e140a244d46b15a2c3e84f68936
1029 02:52:02.611232 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.611327 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1052597}
I20251029 02:52:02.612838 1172 ts_tablet_manager.cc:1434] T f246f6c202cf44d48c9f5e4f335202c5 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
W20251029 02:52:02.612990 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 2e003e140a244d46b15a2c3e84f68936 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2e003e140a244d46b15a2c3e84f68936 already in progress: creating tablet
I20251029 02:52:02.613119 1172 tablet_bootstrap.cc:492] T daa05a34c0684c07a35966290fd7d44c P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:02.613472 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 586) took 1054 ms. Trace:
I20251029 02:52:02.613775 1176 rpcz_store.cc:276] 1029 02:52:01.558770 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:01.559401 (+ 631us) service_pool.cc:225] Handling call
1029 02:52:02.613306 (+1053905us) ts_tablet_manager.cc:1151] Deleting tablet 5bd2163bc4be467fb695629bfbb5a6ca
1029 02:52:02.613317 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.613410 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1053722}
W20251029 02:52:02.615077 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 5bd2163bc4be467fb695629bfbb5a6ca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5bd2163bc4be467fb695629bfbb5a6ca already in progress: creating tablet
I20251029 02:52:02.615518 1174 log.cc:1199] T 9964ad774890422f8e14f6b99f28cb55 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/9964ad774890422f8e14f6b99f28cb55
I20251029 02:52:02.615547 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 587) took 1055 ms. Trace:
I20251029 02:52:02.615630 1176 rpcz_store.cc:276] 1029 02:52:01.559744 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:01.559941 (+ 197us) service_pool.cc:225] Handling call
1029 02:52:02.615369 (+1055428us) ts_tablet_manager.cc:1151] Deleting tablet 19609bc7f06645dc811b4bae8ce506c8
1029 02:52:02.615379 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.615483 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1055236}
I20251029 02:52:02.615871 1166 tablet_bootstrap.cc:492] T 0a47781d036149569e1164dd938a45e9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
W20251029 02:52:02.616036 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 19609bc7f06645dc811b4bae8ce506c8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 19609bc7f06645dc811b4bae8ce506c8 already in progress: creating tablet
I20251029 02:52:02.616084 1166 ts_tablet_manager.cc:1403] T 0a47781d036149569e1164dd938a45e9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.028s user 0.003s sys 0.002s
I20251029 02:52:02.616200 1174 ts_tablet_manager.cc:1950] T 9964ad774890422f8e14f6b99f28cb55 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:02.617010 1166 raft_consensus.cc:359] T 0a47781d036149569e1164dd938a45e9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:02.617267 1166 raft_consensus.cc:385] T 0a47781d036149569e1164dd938a45e9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:02.617378 1166 raft_consensus.cc:740] T 0a47781d036149569e1164dd938a45e9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:02.617681 1166 consensus_queue.cc:260] T 0a47781d036149569e1164dd938a45e9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:02.618342 1166 ts_tablet_manager.cc:1434] T 0a47781d036149569e1164dd938a45e9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 02:52:02.618853 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 9964ad774890422f8e14f6b99f28cb55 (table default.Table_da0be17b8bdc4e579b6c3faff71d9c98 [id=6e06ac70c6814e7d89a008bbaddccc7e]) successfully deleted
I20251029 02:52:02.620501 1166 tablet_bootstrap.cc:492] T 1fc87e2d7b4240e08f2b4ac7277293ff P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:02.621999 1176 tablet_replica.cc:333] T b5d5b737c86844d58c51b2388c0b1705 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:02.622190 1176 raft_consensus.cc:2243] T b5d5b737c86844d58c51b2388c0b1705 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:02.622332 1166 tablet_bootstrap.cc:654] T 1fc87e2d7b4240e08f2b4ac7277293ff P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:02.622392 1176 raft_consensus.cc:2272] T b5d5b737c86844d58c51b2388c0b1705 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:02.623107 1665 raft_consensus.cc:493] T bfec87f690a640c39c58c64e16cc67e7 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:02.623288 1665 raft_consensus.cc:515] T bfec87f690a640c39c58c64e16cc67e7 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:02.623497 1175 ts_tablet_manager.cc:1929] T 7309d3190cfa415281c5253a634fa7f3 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:02.623592 1175 log.cc:1199] T 7309d3190cfa415281c5253a634fa7f3 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/7309d3190cfa415281c5253a634fa7f3
I20251029 02:52:02.623916 1175 ts_tablet_manager.cc:1950] T 7309d3190cfa415281c5253a634fa7f3 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:02.624151 1172 tablet_bootstrap.cc:654] T daa05a34c0684c07a35966290fd7d44c P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:02.624219 1665 leader_election.cc:290] T bfec87f690a640c39c58c64e16cc67e7 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:02.624835 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bfec87f690a640c39c58c64e16cc67e7" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:02.624953 1167 tablet_bootstrap.cc:492] T 5695b5a7b1854c3983e2d4ab684898b0 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:02.625087 1167 ts_tablet_manager.cc:1403] T 5695b5a7b1854c3983e2d4ab684898b0 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.066s user 0.000s sys 0.003s
I20251029 02:52:02.625418 1176 ts_tablet_manager.cc:1916] T b5d5b737c86844d58c51b2388c0b1705 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:02.625672 918 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bfec87f690a640c39c58c64e16cc67e7" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:02.626020 1167 raft_consensus.cc:359] T 5695b5a7b1854c3983e2d4ab684898b0 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:02.626140 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 534) took 2488 ms. Trace:
I20251029 02:52:02.626221 1167 raft_consensus.cc:385] T 5695b5a7b1854c3983e2d4ab684898b0 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251029 02:52:02.626220 712 leader_election.cc:343] T bfec87f690a640c39c58c64e16cc67e7 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:02.626317 1167 raft_consensus.cc:740] T 5695b5a7b1854c3983e2d4ab684898b0 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:02.626271 1175 rpcz_store.cc:276] 1029 02:52:00.137751 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:00.362886 (+225135us) service_pool.cc:225] Handling call
1029 02:52:02.591285 (+2228399us) ts_tablet_manager.cc:1151] Deleting tablet 7309d3190cfa415281c5253a634fa7f3
1029 02:52:02.591293 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.608316 (+ 17023us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:02.622802 (+ 14486us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:02.626119 (+ 3317us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2228190}
I20251029 02:52:02.626591 1167 consensus_queue.cc:260] T 5695b5a7b1854c3983e2d4ab684898b0 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
W20251029 02:52:02.626636 711 leader_election.cc:343] T bfec87f690a640c39c58c64e16cc67e7 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:02.626796 711 leader_election.cc:304] T bfec87f690a640c39c58c64e16cc67e7 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:02.627209 1665 raft_consensus.cc:2749] T bfec87f690a640c39c58c64e16cc67e7 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:02.627907 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 7309d3190cfa415281c5253a634fa7f3 (table default.Table_ef563eb489f947d4a4772678806a0777 [id=7de93d6d48ab4d58b5aed9319d7299a4]) successfully deleted
I20251029 02:52:02.628454 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 540) took 2287 ms. Trace:
I20251029 02:52:02.628870 1175 rpcz_store.cc:276] 1029 02:52:00.340599 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:00.432013 (+ 91414us) service_pool.cc:225] Handling call
1029 02:52:02.628343 (+2196330us) ts_tablet_manager.cc:1151] Deleting tablet 998b9cbe76514665be76c6c387dc75b3
1029 02:52:02.628354 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.628438 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2183169}
W20251029 02:52:02.629531 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 998b9cbe76514665be76c6c387dc75b3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 998b9cbe76514665be76c6c387dc75b3 already in progress: creating tablet
I20251029 02:52:02.629937 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 542) took 2283 ms. Trace:
I20251029 02:52:02.630270 1175 rpcz_store.cc:276] 1029 02:52:00.346696 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:00.445204 (+ 98508us) service_pool.cc:225] Handling call
1029 02:52:02.629833 (+2184629us) ts_tablet_manager.cc:1151] Deleting tablet 4cede3b183494845af91dca39a2096c4
1029 02:52:02.629842 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.629923 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2164869}
W20251029 02:52:02.630803 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 4cede3b183494845af91dca39a2096c4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4cede3b183494845af91dca39a2096c4 already in progress: creating tablet
I20251029 02:52:02.631186 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 539) took 2306 ms. Trace:
I20251029 02:52:02.631461 1167 ts_tablet_manager.cc:1434] T 5695b5a7b1854c3983e2d4ab684898b0 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20251029 02:52:02.631495 1175 rpcz_store.cc:276] 1029 02:52:00.326157 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:00.431976 (+105819us) service_pool.cc:225] Handling call
1029 02:52:02.631098 (+2199122us) ts_tablet_manager.cc:1151] Deleting tablet 97b2ecae572a4b7ea1b9595074556708
1029 02:52:02.631106 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.631173 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2165067}
W20251029 02:52:02.632050 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 97b2ecae572a4b7ea1b9595074556708 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 97b2ecae572a4b7ea1b9595074556708 already in progress: creating tablet
I20251029 02:52:02.632406 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 541) took 2288 ms. Trace:
I20251029 02:52:02.632707 1175 rpcz_store.cc:276] 1029 02:52:00.343949 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:00.438199 (+ 94250us) service_pool.cc:225] Handling call
1029 02:52:02.632320 (+2194121us) ts_tablet_manager.cc:1151] Deleting tablet a7685b0bed174ed0b25f99ef8a46fca3
1029 02:52:02.632328 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.632392 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2165750}
W20251029 02:52:02.633229 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet a7685b0bed174ed0b25f99ef8a46fca3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a7685b0bed174ed0b25f99ef8a46fca3 already in progress: creating tablet
I20251029 02:52:02.633495 1167 tablet_bootstrap.cc:492] T 6cd7009e06a646acad0802918c2b425e P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:02.633556 1175 tablet_replica.cc:333] T dbc2092b63e0417997565c8fadb53ae0 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:02.633970 1175 raft_consensus.cc:2243] T dbc2092b63e0417997565c8fadb53ae0 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:02.634398 1175 raft_consensus.cc:2272] T dbc2092b63e0417997565c8fadb53ae0 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:02.635128 1175 ts_tablet_manager.cc:1916] T dbc2092b63e0417997565c8fadb53ae0 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:02.635401 1167 tablet_bootstrap.cc:654] T 6cd7009e06a646acad0802918c2b425e P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
W20251029 02:52:02.638636 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet c7b78fd458e4414b9fc3aa064a80b206 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c7b78fd458e4414b9fc3aa064a80b206 already in progress: creating tablet
W20251029 02:52:02.639917 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 98bbaec01186435b8bcc7257a1390185 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 98bbaec01186435b8bcc7257a1390185 already in progress: creating tablet
I20251029 02:52:02.643360 1172 tablet_bootstrap.cc:492] T daa05a34c0684c07a35966290fd7d44c P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:02.643503 1172 ts_tablet_manager.cc:1403] T daa05a34c0684c07a35966290fd7d44c P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.002s
I20251029 02:52:02.643853 1166 tablet_bootstrap.cc:492] T 1fc87e2d7b4240e08f2b4ac7277293ff P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:02.643994 1166 ts_tablet_manager.cc:1403] T 1fc87e2d7b4240e08f2b4ac7277293ff P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.001s
I20251029 02:52:02.644274 1172 raft_consensus.cc:359] T daa05a34c0684c07a35966290fd7d44c P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:02.644444 1172 raft_consensus.cc:385] T daa05a34c0684c07a35966290fd7d44c P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:02.644511 1172 raft_consensus.cc:740] T daa05a34c0684c07a35966290fd7d44c P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:02.644759 1172 consensus_queue.cc:260] T daa05a34c0684c07a35966290fd7d44c P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:02.644802 1166 raft_consensus.cc:359] T 1fc87e2d7b4240e08f2b4ac7277293ff P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:02.645125 1166 raft_consensus.cc:385] T 1fc87e2d7b4240e08f2b4ac7277293ff P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:02.645268 1166 raft_consensus.cc:740] T 1fc87e2d7b4240e08f2b4ac7277293ff P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:02.645521 1166 consensus_queue.cc:260] T 1fc87e2d7b4240e08f2b4ac7277293ff P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:02.646214 1166 ts_tablet_manager.cc:1434] T 1fc87e2d7b4240e08f2b4ac7277293ff P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251029 02:52:02.646631 1166 tablet_bootstrap.cc:492] T a5c3a15510cb4321bbb80d61fabd0c0a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:02.647380 1172 ts_tablet_manager.cc:1434] T daa05a34c0684c07a35966290fd7d44c P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251029 02:52:02.647827 1172 tablet_bootstrap.cc:492] T 9e6eb099648549dbbf9969e59865f51f P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:02.648589 1166 tablet_bootstrap.cc:654] T a5c3a15510cb4321bbb80d61fabd0c0a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:02.649428 1172 tablet_bootstrap.cc:654] T 9e6eb099648549dbbf9969e59865f51f P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:02.658334 1176 ts_tablet_manager.cc:1929] T b5d5b737c86844d58c51b2388c0b1705 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:02.658607 1176 log.cc:1199] T b5d5b737c86844d58c51b2388c0b1705 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/b5d5b737c86844d58c51b2388c0b1705
I20251029 02:52:02.659735 1176 ts_tablet_manager.cc:1950] T b5d5b737c86844d58c51b2388c0b1705 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:02.663103 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 589) took 1091 ms. Trace:
I20251029 02:52:02.663301 1176 rpcz_store.cc:276] 1029 02:52:01.571379 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:01.571441 (+ 62us) service_pool.cc:225] Handling call
1029 02:52:02.621910 (+1050469us) ts_tablet_manager.cc:1151] Deleting tablet b5d5b737c86844d58c51b2388c0b1705
1029 02:52:02.621919 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.634219 (+ 12300us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:02.653353 (+ 19134us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:02.663053 (+ 9700us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1050241}
I20251029 02:52:02.664106 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet b5d5b737c86844d58c51b2388c0b1705 (table default.Table_8b39acd343e94119804478e17baaa785 [id=0b43261c80a5422092e96d77c0d2ee9e]) successfully deleted
W20251029 02:52:02.664430 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 7d1408dc544944669de519a1553d9bb7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7d1408dc544944669de519a1553d9bb7 already in progress: creating tablet
W20251029 02:52:02.665849 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet f2d47238761e4ed49c88abdbb677b18a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f2d47238761e4ed49c88abdbb677b18a already in progress: creating tablet
W20251029 02:52:02.666080 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 6142018e44af4d32818adefa2ee43d02 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6142018e44af4d32818adefa2ee43d02 already in progress: creating tablet
W20251029 02:52:02.666244 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 217144957930495aa3e9dfcfde268e71 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 217144957930495aa3e9dfcfde268e71 already in progress: creating tablet
W20251029 02:52:02.666417 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet b3de6670357c4ae59eb77469b2fb434c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b3de6670357c4ae59eb77469b2fb434c already in progress: creating tablet
W20251029 02:52:02.666752 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet d041e87ceb11441983d0e38b7929c3a4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d041e87ceb11441983d0e38b7929c3a4 already in progress: creating tablet
W20251029 02:52:02.666965 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet c64d6761ef924ccaa9c2579f8ba47e33 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c64d6761ef924ccaa9c2579f8ba47e33 already in progress: creating tablet
W20251029 02:52:02.667155 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 5ef87f7a3dff4f00a0e7273eaec98dff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5ef87f7a3dff4f00a0e7273eaec98dff already in progress: creating tablet
W20251029 02:52:02.667341 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet e65eaaae43ac435ebd665546de9d68dd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e65eaaae43ac435ebd665546de9d68dd already in progress: creating tablet
W20251029 02:52:02.667487 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 8f1d5847265b4bb0b29b5e513c35b9bb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8f1d5847265b4bb0b29b5e513c35b9bb already in progress: creating tablet
W20251029 02:52:02.667658 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet e3000183230e4350b47f5085952f2db6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e3000183230e4350b47f5085952f2db6 already in progress: creating tablet
W20251029 02:52:02.667806 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 98bbaec01186435b8bcc7257a1390185 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 98bbaec01186435b8bcc7257a1390185 already in progress: creating tablet
W20251029 02:52:02.667964 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet c7b78fd458e4414b9fc3aa064a80b206 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c7b78fd458e4414b9fc3aa064a80b206 already in progress: creating tablet
I20251029 02:52:02.668423 1175 ts_tablet_manager.cc:1929] T dbc2092b63e0417997565c8fadb53ae0 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:02.668582 1175 log.cc:1199] T dbc2092b63e0417997565c8fadb53ae0 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/dbc2092b63e0417997565c8fadb53ae0
I20251029 02:52:02.492710 1514 rpcz_store.cc:276] 1029 02:52:01.048849 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:01.056108 (+ 7259us) service_pool.cc:225] Handling call
1029 02:52:01.090244 (+ 34136us) raft_consensus.cc:1471] Updating replica for 307 ops
1029 02:52:01.133600 (+ 43356us) raft_consensus.cc:1534] Early marking committed up to index 0
1029 02:52:01.133606 (+ 6us) raft_consensus.cc:1539] Triggering prepare for 307 ops
1029 02:52:01.133760 (+ 154us) write_op.cc:270] Start()
1029 02:52:01.133805 (+ 45us) write_op.cc:276] Timestamp: P: 1761706316550801 usec, L: 0
1029 02:52:01.134020 (+ 215us) write_op.cc:270] Start()
1029 02:52:01.134042 (+ 22us) write_op.cc:276] Timestamp: P: 1761706316633476 usec, L: 0
1029 02:52:01.134187 (+ 145us) write_op.cc:270] Start()
1029 02:52:01.134208 (+ 21us) write_op.cc:276] Timestamp: P: 1761706316634079 usec, L: 0
1029 02:52:01.134347 (+ 139us) write_op.cc:270] Start()
1029 02:52:01.134367 (+ 20us) write_op.cc:276] Timestamp: P: 1761706316634615 usec, L: 0
1029 02:52:01.142891 (+ 8524us) write_op.cc:270] Start()
1029 02:52:01.142937 (+ 46us) write_op.cc:276] Timestamp: P: 1761706316635083 usec, L: 0
1029 02:52:01.152878 (+ 9941us) write_op.cc:270] Start()
1029 02:52:01.152926 (+ 48us) write_op.cc:276] Timestamp: P: 1761706316651313 usec, L: 0
1029 02:52:01.153147 (+ 221us) write_op.cc:270] Start()
1029 02:52:01.153194 (+ 47us) write_op.cc:276] Timestamp: P: 1761706316668388 usec, L: 0
1029 02:52:01.153337 (+ 143us) write_op.cc:270] Start()
1029 02:52:01.153361 (+ 24us) write_op.cc:276] Timestamp: P: 1761706316669451 usec, L: 0
1029 02:52:01.153519 (+ 158us) write_op.cc:270] Start()
1029 02:52:01.153538 (+ 19us) write_op.cc:276] Timestamp: P: 1761706316674223 usec, L: 0
1029 02:52:01.153673 (+ 135us) write_op.cc:270] Start()
1029 02:52:01.153690 (+ 17us) write_op.cc:276] Timestamp: P: 1761706316675240 usec, L: 0
1029 02:52:01.153811 (+ 121us) write_op.cc:270] Start()
1029 02:52:01.153829 (+ 18us) write_op.cc:276] Timestamp: P: 1761706316677678 usec, L: 0
1029 02:52:01.153932 (+ 103us) write_op.cc:270] Start()
1029 02:52:01.153947 (+ 15us) write_op.cc:276] Timestamp: P: 1761706316691348 usec, L: 0
1029 02:52:01.154067 (+ 120us) write_op.cc:270] Start()
1029 02:52:01.154085 (+ 18us) write_op.cc:276] Timestamp: P: 1761706316719595 usec, L: 0
1029 02:52:01.154186 (+ 101us) write_op.cc:270] Start()
1029 02:52:01.154207 (+ 21us) write_op.cc:276] Timestamp: P: 1761706316720435 usec, L: 0
1029 02:52:01.154360 (+ 153us) write_op.cc:270] Start()
1029 02:52:01.154379 (+ 19us) write_op.cc:276] Timestamp: P: 1761706316723831 usec, L: 0
1029 02:52:01.154584 (+ 205us) write_op.cc:270] Start()
1029 02:52:01.154607 (+ 23us) write_op.cc:276] Timestamp: P: 1761706316824211 usec, L: 0
1029 02:52:01.154747 (+ 140us) write_op.cc:270] Start()
1029 02:52:01.154764 (+ 17us) write_op.cc:276] Timestamp: P: 1761706316829465 usec, L: 0
1029 02:52:01.189255 (+ 34491us) write_op.cc:270] Start()
1029 02:52:01.189302 (+ 47us) write_op.cc:276] Timestamp: P: 1761706316830121 usec, L: 0
1029 02:52:01.189524 (+ 222us) write_op.cc:270] Start()
1029 02:52:01.189550 (+ 26us) write_op.cc:276] Timestamp: P: 1761706316836983 usec, L: 0
1029 02:52:01.189695 (+ 145us) write_op.cc:270] Start()
1029 02:52:01.189728 (+ 33us) write_op.cc:276] Timestamp: P: 1761706316837755 usec, L: 0
1029 02:52:01.189922 (+ 194us) write_op.cc:270] Start()
1029 02:52:01.189943 (+ 21us) write_op.cc:276] Timestamp: P: 1761706316849816 usec, L: 0
1029 02:52:01.190584 (+ 641us) write_op.cc:270] Start()
1029 02:52:01.190607 (+ 23us) write_op.cc:276] Timestamp: P: 1761706316850486 usec, L: 0
1029 02:52:01.190734 (+ 127us) write_op.cc:270] Start()
1029 02:52:01.190751 (+ 17us) write_op.cc:276] Timestamp: P: 1761706316853724 usec, L: 0
1029 02:52:01.190862 (+ 111us) write_op.cc:270] Start()
1029 02:52:01.190877 (+ 15us) write_op.cc:276] Timestamp: P: 1761706316855234 usec, L: 0
1029 02:52:01.191027 (+ 150us) write_op.cc:270] Start()
1029 02:52:01.191045 (+ 18us) write_op.cc:276] Timestamp: P: 1761706316857351 usec, L: 0
1029 02:52:01.191167 (+ 122us) write_op.cc:270] Start()
1029 02:52:01.191186 (+ 19us) write_op.cc:276] Timestamp: P: 1761706316907121 usec, L: 0
1029 02:52:01.191333 (+ 147us) write_op.cc:270] Start()
1029 02:52:01.191356 (+ 23us) write_op.cc:276] Timestamp: P: 1761706316924527 usec, L: 0
1029 02:52:01.191479 (+ 123us) write_op.cc:270] Start()
1029 02:52:01.191495 (+ 16us) write_op.cc:276] Timestamp: P: 1761706317027077 usec, L: 0
1029 02:52:01.191611 (+ 116us) write_op.cc:270] Start()
1029 02:52:01.191627 (+ 16us) write_op.cc:276] Timestamp: P: 1761706317038089 usec, L: 0
1029 02:52:01.191744 (+ 117us) write_op.cc:270] Start()
1029 02:52:01.191760 (+ 16us) write_op.cc:276] Timestamp: P: 1761706317045360 usec, L: 0
1029 02:52:01.191866 (+ 106us) write_op.cc:270] Start()
1029 02:52:01.191881 (+ 15us) write_op.cc:276] Timestamp: P: 1761706317059763 usec, L: 0
1029 02:52:01.191998 (+ 117us) write_op.cc:270] Start()
1029 02:52:01.192013 (+ 15us) write_op.cc:276] Timestamp: P: 1761706317060512 usec, L: 0
1029 02:52:01.192129 (+ 116us) write_op.cc:270] Start()
1029 02:52:01.192144 (+ 15us) write_op.cc:276] Timestamp: P: 1761706317061181 usec, L: 0
1029 02:52:01.192235 (+ 91us) write_op.cc:270] Start()
1029 02:52:01.192250 (+ 15us) write_op.cc:276] Timestamp: P: 1761706317061856 usec, L: 0
1029 02:52:01.192368 (+ 118us) write_op.cc:270] Start()
1029 02:52:01.192387 (+ 19us) write_op.cc:276] Timestamp: P: 1761706317074677 usec, L: 0
1029 02:52:01.192496 (+ 109us) write_op.cc:270] Start()
1029 02:52:01.192513 (+ 17us) write_op.cc:276] Timestamp: P: 1761706317095315 usec, L: 0
1029 02:52:01.192633 (+ 120us) write_op.cc:270] Start()
1029 02:52:01.192652 (+ 19us) write_op.cc:276] Timestamp: P: 1761706317097267 usec, L: 0
1029 02:52:01.192782 (+ 130us) write_op.cc:270] Start()
1029 02:52:01.192805 (+ 23us) write_op.cc:276] Timestamp: P: 1761706317099131 usec, L: 0
1029 02:52:01.192965 (+ 160us) write_op.cc:270] Start()
1029 02:52:01.192986 (+ 21us) write_op.cc:276] Timestamp: P: 1761706317143355 usec, L: 0
1029 02:52:01.193127 (+ 141us) write_op.cc:270] Start()
1029 02:52:01.193147 (+ 20us) write_op.cc:276] Timestamp: P: 1761706317144062 usec, L: 0
1029 02:52:01.193309 (+ 162us) write_op.cc:270] Start()
1029 02:52:01.193330 (+ 21us) write_op.cc:276] Timestamp: P: 1761706317147219 usec, L: 0
1029 02:52:01.193450 (+ 120us) write_op.cc:270] Start()
1029 02:52:01.193472 (+ 22us) write_op.cc:276] Timestamp: P: 1761706317147775 usec, L: 0
1029 02:52:01.193593 (+ 121us) write_op.cc:270] Start()
1029 02:52:01.193611 (+ 18us) write_op.cc:276] Timestamp: P: 1761706317148262 usec, L: 0
1029 02:52:01.193732 (+ 121us) write_op.cc:270] Start()
1029 02:52:01.193749 (+ 17us) write_op.cc:276] Timestamp: P: 1761706317161873 usec, L: 0
1029 02:52:01.193877 (+ 128us) write_op.cc:270] Start()
1029 02:52:01.193893 (+ 16us) write_op.cc:276] Timestamp: P: 1761706317162544 usec, L: 0
1029 02:52:01.194027 (+ 134us) write_op.cc:270] Start()
1029 02:52:01.194048 (+ 21us) write_op.cc:276] Timestamp: P: 1761706317163056 usec, L: 0
1029 02:52:01.194218 (+ 170us) write_op.cc:270] Start()
1029 02:52:01.194241 (+ 23us) write_op.cc:276] Timestamp: P: 1761706317163566 usec, L: 0
1029 02:52:01.194392 (+ 151us) write_op.cc:270] Start()
1029 02:52:01.194412 (+ 20us) write_op.cc:276] Timestamp: P: 1761706317168801 usec, L: 0
1029 02:52:01.273471 (+ 79059us) write_op.cc:270] Start()
1029 02:52:01.273525 (+ 54us) write_op.cc:276] Timestamp: P: 1761706317171108 usec, L: 0
1029 02:52:01.273762 (+ 237us) write_op.cc:270] Start()
1029 02:52:01.273785 (+ 23us) write_op.cc:276] Timestamp: P: 1761706317179487 usec, L: 0
1029 02:52:01.273943 (+ 158us) write_op.cc:270] Start()
1029 02:52:01.273961 (+ 18us) write_op.cc:276] Timestamp: P: 1761706317236308 usec, L: 0
1029 02:52:01.274091 (+ 130us) write_op.cc:270] Start()
1029 02:52:01.274108 (+ 17us) write_op.cc:276] Timestamp: P: 1761706317249638 usec, L: 0
1029 02:52:01.274243 (+ 135us) write_op.cc:270] Start()
1029 02:52:01.274264 (+ 21us) write_op.cc:276] Timestamp: P: 1761706317270856 usec, L: 0
1029 02:52:01.274409 (+ 145us) write_op.cc:270] Start()
1029 02:52:01.274435 (+ 26us) write_op.cc:276] Timestamp: P: 1761706317278663 usec, L: 0
1029 02:52:01.329795 (+ 55360us) write_op.cc:270] Start()
1029 02:52:01.329844 (+ 49us) write_op.cc:276] Timestamp: P: 1761706317289975 usec, L: 0
1029 02:52:01.330049 (+ 205us) write_op.cc:270] Start()
1029 02:52:01.330070 (+ 21us) write_op.cc:276] Timestamp: P: 1761706317290773 usec, L: 0
1029 02:52:01.330202 (+ 132us) write_op.cc:270] Start()
1029 02:52:01.330221 (+ 19us) write_op.cc:276] Timestamp: P: 1761706317302199 usec, L: 0
1029 02:52:01.330332 (+ 111us) write_op.cc:270] Start()
1029 02:52:01.330348 (+ 16us) write_op.cc:276] Timestamp: P: 1761706317302762 usec, L: 0
1029 02:52:01.330471 (+ 123us) write_op.cc:270] Start()
1029 02:52:01.330492 (+ 21us) write_op.cc:276] Timestamp: P: 1761706317303811 usec, L: 0
1029 02:52:01.330672 (+ 180us) write_op.cc:270] Start()
1029 02:52:01.330718 (+ 46us) write_op.cc:276] Timestamp: P: 1761706317367319 usec, L: 0
1029 02:52:01.330891 (+ 173us) write_op.cc:270] Start()
1029 02:52:01.330915 (+ 24us) write_op.cc:276] Timestamp: P: 1761706317373901 usec, L: 0
1029 02:52:01.331065 (+ 150us) write_op.cc:270] Start()
1029 02:52:01.331085 (+ 20us) write_op.cc:276] Timestamp: P: 1761706317374694 usec, L: 0
1029 02:52:01.331221 (+ 136us) write_op.cc:270] Start()
1029 02:52:01.331240 (+ 19us) write_op.cc:276] Timestamp: P: 1761706317375430 usec, L: 0
1029 02:52:01.331362 (+ 122us) write_op.cc:270] Start()
1029 02:52:01.331380 (+ 18us) write_op.cc:276] Timestamp: P: 1761706317397770 usec, L: 0
1029 02:52:01.331524 (+ 144us) write_op.cc:270] Start()
1029 02:52:01.331543 (+ 19us) write_op.cc:276] Timestamp: P: 1761706317398597 usec, L: 0
1029 02:52:01.331671 (+ 128us) write_op.cc:270] Start()
1029 02:52:01.331689 (+ 18us) write_op.cc:276] Timestamp: P: 1761706317399275 usec, L: 0
1029 02:52:01.331845 (+ 156us) write_op.cc:270] Start()
1029 02:52:01.331865 (+ 20us) write_op.cc:276] Timestamp: P: 1761706317432546 usec, L: 0
1029 02:52:01.332007 (+ 142us) write_op.cc:270] Start()
1029 02:52:01.332026 (+ 19us) write_op.cc:276] Timestamp: P: 1761706317468014 usec, L: 0
1029 02:52:01.332190 (+ 164us) write_op.cc:270] Start()
1029 02:52:01.332210 (+ 20us) write_op.cc:276] Timestamp: P: 1761706317482250 usec, L: 0
1029 02:52:01.332366 (+ 156us) write_op.cc:270] Start()
1029 02:52:01.332387 (+ 21us) write_op.cc:276] Timestamp: P: 1761706317493255 usec, L: 0
1029 02:52:01.332544 (+ 157us) write_op.cc:270] Start()
1029 02:52:01.332563 (+ 19us) write_op.cc:276] Timestamp: P: 1761706317507164 usec, L: 0
1029 02:52:01.332723 (+ 160us) write_op.cc:270] Start()
1029 02:52:01.332741 (+ 18us) write_op.cc:276] Timestamp: P: 1761706317529243 usec, L: 0
1029 02:52:01.332905 (+ 164us) write_op.cc:270] Start()
1029 02:52:01.332922 (+ 17us) write_op.cc:276] Timestamp: P: 1761706317540035 usec, L: 0
1029 02:52:01.333044 (+ 122us) write_op.cc:270] Start()
1029 02:52:01.333059 (+ 15us) write_op.cc:276] Timestamp: P: 1761706317549236 usec, L: 0
1029 02:52:01.333166 (+ 107us) write_op.cc:270] Start()
1029 02:52:01.333181 (+ 15us) write_op.cc:276] Timestamp: P: 1761706317578543 usec, L: 0
1029 02:52:01.333280 (+ 99us) write_op.cc:270] Start()
1029 02:52:01.333295 (+ 15us) write_op.cc:276] Timestamp: P: 1761706317583473 usec, L: 0
1029 02:52:01.333440 (+ 145us) write_op.cc:270] Start()
1029 02:52:01.333457 (+ 17us) write_op.cc:276] Timestamp: P: 1761706317584060 usec, L: 0
1029 02:52:01.333562 (+ 105us) write_op.cc:270] Start()
1029 02:52:01.333577 (+ 15us) write_op.cc:276] Timestamp: P: 1761706317584610 usec, L: 0
1029 02:52:01.333714 (+ 137us) write_op.cc:270] Start()
1029 02:52:01.333731 (+ 17us) write_op.cc:276] Timestamp: P: 1761706317585111 usec, L: 0
1029 02:52:01.333845 (+ 114us) write_op.cc:270] Start()
1029 02:52:01.333864 (+ 19us) write_op.cc:276] Timestamp: P: 1761706317585556 usec, L: 0
1029 02:52:01.333984 (+ 120us) write_op.cc:270] Start()
1029 02:52:01.398063 (+ 64079us) write_op.cc:276] Timestamp: P: 1761706317592325 usec, L: 0
1029 02:52:01.398308 (+ 245us) write_op.cc:270] Start()
1029 02:52:01.398341 (+ 33us) write_op.cc:276] Timestamp: P: 1761706317595960 usec, L: 0
1029 02:52:01.398470 (+ 129us) write_op.cc:270] Start()
1029 02:52:01.398493 (+ 23us) write_op.cc:276] Timestamp: P: 1761706317621709 usec, L: 0
1029 02:52:01.398643 (+ 150us) write_op.cc:270] Start()
1029 02:52:01.398660 (+ 17us) write_op.cc:276] Timestamp: P: 1761706317640101 usec, L: 0
1029 02:52:01.412386 (+ 13726us) write_op.cc:270] Start()
1029 02:52:01.412422 (+ 36us) write_op.cc:276] Timestamp: P: 1761706317641154 usec, L: 0
1029 02:52:01.412561 (+ 139us) write_op.cc:270] Start()
1029 02:52:01.412580 (+ 19us) write_op.cc:276] Timestamp: P: 1761706317661206 usec, L: 0
1029 02:52:01.412701 (+ 121us) write_op.cc:270] Start()
1029 02:52:01.412720 (+ 19us) write_op.cc:276] Timestamp: P: 1761706317668691 usec, L: 0
1029 02:52:01.412846 (+ 126us) write_op.cc:270] Start()
1029 02:52:01.412865 (+ 19us) write_op.cc:276] Timestamp: P: 1761706317699991 usec, L: 0
1029 02:52:01.412982 (+ 117us) write_op.cc:270] Start()
1029 02:52:01.412998 (+ 16us) write_op.cc:276] Timestamp: P: 1761706317716154 usec, L: 0
1029 02:52:01.413113 (+ 115us) write_op.cc:270] Start()
1029 02:52:01.413129 (+ 16us) write_op.cc:276] Timestamp: P: 1761706317719014 usec, L: 0
1029 02:52:01.413248 (+ 119us) write_op.cc:270] Start()
1029 02:52:01.413287 (+ 39us) write_op.cc:276] Timestamp: P: 1761706317733163 usec, L: 0
1029 02:52:01.413403 (+ 116us) write_op.cc:270] Start()
1029 02:52:01.413419 (+ 16us) write_op.cc:276] Timestamp: P: 1761706317751733 usec, L: 0
1029 02:52:01.413572 (+ 153us) write_op.cc:270] Start()
1029 02:52:01.413591 (+ 19us) write_op.cc:276] Timestamp: P: 1761706317767484 usec, L: 0
1029 02:52:01.413717 (+ 126us) write_op.cc:270] Start()
1029 02:52:01.413737 (+ 20us) write_op.cc:276] Timestamp: P: 1761706317779531 usec, L: 0
1029 02:52:01.413876 (+ 139us) write_op.cc:270] Start()
1029 02:52:01.413895 (+ 19us) write_op.cc:276] Timestamp: P: 1761706317794718 usec, L: 0
1029 02:52:01.414048 (+ 153us) write_op.cc:270] Start()
1029 02:52:01.414065 (+ 17us) write_op.cc:276] Timestamp: P: 1761706317810277 usec, L: 0
1029 02:52:01.414197 (+ 132us) write_op.cc:270] Start()
1029 02:52:01.414212 (+ 15us) write_op.cc:276] Timestamp: P: 1761706317820592 usec, L: 0
1029 02:52:01.414316 (+ 104us) write_op.cc:270] Start()
1029 02:52:01.414342 (+ 26us) write_op.cc:276] Timestamp: P: 1761706317835854 usec, L: 0
1029 02:52:01.414447 (+ 105us) write_op.cc:270] Start()
1029 02:52:01.414463 (+ 16us) write_op.cc:276] Timestamp: P: 1761706317850190 usec, L: 0
1029 02:52:01.473823 (+ 59360us) write_op.cc:270] Start()
1029 02:52:01.473867 (+ 44us) write_op.cc:276] Timestamp: P: 1761706317860631 usec, L: 0
1029 02:52:01.474085 (+ 218us) write_op.cc:270] Start()
1029 02:52:01.474123 (+ 38us) write_op.cc:276] Timestamp: P: 1761706317878837 usec, L: 0
1029 02:52:01.474247 (+ 124us) write_op.cc:270] Start()
1029 02:52:01.474264 (+ 17us) write_op.cc:276] Timestamp: P: 1761706317896469 usec, L: 0
1029 02:52:01.474376 (+ 112us) write_op.cc:270] Start()
1029 02:52:01.474393 (+ 17us) write_op.cc:276] Timestamp: P: 1761706317912951 usec, L: 0
1029 02:52:01.474535 (+ 142us) write_op.cc:270] Start()
1029 02:52:01.474555 (+ 20us) write_op.cc:276] Timestamp: P: 1761706317924984 usec, L: 0
1029 02:52:01.474716 (+ 161us) write_op.cc:270] Start()
1029 02:52:01.474736 (+ 20us) write_op.cc:276] Timestamp: P: 1761706317940158 usec, L: 0
1029 02:52:01.474864 (+ 128us) write_op.cc:270] Start()
1029 02:52:01.619817 (+144953us) write_op.cc:276] Timestamp: P: 1761706317946221 usec, L: 0
1029 02:52:01.620121 (+ 304us) write_op.cc:270] Start()
1029 02:52:01.620156 (+ 35us) write_op.cc:276] Timestamp: P: 1761706317966506 usec, L: 0
1029 02:52:01.620307 (+ 151us) write_op.cc:270] Start()
1029 02:52:01.620326 (+ 19us) write_op.cc:276] Timestamp: P: 1761706317996052 usec, L: 0
1029 02:52:01.620449 (+ 123us) write_op.cc:270] Start()
1029 02:52:01.620469 (+ 20us) write_op.cc:276] Timestamp: P: 1761706318010942 usec, L: 0
1029 02:52:01.620591 (+ 122us) write_op.cc:270] Start()
1029 02:52:01.620609 (+ 18us) write_op.cc:276] Timestamp: P: 1761706318024102 usec, L: 0
1029 02:52:01.620766 (+ 157us) write_op.cc:270] Start()
1029 02:52:01.620804 (+ 38us) write_op.cc:276] Timestamp: P: 1761706318042732 usec, L: 0
1029 02:52:01.620970 (+ 166us) write_op.cc:270] Start()
1029 02:52:01.620993 (+ 23us) write_op.cc:276] Timestamp: P: 1761706318058827 usec, L: 0
1029 02:52:01.621125 (+ 132us) write_op.cc:270] Start()
1029 02:52:01.621147 (+ 22us) write_op.cc:276] Timestamp: P: 1761706318085333 usec, L: 0
1029 02:52:01.621270 (+ 123us) write_op.cc:270] Start()
1029 02:52:01.621290 (+ 20us) write_op.cc:276] Timestamp: P: 1761706318090790 usec, L: 0
1029 02:52:01.621426 (+ 136us) write_op.cc:270] Start()
1029 02:52:01.621447 (+ 21us) write_op.cc:276] Timestamp: P: 1761706318100018 usec, L: 0
1029 02:52:01.621563 (+ 116us) write_op.cc:270] Start()
1029 02:52:01.621579 (+ 16us) write_op.cc:276] Timestamp: P: 1761706318100563 usec, L: 0
1029 02:52:01.621706 (+ 127us) write_op.cc:270] Start()
1029 02:52:01.621726 (+ 20us) write_op.cc:276] Timestamp: P: 1761706318101050 usec, L: 0
1029 02:52:01.621873 (+ 147us) write_op.cc:270] Start()
1029 02:52:01.621893 (+ 20us) write_op.cc:276] Timestamp: P: 1761706318101565 usec, L: 0
1029 02:52:01.696845 (+ 74952us) write_op.cc:270] Start()
1029 02:52:01.696895 (+ 50us) write_op.cc:276] Timestamp: P: 1761706318102076 usec, L: 0
1029 02:52:01.697113 (+ 218us) write_op.cc:270] Start()
1029 02:52:01.697138 (+ 25us) write_op.cc:276] Timestamp: P: 1761706318114968 usec, L: 0
1029 02:52:01.697507 (+ 369us) write_op.cc:270] Start()
1029 02:52:01.697537 (+ 30us) write_op.cc:276] Timestamp: P: 1761706318119542 usec, L: 0
1029 02:52:01.697705 (+ 168us) write_op.cc:270] Start()
1029 02:52:01.697728 (+ 23us) write_op.cc:276] Timestamp: P: 1761706318120068 usec, L: 0
1029 02:52:01.697927 (+ 199us) write_op.cc:270] Start()
1029 02:52:01.697952 (+ 25us) write_op.cc:276] Timestamp: P: 1761706318125925 usec, L: 0
1029 02:52:01.698122 (+ 170us) write_op.cc:270] Start()
1029 02:52:01.698146 (+ 24us) write_op.cc:276] Timestamp: P: 1761706318137359 usec, L: 0
1029 02:52:01.698310 (+ 164us) write_op.cc:270] Start()
1029 02:52:01.698333 (+ 23us) write_op.cc:276] Timestamp: P: 1761706318168196 usec, L: 0
1029 02:52:01.698483 (+ 150us) write_op.cc:270] Start()
1029 02:52:01.698535 (+ 52us) write_op.cc:276] Timestamp: P: 1761706318169559 usec, L: 0
1029 02:52:01.698711 (+ 176us) write_op.cc:270] Start()
1029 02:52:01.698737 (+ 26us) write_op.cc:276] Timestamp: P: 1761706318174197 usec, L: 0
1029 02:52:01.698888 (+ 151us) write_op.cc:270] Start()
1029 02:52:01.698911 (+ 23us) write_op.cc:276] Timestamp: P: 1761706318216884 usec, L: 0
1029 02:52:01.699095 (+ 184us) write_op.cc:270] Start()
1029 02:52:01.699118 (+ 23us) write_op.cc:276] Timestamp: P: 1761706318261516 usec, L: 0
1029 02:52:01.699336 (+ 218us) write_op.cc:270] Start()
1029 02:52:01.699360 (+ 24us) write_op.cc:276] Timestamp: P: 1761706318270828 usec, L: 0
1029 02:52:01.699516 (+ 156us) write_op.cc:270] Start()
1029 02:52:01.699648 (+ 132us) write_op.cc:276] Timestamp: P: 1761706318273143 usec, L: 0
1029 02:52:01.699826 (+ 178us) write_op.cc:270] Start()
1029 02:52:01.699846 (+ 20us) write_op.cc:276] Timestamp: P: 1761706318273760 usec, L: 0
1029 02:52:01.699968 (+ 122us) write_op.cc:270] Start()
1029 02:52:01.699985 (+ 17us) write_op.cc:276] Timestamp: P: 1761706318281364 usec, L: 0
1029 02:52:01.700155 (+ 170us) write_op.cc:270] Start()
1029 02:52:01.700175 (+ 20us) write_op.cc:276] Timestamp: P: 1761706318302426 usec, L: 0
1029 02:52:01.700306 (+ 131us) write_op.cc:270] Start()
1029 02:52:01.700386 (+ 80us) write_op.cc:276] Timestamp: P: 1761706318318339 usec, L: 0
1029 02:52:01.700511 (+ 125us) write_op.cc:270] Start()
1029 02:52:01.700548 (+ 37us) write_op.cc:276] Timestamp: P: 1761706318328897 usec, L: 0
1029 02:52:01.700728 (+ 180us) write_op.cc:270] Start()
1029 02:52:01.700754 (+ 26us) write_op.cc:276] Timestamp: P: 1761706318329558 usec, L: 0
1029 02:52:01.708961 (+ 8207us) write_op.cc:270] Start()
1029 02:52:01.708990 (+ 29us) write_op.cc:276] Timestamp: P: 1761706318330155 usec, L: 0
1029 02:52:01.709160 (+ 170us) write_op.cc:270] Start()
1029 02:52:01.709183 (+ 23us) write_op.cc:276] Timestamp: P: 1761706318330808 usec, L: 0
1029 02:52:01.709326 (+ 143us) write_op.cc:270] Start()
1029 02:52:01.709352 (+ 26us) write_op.cc:276] Timestamp: P: 1761706318331405 usec, L: 0
1029 02:52:01.709482 (+ 130us) write_op.cc:270] Start()
1029 02:52:01.709500 (+ 18us) write_op.cc:276] Timestamp: P: 1761706318403934 usec, L: 0
1029 02:52:01.709670 (+ 170us) write_op.cc:270] Start()
1029 02:52:01.709692 (+ 22us) write_op.cc:276] Timestamp: P: 1761706318485469 usec, L: 0
1029 02:52:01.709838 (+ 146us) write_op.cc:270] Start()
1029 02:52:01.709862 (+ 24us) write_op.cc:276] Timestamp: P: 1761706318489784 usec, L: 0
1029 02:52:01.710018 (+ 156us) write_op.cc:270] Start()
1029 02:52:01.710039 (+ 21us) write_op.cc:276] Timestamp: P: 1761706318506318 usec, L: 0
1029 02:52:01.710187 (+ 148us) write_op.cc:270] Start()
1029 02:52:01.710209 (+ 22us) write_op.cc:276] Timestamp: P: 1761706318507096 usec, L: 0
1029 02:52:01.710339 (+ 130us) write_op.cc:270] Start()
1029 02:52:01.710357 (+ 18us) write_op.cc:276] Timestamp: P: 1761706318507719 usec, L: 0
1029 02:52:01.710547 (+ 190us) write_op.cc:270] Start()
1029 02:52:01.710570 (+ 23us) write_op.cc:276] Timestamp: P: 1761706318508213 usec, L: 0
1029 02:52:01.710739 (+ 169us) write_op.cc:270] Start()
1029 02:52:01.710757 (+ 18us) write_op.cc:276] Timestamp: P: 1761706318508808 usec, L: 0
1029 02:52:01.710881 (+ 124us) write_op.cc:270] Start()
1029 02:52:01.710899 (+ 18us) write_op.cc:276] Timestamp: P: 1761706318509331 usec, L: 0
1029 02:52:01.711044 (+ 145us) write_op.cc:270] Start()
1029 02:52:01.711069 (+ 25us) write_op.cc:276] Timestamp: P: 1761706318533502 usec, L: 0
1029 02:52:01.711196 (+ 127us) write_op.cc:270] Start()
1029 02:52:01.711213 (+ 17us) write_op.cc:276] Timestamp: P: 1761706318536631 usec, L: 0
1029 02:52:01.711338 (+ 125us) write_op.cc:270] Start()
1029 02:52:01.711357 (+ 19us) write_op.cc:276] Timestamp: P: 1761706318537750 usec, L: 0
1029 02:52:01.711486 (+ 129us) write_op.cc:270] Start()
1029 02:52:01.711509 (+ 23us) write_op.cc:276] Timestamp: P: 1761706318538798 usec, L: 0
1029 02:52:01.711652 (+ 143us) write_op.cc:270] Start()
1029 02:52:01.711668 (+ 16us) write_op.cc:276] Timestamp: P: 1761706318539728 usec, L: 0
1029 02:52:01.711813 (+ 145us) write_op.cc:270] Start()
1029 02:52:01.711831 (+ 18us) write_op.cc:276] Timestamp: P: 1761706318540513 usec, L: 0
1029 02:52:01.722309 (+ 10478us) write_op.cc:270] Start()
1029 02:52:01.722358 (+ 49us) write_op.cc:276] Timestamp: P: 1761706318549039 usec, L: 0
1029 02:52:01.722573 (+ 215us) write_op.cc:270] Start()
1029 02:52:01.722593 (+ 20us) write_op.cc:276] Timestamp: P: 1761706318595941 usec, L: 0
1029 02:52:01.755134 (+ 32541us) write_op.cc:270] Start()
1029 02:52:01.755526 (+ 392us) write_op.cc:276] Timestamp: P: 1761706318600072 usec, L: 0
1029 02:52:01.851512 (+ 95986us) write_op.cc:270] Start()
1029 02:52:01.851567 (+ 55us) write_op.cc:276] Timestamp: P: 1761706318612884 usec, L: 0
1029 02:52:01.851763 (+ 196us) write_op.cc:270] Start()
1029 02:52:01.851790 (+ 27us) write_op.cc:276] Timestamp: P: 1761706318618495 usec, L: 0
1029 02:52:01.851947 (+ 157us) write_op.cc:270] Start()
1029 02:52:01.851967 (+ 20us) write_op.cc:276] Timestamp: P: 1761706318630571 usec, L: 0
1029 02:52:01.852103 (+ 136us) write_op.cc:270] Start()
1029 02:52:01.852129 (+ 26us) write_op.cc:276] Timestamp: P: 1761706318646333 usec, L: 0
1029 02:52:01.859871 (+ 7742us) write_op.cc:270] Start()
1029 02:52:01.859900 (+ 29us) write_op.cc:276] Timestamp: P: 1761706318688219 usec, L: 0
1029 02:52:01.860086 (+ 186us) write_op.cc:270] Start()
1029 02:52:01.860110 (+ 24us) write_op.cc:276] Timestamp: P: 1761706318709444 usec, L: 0
1029 02:52:01.860236 (+ 126us) write_op.cc:270] Start()
1029 02:52:01.860255 (+ 19us) write_op.cc:276] Timestamp: P: 1761706318720388 usec, L: 0
1029 02:52:01.860389 (+ 134us) write_op.cc:270] Start()
1029 02:52:01.860407 (+ 18us) write_op.cc:276] Timestamp: P: 1761706318743258 usec, L: 0
1029 02:52:01.860567 (+ 160us) write_op.cc:270] Start()
1029 02:52:01.860586 (+ 19us) write_op.cc:276] Timestamp: P: 1761706318746244 usec, L: 0
1029 02:52:01.860702 (+ 116us) write_op.cc:270] Start()
1029 02:52:01.860722 (+ 20us) write_op.cc:276] Timestamp: P: 1761706318751655 usec, L: 0
1029 02:52:01.860865 (+ 143us) write_op.cc:270] Start()
1029 02:52:01.860884 (+ 19us) write_op.cc:276] Timestamp: P: 1761706318752645 usec, L: 0
1029 02:52:01.861037 (+ 153us) write_op.cc:270] Start()
1029 02:52:01.861058 (+ 21us) write_op.cc:276] Timestamp: P: 1761706318753229 usec, L: 0
1029 02:52:01.861199 (+ 141us) write_op.cc:270] Start()
1029 02:52:01.861219 (+ 20us) write_op.cc:276] Timestamp: P: 1761706318753876 usec, L: 0
1029 02:52:01.861358 (+ 139us) write_op.cc:270] Start()
1029 02:52:01.861379 (+ 21us) write_op.cc:276] Timestamp: P: 1761706318780674 usec, L: 0
1029 02:52:01.861505 (+ 126us) write_op.cc:270] Start()
1029 02:52:01.861525 (+ 20us) write_op.cc:276] Timestamp: P: 1761706318789717 usec, L: 0
1029 02:52:01.861705 (+ 180us) write_op.cc:270] Start()
1029 02:52:01.861737 (+ 32us) write_op.cc:276] Timestamp: P: 1761706318790325 usec, L: 0
1029 02:52:01.861914 (+ 177us) write_op.cc:270] Start()
1029 02:52:01.861934 (+ 20us) write_op.cc:276] Timestamp: P: 1761706318813557 usec, L: 0
1029 02:52:01.862060 (+ 126us) write_op.cc:270] Start()
1029 02:52:01.862078 (+ 18us) write_op.cc:276] Timestamp: P: 1761706318814151 usec, L: 0
1029 02:52:01.862242 (+ 164us) write_op.cc:270] Start()
1029 02:52:01.862261 (+ 19us) write_op.cc:276] Timestamp: P: 1761706318832945 usec, L: 0
1029 02:52:01.862394 (+ 133us) write_op.cc:270] Start()
1029 02:52:01.862413 (+ 19us) write_op.cc:276] Timestamp: P: 1761706318833499 usec, L: 0
1029 02:52:01.862588 (+ 175us) write_op.cc:270] Start()
1029 02:52:01.862609 (+ 21us) write_op.cc:276] Timestamp: P: 1761706318845715 usec, L: 0
1029 02:52:01.862753 (+ 144us) write_op.cc:270] Start()
1029 02:52:01.862776 (+ 23us) write_op.cc:276] Timestamp: P: 1761706318852940 usec, L: 0
1029 02:52:01.862904 (+ 128us) write_op.cc:270] Start()
1029 02:52:01.862924 (+ 20us) write_op.cc:276] Timestamp: P: 1761706318861607 usec, L: 0
1029 02:52:01.863052 (+ 128us) write_op.cc:270] Start()
1029 02:52:01.863072 (+ 20us) write_op.cc:276] Timestamp: P: 1761706318872323 usec, L: 0
1029 02:52:01.863211 (+ 139us) write_op.cc:270] Start()
1029 02:52:01.863230 (+ 19us) write_op.cc:276] Timestamp: P: 1761706318882173 usec, L: 0
1029 02:52:01.863361 (+ 131us) write_op.cc:270] Start()
1029 02:52:01.863382 (+ 21us) write_op.cc:276] Timestamp: P: 1761706318892571 usec, L: 0
1029 02:52:01.863507 (+ 125us) write_op.cc:270] Start()
1029 02:52:01.863527 (+ 20us) write_op.cc:276] Timestamp: P: 1761706318921842 usec, L: 0
1029 02:52:01.863668 (+ 141us) write_op.cc:270] Start()
1029 02:52:01.863694 (+ 26us) write_op.cc:276] Timestamp: P: 1761706318932331 usec, L: 0
1029 02:52:01.863853 (+ 159us) write_op.cc:270] Start()
1029 02:52:01.863875 (+ 22us) write_op.cc:276] Timestamp: P: 1761706318941515 usec, L: 0
1029 02:52:01.864022 (+ 147us) write_op.cc:270] Start()
1029 02:52:01.864044 (+ 22us) write_op.cc:276] Timestamp: P: 1761706318960533 usec, L: 0
1029 02:52:01.864192 (+ 148us) write_op.cc:270] Start()
1029 02:52:01.864212 (+ 20us) write_op.cc:276] Timestamp: P: 1761706318966970 usec, L: 0
1029 02:52:01.864353 (+ 141us) write_op.cc:270] Start()
1029 02:52:01.864372 (+ 19us) write_op.cc:276] Timestamp: P: 1761706318979087 usec, L: 0
1029 02:52:01.864503 (+ 131us) write_op.cc:270] Start()
1029 02:52:01.864525 (+ 22us) write_op.cc:276] Timestamp: P: 1761706318979760 usec, L: 0
1029 02:52:01.864662 (+ 137us) write_op.cc:270] Start()
1029 02:52:01.864684 (+ 22us) write_op.cc:276] Timestamp: P: 1761706318980330 usec, L: 0
1029 02:52:01.864843 (+ 159us) write_op.cc:270] Start()
1029 02:52:01.864865 (+ 22us) write_op.cc:276] Timestamp: P: 1761706318984326 usec, L: 0
1029 02:52:01.865030 (+ 165us) write_op.cc:270] Start()
1029 02:52:01.865053 (+ 23us) write_op.cc:276] Timestamp: P: 1761706319002948 usec, L: 0
1029 02:52:01.865188 (+ 135us) write_op.cc:270] Start()
1029 02:52:01.865209 (+ 21us) write_op.cc:276] Timestamp: P: 1761706319037139 usec, L: 0
1029 02:52:01.865345 (+ 136us) write_op.cc:270] Start()
1029 02:52:01.865364 (+ 19us) write_op.cc:276] Timestamp: P: 1761706319037788 usec, L: 0
1029 02:52:01.865484 (+ 120us) write_op.cc:270] Start()
1029 02:52:01.865504 (+ 20us) write_op.cc:276] Timestamp: P: 17617063
I20251029 02:52:02.675063 1175 ts_tablet_manager.cc:1950] T dbc2092b63e0417997565c8fadb53ae0 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:02.675699 1166 tablet_bootstrap.cc:492] T a5c3a15510cb4321bbb80d61fabd0c0a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:02.675872 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 543) took 2150 ms. Trace:
I20251029 02:52:02.675920 1166 ts_tablet_manager.cc:1403] T a5c3a15510cb4321bbb80d61fabd0c0a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.000s
I20251029 02:52:02.676010 1175 rpcz_store.cc:276] 1029 02:52:00.525630 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:00.526325 (+ 695us) service_pool.cc:225] Handling call
1029 02:52:02.633513 (+2107188us) ts_tablet_manager.cc:1151] Deleting tablet dbc2092b63e0417997565c8fadb53ae0
1029 02:52:02.633521 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.665059 (+ 31538us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:02.668316 (+ 3257us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:02.675841 (+ 7525us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2097034}
I20251029 02:52:02.676455 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 558) took 1754 ms. Trace:
I20251029 02:52:02.676689 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet dbc2092b63e0417997565c8fadb53ae0 (table default.Table_a873546f69334d12b0cbecce2cd489fb [id=1358e2055c254030a51e2de446eb742f]) successfully deleted
I20251029 02:52:02.676723 1166 raft_consensus.cc:359] T a5c3a15510cb4321bbb80d61fabd0c0a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:02.676923 1166 raft_consensus.cc:385] T a5c3a15510cb4321bbb80d61fabd0c0a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:02.677006 1166 raft_consensus.cc:740] T a5c3a15510cb4321bbb80d61fabd0c0a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:02.676959 1175 rpcz_store.cc:276] 1029 02:52:00.922356 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:00.922400 (+ 44us) service_pool.cc:225] Handling call
1029 02:52:02.676318 (+1753918us) ts_tablet_manager.cc:1151] Deleting tablet 2b278db5b2b04344b864797055a5f665
1029 02:52:02.676328 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.676439 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1749600}
I20251029 02:52:02.677273 1166 consensus_queue.cc:260] T a5c3a15510cb4321bbb80d61fabd0c0a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
W20251029 02:52:02.677511 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 2b278db5b2b04344b864797055a5f665 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2b278db5b2b04344b864797055a5f665 already in progress: creating tablet
I20251029 02:52:02.677899 1166 ts_tablet_manager.cc:1434] T a5c3a15510cb4321bbb80d61fabd0c0a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 02:52:02.678274 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 556) took 1756 ms. Trace:
I20251029 02:52:02.678369 1175 rpcz_store.cc:276] 1029 02:52:00.922057 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:00.923952 (+ 1895us) service_pool.cc:225] Handling call
1029 02:52:02.678154 (+1754202us) ts_tablet_manager.cc:1151] Deleting tablet d31e548feae24b739087c1e211d514f9
1029 02:52:02.678163 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.678257 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1750077}
I20251029 02:52:02.678609 1172 tablet_bootstrap.cc:492] T 9e6eb099648549dbbf9969e59865f51f P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:02.678737 1172 ts_tablet_manager.cc:1403] T 9e6eb099648549dbbf9969e59865f51f P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.001s
W20251029 02:52:02.678881 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet d31e548feae24b739087c1e211d514f9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d31e548feae24b739087c1e211d514f9 already in progress: creating tablet
I20251029 02:52:02.679315 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 555) took 1784 ms. Trace:
I20251029 02:52:02.679410 1175 rpcz_store.cc:276] 1029 02:52:00.894581 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:00.896888 (+ 2307us) service_pool.cc:225] Handling call
1029 02:52:02.679190 (+1782302us) ts_tablet_manager.cc:1151] Deleting tablet 2089b058d0e94d99b40ad2718518353e
1029 02:52:02.679199 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.679300 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1741267}
I20251029 02:52:02.679488 1172 raft_consensus.cc:359] T 9e6eb099648549dbbf9969e59865f51f P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:02.679658 1172 raft_consensus.cc:385] T 9e6eb099648549dbbf9969e59865f51f P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:02.679723 1172 raft_consensus.cc:740] T 9e6eb099648549dbbf9969e59865f51f P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:02.679883 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 561) took 1741 ms. Trace:
W20251029 02:52:02.679873 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 2089b058d0e94d99b40ad2718518353e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2089b058d0e94d99b40ad2718518353e already in progress: creating tablet
I20251029 02:52:02.679929 1172 consensus_queue.cc:260] T 9e6eb099648549dbbf9969e59865f51f P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:02.679971 1175 rpcz_store.cc:276] 1029 02:52:00.938891 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:00.939083 (+ 192us) service_pool.cc:225] Handling call
1029 02:52:02.679698 (+1740615us) ts_tablet_manager.cc:1151] Deleting tablet 44ae2b07edca44afbe1b1fb3a5b1115d
1029 02:52:02.679707 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.679827 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1740426}
I20251029 02:52:02.680406 1172 ts_tablet_manager.cc:1434] T 9e6eb099648549dbbf9969e59865f51f P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251029 02:52:02.680430 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 560) took 1743 ms. Trace:
I20251029 02:52:02.680856 1167 tablet_bootstrap.cc:492] T 6cd7009e06a646acad0802918c2b425e P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
W20251029 02:52:02.680878 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 44ae2b07edca44afbe1b1fb3a5b1115d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 44ae2b07edca44afbe1b1fb3a5b1115d already in progress: creating tablet
I20251029 02:52:02.681078 1167 ts_tablet_manager.cc:1403] T 6cd7009e06a646acad0802918c2b425e P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.048s user 0.004s sys 0.000s
I20251029 02:52:02.681262 1166 tablet_bootstrap.cc:492] T 2242f5b2c20346d9b753d7cd29242441 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:02.681871 1167 raft_consensus.cc:359] T 6cd7009e06a646acad0802918c2b425e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:02.682096 1167 raft_consensus.cc:385] T 6cd7009e06a646acad0802918c2b425e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:02.682185 1175 rpcz_store.cc:276] 1029 02:52:00.937429 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:00.955249 (+ 17820us) service_pool.cc:225] Handling call
1029 02:52:02.680328 (+1725079us) ts_tablet_manager.cc:1151] Deleting tablet d29cc2a0c8b84e3ebadc7f571d666847
1029 02:52:02.680337 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.680416 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1724723}
I20251029 02:52:02.682547 1175 tablet_replica.cc:333] T 576c4e6e70e24b0399ac9597a30f0569 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:02.682751 1175 raft_consensus.cc:2243] T 576c4e6e70e24b0399ac9597a30f0569 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
W20251029 02:52:02.682868 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet d29cc2a0c8b84e3ebadc7f571d666847 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d29cc2a0c8b84e3ebadc7f571d666847 already in progress: creating tablet
I20251029 02:52:02.682924 1175 raft_consensus.cc:2272] T 576c4e6e70e24b0399ac9597a30f0569 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:02.681563 1172 tablet_bootstrap.cc:492] T 4aa78f4ac38e4919864c924e50f1aaf4 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:02.683259 1167 raft_consensus.cc:740] T 6cd7009e06a646acad0802918c2b425e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:02.683511 1167 consensus_queue.cc:260] T 6cd7009e06a646acad0802918c2b425e P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:02.684767 1172 tablet_bootstrap.cc:654] T 4aa78f4ac38e4919864c924e50f1aaf4 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:02.685511 1167 ts_tablet_manager.cc:1434] T 6cd7009e06a646acad0802918c2b425e P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20251029 02:52:02.685727 1175 ts_tablet_manager.cc:1916] T 576c4e6e70e24b0399ac9597a30f0569 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:02.685856 1167 tablet_bootstrap.cc:492] T 8afc01d52d784f2fa559cab7ce07c8be P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:02.694352 1167 tablet_bootstrap.cc:654] T 8afc01d52d784f2fa559cab7ce07c8be P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:02.702737 1166 tablet_bootstrap.cc:654] T 2242f5b2c20346d9b753d7cd29242441 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:02.708370 1172 tablet_bootstrap.cc:492] T 4aa78f4ac38e4919864c924e50f1aaf4 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:02.708567 1172 ts_tablet_manager.cc:1403] T 4aa78f4ac38e4919864c924e50f1aaf4 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.027s user 0.005s sys 0.000s
I20251029 02:52:02.709494 1172 raft_consensus.cc:359] T 4aa78f4ac38e4919864c924e50f1aaf4 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:02.709668 1172 raft_consensus.cc:385] T 4aa78f4ac38e4919864c924e50f1aaf4 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:02.709723 1172 raft_consensus.cc:740] T 4aa78f4ac38e4919864c924e50f1aaf4 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:02.709977 1172 consensus_queue.cc:260] T 4aa78f4ac38e4919864c924e50f1aaf4 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:02.710619 1172 ts_tablet_manager.cc:1434] T 4aa78f4ac38e4919864c924e50f1aaf4 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20251029 02:52:02.710964 1172 tablet_bootstrap.cc:492] T 6ca9bd37de9e4be388501b15af4bda31 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:02.712622 1172 tablet_bootstrap.cc:654] T 6ca9bd37de9e4be388501b15af4bda31 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:02.719218 1175 ts_tablet_manager.cc:1929] T 576c4e6e70e24b0399ac9597a30f0569 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:02.719326 1175 log.cc:1199] T 576c4e6e70e24b0399ac9597a30f0569 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/576c4e6e70e24b0399ac9597a30f0569
I20251029 02:52:02.719686 1175 ts_tablet_manager.cc:1950] T 576c4e6e70e24b0399ac9597a30f0569 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:02.720629 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 563) took 1338 ms. Trace:
I20251029 02:52:02.720742 1175 rpcz_store.cc:276] 1029 02:52:01.382290 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:01.382552 (+ 262us) service_pool.cc:225] Handling call
1029 02:52:02.682484 (+1299932us) ts_tablet_manager.cc:1151] Deleting tablet 576c4e6e70e24b0399ac9597a30f0569
1029 02:52:02.682492 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.689153 (+ 6661us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:02.719116 (+ 29963us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:02.720604 (+ 1488us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1299657}
I20251029 02:52:02.721128 1175 tablet_replica.cc:333] T 1f5b40c0403d4a49a4bfde222d8c947a P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:02.721352 1175 raft_consensus.cc:2243] T 1f5b40c0403d4a49a4bfde222d8c947a P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:02.721331 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 576c4e6e70e24b0399ac9597a30f0569 (table default.Table_e8685346a99d41078ecd38cd4ce4fec5 [id=7774a12db22f414f9266236d92b3b511]) successfully deleted
I20251029 02:52:02.721529 1175 raft_consensus.cc:2272] T 1f5b40c0403d4a49a4bfde222d8c947a P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:02.722148 1175 ts_tablet_manager.cc:1916] T 1f5b40c0403d4a49a4bfde222d8c947a P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:02.722820 1167 tablet_bootstrap.cc:492] T 8afc01d52d784f2fa559cab7ce07c8be P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:02.723039 1167 ts_tablet_manager.cc:1403] T 8afc01d52d784f2fa559cab7ce07c8be P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.037s user 0.001s sys 0.004s
I20251029 02:52:02.723955 1167 raft_consensus.cc:359] T 8afc01d52d784f2fa559cab7ce07c8be P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:02.724202 1167 raft_consensus.cc:385] T 8afc01d52d784f2fa559cab7ce07c8be P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:02.724334 1167 raft_consensus.cc:740] T 8afc01d52d784f2fa559cab7ce07c8be P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:02.724596 1167 consensus_queue.cc:260] T 8afc01d52d784f2fa559cab7ce07c8be P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:02.726635 1167 ts_tablet_manager.cc:1434] T 8afc01d52d784f2fa559cab7ce07c8be P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251029 02:52:02.726967 1167 tablet_bootstrap.cc:492] T faa835de84ff45b19525d1d603678987 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:02.728686 1167 tablet_bootstrap.cc:654] T faa835de84ff45b19525d1d603678987 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:02.732156 1665 raft_consensus.cc:493] T c9e923724ca9402682f13b252dbc37aa P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:02.732287 1665 raft_consensus.cc:515] T c9e923724ca9402682f13b252dbc37aa P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:02.732895 1665 leader_election.cc:290] T c9e923724ca9402682f13b252dbc37aa P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:02.734118 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9e923724ca9402682f13b252dbc37aa" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:52:02.734921 712 leader_election.cc:343] T c9e923724ca9402682f13b252dbc37aa P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:02.733776 918 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9e923724ca9402682f13b252dbc37aa" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:02.740367 711 leader_election.cc:343] T c9e923724ca9402682f13b252dbc37aa P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:02.740480 711 leader_election.cc:304] T c9e923724ca9402682f13b252dbc37aa P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:02.740840 1665 raft_consensus.cc:2749] T c9e923724ca9402682f13b252dbc37aa P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:02.744686 1175 ts_tablet_manager.cc:1929] T 1f5b40c0403d4a49a4bfde222d8c947a P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:02.749317 1166 tablet_bootstrap.cc:492] T 2242f5b2c20346d9b753d7cd29242441 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:02.749506 1166 ts_tablet_manager.cc:1403] T 2242f5b2c20346d9b753d7cd29242441 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.069s user 0.012s sys 0.008s
I20251029 02:52:02.756350 1166 raft_consensus.cc:359] T 2242f5b2c20346d9b753d7cd29242441 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:02.756577 1166 raft_consensus.cc:385] T 2242f5b2c20346d9b753d7cd29242441 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:02.756669 1166 raft_consensus.cc:740] T 2242f5b2c20346d9b753d7cd29242441 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:02.756906 1166 consensus_queue.cc:260] T 2242f5b2c20346d9b753d7cd29242441 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:02.762480 1167 tablet_bootstrap.cc:492] T faa835de84ff45b19525d1d603678987 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:02.762725 1167 ts_tablet_manager.cc:1403] T faa835de84ff45b19525d1d603678987 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.036s user 0.004s sys 0.000s
I20251029 02:52:02.763617 1166 ts_tablet_manager.cc:1434] T 2242f5b2c20346d9b753d7cd29242441 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.014s user 0.001s sys 0.001s
I20251029 02:52:02.763562 1167 raft_consensus.cc:359] T faa835de84ff45b19525d1d603678987 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:02.751456 1172 tablet_bootstrap.cc:492] T 6ca9bd37de9e4be388501b15af4bda31 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:02.763779 1167 raft_consensus.cc:385] T faa835de84ff45b19525d1d603678987 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:02.763852 1167 raft_consensus.cc:740] T faa835de84ff45b19525d1d603678987 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:02.763952 1172 ts_tablet_manager.cc:1403] T 6ca9bd37de9e4be388501b15af4bda31 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.053s user 0.005s sys 0.002s
I20251029 02:52:02.763988 1166 tablet_bootstrap.cc:492] T faa835de84ff45b19525d1d603678987 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:02.763968 1175 log.cc:1199] T 1f5b40c0403d4a49a4bfde222d8c947a P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/1f5b40c0403d4a49a4bfde222d8c947a
I20251029 02:52:02.764098 1167 consensus_queue.cc:260] T faa835de84ff45b19525d1d603678987 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:02.764520 1175 ts_tablet_manager.cc:1950] T 1f5b40c0403d4a49a4bfde222d8c947a P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:02.765646 1166 tablet_bootstrap.cc:654] T faa835de84ff45b19525d1d603678987 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:02.764773 1172 raft_consensus.cc:359] T 6ca9bd37de9e4be388501b15af4bda31 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:02.765844 1172 raft_consensus.cc:385] T 6ca9bd37de9e4be388501b15af4bda31 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:02.765923 1172 raft_consensus.cc:740] T 6ca9bd37de9e4be388501b15af4bda31 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:02.766157 1172 consensus_queue.cc:260] T 6ca9bd37de9e4be388501b15af4bda31 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:02.766669 1172 ts_tablet_manager.cc:1434] T 6ca9bd37de9e4be388501b15af4bda31 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251029 02:52:02.766816 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 564) took 1384 ms. Trace:
I20251029 02:52:02.766942 1175 rpcz_store.cc:276] 1029 02:52:01.382427 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:01.382869 (+ 442us) service_pool.cc:225] Handling call
1029 02:52:02.721068 (+1338199us) ts_tablet_manager.cc:1151] Deleting tablet 1f5b40c0403d4a49a4bfde222d8c947a
1029 02:52:02.721081 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.741554 (+ 20473us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:02.744603 (+ 3049us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:02.766795 (+ 22192us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1337984}
I20251029 02:52:02.766953 1172 tablet_bootstrap.cc:492] T d5741dde463a43d6a776ab92dcfb04bf P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:02.767969 1665 raft_consensus.cc:493] T 6bd3fac2502e479aa9f4ceaabbb39504 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:02.768167 1665 raft_consensus.cc:515] T 6bd3fac2502e479aa9f4ceaabbb39504 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:02.768793 1665 leader_election.cc:290] T 6bd3fac2502e479aa9f4ceaabbb39504 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:02.769668 1172 tablet_bootstrap.cc:654] T d5741dde463a43d6a776ab92dcfb04bf P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:02.769814 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 1f5b40c0403d4a49a4bfde222d8c947a (table default.Table_72f8739f5f654172b10d619a9ac306bd [id=1565d03ae28649e48c2a614a8d067227]) successfully deleted
I20251029 02:52:02.769897 1175 tablet_replica.cc:333] T 24cb2a71ba3841d4a5e3e5c6fe99ae2f P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:02.770179 1175 raft_consensus.cc:2243] T 24cb2a71ba3841d4a5e3e5c6fe99ae2f P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:02.770380 1167 ts_tablet_manager.cc:1434] T faa835de84ff45b19525d1d603678987 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20251029 02:52:02.770622 1175 raft_consensus.cc:2272] T 24cb2a71ba3841d4a5e3e5c6fe99ae2f P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:02.770802 1167 tablet_bootstrap.cc:492] T 1fc87e2d7b4240e08f2b4ac7277293ff P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:02.771199 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6bd3fac2502e479aa9f4ceaabbb39504" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:02.771358 1175 ts_tablet_manager.cc:1916] T 24cb2a71ba3841d4a5e3e5c6fe99ae2f P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:02.770750 918 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6bd3fac2502e479aa9f4ceaabbb39504" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:02.771754 712 leader_election.cc:343] T 6bd3fac2502e479aa9f4ceaabbb39504 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
W20251029 02:52:02.772159 711 leader_election.cc:343] T 6bd3fac2502e479aa9f4ceaabbb39504 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:02.772300 711 leader_election.cc:304] T 6bd3fac2502e479aa9f4ceaabbb39504 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:02.772794 1665 raft_consensus.cc:2749] T 6bd3fac2502e479aa9f4ceaabbb39504 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:02.773087 1167 tablet_bootstrap.cc:654] T 1fc87e2d7b4240e08f2b4ac7277293ff P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:02.777118 1665 raft_consensus.cc:493] T cedfb53078d94c81b5e294ce7438325e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:02.777279 1665 raft_consensus.cc:515] T cedfb53078d94c81b5e294ce7438325e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:02.778079 1665 leader_election.cc:290] T cedfb53078d94c81b5e294ce7438325e P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:52:02.779210 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cedfb53078d94c81b5e294ce7438325e" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:52:02.779709 712 leader_election.cc:343] T cedfb53078d94c81b5e294ce7438325e P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:02.779042 918 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cedfb53078d94c81b5e294ce7438325e" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:02.781875 711 leader_election.cc:343] T cedfb53078d94c81b5e294ce7438325e P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:02.781978 711 leader_election.cc:304] T cedfb53078d94c81b5e294ce7438325e P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:02.782326 1665 raft_consensus.cc:2749] T cedfb53078d94c81b5e294ce7438325e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:02.785266 1172 tablet_bootstrap.cc:492] T d5741dde463a43d6a776ab92dcfb04bf P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:02.785408 1172 ts_tablet_manager.cc:1403] T d5741dde463a43d6a776ab92dcfb04bf P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.005s
I20251029 02:52:02.786676 1172 raft_consensus.cc:359] T d5741dde463a43d6a776ab92dcfb04bf P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:02.786841 1172 raft_consensus.cc:385] T d5741dde463a43d6a776ab92dcfb04bf P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:02.786895 1172 raft_consensus.cc:740] T d5741dde463a43d6a776ab92dcfb04bf P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:02.787114 1172 consensus_queue.cc:260] T d5741dde463a43d6a776ab92dcfb04bf P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:02.787776 1172 ts_tablet_manager.cc:1434] T d5741dde463a43d6a776ab92dcfb04bf P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 02:52:02.788070 1172 tablet_bootstrap.cc:492] T c764062d3ba04d4ba8b2e367ad8ed433 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:02.789667 1172 tablet_bootstrap.cc:654] T c764062d3ba04d4ba8b2e367ad8ed433 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:02.793166 1167 tablet_bootstrap.cc:492] T 1fc87e2d7b4240e08f2b4ac7277293ff P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:02.793390 1167 ts_tablet_manager.cc:1403] T 1fc87e2d7b4240e08f2b4ac7277293ff P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.023s user 0.006s sys 0.000s
I20251029 02:52:02.794360 1167 raft_consensus.cc:359] T 1fc87e2d7b4240e08f2b4ac7277293ff P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:02.794624 1167 raft_consensus.cc:385] T 1fc87e2d7b4240e08f2b4ac7277293ff P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:02.794749 1167 raft_consensus.cc:740] T 1fc87e2d7b4240e08f2b4ac7277293ff P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:02.795044 1167 consensus_queue.cc:260] T 1fc87e2d7b4240e08f2b4ac7277293ff P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:02.795764 1167 ts_tablet_manager.cc:1434] T 1fc87e2d7b4240e08f2b4ac7277293ff P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 02:52:02.796089 1167 tablet_bootstrap.cc:492] T 6feec2c8e030447fae9c292a626dcd1d P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:02.797868 1167 tablet_bootstrap.cc:654] T 6feec2c8e030447fae9c292a626dcd1d P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:02.799896 1686 raft_consensus.cc:493] T 9964ad774890422f8e14f6b99f28cb55 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:02.800066 1686 raft_consensus.cc:515] T 9964ad774890422f8e14f6b99f28cb55 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:02.800668 1686 leader_election.cc:290] T 9964ad774890422f8e14f6b99f28cb55 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
W20251029 02:52:02.803364 990 leader_election.cc:343] T 9964ad774890422f8e14f6b99f28cb55 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Not found: Tablet not found: 9964ad774890422f8e14f6b99f28cb55
W20251029 02:52:02.803803 989 leader_election.cc:343] T 9964ad774890422f8e14f6b99f28cb55 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 9964ad774890422f8e14f6b99f28cb55
I20251029 02:52:02.803941 989 leader_election.cc:304] T 9964ad774890422f8e14f6b99f28cb55 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5; no voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:02.804378 1686 raft_consensus.cc:2749] T 9964ad774890422f8e14f6b99f28cb55 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:02.811506 1166 tablet_bootstrap.cc:492] T faa835de84ff45b19525d1d603678987 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:02.811782 1166 ts_tablet_manager.cc:1403] T faa835de84ff45b19525d1d603678987 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.048s user 0.002s sys 0.002s
I20251029 02:52:02.812693 1166 raft_consensus.cc:359] T faa835de84ff45b19525d1d603678987 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:02.812994 1166 raft_consensus.cc:385] T faa835de84ff45b19525d1d603678987 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:02.813163 1166 raft_consensus.cc:740] T faa835de84ff45b19525d1d603678987 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:02.813520 1167 tablet_bootstrap.cc:492] T 6feec2c8e030447fae9c292a626dcd1d P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:02.813520 1166 consensus_queue.cc:260] T faa835de84ff45b19525d1d603678987 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:02.813694 1167 ts_tablet_manager.cc:1403] T 6feec2c8e030447fae9c292a626dcd1d P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.001s
I20251029 02:52:02.814262 1166 ts_tablet_manager.cc:1434] T faa835de84ff45b19525d1d603678987 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 02:52:02.814471 1167 raft_consensus.cc:359] T 6feec2c8e030447fae9c292a626dcd1d P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:02.814817 1166 tablet_bootstrap.cc:492] T 3362e95063af41c8a24e935f547e3089 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:02.815457 1167 raft_consensus.cc:385] T 6feec2c8e030447fae9c292a626dcd1d P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:02.816445 1172 tablet_bootstrap.cc:492] T c764062d3ba04d4ba8b2e367ad8ed433 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:02.816642 1172 ts_tablet_manager.cc:1403] T c764062d3ba04d4ba8b2e367ad8ed433 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.004s
I20251029 02:52:02.817423 1172 raft_consensus.cc:359] T c764062d3ba04d4ba8b2e367ad8ed433 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:02.817641 1172 raft_consensus.cc:385] T c764062d3ba04d4ba8b2e367ad8ed433 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:02.817746 1172 raft_consensus.cc:740] T c764062d3ba04d4ba8b2e367ad8ed433 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:02.817991 1172 consensus_queue.cc:260] T c764062d3ba04d4ba8b2e367ad8ed433 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:02.818568 1172 ts_tablet_manager.cc:1434] T c764062d3ba04d4ba8b2e367ad8ed433 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 02:52:02.818728 1167 raft_consensus.cc:740] T 6feec2c8e030447fae9c292a626dcd1d P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:02.819063 1172 tablet_bootstrap.cc:492] T 0cc1957a845f4e9cb703a553b27d1347 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:02.820631 1172 tablet_bootstrap.cc:654] T 0cc1957a845f4e9cb703a553b27d1347 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:02.827555 1166 tablet_bootstrap.cc:654] T 3362e95063af41c8a24e935f547e3089 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:02.830883 1167 consensus_queue.cc:260] T 6feec2c8e030447fae9c292a626dcd1d P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:02.831717 1167 ts_tablet_manager.cc:1434] T 6feec2c8e030447fae9c292a626dcd1d P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.018s user 0.000s sys 0.003s
I20251029 02:52:02.832916 1167 tablet_bootstrap.cc:492] T 392a36a6b59a42f78e3ac7c8fb63214b P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:02.835309 1167 tablet_bootstrap.cc:654] T 392a36a6b59a42f78e3ac7c8fb63214b P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:02.836898 1686 raft_consensus.cc:493] T 1fc87e2d7b4240e08f2b4ac7277293ff P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:02.837116 1686 raft_consensus.cc:515] T 1fc87e2d7b4240e08f2b4ac7277293ff P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:02.837994 1686 leader_election.cc:290] T 1fc87e2d7b4240e08f2b4ac7277293ff P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:02.838343 918 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1fc87e2d7b4240e08f2b4ac7277293ff" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:02.838609 918 raft_consensus.cc:2468] T 1fc87e2d7b4240e08f2b4ac7277293ff P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 0.
I20251029 02:52:02.839054 990 leader_election.cc:304] T 1fc87e2d7b4240e08f2b4ac7277293ff P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters:
W20251029 02:52:02.839957 989 leader_election.cc:343] T 1fc87e2d7b4240e08f2b4ac7277293ff P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 1fc87e2d7b4240e08f2b4ac7277293ff
I20251029 02:52:02.840574 1686 raft_consensus.cc:2804] T 1fc87e2d7b4240e08f2b4ac7277293ff P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:02.840682 1686 raft_consensus.cc:493] T 1fc87e2d7b4240e08f2b4ac7277293ff P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:02.840798 1686 raft_consensus.cc:3060] T 1fc87e2d7b4240e08f2b4ac7277293ff P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:02.851384 1841 raft_consensus.cc:493] T 0a47781d036149569e1164dd938a45e9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:02.851495 1686 raft_consensus.cc:515] T 1fc87e2d7b4240e08f2b4ac7277293ff P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:02.852618 1686 leader_election.cc:290] T 1fc87e2d7b4240e08f2b4ac7277293ff P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Requested vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:02.852876 1841 raft_consensus.cc:515] T 0a47781d036149569e1164dd938a45e9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
W20251029 02:52:02.853471 989 leader_election.cc:343] T 1fc87e2d7b4240e08f2b4ac7277293ff P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 1fc87e2d7b4240e08f2b4ac7277293ff
I20251029 02:52:02.853698 1841 leader_election.cc:290] T 0a47781d036149569e1164dd938a45e9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:02.854178 774 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0a47781d036149569e1164dd938a45e9" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:02.854455 774 raft_consensus.cc:2393] T 0a47781d036149569e1164dd938a45e9 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in current term 1: Already voted for candidate c756528642a543eeacd93cef3e335a7d in this term.
I20251029 02:52:02.855625 1167 tablet_bootstrap.cc:492] T 392a36a6b59a42f78e3ac7c8fb63214b P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:02.855803 1167 ts_tablet_manager.cc:1403] T 392a36a6b59a42f78e3ac7c8fb63214b P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.023s user 0.000s sys 0.009s
I20251029 02:52:02.857564 1167 raft_consensus.cc:359] T 392a36a6b59a42f78e3ac7c8fb63214b P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:02.857817 1167 raft_consensus.cc:385] T 392a36a6b59a42f78e3ac7c8fb63214b P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:02.857913 1167 raft_consensus.cc:740] T 392a36a6b59a42f78e3ac7c8fb63214b P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:02.858189 1167 consensus_queue.cc:260] T 392a36a6b59a42f78e3ac7c8fb63214b P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:02.826936 1175 ts_tablet_manager.cc:1929] T 24cb2a71ba3841d4a5e3e5c6fe99ae2f P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:02.861256 1175 log.cc:1199] T 24cb2a71ba3841d4a5e3e5c6fe99ae2f P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/24cb2a71ba3841d4a5e3e5c6fe99ae2f
I20251029 02:52:02.861882 1175 ts_tablet_manager.cc:1950] T 24cb2a71ba3841d4a5e3e5c6fe99ae2f P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:02.864181 1167 ts_tablet_manager.cc:1434] T 392a36a6b59a42f78e3ac7c8fb63214b P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.008s user 0.002s sys 0.002s
I20251029 02:52:02.864351 526 master_service.cc:954] ReplaceTablet: received request to replace tablet f187dbd8a55b45b292ea61db43caf47d from {username='slave'} at 127.0.0.1:35314
I20251029 02:52:02.864509 1167 tablet_bootstrap.cc:492] T 2242f5b2c20346d9b753d7cd29242441 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:02.864784 1682 raft_consensus.cc:493] T 65fdec7cf5d143208d78d4c3806b1352 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:02.864918 1682 raft_consensus.cc:515] T 65fdec7cf5d143208d78d4c3806b1352 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:02.865613 1682 leader_election.cc:290] T 65fdec7cf5d143208d78d4c3806b1352 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:02.866153 1682 raft_consensus.cc:493] T 8afc01d52d784f2fa559cab7ce07c8be P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:02.866312 1682 raft_consensus.cc:515] T 8afc01d52d784f2fa559cab7ce07c8be P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:02.866878 1682 leader_election.cc:290] T 8afc01d52d784f2fa559cab7ce07c8be P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:02.866842 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "65fdec7cf5d143208d78d4c3806b1352" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:02.867092 1060 raft_consensus.cc:2468] T 65fdec7cf5d143208d78d4c3806b1352 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
W20251029 02:52:02.867935 847 leader_election.cc:343] T 8afc01d52d784f2fa559cab7ce07c8be P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 8afc01d52d784f2fa559cab7ce07c8be
I20251029 02:52:02.868330 849 leader_election.cc:304] T 65fdec7cf5d143208d78d4c3806b1352 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:02.868686 1682 raft_consensus.cc:2804] T 65fdec7cf5d143208d78d4c3806b1352 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:02.868772 1682 raft_consensus.cc:493] T 65fdec7cf5d143208d78d4c3806b1352 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:02.868791 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8afc01d52d784f2fa559cab7ce07c8be" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:02.868887 1682 raft_consensus.cc:3060] T 65fdec7cf5d143208d78d4c3806b1352 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
W20251029 02:52:02.868849 847 leader_election.cc:343] T 65fdec7cf5d143208d78d4c3806b1352 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 65fdec7cf5d143208d78d4c3806b1352
I20251029 02:52:02.869699 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1fc87e2d7b4240e08f2b4ac7277293ff" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
I20251029 02:52:02.869838 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0a47781d036149569e1164dd938a45e9" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:02.869903 917 raft_consensus.cc:3060] T 1fc87e2d7b4240e08f2b4ac7277293ff P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
W20251029 02:52:02.870501 849 leader_election.cc:343] T 8afc01d52d784f2fa559cab7ce07c8be P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:02.870668 849 leader_election.cc:304] T 8afc01d52d784f2fa559cab7ce07c8be P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9
I20251029 02:52:02.871089 990 leader_election.cc:304] T 0a47781d036149569e1164dd938a45e9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5; no voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:02.872120 1682 raft_consensus.cc:515] T 65fdec7cf5d143208d78d4c3806b1352 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:02.872701 1773 raft_consensus.cc:2749] T 8afc01d52d784f2fa559cab7ce07c8be P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:02.872784 1682 leader_election.cc:290] T 65fdec7cf5d143208d78d4c3806b1352 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:02.872906 1773 raft_consensus.cc:493] T 5695b5a7b1854c3983e2d4ab684898b0 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:02.873028 1773 raft_consensus.cc:515] T 5695b5a7b1854c3983e2d4ab684898b0 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:02.873150 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "65fdec7cf5d143208d78d4c3806b1352" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:52:02.873298 1060 raft_consensus.cc:3060] T 65fdec7cf5d143208d78d4c3806b1352 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:02.873659 1773 leader_election.cc:290] T 5695b5a7b1854c3983e2d4ab684898b0 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:02.873746 917 raft_consensus.cc:2468] T 1fc87e2d7b4240e08f2b4ac7277293ff P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 1.
I20251029 02:52:02.874507 1167 tablet_bootstrap.cc:654] T 2242f5b2c20346d9b753d7cd29242441 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:02.875422 1058 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5695b5a7b1854c3983e2d4ab684898b0" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:52:02.875949 849 leader_election.cc:343] T 5695b5a7b1854c3983e2d4ab684898b0 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
W20251029 02:52:02.876574 847 leader_election.cc:343] T 5695b5a7b1854c3983e2d4ab684898b0 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 5695b5a7b1854c3983e2d4ab684898b0
I20251029 02:52:02.876694 847 leader_election.cc:304] T 5695b5a7b1854c3983e2d4ab684898b0 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9
I20251029 02:52:02.877039 1774 raft_consensus.cc:2749] T 5695b5a7b1854c3983e2d4ab684898b0 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 02:52:02.877589 847 leader_election.cc:343] T 65fdec7cf5d143208d78d4c3806b1352 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 65fdec7cf5d143208d78d4c3806b1352
I20251029 02:52:02.877839 1060 raft_consensus.cc:2468] T 65fdec7cf5d143208d78d4c3806b1352 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:02.878296 849 leader_election.cc:304] T 65fdec7cf5d143208d78d4c3806b1352 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters: b495a49349324f7d81908f2da700f0d9
I20251029 02:52:02.878701 1841 raft_consensus.cc:3060] T 0a47781d036149569e1164dd938a45e9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:02.878732 1774 raft_consensus.cc:2804] T 65fdec7cf5d143208d78d4c3806b1352 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:02.878906 1774 raft_consensus.cc:697] T 65fdec7cf5d143208d78d4c3806b1352 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:52:02.879186 1774 consensus_queue.cc:237] T 65fdec7cf5d143208d78d4c3806b1352 P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:02.879495 1686 raft_consensus.cc:493] T a5c3a15510cb4321bbb80d61fabd0c0a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:02.879642 1686 raft_consensus.cc:515] T a5c3a15510cb4321bbb80d61fabd0c0a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:02.884733 520 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 628a7d0419cb434f8171faf500d585d8
I20251029 02:52:02.884740 990 leader_election.cc:304] T 1fc87e2d7b4240e08f2b4ac7277293ff P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters: b495a49349324f7d81908f2da700f0d9
W20251029 02:52:02.886364 521 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b41e114ed6e94f5895bdec9bee836d2d consensus state has no leader
I20251029 02:52:02.886323 756 tablet_service.cc:1552] Processing DeleteTablet for tablet 628a7d0419cb434f8171faf500d585d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:02 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:02.886693 521 master_service.cc:954] ReplaceTablet: received request to replace tablet a47821980eb4413b8af40eb5415b5d72 from {username='slave'} at 127.0.0.1:35314
I20251029 02:52:02.886690 884 tablet_service.cc:1552] Processing DeleteTablet for tablet 628a7d0419cb434f8171faf500d585d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:02 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:02.887440 1041 tablet_service.cc:1552] Processing DeleteTablet for tablet 628a7d0419cb434f8171faf500d585d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:02 UTC) from {username='slave'} at 127.0.0.1:37626
W20251029 02:52:02.887600 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 628a7d0419cb434f8171faf500d585d8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 628a7d0419cb434f8171faf500d585d8 already in progress: creating tablet
W20251029 02:52:02.888712 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 628a7d0419cb434f8171faf500d585d8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 628a7d0419cb434f8171faf500d585d8 already in progress: creating tablet
I20251029 02:52:02.889032 1839 raft_consensus.cc:2804] T 1fc87e2d7b4240e08f2b4ac7277293ff P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:02.889698 1839 raft_consensus.cc:697] T 1fc87e2d7b4240e08f2b4ac7277293ff P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Becoming Leader. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Running, Role: LEADER
I20251029 02:52:02.890012 1839 consensus_queue.cc:237] T 1fc87e2d7b4240e08f2b4ac7277293ff P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:02.891553 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_ca44b863eb7a446693602cd5053ab82b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:02.892313 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ca44b863eb7a446693602cd5053ab82b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:02.894018 526 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_244839a45b634a3b86f305b2c1e337c6" } new_table_name: "default.Table_92b1e1bb0c8e445aab37bb66b9dba064" modify_external_catalogs: true
I20251029 02:52:02.894240 523 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_743b2905866a4205b15f193d6d404b17" } modify_external_catalogs: true
I20251029 02:52:02.894430 523 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_743b2905866a4205b15f193d6d404b17" } modify_external_catalogs: true
I20251029 02:52:02.894796 520 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_691548aa805841a281c6b0bbff6af9cc" } modify_external_catalogs: true
I20251029 02:52:02.894878 518 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_7c3fa8fa8b3e4f3c91739b490c0e807e" } modify_external_catalogs: true
I20251029 02:52:02.894969 520 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_691548aa805841a281c6b0bbff6af9cc" } modify_external_catalogs: true
I20251029 02:52:02.895061 518 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_7c3fa8fa8b3e4f3c91739b490c0e807e" } modify_external_catalogs: true
I20251029 02:52:02.895665 517 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_64d89ed6a9af4c4880c3203fddf36126" } modify_external_catalogs: true
I20251029 02:52:02.895810 517 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_64d89ed6a9af4c4880c3203fddf36126" } modify_external_catalogs: true
I20251029 02:52:02.899123 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_1a03f74b2fc443fa914baee76bc046f9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:02.899878 756 tablet_service.cc:1552] Processing DeleteTablet for tablet b2b2d64dcf964da8be4b544dbbe31977 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:53 UTC) from {username='slave'} at 127.0.0.1:44580
W20251029 02:52:02.900319 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1a03f74b2fc443fa914baee76bc046f9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:02.900920 1841 raft_consensus.cc:2749] T 0a47781d036149569e1164dd938a45e9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:02.916249 1167 tablet_bootstrap.cc:492] T 2242f5b2c20346d9b753d7cd29242441 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:02.916451 1167 ts_tablet_manager.cc:1403] T 2242f5b2c20346d9b753d7cd29242441 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.052s user 0.003s sys 0.001s
I20251029 02:52:02.917261 1167 raft_consensus.cc:359] T 2242f5b2c20346d9b753d7cd29242441 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:02.917466 1167 raft_consensus.cc:385] T 2242f5b2c20346d9b753d7cd29242441 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:02.917534 1167 raft_consensus.cc:740] T 2242f5b2c20346d9b753d7cd29242441 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:02.917586 1841 raft_consensus.cc:493] T 44784a3a760946e0be24743bc2fc8138 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:02.917735 1841 raft_consensus.cc:515] T 44784a3a760946e0be24743bc2fc8138 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:02.917779 1167 consensus_queue.cc:260] T 2242f5b2c20346d9b753d7cd29242441 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:02.918561 1841 leader_election.cc:290] T 44784a3a760946e0be24743bc2fc8138 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
W20251029 02:52:02.919359 989 leader_election.cc:343] T 44784a3a760946e0be24743bc2fc8138 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 44784a3a760946e0be24743bc2fc8138
I20251029 02:52:02.919878 918 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "44784a3a760946e0be24743bc2fc8138" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:02.920074 918 raft_consensus.cc:2468] T 44784a3a760946e0be24743bc2fc8138 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 0.
I20251029 02:52:02.921267 990 leader_election.cc:304] T 44784a3a760946e0be24743bc2fc8138 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters: b495a49349324f7d81908f2da700f0d9
I20251029 02:52:02.923987 1167 ts_tablet_manager.cc:1434] T 2242f5b2c20346d9b753d7cd29242441 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20251029 02:52:02.925007 1167 tablet_bootstrap.cc:492] T a5c3a15510cb4321bbb80d61fabd0c0a P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:02.926865 1167 tablet_bootstrap.cc:654] T a5c3a15510cb4321bbb80d61fabd0c0a P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:02.936022 1841 raft_consensus.cc:2804] T 44784a3a760946e0be24743bc2fc8138 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:02.936147 1841 raft_consensus.cc:493] T 44784a3a760946e0be24743bc2fc8138 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:02.936223 1841 raft_consensus.cc:3060] T 44784a3a760946e0be24743bc2fc8138 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
W20251029 02:52:02.938552 989 leader_election.cc:343] T a5c3a15510cb4321bbb80d61fabd0c0a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: a5c3a15510cb4321bbb80d61fabd0c0a
I20251029 02:52:02.939211 1841 raft_consensus.cc:515] T 44784a3a760946e0be24743bc2fc8138 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:02.940007 1167 tablet_bootstrap.cc:492] T a5c3a15510cb4321bbb80d61fabd0c0a P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:02.940023 1841 leader_election.cc:290] T 44784a3a760946e0be24743bc2fc8138 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Requested vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:02.940176 1167 ts_tablet_manager.cc:1403] T a5c3a15510cb4321bbb80d61fabd0c0a P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.000s
W20251029 02:52:02.940668 989 leader_election.cc:343] T 44784a3a760946e0be24743bc2fc8138 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 44784a3a760946e0be24743bc2fc8138
I20251029 02:52:02.941171 1167 raft_consensus.cc:359] T a5c3a15510cb4321bbb80d61fabd0c0a P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:02.941412 1167 raft_consensus.cc:385] T a5c3a15510cb4321bbb80d61fabd0c0a P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:02.941531 1167 raft_consensus.cc:740] T a5c3a15510cb4321bbb80d61fabd0c0a P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:02.941802 1167 consensus_queue.cc:260] T a5c3a15510cb4321bbb80d61fabd0c0a P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:02.942376 1686 leader_election.cc:290] T a5c3a15510cb4321bbb80d61fabd0c0a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:52:02.945513 1167 ts_tablet_manager.cc:1434] T a5c3a15510cb4321bbb80d61fabd0c0a P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20251029 02:52:02.951011 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "44784a3a760946e0be24743bc2fc8138" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
I20251029 02:52:02.951166 1167 tablet_bootstrap.cc:492] T 3cded156380f45cc95893825c71ba0e9 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:02.951264 917 raft_consensus.cc:3060] T 44784a3a760946e0be24743bc2fc8138 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:02.953132 1167 tablet_bootstrap.cc:654] T 3cded156380f45cc95893825c71ba0e9 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
W20251029 02:52:02.954361 513 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet b41e114ed6e94f5895bdec9bee836d2d (table default.Table_18fc160d4f4a48c792cfda3695674974 [id=825464f47f3a45f79eb319c3ca3e54e3], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet b41e114ed6e94f5895bdec9bee836d2d consensus state has no leader
I20251029 02:52:02.957333 917 raft_consensus.cc:2468] T 44784a3a760946e0be24743bc2fc8138 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 1.
I20251029 02:52:02.959525 1665 raft_consensus.cc:493] T 9e6eb099648549dbbf9969e59865f51f P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:02.959694 1665 raft_consensus.cc:515] T 9e6eb099648549dbbf9969e59865f51f P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:02.959901 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a5c3a15510cb4321bbb80d61fabd0c0a" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:02.960079 917 raft_consensus.cc:2468] T a5c3a15510cb4321bbb80d61fabd0c0a P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 0.
I20251029 02:52:02.960431 1665 leader_election.cc:290] T 9e6eb099648549dbbf9969e59865f51f P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:02.961659 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e6eb099648549dbbf9969e59865f51f" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:02.961608 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e6eb099648549dbbf9969e59865f51f" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:02.962353 711 leader_election.cc:343] T 9e6eb099648549dbbf9969e59865f51f P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
W20251029 02:52:02.962738 712 leader_election.cc:343] T 9e6eb099648549dbbf9969e59865f51f P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:02.962857 712 leader_election.cc:304] T 9e6eb099648549dbbf9969e59865f51f P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:02.963310 1665 raft_consensus.cc:2749] T 9e6eb099648549dbbf9969e59865f51f P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:02.965855 1862 tablet.cc:1722] T aacb6f4b9284457bb6607e3a85d5c809 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251029 02:52:02.976569 1167 tablet_bootstrap.cc:492] T 3cded156380f45cc95893825c71ba0e9 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:02.977166 1167 ts_tablet_manager.cc:1403] T 3cded156380f45cc95893825c71ba0e9 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.026s user 0.000s sys 0.004s
I20251029 02:52:02.978246 1167 raft_consensus.cc:359] T 3cded156380f45cc95893825c71ba0e9 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:02.978535 1167 raft_consensus.cc:385] T 3cded156380f45cc95893825c71ba0e9 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:02.978618 1167 raft_consensus.cc:740] T 3cded156380f45cc95893825c71ba0e9 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:02.978868 1167 consensus_queue.cc:260] T 3cded156380f45cc95893825c71ba0e9 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:02.979378 1863 tablet.cc:1722] T aacb6f4b9284457bb6607e3a85d5c809 P c756528642a543eeacd93cef3e335a7d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251029 02:52:02.981966 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 562) took 1599 ms. Trace:
I20251029 02:52:02.982079 1175 rpcz_store.cc:276] 1029 02:52:01.382167 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:01.382481 (+ 314us) service_pool.cc:225] Handling call
1029 02:52:02.769822 (+1387341us) ts_tablet_manager.cc:1151] Deleting tablet 24cb2a71ba3841d4a5e3e5c6fe99ae2f
1029 02:52:02.769834 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:02.789141 (+ 19307us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:02.826827 (+ 37686us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:02.981926 (+155099us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1386564}
I20251029 02:52:02.982551 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 24cb2a71ba3841d4a5e3e5c6fe99ae2f (table default.Table_4350e92f14de4ba8a04afbe12ea7422a [id=226a2d412a7b40e59b6f5340b5a9b739]) successfully deleted
I20251029 02:52:02.982705 1167 ts_tablet_manager.cc:1434] T 3cded156380f45cc95893825c71ba0e9 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20251029 02:52:02.982766 1774 consensus_queue.cc:1048] T 0a47781d036149569e1164dd938a45e9 P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:02.983034 1167 tablet_bootstrap.cc:492] T 3362e95063af41c8a24e935f547e3089 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:02.983335 1774 raft_consensus.cc:493] T 6cd7009e06a646acad0802918c2b425e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:02.983487 1774 raft_consensus.cc:515] T 6cd7009e06a646acad0802918c2b425e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:02.984233 1774 leader_election.cc:290] T 6cd7009e06a646acad0802918c2b425e P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:02.984504 1058 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6cd7009e06a646acad0802918c2b425e" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:52:02.985013 849 leader_election.cc:343] T 6cd7009e06a646acad0802918c2b425e P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
W20251029 02:52:02.986058 847 leader_election.cc:343] T 6cd7009e06a646acad0802918c2b425e P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 6cd7009e06a646acad0802918c2b425e
I20251029 02:52:02.986186 847 leader_election.cc:304] T 6cd7009e06a646acad0802918c2b425e P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9
I20251029 02:52:02.986784 1774 raft_consensus.cc:2749] T 6cd7009e06a646acad0802918c2b425e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:02.988718 1167 tablet_bootstrap.cc:654] T 3362e95063af41c8a24e935f547e3089 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:02.988895 990 leader_election.cc:304] T 44784a3a760946e0be24743bc2fc8138 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters: b495a49349324f7d81908f2da700f0d9
I20251029 02:52:02.989231 1686 raft_consensus.cc:2804] T 44784a3a760946e0be24743bc2fc8138 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:02.989347 1686 raft_consensus.cc:697] T 44784a3a760946e0be24743bc2fc8138 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Becoming Leader. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Running, Role: LEADER
I20251029 02:52:02.989670 990 leader_election.cc:304] T a5c3a15510cb4321bbb80d61fabd0c0a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters: b495a49349324f7d81908f2da700f0d9
I20251029 02:52:02.989601 1686 consensus_queue.cc:237] T 44784a3a760946e0be24743bc2fc8138 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:02.990206 1839 raft_consensus.cc:2804] T a5c3a15510cb4321bbb80d61fabd0c0a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:02.990286 1839 raft_consensus.cc:493] T a5c3a15510cb4321bbb80d61fabd0c0a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:02.990368 1839 raft_consensus.cc:3060] T a5c3a15510cb4321bbb80d61fabd0c0a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:02.994015 1839 raft_consensus.cc:515] T a5c3a15510cb4321bbb80d61fabd0c0a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:02.995023 1839 leader_election.cc:290] T a5c3a15510cb4321bbb80d61fabd0c0a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Requested vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
W20251029 02:52:02.996003 989 leader_election.cc:343] T a5c3a15510cb4321bbb80d61fabd0c0a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: a5c3a15510cb4321bbb80d61fabd0c0a
W20251029 02:52:03.010329 526 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4e82c574dfa14ffbabf162a17355f841 consensus state has no leader
I20251029 02:52:03.010735 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_90cf8deabd5c47c1b162ce513d4c1fe4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:03.010103 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_021cde575a7a44f8a24392dcf558996a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:03.011319 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_90cf8deabd5c47c1b162ce513d4c1fe4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:52:03.011685 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_021cde575a7a44f8a24392dcf558996a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:03.014268 756 tablet_service.cc:1505] Processing CreateTablet for tablet ca7f7e8323f94ef1b106e14e6d87a602 (DEFAULT_TABLE table=default.Table_7c3fa8fa8b3e4f3c91739b490c0e807e [id=e8fd5755415743939ce563c0522be2a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:03.014743 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca7f7e8323f94ef1b106e14e6d87a602. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:03.016240 884 tablet_service.cc:1505] Processing CreateTablet for tablet ca7f7e8323f94ef1b106e14e6d87a602 (DEFAULT_TABLE table=default.Table_7c3fa8fa8b3e4f3c91739b490c0e807e [id=e8fd5755415743939ce563c0522be2a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:03.016650 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca7f7e8323f94ef1b106e14e6d87a602. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:03.018852 879 tablet_service.cc:1505] Processing CreateTablet for tablet 72e91c64a9b54cbda52ca9dfcee2ce22 (DEFAULT_TABLE table=default.Table_877abcb0d221493c8875368e1c75382b [id=6ea46a01a86140b9ae63e33b90b03dd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:03.019214 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72e91c64a9b54cbda52ca9dfcee2ce22. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:03.019568 895 tablet_service.cc:1505] Processing CreateTablet for tablet c47b7ad465c24c38a3777ff9f82223d3 (DEFAULT_TABLE table=default.Table_2499efd3a69d41bd8d6ddcb0987c98b4 [id=e49a1e567ce04ce197c7bfa20bf5ffab]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:03.019812 892 tablet_service.cc:1505] Processing CreateTablet for tablet 00bc50e2c9df4f2285d91ef2ef6452af (DEFAULT_TABLE table=default.Table_fd1d23f31ebf47b884897c68eac9004d [id=1c99ebd287ff47d08e817853a628256c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:03.019941 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c47b7ad465c24c38a3777ff9f82223d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:03.020148 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00bc50e2c9df4f2285d91ef2ef6452af. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:03.023061 891 tablet_service.cc:1505] Processing CreateTablet for tablet 4c17d21145e54cc78134cab37059ce1e (DEFAULT_TABLE table=default.Table_9bed90ff07fb4ea28bc7a94b6ebfc125 [id=06a02e7a32744fe7b512b4b8b49f82a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:03.023406 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c17d21145e54cc78134cab37059ce1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:03.025545 1041 tablet_service.cc:1505] Processing CreateTablet for tablet 72e91c64a9b54cbda52ca9dfcee2ce22 (DEFAULT_TABLE table=default.Table_877abcb0d221493c8875368e1c75382b [id=6ea46a01a86140b9ae63e33b90b03dd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:03.025913 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72e91c64a9b54cbda52ca9dfcee2ce22. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:03.030864 1032 tablet_service.cc:1505] Processing CreateTablet for tablet 4c17d21145e54cc78134cab37059ce1e (DEFAULT_TABLE table=default.Table_9bed90ff07fb4ea28bc7a94b6ebfc125 [id=06a02e7a32744fe7b512b4b8b49f82a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:03.031255 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c17d21145e54cc78134cab37059ce1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:03.034842 1028 tablet_service.cc:1505] Processing CreateTablet for tablet c47b7ad465c24c38a3777ff9f82223d3 (DEFAULT_TABLE table=default.Table_2499efd3a69d41bd8d6ddcb0987c98b4 [id=e49a1e567ce04ce197c7bfa20bf5ffab]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:03.035241 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c47b7ad465c24c38a3777ff9f82223d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:03.039278 1665 raft_consensus.cc:493] T f81aecb20d6e498bb9b498698886d65e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:03.039420 1665 raft_consensus.cc:515] T f81aecb20d6e498bb9b498698886d65e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:03.040001 1665 leader_election.cc:290] T f81aecb20d6e498bb9b498698886d65e P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:03.040398 742 tablet_service.cc:1505] Processing CreateTablet for tablet 72e91c64a9b54cbda52ca9dfcee2ce22 (DEFAULT_TABLE table=default.Table_877abcb0d221493c8875368e1c75382b [id=6ea46a01a86140b9ae63e33b90b03dd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:03.040755 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72e91c64a9b54cbda52ca9dfcee2ce22. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:03.041460 749 tablet_service.cc:1505] Processing CreateTablet for tablet c47b7ad465c24c38a3777ff9f82223d3 (DEFAULT_TABLE table=default.Table_2499efd3a69d41bd8d6ddcb0987c98b4 [id=e49a1e567ce04ce197c7bfa20bf5ffab]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:03.041841 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c47b7ad465c24c38a3777ff9f82223d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:03.042428 755 tablet_service.cc:1505] Processing CreateTablet for tablet 00bc50e2c9df4f2285d91ef2ef6452af (DEFAULT_TABLE table=default.Table_fd1d23f31ebf47b884897c68eac9004d [id=1c99ebd287ff47d08e817853a628256c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:03.042869 748 tablet_service.cc:1505] Processing CreateTablet for tablet 4c17d21145e54cc78134cab37059ce1e (DEFAULT_TABLE table=default.Table_9bed90ff07fb4ea28bc7a94b6ebfc125 [id=06a02e7a32744fe7b512b4b8b49f82a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:03.025568 1039 tablet_service.cc:1505] Processing CreateTablet for tablet 00bc50e2c9df4f2285d91ef2ef6452af (DEFAULT_TABLE table=default.Table_fd1d23f31ebf47b884897c68eac9004d [id=1c99ebd287ff47d08e817853a628256c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:03.043588 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00bc50e2c9df4f2285d91ef2ef6452af. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:03.051383 1682 consensus_queue.cc:1048] T 0a47781d036149569e1164dd938a45e9 P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20251029 02:52:03.052304 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00bc50e2c9df4f2285d91ef2ef6452af. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:03.054680 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c17d21145e54cc78134cab37059ce1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:03.063169 1665 raft_consensus.cc:493] T 6ca9bd37de9e4be388501b15af4bda31 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:03.063335 1665 raft_consensus.cc:515] T 6ca9bd37de9e4be388501b15af4bda31 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:03.064039 1665 leader_election.cc:290] T 6ca9bd37de9e4be388501b15af4bda31 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:03.071801 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f81aecb20d6e498bb9b498698886d65e" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:03.072228 1058 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ca9bd37de9e4be388501b15af4bda31" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:52:03.074608 712 leader_election.cc:343] T f81aecb20d6e498bb9b498698886d65e P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
W20251029 02:52:03.074836 712 leader_election.cc:343] T 6ca9bd37de9e4be388501b15af4bda31 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:03.083213 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ca9bd37de9e4be388501b15af4bda31" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:03.085736 1682 consensus_queue.cc:1048] T 078ace5eada044c88d2462efb2b57aa8 P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251029 02:52:03.086475 711 leader_election.cc:343] T 6ca9bd37de9e4be388501b15af4bda31 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:03.086629 711 leader_election.cc:304] T 6ca9bd37de9e4be388501b15af4bda31 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:03.088414 1665 raft_consensus.cc:2749] T 6ca9bd37de9e4be388501b15af4bda31 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:03.092752 1682 raft_consensus.cc:493] T fd0f89180aba43eabb20f926abccbaa6 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:03.092971 1682 raft_consensus.cc:515] T fd0f89180aba43eabb20f926abccbaa6 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:03.094292 1682 leader_election.cc:290] T fd0f89180aba43eabb20f926abccbaa6 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
W20251029 02:52:03.095523 847 leader_election.cc:343] T fd0f89180aba43eabb20f926abccbaa6 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: fd0f89180aba43eabb20f926abccbaa6
I20251029 02:52:03.098843 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd0f89180aba43eabb20f926abccbaa6" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:03.099093 1060 raft_consensus.cc:2468] T fd0f89180aba43eabb20f926abccbaa6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:03.100818 1665 raft_consensus.cc:493] T f246f6c202cf44d48c9f5e4f335202c5 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:03.100948 1665 raft_consensus.cc:515] T f246f6c202cf44d48c9f5e4f335202c5 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:03.101531 1665 leader_election.cc:290] T f246f6c202cf44d48c9f5e4f335202c5 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:03.102087 849 leader_election.cc:304] T fd0f89180aba43eabb20f926abccbaa6 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters: b495a49349324f7d81908f2da700f0d9
I20251029 02:52:03.102185 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f246f6c202cf44d48c9f5e4f335202c5" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:03.102828 711 leader_election.cc:343] T f246f6c202cf44d48c9f5e4f335202c5 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:03.103286 1682 raft_consensus.cc:2804] T fd0f89180aba43eabb20f926abccbaa6 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:03.103392 1682 raft_consensus.cc:493] T fd0f89180aba43eabb20f926abccbaa6 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:03.103482 1682 raft_consensus.cc:3060] T fd0f89180aba43eabb20f926abccbaa6 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:03.117082 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f81aecb20d6e498bb9b498698886d65e" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:03.122548 711 leader_election.cc:343] T f81aecb20d6e498bb9b498698886d65e P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:03.122731 711 leader_election.cc:304] T f81aecb20d6e498bb9b498698886d65e P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:03.123351 1665 raft_consensus.cc:2749] T f81aecb20d6e498bb9b498698886d65e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:03.125708 1773 consensus_queue.cc:1048] T 078ace5eada044c88d2462efb2b57aa8 P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:03.132455 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f246f6c202cf44d48c9f5e4f335202c5" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:52:03.133868 712 leader_election.cc:343] T f246f6c202cf44d48c9f5e4f335202c5 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:03.134039 712 leader_election.cc:304] T f246f6c202cf44d48c9f5e4f335202c5 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:03.134672 1665 raft_consensus.cc:2749] T f246f6c202cf44d48c9f5e4f335202c5 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 02:52:03.147055 416 external_mini_cluster.cc:1715] Unable to check leaks on /tmp/dist-test-taskYVokEG/build/asan/bin/kudu pid 424: Timed out: CheckLeaks RPC to 127.0.104.62:42343 timed out after 1.000s (SENT)
I20251029 02:52:03.147282 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskYVokEG/build/asan/bin/kudu with pid 424
I20251029 02:52:02.982626 1175 tablet_replica.cc:333] T 51effc9324034ae79e2b25cb57a996c1 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:03.148216 1175 raft_consensus.cc:2243] T 51effc9324034ae79e2b25cb57a996c1 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:03.148415 1175 raft_consensus.cc:2272] T 51effc9324034ae79e2b25cb57a996c1 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:03.148687 1682 raft_consensus.cc:515] T fd0f89180aba43eabb20f926abccbaa6 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:03.149061 1175 ts_tablet_manager.cc:1916] T 51effc9324034ae79e2b25cb57a996c1 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 02:52:03.156136 990 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:41701, user_credentials={real_user=slave}} blocked reactor thread for 166458us
I20251029 02:52:03.157929 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a5c3a15510cb4321bbb80d61fabd0c0a" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
I20251029 02:52:03.158329 917 raft_consensus.cc:3060] T a5c3a15510cb4321bbb80d61fabd0c0a P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:03.163743 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fd0f89180aba43eabb20f926abccbaa6" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:52:03.164001 1060 raft_consensus.cc:3060] T fd0f89180aba43eabb20f926abccbaa6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
W20251029 02:52:03.164165 847 leader_election.cc:343] T fd0f89180aba43eabb20f926abccbaa6 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: fd0f89180aba43eabb20f926abccbaa6
I20251029 02:52:03.167572 1060 raft_consensus.cc:2468] T fd0f89180aba43eabb20f926abccbaa6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:03.168051 849 leader_election.cc:304] T fd0f89180aba43eabb20f926abccbaa6 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters: b495a49349324f7d81908f2da700f0d9
I20251029 02:52:03.168818 1682 leader_election.cc:290] T fd0f89180aba43eabb20f926abccbaa6 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:03.172217 1172 tablet_bootstrap.cc:492] T 0cc1957a845f4e9cb703a553b27d1347 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:03.172230 917 raft_consensus.cc:2468] T a5c3a15510cb4321bbb80d61fabd0c0a P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 1.
I20251029 02:52:03.172411 1172 ts_tablet_manager.cc:1403] T 0cc1957a845f4e9cb703a553b27d1347 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.353s user 0.020s sys 0.049s
I20251029 02:52:03.172827 1773 raft_consensus.cc:2804] T fd0f89180aba43eabb20f926abccbaa6 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:03.173063 1773 raft_consensus.cc:697] T fd0f89180aba43eabb20f926abccbaa6 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:52:03.173477 1773 consensus_queue.cc:237] T fd0f89180aba43eabb20f926abccbaa6 P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:03.173233 1172 raft_consensus.cc:359] T 0cc1957a845f4e9cb703a553b27d1347 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:03.173856 1172 raft_consensus.cc:385] T 0cc1957a845f4e9cb703a553b27d1347 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:03.173951 1172 raft_consensus.cc:740] T 0cc1957a845f4e9cb703a553b27d1347 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:03.174180 1665 raft_consensus.cc:493] T f187dbd8a55b45b292ea61db43caf47d P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:03.174443 990 leader_election.cc:304] T a5c3a15510cb4321bbb80d61fabd0c0a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters: b495a49349324f7d81908f2da700f0d9
I20251029 02:52:03.174422 1665 raft_consensus.cc:515] T f187dbd8a55b45b292ea61db43caf47d P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:03.174816 1839 raft_consensus.cc:2804] T a5c3a15510cb4321bbb80d61fabd0c0a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:03.174989 1839 raft_consensus.cc:697] T a5c3a15510cb4321bbb80d61fabd0c0a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Becoming Leader. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Running, Role: LEADER
I20251029 02:52:03.175262 1839 consensus_queue.cc:237] T a5c3a15510cb4321bbb80d61fabd0c0a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:03.175657 1037 tablet_service.cc:1505] Processing CreateTablet for tablet ca7f7e8323f94ef1b106e14e6d87a602 (DEFAULT_TABLE table=default.Table_7c3fa8fa8b3e4f3c91739b490c0e807e [id=e8fd5755415743939ce563c0522be2a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:03.175725 1665 leader_election.cc:290] T f187dbd8a55b45b292ea61db43caf47d P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:03.176244 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f187dbd8a55b45b292ea61db43caf47d" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:52:03.176782 712 leader_election.cc:343] T f187dbd8a55b45b292ea61db43caf47d P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:03.177044 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca7f7e8323f94ef1b106e14e6d87a602. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:03.177083 918 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f187dbd8a55b45b292ea61db43caf47d" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:03.177464 1862 tablet.cc:1722] T d2990e9f626c40618c118266bc595684 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20251029 02:52:03.178047 711 leader_election.cc:343] T f187dbd8a55b45b292ea61db43caf47d P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:03.178237 711 leader_election.cc:304] T f187dbd8a55b45b292ea61db43caf47d P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:03.180202 1665 raft_consensus.cc:2749] T f187dbd8a55b45b292ea61db43caf47d P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 02:52:03.182821 1412 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.104.62:42343 timed out after 0.643s (SENT)
I20251029 02:52:03.185305 1863 tablet.cc:1722] T d2990e9f626c40618c118266bc595684 P c756528642a543eeacd93cef3e335a7d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 02:52:03.188648 522 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet 9e6eb099648549dbbf9969e59865f51f (table default.Table_683d67689da74097b69f31b3d9461fe2 [id=0c5d3e4d34d045dc99ffc88e101cae1d]). Expected version 1 got 0
I20251029 02:52:03.190282 522 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet d5741dde463a43d6a776ab92dcfb04bf (table default.Table_966779a0d37a4dc4ab3f89d2bf7c6b06 [id=08e3b2ebcf37415d9a2886ad5bbda068]). Expected version 1 got 0
I20251029 02:52:03.194217 522 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet c9e923724ca9402682f13b252dbc37aa (table default.Table_443f33587b8f428484b99e07355d69ca [id=03a25420ce6a46fc93a9fb753246bc7b]). Expected version 3 got 0
I20251029 02:52:03.194711 522 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet cedfb53078d94c81b5e294ce7438325e (table default.Table_e75db5cf8f5f49ea837cb6d63f4b7ebf [id=86195563fb4a4d7abd746f4ac7cd241c]). Expected version 1 got 0
I20251029 02:52:03.199730 522 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet da2a21c3236a4bd99872b89b12a36508 (table default.Table_2a63af6894694f16b45f4bf48f73a68f [id=fb42af08ec1f48498ee5e91cce62333e]). Expected version 5 got 0
I20251029 02:52:03.200096 520 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 1fa8cc0c60cd43cdaa7a7646f70cfb97
I20251029 02:52:03.200645 523 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 699ee74f39b342639ab84d0d65e53a70
I20251029 02:52:03.200994 525 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet aacb6f4b9284457bb6607e3a85d5c809 (table default.Table_bb4222ae83ec442f8f62fbb51e9ab8eb [id=9f95160639ad40e995a6c89e297a0fad]). Expected version 3 got 2
I20251029 02:52:03.201149 521 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet a47821980eb4413b8af40eb5415b5d72
I20251029 02:52:03.201293 523 master_service.cc:954] ReplaceTablet: received request to replace tablet 70e8cfe5eaa9492c85874dc3214a1e0b from {username='slave'} at 127.0.0.1:35314
I20251029 02:52:03.201524 521 catalog_manager.cc:6719] ReplaceTablet: tablet a47821980eb4413b8af40eb5415b5d72 deleted and replaced by tablet 932f69a7ac8c4a0eb2a436c6b3e6e83f
I20251029 02:52:03.201375 525 catalog_manager.cc:5649] T 078ace5eada044c88d2462efb2b57aa8 P c756528642a543eeacd93cef3e335a7d reported cstate change: term changed from 0 to 1, leader changed from <none> to c756528642a543eeacd93cef3e335a7d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "c756528642a543eeacd93cef3e335a7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } health_report { overall_health: UNKNOWN } } }
I20251029 02:52:03.201732 521 master_service.cc:954] ReplaceTablet: received request to replace tablet fcb3db8399aa4cf188b07dda220f7f5b from {username='slave'} at 127.0.0.1:35314
I20251029 02:52:03.202688 748 tablet_service.cc:1552] Processing DeleteTablet for tablet 1fa8cc0c60cd43cdaa7a7646f70cfb97 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:02 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:03.203636 879 tablet_service.cc:1552] Processing DeleteTablet for tablet 1fa8cc0c60cd43cdaa7a7646f70cfb97 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:02 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:03.203918 891 tablet_service.cc:1552] Processing DeleteTablet for tablet 699ee74f39b342639ab84d0d65e53a70 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:02 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:03.204149 884 tablet_service.cc:1552] Processing DeleteTablet for tablet a47821980eb4413b8af40eb5415b5d72 with delete_type TABLET_DATA_DELETED (replaced by tablet 932f69a7ac8c4a0eb2a436c6b3e6e83f) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:03.205005 1841 raft_consensus.cc:493] T 2242f5b2c20346d9b753d7cd29242441 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:03.205156 1841 raft_consensus.cc:515] T 2242f5b2c20346d9b753d7cd29242441 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:03.205698 748 tablet_service.cc:1552] Processing DeleteTablet for tablet 699ee74f39b342639ab84d0d65e53a70 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:02 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:03.206142 518 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet ca7f7e8323f94ef1b106e14e6d87a602
I20251029 02:52:03.206223 748 tablet_service.cc:1552] Processing DeleteTablet for tablet a47821980eb4413b8af40eb5415b5d72 with delete_type TABLET_DATA_DELETED (replaced by tablet 932f69a7ac8c4a0eb2a436c6b3e6e83f) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:03.206838 518 master_service.cc:954] ReplaceTablet: received request to replace tablet d1866d6dcb0349bba5db86163375cada from {username='slave'} at 127.0.0.1:35314
I20251029 02:52:03.208396 748 tablet_service.cc:1552] Processing DeleteTablet for tablet ca7f7e8323f94ef1b106e14e6d87a602 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:02 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:03.208658 520 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_4df27f8e1a5d4e1596b3c52a1f6ab58d" } new_table_name: "default.Table_23772dc0c0cf4ab08ba7f164661de878" modify_external_catalogs: true
I20251029 02:52:03.208690 1841 leader_election.cc:290] T 2242f5b2c20346d9b753d7cd29242441 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:03.211046 521 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_8b0c5038160542c49a0bac581c1189ca" } new_table_name: "default.Table_3800864035b741c29cf00bfe2b35a32f" modify_external_catalogs: true
W20251029 02:52:03.212721 989 leader_election.cc:343] T 2242f5b2c20346d9b753d7cd29242441 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 2242f5b2c20346d9b753d7cd29242441
I20251029 02:52:03.212919 1039 tablet_service.cc:1552] Processing DeleteTablet for tablet 1fa8cc0c60cd43cdaa7a7646f70cfb97 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:02 UTC) from {username='slave'} at 127.0.0.1:37626
W20251029 02:52:03.213282 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 1fa8cc0c60cd43cdaa7a7646f70cfb97 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1fa8cc0c60cd43cdaa7a7646f70cfb97 already in progress: creating tablet
I20251029 02:52:03.213465 884 tablet_service.cc:1552] Processing DeleteTablet for tablet ca7f7e8323f94ef1b106e14e6d87a602 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:02 UTC) from {username='slave'} at 127.0.0.1:35702
W20251029 02:52:03.213572 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 699ee74f39b342639ab84d0d65e53a70 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 699ee74f39b342639ab84d0d65e53a70 already in progress: creating tablet
W20251029 02:52:03.213776 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet a47821980eb4413b8af40eb5415b5d72 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a47821980eb4413b8af40eb5415b5d72 already in progress: creating tablet
I20251029 02:52:03.214941 1166 tablet_bootstrap.cc:492] T 3362e95063af41c8a24e935f547e3089 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
W20251029 02:52:03.215123 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 1fa8cc0c60cd43cdaa7a7646f70cfb97 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1fa8cc0c60cd43cdaa7a7646f70cfb97 already in progress: creating tablet
I20251029 02:52:03.215073 1039 tablet_service.cc:1552] Processing DeleteTablet for tablet ca7f7e8323f94ef1b106e14e6d87a602 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:02 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:03.215595 918 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2242f5b2c20346d9b753d7cd29242441" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:03.215857 513 catalog_manager.cc:4952] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet ca7f7e8323f94ef1b106e14e6d87a602 because the tablet was not found. No further retry: Not found: Tablet not found: ca7f7e8323f94ef1b106e14e6d87a602
I20251029 02:52:03.215896 918 raft_consensus.cc:2468] T 2242f5b2c20346d9b753d7cd29242441 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 0.
W20251029 02:52:03.217157 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet ca7f7e8323f94ef1b106e14e6d87a602 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ca7f7e8323f94ef1b106e14e6d87a602 already in progress: creating tablet
I20251029 02:52:03.215083 1166 ts_tablet_manager.cc:1403] T 3362e95063af41c8a24e935f547e3089 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.400s user 0.003s sys 0.003s
I20251029 02:52:03.219333 517 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet fe24183d24e34579b96e6fb6bc0a914e
I20251029 02:52:03.219798 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_51d1934e7b1e438ebea90a3ff5da2ed5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:03.220083 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_e24d2641727d40a79dd72860f4d9d272"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:03.220599 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e24d2641727d40a79dd72860f4d9d272 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:52:03.220621 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_51d1934e7b1e438ebea90a3ff5da2ed5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:03.221167 1037 tablet_service.cc:1552] Processing DeleteTablet for tablet fe24183d24e34579b96e6fb6bc0a914e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:02 UTC) from {username='slave'} at 127.0.0.1:37626
W20251029 02:52:03.222023 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet fe24183d24e34579b96e6fb6bc0a914e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fe24183d24e34579b96e6fb6bc0a914e already in progress: creating tablet
I20251029 02:52:03.222224 748 tablet_service.cc:1552] Processing DeleteTablet for tablet fe24183d24e34579b96e6fb6bc0a914e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:02 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:03.223273 884 tablet_service.cc:1552] Processing DeleteTablet for tablet fe24183d24e34579b96e6fb6bc0a914e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:02 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:03.216753 990 leader_election.cc:304] T 2242f5b2c20346d9b753d7cd29242441 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters: b495a49349324f7d81908f2da700f0d9
W20251029 02:52:03.223929 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet fe24183d24e34579b96e6fb6bc0a914e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fe24183d24e34579b96e6fb6bc0a914e already in progress: creating tablet
I20251029 02:52:03.224182 1841 raft_consensus.cc:2804] T 2242f5b2c20346d9b753d7cd29242441 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:03.224263 1841 raft_consensus.cc:493] T 2242f5b2c20346d9b753d7cd29242441 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:03.224325 1841 raft_consensus.cc:3060] T 2242f5b2c20346d9b753d7cd29242441 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:03.226028 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet a47821980eb4413b8af40eb5415b5d72 with delete_type TABLET_DATA_DELETED (replaced by tablet 932f69a7ac8c4a0eb2a436c6b3e6e83f) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:03.226658 1032 tablet_service.cc:1552] Processing DeleteTablet for tablet 699ee74f39b342639ab84d0d65e53a70 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:02 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:03.227469 1841 raft_consensus.cc:515] T 2242f5b2c20346d9b753d7cd29242441 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:03.228134 1841 leader_election.cc:290] T 2242f5b2c20346d9b753d7cd29242441 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Requested vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
W20251029 02:52:03.229046 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet a47821980eb4413b8af40eb5415b5d72 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a47821980eb4413b8af40eb5415b5d72 already in progress: creating tablet
I20251029 02:52:03.229961 918 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2242f5b2c20346d9b753d7cd29242441" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
I20251029 02:52:03.230166 918 raft_consensus.cc:3060] T 2242f5b2c20346d9b753d7cd29242441 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
W20251029 02:52:03.231279 989 leader_election.cc:343] T 2242f5b2c20346d9b753d7cd29242441 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 2242f5b2c20346d9b753d7cd29242441
W20251029 02:52:03.231933 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 699ee74f39b342639ab84d0d65e53a70 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 699ee74f39b342639ab84d0d65e53a70 already in progress: creating tablet
I20251029 02:52:03.223816 1166 raft_consensus.cc:359] T 3362e95063af41c8a24e935f547e3089 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:03.234155 1166 raft_consensus.cc:385] T 3362e95063af41c8a24e935f547e3089 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:03.234288 1166 raft_consensus.cc:740] T 3362e95063af41c8a24e935f547e3089 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:03.234673 1166 consensus_queue.cc:260] T 3362e95063af41c8a24e935f547e3089 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:03.235620 1166 ts_tablet_manager.cc:1434] T 3362e95063af41c8a24e935f547e3089 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
I20251029 02:52:03.235703 1878 tablet.cc:1722] T d2990e9f626c40618c118266bc595684 P b495a49349324f7d81908f2da700f0d9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 02:52:03.237119 1877 tablet.cc:1722] T aacb6f4b9284457bb6607e3a85d5c809 P b495a49349324f7d81908f2da700f0d9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251029 02:52:03.236721 1166 tablet_bootstrap.cc:492] T 1bab90fe904944af948dab3db06b7e34 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
W20251029 02:52:03.238164 1412 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.62:42343, user_credentials={real_user=slave}} blocked reactor thread for 55404us
I20251029 02:52:03.243248 523 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 70e8cfe5eaa9492c85874dc3214a1e0b
I20251029 02:52:03.243842 523 catalog_manager.cc:6719] ReplaceTablet: tablet 70e8cfe5eaa9492c85874dc3214a1e0b deleted and replaced by tablet a2d08c252a934cf9a45d2f60e62f76aa
I20251029 02:52:03.244222 523 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_18fc160d4f4a48c792cfda3695674974" } new_table_name: "default.Table_2598d9a14a4e46539f1f775098559c84" modify_external_catalogs: true
I20251029 02:52:03.245469 1032 tablet_service.cc:1552] Processing DeleteTablet for tablet 70e8cfe5eaa9492c85874dc3214a1e0b with delete_type TABLET_DATA_DELETED (replaced by tablet a2d08c252a934cf9a45d2f60e62f76aa) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:03.245883 884 tablet_service.cc:1552] Processing DeleteTablet for tablet 70e8cfe5eaa9492c85874dc3214a1e0b with delete_type TABLET_DATA_DELETED (replaced by tablet a2d08c252a934cf9a45d2f60e62f76aa) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:03.245967 1176 tablet_replica.cc:333] T 70e8cfe5eaa9492c85874dc3214a1e0b P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:03.246146 1176 raft_consensus.cc:2243] T 70e8cfe5eaa9492c85874dc3214a1e0b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:03.246246 1174 tablet_replica.cc:333] T 70e8cfe5eaa9492c85874dc3214a1e0b P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:03.246344 1176 raft_consensus.cc:2272] T 70e8cfe5eaa9492c85874dc3214a1e0b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:03.246551 1174 raft_consensus.cc:2243] T 70e8cfe5eaa9492c85874dc3214a1e0b P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:03.246791 1174 raft_consensus.cc:2272] T 70e8cfe5eaa9492c85874dc3214a1e0b P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:03.246966 1176 ts_tablet_manager.cc:1916] T 70e8cfe5eaa9492c85874dc3214a1e0b P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:03.247416 1174 ts_tablet_manager.cc:1916] T 70e8cfe5eaa9492c85874dc3214a1e0b P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:03.243266 518 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet d1866d6dcb0349bba5db86163375cada
I20251029 02:52:03.249260 518 catalog_manager.cc:6719] ReplaceTablet: tablet d1866d6dcb0349bba5db86163375cada deleted and replaced by tablet 313b963f9d63437ba0dd32385b21f5a6
I20251029 02:52:03.249676 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_4d82d8b113274cf1841f2141772815e0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:03.255118 748 tablet_service.cc:1552] Processing DeleteTablet for tablet 70e8cfe5eaa9492c85874dc3214a1e0b with delete_type TABLET_DATA_DELETED (replaced by tablet a2d08c252a934cf9a45d2f60e62f76aa) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:03.255187 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet d1866d6dcb0349bba5db86163375cada with delete_type TABLET_DATA_DELETED (replaced by tablet 313b963f9d63437ba0dd32385b21f5a6) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:03.255525 748 tablet_service.cc:1552] Processing DeleteTablet for tablet d1866d6dcb0349bba5db86163375cada with delete_type TABLET_DATA_DELETED (replaced by tablet 313b963f9d63437ba0dd32385b21f5a6) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:03.256337 884 tablet_service.cc:1552] Processing DeleteTablet for tablet d1866d6dcb0349bba5db86163375cada with delete_type TABLET_DATA_DELETED (replaced by tablet 313b963f9d63437ba0dd32385b21f5a6) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:03.257058 1167 tablet_bootstrap.cc:492] T 3362e95063af41c8a24e935f547e3089 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:03.257222 1167 ts_tablet_manager.cc:1403] T 3362e95063af41c8a24e935f547e3089 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.274s user 0.010s sys 0.025s
I20251029 02:52:03.258406 1176 ts_tablet_manager.cc:1929] T 70e8cfe5eaa9492c85874dc3214a1e0b P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20251029 02:52:03.258621 1176 log.cc:1199] T 70e8cfe5eaa9492c85874dc3214a1e0b P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/70e8cfe5eaa9492c85874dc3214a1e0b
I20251029 02:52:03.258956 1665 raft_consensus.cc:493] T c764062d3ba04d4ba8b2e367ad8ed433 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:03.259094 1665 raft_consensus.cc:515] T c764062d3ba04d4ba8b2e367ad8ed433 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:03.259467 1176 ts_tablet_manager.cc:1950] T 70e8cfe5eaa9492c85874dc3214a1e0b P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
W20251029 02:52:03.259735 521 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d7abaf84ba5d445aa17433b3be891548 consensus state has no leader
I20251029 02:52:03.259810 1665 leader_election.cc:290] T c764062d3ba04d4ba8b2e367ad8ed433 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:03.260167 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_18581b0529cb451d9bda8bf2c84a8144"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:03.260632 1166 tablet_bootstrap.cc:654] T 1bab90fe904944af948dab3db06b7e34 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:03.262627 1058 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c764062d3ba04d4ba8b2e367ad8ed433" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:03.264302 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 70e8cfe5eaa9492c85874dc3214a1e0b (table default.Table_9bf4e25d441f4d4dbd6651386eec512e [id=82df85af001340efb570040e047fa136]) successfully deleted
W20251029 02:52:03.265450 712 leader_election.cc:343] T c764062d3ba04d4ba8b2e367ad8ed433 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
W20251029 02:52:03.266635 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet d1866d6dcb0349bba5db86163375cada because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d1866d6dcb0349bba5db86163375cada already in progress: creating tablet
I20251029 02:52:03.267889 519 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_e0a153971f6f4af98ee02f8bbc615620" } modify_external_catalogs: true
I20251029 02:52:03.268095 519 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_e0a153971f6f4af98ee02f8bbc615620" } modify_external_catalogs: true
I20251029 02:52:03.267277 1174 ts_tablet_manager.cc:1929] T 70e8cfe5eaa9492c85874dc3214a1e0b P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20251029 02:52:03.270126 1174 log.cc:1199] T 70e8cfe5eaa9492c85874dc3214a1e0b P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/70e8cfe5eaa9492c85874dc3214a1e0b
I20251029 02:52:03.270988 1174 ts_tablet_manager.cc:1950] T 70e8cfe5eaa9492c85874dc3214a1e0b P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
W20251029 02:52:03.272356 520 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9269622ca92645d5aa8e73341201ccbb consensus state has no leader
I20251029 02:52:03.272872 520 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_3b26d01c7bce4d678c1b65f1753e3662" } new_table_name: "default.Table_ad2ac3ba98784d68a94eb4eb03e530a4" modify_external_catalogs: true
I20251029 02:52:03.273334 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 70e8cfe5eaa9492c85874dc3214a1e0b (table default.Table_9bf4e25d441f4d4dbd6651386eec512e [id=82df85af001340efb570040e047fa136]) successfully deleted
W20251029 02:52:03.274207 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet d1866d6dcb0349bba5db86163375cada because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d1866d6dcb0349bba5db86163375cada already in progress: creating tablet
I20251029 02:52:03.275882 1665 raft_consensus.cc:493] T d5741dde463a43d6a776ab92dcfb04bf P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:03.276089 1665 raft_consensus.cc:515] T d5741dde463a43d6a776ab92dcfb04bf P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:03.277047 1665 leader_election.cc:290] T d5741dde463a43d6a776ab92dcfb04bf P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:03.277801 1058 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d5741dde463a43d6a776ab92dcfb04bf" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:03.277900 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d5741dde463a43d6a776ab92dcfb04bf" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:03.174165 1172 consensus_queue.cc:260] T 0cc1957a845f4e9cb703a553b27d1347 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
W20251029 02:52:03.278465 711 leader_election.cc:343] T d5741dde463a43d6a776ab92dcfb04bf P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
W20251029 02:52:03.278915 712 leader_election.cc:343] T d5741dde463a43d6a776ab92dcfb04bf P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:03.279145 712 leader_election.cc:304] T d5741dde463a43d6a776ab92dcfb04bf P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:03.279345 1172 ts_tablet_manager.cc:1434] T 0cc1957a845f4e9cb703a553b27d1347 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.107s user 0.001s sys 0.002s
W20251029 02:52:03.279985 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_18581b0529cb451d9bda8bf2c84a8144 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:03.281301 1665 raft_consensus.cc:2749] T d5741dde463a43d6a776ab92dcfb04bf P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 02:52:03.281769 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4d82d8b113274cf1841f2141772815e0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:03.290870 1166 tablet_bootstrap.cc:492] T 1bab90fe904944af948dab3db06b7e34 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:03.291038 1166 ts_tablet_manager.cc:1403] T 1bab90fe904944af948dab3db06b7e34 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.054s user 0.010s sys 0.000s
I20251029 02:52:03.292042 1166 raft_consensus.cc:359] T 1bab90fe904944af948dab3db06b7e34 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:03.292245 1166 raft_consensus.cc:385] T 1bab90fe904944af948dab3db06b7e34 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:03.292304 1166 raft_consensus.cc:740] T 1bab90fe904944af948dab3db06b7e34 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:03.292502 1166 consensus_queue.cc:260] T 1bab90fe904944af948dab3db06b7e34 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:03.292986 1166 ts_tablet_manager.cc:1434] T 1bab90fe904944af948dab3db06b7e34 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251029 02:52:03.295840 1166 tablet_bootstrap.cc:492] T 8afc01d52d784f2fa559cab7ce07c8be P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:03.297647 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c764062d3ba04d4ba8b2e367ad8ed433" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:03.298355 1172 tablet_bootstrap.cc:492] T 519754f5d8f7484480f973c53708c5f3 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:03.298465 1166 tablet_bootstrap.cc:654] T 8afc01d52d784f2fa559cab7ce07c8be P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
W20251029 02:52:03.307152 711 leader_election.cc:343] T c764062d3ba04d4ba8b2e367ad8ed433 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:03.307403 711 leader_election.cc:304] T c764062d3ba04d4ba8b2e367ad8ed433 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:03.307915 1665 raft_consensus.cc:2749] T c764062d3ba04d4ba8b2e367ad8ed433 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:03.257992 1167 raft_consensus.cc:359] T 3362e95063af41c8a24e935f547e3089 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:03.309854 1167 raft_consensus.cc:385] T 3362e95063af41c8a24e935f547e3089 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:03.310019 1167 raft_consensus.cc:740] T 3362e95063af41c8a24e935f547e3089 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:03.310732 1167 consensus_queue.cc:260] T 3362e95063af41c8a24e935f547e3089 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:03.311374 1167 ts_tablet_manager.cc:1434] T 3362e95063af41c8a24e935f547e3089 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.054s user 0.001s sys 0.001s
I20251029 02:52:03.315435 1166 tablet_bootstrap.cc:492] T 8afc01d52d784f2fa559cab7ce07c8be P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:03.315601 1166 ts_tablet_manager.cc:1403] T 8afc01d52d784f2fa559cab7ce07c8be P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.000s
I20251029 02:52:03.316733 1166 raft_consensus.cc:359] T 8afc01d52d784f2fa559cab7ce07c8be P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:03.316913 1166 raft_consensus.cc:385] T 8afc01d52d784f2fa559cab7ce07c8be P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:03.316967 1166 raft_consensus.cc:740] T 8afc01d52d784f2fa559cab7ce07c8be P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:03.317198 1166 consensus_queue.cc:260] T 8afc01d52d784f2fa559cab7ce07c8be P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:03.317683 1166 ts_tablet_manager.cc:1434] T 8afc01d52d784f2fa559cab7ce07c8be P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 02:52:03.317970 1166 tablet_bootstrap.cc:492] T c65469173762446e9725262ae998dbf3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
W20251029 02:52:03.318818 523 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b41e114ed6e94f5895bdec9bee836d2d consensus state has no leader
I20251029 02:52:03.319231 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_aa561368dec842f2a73780eef725fd8d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:03.319698 1166 tablet_bootstrap.cc:654] T c65469173762446e9725262ae998dbf3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
W20251029 02:52:03.319832 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aa561368dec842f2a73780eef725fd8d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:03.323782 1172 tablet_bootstrap.cc:654] T 519754f5d8f7484480f973c53708c5f3 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:03.323912 1167 tablet_bootstrap.cc:492] T c65469173762446e9725262ae998dbf3 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:03.324554 918 raft_consensus.cc:2468] T 2242f5b2c20346d9b753d7cd29242441 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 1.
I20251029 02:52:03.329459 990 leader_election.cc:304] T 2242f5b2c20346d9b753d7cd29242441 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters: b495a49349324f7d81908f2da700f0d9
I20251029 02:52:03.330037 1839 raft_consensus.cc:2804] T 2242f5b2c20346d9b753d7cd29242441 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:03.330212 1839 raft_consensus.cc:697] T 2242f5b2c20346d9b753d7cd29242441 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Becoming Leader. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Running, Role: LEADER
I20251029 02:52:03.330494 1839 consensus_queue.cc:237] T 2242f5b2c20346d9b753d7cd29242441 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:03.339701 1175 ts_tablet_manager.cc:1929] T 51effc9324034ae79e2b25cb57a996c1 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:03.345063 1175 log.cc:1199] T 51effc9324034ae79e2b25cb57a996c1 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/51effc9324034ae79e2b25cb57a996c1
I20251029 02:52:03.345412 1166 tablet_bootstrap.cc:492] T c65469173762446e9725262ae998dbf3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:03.345592 1166 ts_tablet_manager.cc:1403] T c65469173762446e9725262ae998dbf3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.028s user 0.016s sys 0.004s
I20251029 02:52:03.347071 519 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 313b963f9d63437ba0dd32385b21f5a6
W20251029 02:52:03.347617 520 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 42cfdf88472b44979d86836561005ed1 consensus state has no leader
I20251029 02:52:03.347504 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_dd632ee443af4a59905aee8a89d0a4bb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:03.347941 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_de3500a0d5f64dd788359ae260ffd539"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:03.348102 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dd632ee443af4a59905aee8a89d0a4bb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:52:03.348426 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_de3500a0d5f64dd788359ae260ffd539 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:03.350687 1175 ts_tablet_manager.cc:1950] T 51effc9324034ae79e2b25cb57a996c1 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
W20251029 02:52:03.351516 847 consensus_peers.cc:597] T 69597d48631d43b6a5b92abbf60d5ed3 P c756528642a543eeacd93cef3e335a7d -> Peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Couldn't send request to peer b495a49349324f7d81908f2da700f0d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 69597d48631d43b6a5b92abbf60d5ed3. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:03.352098 1167 tablet_bootstrap.cc:654] T c65469173762446e9725262ae998dbf3 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:03.353956 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 565) took 1970 ms. Trace:
I20251029 02:52:03.354056 1175 rpcz_store.cc:276] 1029 02:52:01.383597 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:01.383650 (+ 53us) service_pool.cc:225] Handling call
1029 02:52:02.982569 (+1598919us) ts_tablet_manager.cc:1151] Deleting tablet 51effc9324034ae79e2b25cb57a996c1
1029 02:52:02.982581 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:03.254249 (+271668us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:03.339574 (+ 85325us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:03.353933 (+ 14359us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1598727}
I20251029 02:52:03.354532 1175 tablet_replica.cc:333] T 24016ceaa84147eca0b94b3ad288e772 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:03.354784 1175 raft_consensus.cc:2243] T 24016ceaa84147eca0b94b3ad288e772 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:03.354961 1175 raft_consensus.cc:2272] T 24016ceaa84147eca0b94b3ad288e772 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:03.355013 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 51effc9324034ae79e2b25cb57a996c1 (table default.Table_ee95bb03ab1246e8826f28083411050d [id=09fbf31759f047be924a5ef1f0030765]) successfully deleted
I20251029 02:52:03.355480 1175 ts_tablet_manager.cc:1916] T 24016ceaa84147eca0b94b3ad288e772 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:03.356637 1166 raft_consensus.cc:359] T c65469173762446e9725262ae998dbf3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:03.356839 1166 raft_consensus.cc:385] T c65469173762446e9725262ae998dbf3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:03.356886 1166 raft_consensus.cc:740] T c65469173762446e9725262ae998dbf3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:03.357095 1166 consensus_queue.cc:260] T c65469173762446e9725262ae998dbf3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:03.357621 1166 ts_tablet_manager.cc:1434] T c65469173762446e9725262ae998dbf3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20251029 02:52:03.357944 1166 tablet_bootstrap.cc:492] T 6feec2c8e030447fae9c292a626dcd1d P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:03.358296 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_8015f6519aff4d269aa0f16e8cdd96dd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:03.358922 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8015f6519aff4d269aa0f16e8cdd96dd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:03.359920 1172 tablet_bootstrap.cc:492] T 519754f5d8f7484480f973c53708c5f3 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:03.359965 1166 tablet_bootstrap.cc:654] T 6feec2c8e030447fae9c292a626dcd1d P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:03.360045 1172 ts_tablet_manager.cc:1403] T 519754f5d8f7484480f973c53708c5f3 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.062s user 0.004s sys 0.000s
I20251029 02:52:03.360941 1172 raft_consensus.cc:359] T 519754f5d8f7484480f973c53708c5f3 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:03.363296 1166 tablet_bootstrap.cc:492] T 6feec2c8e030447fae9c292a626dcd1d P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:03.363420 1166 ts_tablet_manager.cc:1403] T 6feec2c8e030447fae9c292a626dcd1d P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20251029 02:52:03.364136 1166 raft_consensus.cc:359] T 6feec2c8e030447fae9c292a626dcd1d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:03.364290 1166 raft_consensus.cc:385] T 6feec2c8e030447fae9c292a626dcd1d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:03.364334 1166 raft_consensus.cc:740] T 6feec2c8e030447fae9c292a626dcd1d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:03.364498 1166 consensus_queue.cc:260] T 6feec2c8e030447fae9c292a626dcd1d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:03.365124 1166 ts_tablet_manager.cc:1434] T 6feec2c8e030447fae9c292a626dcd1d P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 02:52:03.365382 1166 tablet_bootstrap.cc:492] T 5695b5a7b1854c3983e2d4ab684898b0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:03.361125 1172 raft_consensus.cc:385] T 519754f5d8f7484480f973c53708c5f3 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:03.366602 1172 raft_consensus.cc:740] T 519754f5d8f7484480f973c53708c5f3 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:03.366866 1172 consensus_queue.cc:260] T 519754f5d8f7484480f973c53708c5f3 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:03.367365 1166 tablet_bootstrap.cc:654] T 5695b5a7b1854c3983e2d4ab684898b0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:03.369213 1172 ts_tablet_manager.cc:1434] T 519754f5d8f7484480f973c53708c5f3 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20251029 02:52:03.369529 1172 tablet_bootstrap.cc:492] T dd6ad641fbda472fab07aa35c5e4523a P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:03.371275 1172 tablet_bootstrap.cc:654] T dd6ad641fbda472fab07aa35c5e4523a P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:03.374567 1167 tablet_bootstrap.cc:492] T c65469173762446e9725262ae998dbf3 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:03.374761 1167 ts_tablet_manager.cc:1403] T c65469173762446e9725262ae998dbf3 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.059s user 0.002s sys 0.002s
I20251029 02:52:03.376957 1166 tablet_bootstrap.cc:492] T 5695b5a7b1854c3983e2d4ab684898b0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:03.377104 1166 ts_tablet_manager.cc:1403] T 5695b5a7b1854c3983e2d4ab684898b0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.006s
I20251029 02:52:03.377837 1166 raft_consensus.cc:359] T 5695b5a7b1854c3983e2d4ab684898b0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:03.378007 1166 raft_consensus.cc:385] T 5695b5a7b1854c3983e2d4ab684898b0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:03.378055 1166 raft_consensus.cc:740] T 5695b5a7b1854c3983e2d4ab684898b0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:03.378262 1166 consensus_queue.cc:260] T 5695b5a7b1854c3983e2d4ab684898b0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:03.378760 1166 ts_tablet_manager.cc:1434] T 5695b5a7b1854c3983e2d4ab684898b0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 02:52:03.379204 1166 tablet_bootstrap.cc:492] T 392a36a6b59a42f78e3ac7c8fb63214b P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:03.379142 1167 raft_consensus.cc:359] T c65469173762446e9725262ae998dbf3 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:03.379326 1167 raft_consensus.cc:385] T c65469173762446e9725262ae998dbf3 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:03.379408 1167 raft_consensus.cc:740] T c65469173762446e9725262ae998dbf3 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:03.379657 1167 consensus_queue.cc:260] T c65469173762446e9725262ae998dbf3 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:03.380643 1167 ts_tablet_manager.cc:1434] T c65469173762446e9725262ae998dbf3 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251029 02:52:03.380867 1166 tablet_bootstrap.cc:654] T 392a36a6b59a42f78e3ac7c8fb63214b P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:03.380977 1167 tablet_bootstrap.cc:492] T 1bab90fe904944af948dab3db06b7e34 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:03.381224 1682 consensus_queue.cc:1048] T 69597d48631d43b6a5b92abbf60d5ed3 P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:03.388111 1167 tablet_bootstrap.cc:654] T 1bab90fe904944af948dab3db06b7e34 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:03.389117 1166 tablet_bootstrap.cc:492] T 392a36a6b59a42f78e3ac7c8fb63214b P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:03.389250 1166 ts_tablet_manager.cc:1403] T 392a36a6b59a42f78e3ac7c8fb63214b P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20251029 02:52:03.389901 1166 raft_consensus.cc:359] T 392a36a6b59a42f78e3ac7c8fb63214b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:03.390055 1166 raft_consensus.cc:385] T 392a36a6b59a42f78e3ac7c8fb63214b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:03.390096 1166 raft_consensus.cc:740] T 392a36a6b59a42f78e3ac7c8fb63214b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:03.390581 1166 consensus_queue.cc:260] T 392a36a6b59a42f78e3ac7c8fb63214b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:03.391728 1166 ts_tablet_manager.cc:1434] T 392a36a6b59a42f78e3ac7c8fb63214b P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 02:52:03.392007 1166 tablet_bootstrap.cc:492] T 6cd7009e06a646acad0802918c2b425e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:03.393795 1166 tablet_bootstrap.cc:654] T 6cd7009e06a646acad0802918c2b425e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:03.394083 518 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_3d0a6e32dc914ba2b0263680b2587277" } new_table_name: "default.Table_74a966e440094ced83dd557daa4c671a" modify_external_catalogs: true
I20251029 02:52:03.395401 1172 tablet_bootstrap.cc:492] T dd6ad641fbda472fab07aa35c5e4523a P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:03.395550 1172 ts_tablet_manager.cc:1403] T dd6ad641fbda472fab07aa35c5e4523a P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.000s
I20251029 02:52:03.396201 1175 ts_tablet_manager.cc:1929] T 24016ceaa84147eca0b94b3ad288e772 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251029 02:52:03.396324 1175 log.cc:1199] T 24016ceaa84147eca0b94b3ad288e772 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/24016ceaa84147eca0b94b3ad288e772
I20251029 02:52:03.396250 1172 raft_consensus.cc:359] T dd6ad641fbda472fab07aa35c5e4523a P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:03.396394 1172 raft_consensus.cc:385] T dd6ad641fbda472fab07aa35c5e4523a P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:03.396448 1172 raft_consensus.cc:740] T dd6ad641fbda472fab07aa35c5e4523a P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:03.396644 1172 consensus_queue.cc:260] T dd6ad641fbda472fab07aa35c5e4523a P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:03.398998 1172 ts_tablet_manager.cc:1434] T dd6ad641fbda472fab07aa35c5e4523a P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251029 02:52:03.399307 1172 tablet_bootstrap.cc:492] T cc74e83833754f0ca19046031a24ebb9 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:03.401100 1172 tablet_bootstrap.cc:654] T cc74e83833754f0ca19046031a24ebb9 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:03.401660 1175 ts_tablet_manager.cc:1950] T 24016ceaa84147eca0b94b3ad288e772 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:03.402483 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 566) took 1972 ms. Trace:
I20251029 02:52:03.403172 1166 tablet_bootstrap.cc:492] T 6cd7009e06a646acad0802918c2b425e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:03.403348 1166 ts_tablet_manager.cc:1403] T 6cd7009e06a646acad0802918c2b425e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20251029 02:52:03.403620 1175 rpcz_store.cc:276] 1029 02:52:01.430146 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:01.430211 (+ 65us) service_pool.cc:225] Handling call
1029 02:52:03.354482 (+1924271us) ts_tablet_manager.cc:1151] Deleting tablet 24016ceaa84147eca0b94b3ad288e772
1029 02:52:03.354492 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:03.375652 (+ 21160us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:03.396086 (+ 20434us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:03.402426 (+ 6340us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1924041}
I20251029 02:52:03.404212 1166 raft_consensus.cc:359] T 6cd7009e06a646acad0802918c2b425e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:03.404392 1166 raft_consensus.cc:385] T 6cd7009e06a646acad0802918c2b425e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:03.404456 1166 raft_consensus.cc:740] T 6cd7009e06a646acad0802918c2b425e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:03.404456 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 24016ceaa84147eca0b94b3ad288e772 (table default.Table_2662f20eb8d843b8bf6c841e49863c85 [id=bd279b3ce623477ca20e8d35e3b7c0df]) successfully deleted
I20251029 02:52:03.405628 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 567) took 1965 ms. Trace:
I20251029 02:52:03.405989 1175 rpcz_store.cc:276] 1029 02:52:01.439837 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:01.439903 (+ 66us) service_pool.cc:225] Handling call
1029 02:52:03.404261 (+1964358us) ts_tablet_manager.cc:1151] Deleting tablet 1df0d944717d487a98ea83a8ec04bd46
1029 02:52:03.404272 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:03.405570 (+ 1298us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1964129}
I20251029 02:52:03.407014 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 568) took 1963 ms. Trace:
W20251029 02:52:03.407063 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 1df0d944717d487a98ea83a8ec04bd46 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1df0d944717d487a98ea83a8ec04bd46 already in progress: creating tablet
I20251029 02:52:03.408077 1854 raft_consensus.cc:493] T 1bab90fe904944af948dab3db06b7e34 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:03.404666 1166 consensus_queue.cc:260] T 6cd7009e06a646acad0802918c2b425e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:03.408219 1854 raft_consensus.cc:515] T 1bab90fe904944af948dab3db06b7e34 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:03.408864 1854 leader_election.cc:290] T 1bab90fe904944af948dab3db06b7e34 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:52:03.408989 1166 ts_tablet_manager.cc:1434] T 6cd7009e06a646acad0802918c2b425e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20251029 02:52:03.409279 1166 tablet_bootstrap.cc:492] T 3cded156380f45cc95893825c71ba0e9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:03.410995 1166 tablet_bootstrap.cc:654] T 3cded156380f45cc95893825c71ba0e9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:03.412472 918 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1bab90fe904944af948dab3db06b7e34" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:03.413080 990 leader_election.cc:343] T 1bab90fe904944af948dab3db06b7e34 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:03.413846 1166 tablet_bootstrap.cc:492] T 3cded156380f45cc95893825c71ba0e9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:03.413965 1166 ts_tablet_manager.cc:1403] T 3cded156380f45cc95893825c71ba0e9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20251029 02:52:03.414611 1166 raft_consensus.cc:359] T 3cded156380f45cc95893825c71ba0e9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:03.414760 1166 raft_consensus.cc:385] T 3cded156380f45cc95893825c71ba0e9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:03.414834 1166 raft_consensus.cc:740] T 3cded156380f45cc95893825c71ba0e9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
W20251029 02:52:03.420388 989 leader_election.cc:343] T 1bab90fe904944af948dab3db06b7e34 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 1bab90fe904944af948dab3db06b7e34
I20251029 02:52:03.420543 989 leader_election.cc:304] T 1bab90fe904944af948dab3db06b7e34 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5; no voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:03.421196 1839 raft_consensus.cc:2749] T 1bab90fe904944af948dab3db06b7e34 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:03.421900 1166 consensus_queue.cc:260] T 3cded156380f45cc95893825c71ba0e9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:03.422415 1166 ts_tablet_manager.cc:1434] T 3cded156380f45cc95893825c71ba0e9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20251029 02:52:03.422712 1166 tablet_bootstrap.cc:492] T 7309d3190cfa415281c5253a634fa7f3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:03.424290 1166 tablet_bootstrap.cc:654] T 7309d3190cfa415281c5253a634fa7f3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:03.425577 526 catalog_manager.cc:5649] T 0a47781d036149569e1164dd938a45e9 P c756528642a543eeacd93cef3e335a7d reported cstate change: term changed from 0 to 1, leader changed from <none> to c756528642a543eeacd93cef3e335a7d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "c756528642a543eeacd93cef3e335a7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } health_report { overall_health: UNKNOWN } } }
I20251029 02:52:03.429399 1032 tablet_service.cc:1505] Processing CreateTablet for tablet dd6ca9ff8616413ab3ee86dfb4b605dd (DEFAULT_TABLE table=default.Table_1a03f74b2fc443fa914baee76bc046f9 [id=75dce05ee66345a9aa8120caa59956f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:03.429817 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd6ca9ff8616413ab3ee86dfb4b605dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:03.412899 1172 tablet_bootstrap.cc:492] T cc74e83833754f0ca19046031a24ebb9 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:03.438457 1172 ts_tablet_manager.cc:1403] T cc74e83833754f0ca19046031a24ebb9 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.039s user 0.004s sys 0.000s
I20251029 02:52:03.439289 1172 raft_consensus.cc:359] T cc74e83833754f0ca19046031a24ebb9 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:03.439515 1172 raft_consensus.cc:385] T cc74e83833754f0ca19046031a24ebb9 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:03.439608 1172 raft_consensus.cc:740] T cc74e83833754f0ca19046031a24ebb9 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:03.439859 1172 consensus_queue.cc:260] T cc74e83833754f0ca19046031a24ebb9 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:03.436481 1880 raft_consensus.cc:493] T 74873a05ef634867a30aa63b31fcfcc7 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:03.440312 1880 raft_consensus.cc:515] T 74873a05ef634867a30aa63b31fcfcc7 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:03.441006 1880 leader_election.cc:290] T 74873a05ef634867a30aa63b31fcfcc7 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:03.441610 918 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "74873a05ef634867a30aa63b31fcfcc7" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:03.442229 711 leader_election.cc:343] T 74873a05ef634867a30aa63b31fcfcc7 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:03.437074 1665 raft_consensus.cc:493] T daa05a34c0684c07a35966290fd7d44c P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:03.442898 1665 raft_consensus.cc:515] T daa05a34c0684c07a35966290fd7d44c P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:03.443490 1665 leader_election.cc:290] T daa05a34c0684c07a35966290fd7d44c P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:03.444169 1058 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "74873a05ef634867a30aa63b31fcfcc7" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:03.444418 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "daa05a34c0684c07a35966290fd7d44c" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:52:03.444970 712 leader_election.cc:343] T 74873a05ef634867a30aa63b31fcfcc7 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:03.445092 712 leader_election.cc:304] T 74873a05ef634867a30aa63b31fcfcc7 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
W20251029 02:52:03.445515 712 leader_election.cc:343] T daa05a34c0684c07a35966290fd7d44c P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:03.445448 918 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "daa05a34c0684c07a35966290fd7d44c" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:03.445731 1665 raft_consensus.cc:2749] T 74873a05ef634867a30aa63b31fcfcc7 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 02:52:03.446039 711 leader_election.cc:343] T daa05a34c0684c07a35966290fd7d44c P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:03.446201 711 leader_election.cc:304] T daa05a34c0684c07a35966290fd7d44c P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:03.446764 1665 raft_consensus.cc:2749] T daa05a34c0684c07a35966290fd7d44c P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:03.455690 884 tablet_service.cc:1505] Processing CreateTablet for tablet dd6ca9ff8616413ab3ee86dfb4b605dd (DEFAULT_TABLE table=default.Table_1a03f74b2fc443fa914baee76bc046f9 [id=75dce05ee66345a9aa8120caa59956f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:03.456182 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd6ca9ff8616413ab3ee86dfb4b605dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:03.458839 748 tablet_service.cc:1505] Processing CreateTablet for tablet dd6ca9ff8616413ab3ee86dfb4b605dd (DEFAULT_TABLE table=default.Table_1a03f74b2fc443fa914baee76bc046f9 [id=75dce05ee66345a9aa8120caa59956f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:03.459123 1028 tablet_service.cc:1505] Processing CreateTablet for tablet 18632de3bbbd491a86460497544580c1 (DEFAULT_TABLE table=default.Table_37dad1121b2144ee9eeb3ab7c38a0609 [id=be64f276351541b3b926db57c2af7378]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:03.459278 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dd6ca9ff8616413ab3ee86dfb4b605dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:03.461525 1032 tablet_service.cc:1505] Processing CreateTablet for tablet 4e89d72dd274498783dc815f6f2ee031 (DEFAULT_TABLE table=default.Table_94ed5141b07a4cd7b94c14174c30ed99 [id=75154e53c33948d2983f099237706894]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:03.462893 891 tablet_service.cc:1505] Processing CreateTablet for tablet 4e89d72dd274498783dc815f6f2ee031 (DEFAULT_TABLE table=default.Table_94ed5141b07a4cd7b94c14174c30ed99 [id=75154e53c33948d2983f099237706894]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:03.463330 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e89d72dd274498783dc815f6f2ee031. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:03.466886 742 tablet_service.cc:1505] Processing CreateTablet for tablet 4e89d72dd274498783dc815f6f2ee031 (DEFAULT_TABLE table=default.Table_94ed5141b07a4cd7b94c14174c30ed99 [id=75154e53c33948d2983f099237706894]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:03.467320 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e89d72dd274498783dc815f6f2ee031. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:03.469900 520 master_service.cc:954] ReplaceTablet: received request to replace tablet 38f83ebcb2a440bc8e6a8d36607e6666 from {username='slave'} at 127.0.0.1:35314
I20251029 02:52:03.470583 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_99f14277f09b47438fb0522c896dd4b4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:03.470732 879 tablet_service.cc:1505] Processing CreateTablet for tablet 18632de3bbbd491a86460497544580c1 (DEFAULT_TABLE table=default.Table_37dad1121b2144ee9eeb3ab7c38a0609 [id=be64f276351541b3b926db57c2af7378]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 02:52:03.471117 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_99f14277f09b47438fb0522c896dd4b4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:03.471151 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18632de3bbbd491a86460497544580c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:03.471408 1037 tablet_service.cc:1505] Processing CreateTablet for tablet 6dc30c614be64b8ba629d9884d450290 (DEFAULT_TABLE table=default.Table_35d17629e08e4b41bf6621b43343ed6b [id=69f28220165541a388a16c079fc88733]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:03.474443 892 tablet_service.cc:1505] Processing CreateTablet for tablet 6dc30c614be64b8ba629d9884d450290 (DEFAULT_TABLE table=default.Table_35d17629e08e4b41bf6621b43343ed6b [id=69f28220165541a388a16c079fc88733]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:03.474927 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6dc30c614be64b8ba629d9884d450290. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:03.474838 755 tablet_service.cc:1505] Processing CreateTablet for tablet 18632de3bbbd491a86460497544580c1 (DEFAULT_TABLE table=default.Table_37dad1121b2144ee9eeb3ab7c38a0609 [id=be64f276351541b3b926db57c2af7378]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:03.475821 523 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_9dfd24b5c34c4b9f965292b798351bc1" } modify_external_catalogs: true
I20251029 02:52:03.476022 523 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_9dfd24b5c34c4b9f965292b798351bc1" } modify_external_catalogs: true
I20251029 02:52:03.477330 749 tablet_service.cc:1505] Processing CreateTablet for tablet 6dc30c614be64b8ba629d9884d450290 (DEFAULT_TABLE table=default.Table_35d17629e08e4b41bf6621b43343ed6b [id=69f28220165541a388a16c079fc88733]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:03.479836 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18632de3bbbd491a86460497544580c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:03.481777 1039 tablet_service.cc:1505] Processing CreateTablet for tablet bf67a67854d048a49a96cccccf500ab6 (DEFAULT_TABLE table=default.Table_6f4e7e4f715747b492f4d5b8e3ab8c5a [id=7aa2aa99bf784acc9a14094068f39c2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:03.483182 895 tablet_service.cc:1505] Processing CreateTablet for tablet bf67a67854d048a49a96cccccf500ab6 (DEFAULT_TABLE table=default.Table_6f4e7e4f715747b492f4d5b8e3ab8c5a [id=7aa2aa99bf784acc9a14094068f39c2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:03.483516 519 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_0bb20faeb14644ca8e2d5acd4af09f85" } modify_external_catalogs: true
I20251029 02:52:03.483712 519 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_0bb20faeb14644ca8e2d5acd4af09f85" } modify_external_catalogs: true
I20251029 02:52:03.484273 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf67a67854d048a49a96cccccf500ab6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:03.485004 756 tablet_service.cc:1505] Processing CreateTablet for tablet bf67a67854d048a49a96cccccf500ab6 (DEFAULT_TABLE table=default.Table_6f4e7e4f715747b492f4d5b8e3ab8c5a [id=7aa2aa99bf784acc9a14094068f39c2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:03.486723 1041 tablet_service.cc:1505] Processing CreateTablet for tablet 9fc7d7c62ebc4ff29db9c1a925f7acfe (DEFAULT_TABLE table=default.Table_bc99b82ab1b84af899404ca0f6ec25ad [id=247a6298141541f484ddc896d8f8e655]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:03.487605 899 tablet_service.cc:1505] Processing CreateTablet for tablet 9fc7d7c62ebc4ff29db9c1a925f7acfe (DEFAULT_TABLE table=default.Table_bc99b82ab1b84af899404ca0f6ec25ad [id=247a6298141541f484ddc896d8f8e655]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:03.488004 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fc7d7c62ebc4ff29db9c1a925f7acfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:03.489329 521 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_f8c6e4111f9945dfb9e1a7dadc4f5381" } modify_external_catalogs: true
I20251029 02:52:03.489517 521 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_f8c6e4111f9945dfb9e1a7dadc4f5381" } modify_external_catalogs: true
I20251029 02:52:03.494702 1773 raft_consensus.cc:493] T 3cded156380f45cc95893825c71ba0e9 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:03.494892 1773 raft_consensus.cc:515] T 3cded156380f45cc95893825c71ba0e9 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:03.495613 1773 leader_election.cc:290] T 3cded156380f45cc95893825c71ba0e9 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
W20251029 02:52:03.496225 518 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 786d9148fb1d417a8c2c8b8c53665899 consensus state has no leader
I20251029 02:52:03.496508 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cded156380f45cc95893825c71ba0e9" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:03.496661 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_8746e5337ced40038d965fae5bcb4acd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:03.496728 1060 raft_consensus.cc:2468] T 3cded156380f45cc95893825c71ba0e9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
W20251029 02:52:03.497213 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8746e5337ced40038d965fae5bcb4acd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:03.497328 849 leader_election.cc:304] T 3cded156380f45cc95893825c71ba0e9 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:03.498871 1773 raft_consensus.cc:2804] T 3cded156380f45cc95893825c71ba0e9 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:03.498965 1773 raft_consensus.cc:493] T 3cded156380f45cc95893825c71ba0e9 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:03.499092 1773 raft_consensus.cc:3060] T 3cded156380f45cc95893825c71ba0e9 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:03.504369 524 catalog_manager.cc:6194] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: default.Table_bafe26b1fca14cc9b908984b7d8a2442 [id=046010a03d774e91b0ad10df97848c56] alter complete (version 1)
W20251029 02:52:03.505014 524 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.3:48627 (request call id 76) took 1329 ms (client timeout 1000 ms). Trace:
I20251029 02:52:03.505340 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet 9964ad774890422f8e14f6b99f28cb55 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:53 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:03.505628 1176 tablet_replica.cc:333] T 9964ad774890422f8e14f6b99f28cb55 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:03.505874 1176 raft_consensus.cc:2243] T 9964ad774890422f8e14f6b99f28cb55 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:03.506045 1176 raft_consensus.cc:2272] T 9964ad774890422f8e14f6b99f28cb55 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251029 02:52:03.505147 524 rpcz_store.cc:269] 1029 02:52:02.175138 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:02.843693 (+668555us) service_pool.cc:225] Handling call
1029 02:52:02.896045 (+ 52352us) spinlock_profiling.cc:243] Waited 51.8 ms on lock 0x616000041254. stack: 00007f4b1e4899f6 00007f4b1e4897b0 00007f4b1d4653ad 0000000000983f04 0000000000983ecd 00007f4b2d7906bd 00007f4b2d7907b3 00007f4b2a357441 00007f4b320119dd 00007f4b32198aa8 00007f4b28cabca6 00007f4b28cab9d5 00007f4b2542827e 00007f4b2542715e 00007f4b2542d395 00007f4b2542dcbf
1029 02:52:03.504939 (+608894us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1029 02:52:03.000576 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1029 02:52:03.000786 (+ 210us) write_op.cc:432] Acquiring schema lock in shared mode
1029 02:52:03.000795 (+ 9us) write_op.cc:435] Acquired schema lock
1029 02:52:03.000799 (+ 4us) tablet.cc:660] Decoding operations
1029 02:52:03.000893 (+ 94us) write_op.cc:620] Acquiring the partition lock for write op
1029 02:52:03.000904 (+ 11us) write_op.cc:641] Partition lock acquired for write op
1029 02:52:03.000912 (+ 8us) tablet.cc:683] Acquiring locks for 1 operations
1029 02:52:03.000970 (+ 58us) tablet.cc:699] Row locks acquired
1029 02:52:03.000974 (+ 4us) write_op.cc:260] PREPARE: finished
1029 02:52:03.001008 (+ 34us) write_op.cc:270] Start()
1029 02:52:03.001040 (+ 32us) write_op.cc:276] Timestamp: P: 1761706323000979 usec, L: 0
1029 02:52:03.001045 (+ 5us) op_driver.cc:348] REPLICATION: starting
1029 02:52:03.001214 (+ 169us) log.cc:844] Serialized 356 byte log entry
1029 02:52:03.008757 (+ 7543us) op_driver.cc:464] REPLICATION: finished
1029 02:52:03.008952 (+ 195us) write_op.cc:301] APPLY: starting
1029 02:52:03.008966 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
1029 02:52:03.008986 (+ 20us) tablet.cc:1368] finished BulkCheckPresence
1029 02:52:03.009006 (+ 20us) tablet.cc:1370] starting ApplyRowOperation cycle
1029 02:52:03.009060 (+ 54us) tablet.cc:1381] finished ApplyRowOperation cycle
1029 02:52:03.009085 (+ 25us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1029 02:52:03.009092 (+ 7us) write_op.cc:312] APPLY: finished
1029 02:52:03.009180 (+ 88us) log.cc:844] Serialized 23 byte log entry
1029 02:52:03.018645 (+ 9465us) write_op.cc:489] Releasing partition, row and schema locks
1029 02:52:03.018664 (+ 19us) write_op.cc:454] Released schema lock
1029 02:52:03.018670 (+ 6us) write_op.cc:341] FINISH: Updating metrics
1029 02:52:03.018736 (+ 66us) write_op.cc:489] Releasing partition, row and schema locks
1029 02:52:03.018739 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":2,"spinlock_wait_cycles":119088000,"child_traces":[["op",{"apply.queue_time_us":157,"apply.run_cpu_time_us":404,"apply.run_wall_time_us":9778,"mutex_wait_us":9386,"num_ops":1,"prepare.queue_time_us":103113,"prepare.run_cpu_time_us":702,"prepare.run_wall_time_us":702,"replication_time_us":7694,"spinlock_wait_cycles":768}]]}
I20251029 02:52:03.506719 1176 ts_tablet_manager.cc:1916] T 9964ad774890422f8e14f6b99f28cb55 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 02:52:03.508231 525 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.2:37633 (request call id 78) took 1324 ms (client timeout 1000 ms). Trace:
W20251029 02:52:03.508359 525 rpcz_store.cc:269] 1029 02:52:02.183651 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:02.845418 (+661767us) service_pool.cc:225] Handling call
1029 02:52:02.895131 (+ 49713us) spinlock_profiling.cc:243] Waited 49 ms on lock 0x616000041254. stack: 00007f4b1e4899f6 00007f4b1e4897b0 00007f4b1d4653ad 0000000000983f04 0000000000983ecd 00007f4b2d7906bd 00007f4b2d7907b3 00007f4b2a357441 00007f4b320119dd 00007f4b32198aa8 00007f4b28cabca6 00007f4b28cab9d5 00007f4b2542827e 00007f4b2542715e 00007f4b2542d395 00007f4b2542dcbf
1029 02:52:03.508159 (+613028us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1029 02:52:03.202909 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1029 02:52:03.203010 (+ 101us) write_op.cc:432] Acquiring schema lock in shared mode
1029 02:52:03.203015 (+ 5us) write_op.cc:435] Acquired schema lock
1029 02:52:03.203018 (+ 3us) tablet.cc:660] Decoding operations
1029 02:52:03.203105 (+ 87us) write_op.cc:620] Acquiring the partition lock for write op
1029 02:52:03.203114 (+ 9us) write_op.cc:641] Partition lock acquired for write op
1029 02:52:03.203118 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
1029 02:52:03.203170 (+ 52us) tablet.cc:699] Row locks acquired
1029 02:52:03.203173 (+ 3us) write_op.cc:260] PREPARE: finished
1029 02:52:03.203204 (+ 31us) write_op.cc:270] Start()
1029 02:52:03.203233 (+ 29us) write_op.cc:276] Timestamp: P: 1761706323203156 usec, L: 0
1029 02:52:03.203237 (+ 4us) op_driver.cc:348] REPLICATION: starting
1029 02:52:03.227508 (+ 24271us) log.cc:844] Serialized 527 byte log entry
1029 02:52:03.228247 (+ 739us) spinlock_profiling.cc:243] Waited 24 ms on lock 0x616000041254. stack: 00007f4b1e4899f6 00007f4b1e4897b0 00007f4b1d4653ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f4b2a329862 00007f4b2b70c2a5 00007f4b2b70adc9 00007f4b2b7131f0 00007f4b2b712ffd 0000000000a8867c 00007f4b1e4cfd7f 00007f4b1e4d304b 00007f4b1e4d2ec8 0000000000a8867c
1029 02:52:03.231849 (+ 3602us) op_driver.cc:464] REPLICATION: finished
1029 02:52:03.231976 (+ 127us) write_op.cc:301] APPLY: starting
1029 02:52:03.231994 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
1029 02:52:03.232039 (+ 45us) tablet.cc:1368] finished BulkCheckPresence
1029 02:52:03.232043 (+ 4us) tablet.cc:1370] starting ApplyRowOperation cycle
1029 02:52:03.232099 (+ 56us) tablet.cc:1381] finished ApplyRowOperation cycle
1029 02:52:03.232141 (+ 42us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1029 02:52:03.232153 (+ 12us) write_op.cc:312] APPLY: finished
1029 02:52:03.232228 (+ 75us) log.cc:844] Serialized 23 byte log entry
1029 02:52:03.232287 (+ 59us) write_op.cc:489] Releasing partition, row and schema locks
1029 02:52:03.232306 (+ 19us) write_op.cc:454] Released schema lock
1029 02:52:03.232314 (+ 8us) write_op.cc:341] FINISH: Updating metrics
1029 02:52:03.232395 (+ 81us) write_op.cc:489] Releasing partition, row and schema locks
1029 02:52:03.232402 (+ 7us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":5,"spinlock_wait_cycles":112812800,"child_traces":[["op",{"apply.queue_time_us":71,"apply.run_cpu_time_us":407,"apply.run_wall_time_us":407,"num_ops":1,"prepare.queue_time_us":527,"prepare.run_cpu_time_us":998,"prepare.run_wall_time_us":26127,"raft.queue_time_us":42,"raft.run_cpu_time_us":282,"raft.run_wall_time_us":282,"replication_time_us":28568,"spinlock_wait_cycles":55288576}]]}
I20251029 02:52:03.512540 1773 raft_consensus.cc:515] T 3cded156380f45cc95893825c71ba0e9 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:03.512699 883 tablet_service.cc:1552] Processing DeleteTablet for tablet 9e07291d6b5645feb4d5890f58e1ef74 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:57 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:03.513013 1174 tablet_replica.cc:333] T 9e07291d6b5645feb4d5890f58e1ef74 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:03.512976 881 tablet_service.cc:1552] Processing DeleteTablet for tablet b5d5b737c86844d58c51b2388c0b1705 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:54 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:03.513218 1174 raft_consensus.cc:2243] T 9e07291d6b5645feb4d5890f58e1ef74 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:03.513399 1174 raft_consensus.cc:2272] T 9e07291d6b5645feb4d5890f58e1ef74 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:03.514065 1174 ts_tablet_manager.cc:1916] T 9e07291d6b5645feb4d5890f58e1ef74 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:03.518599 524 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_3975b6f24b5b4eb88958a46683a75c43" } modify_external_catalogs: true
I20251029 02:52:03.518818 524 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_3975b6f24b5b4eb88958a46683a75c43" } modify_external_catalogs: true
I20251029 02:52:03.518795 1773 leader_election.cc:290] T 3cded156380f45cc95893825c71ba0e9 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:03.518925 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3cded156380f45cc95893825c71ba0e9" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:52:03.519126 1060 raft_consensus.cc:3060] T 3cded156380f45cc95893825c71ba0e9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:03.520572 525 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_b131ed8021574ae885778d1e8da0ad8e" } new_table_name: "default.Table_47dc7c63ef104aec894c06e91376a212" modify_external_catalogs: true
I20251029 02:52:03.523772 1029 tablet_service.cc:1505] Processing CreateTablet for tablet 1e28a07b1a6242cd919b6ab2cb9b8a92 (DEFAULT_TABLE table=default.Table_ca44b863eb7a446693602cd5053ab82b [id=9b6853a5b9fd470cabc9323228567e14]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:03.526448 892 tablet_service.cc:1505] Processing CreateTablet for tablet 1e28a07b1a6242cd919b6ab2cb9b8a92 (DEFAULT_TABLE table=default.Table_ca44b863eb7a446693602cd5053ab82b [id=9b6853a5b9fd470cabc9323228567e14]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:03.527251 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_df7d932b50b64537b4dbc68a846665fc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:03.527945 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_df7d932b50b64537b4dbc68a846665fc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:03.530313 1025 tablet_service.cc:1505] Processing CreateTablet for tablet 3c3ef62323d643dda28f0ca269274600 (DEFAULT_TABLE table=default.Table_f6666e856c244ff0b9b28f1ffd730bf9 [id=8868237aaef64e8b821a9088ca5dd4d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:03.532595 895 tablet_service.cc:1505] Processing CreateTablet for tablet 3c3ef62323d643dda28f0ca269274600 (DEFAULT_TABLE table=default.Table_f6666e856c244ff0b9b28f1ffd730bf9 [id=8868237aaef64e8b821a9088ca5dd4d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:03.535471 520 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_62644549a49a4f039492b17deac6113d" } new_table_name: "default.Table_f54f62cc94f34996970dcaddc2d91d6f" modify_external_catalogs: true
I20251029 02:52:03.537230 523 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet a27496be3431446da9cb83452a3893e3
I20251029 02:52:03.538004 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_ab2b0e5b127542b6ab30655507c3e682"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:03.538611 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ab2b0e5b127542b6ab30655507c3e682 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:03.540386 1038 tablet_service.cc:1552] Processing DeleteTablet for tablet a27496be3431446da9cb83452a3893e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:03 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:03.541095 881 tablet_service.cc:1552] Processing DeleteTablet for tablet a27496be3431446da9cb83452a3893e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:03 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:03.548444 1166 tablet_bootstrap.cc:492] T 7309d3190cfa415281c5253a634fa7f3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:03.549118 1166 ts_tablet_manager.cc:1403] T 7309d3190cfa415281c5253a634fa7f3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.126s user 0.027s sys 0.015s
I20251029 02:52:03.552524 1166 raft_consensus.cc:359] T 7309d3190cfa415281c5253a634fa7f3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:03.553233 1166 raft_consensus.cc:385] T 7309d3190cfa415281c5253a634fa7f3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:03.553489 1166 raft_consensus.cc:740] T 7309d3190cfa415281c5253a634fa7f3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:03.554010 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18632de3bbbd491a86460497544580c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:03.554200 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c3ef62323d643dda28f0ca269274600. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:03.554378 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fc7d7c62ebc4ff29db9c1a925f7acfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:03.554548 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e28a07b1a6242cd919b6ab2cb9b8a92. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:03.554697 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf67a67854d048a49a96cccccf500ab6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:03.555071 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6dc30c614be64b8ba629d9884d450290. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:03.555552 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4e89d72dd274498783dc815f6f2ee031. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:03.561817 1166 consensus_queue.cc:260] T 7309d3190cfa415281c5253a634fa7f3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:03.562654 1166 ts_tablet_manager.cc:1434] T 7309d3190cfa415281c5253a634fa7f3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.013s user 0.000s sys 0.002s
I20251029 02:52:03.562424 1773 raft_consensus.cc:493] T faa835de84ff45b19525d1d603678987 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:03.563292 1773 raft_consensus.cc:515] T faa835de84ff45b19525d1d603678987 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:03.564230 1166 tablet_bootstrap.cc:492] T 24cb2a71ba3841d4a5e3e5c6fe99ae2f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:03.565838 1166 tablet_bootstrap.cc:654] T 24cb2a71ba3841d4a5e3e5c6fe99ae2f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:03.584781 751 tablet_service.cc:1505] Processing CreateTablet for tablet 9fc7d7c62ebc4ff29db9c1a925f7acfe (DEFAULT_TABLE table=default.Table_bc99b82ab1b84af899404ca0f6ec25ad [id=247a6298141541f484ddc896d8f8e655]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:03.586503 526 catalog_manager.cc:6194] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: default.Table_bb4222ae83ec442f8f62fbb51e9ab8eb [id=9f95160639ad40e995a6c89e297a0fad] alter complete (version 3)
I20251029 02:52:03.586580 1172 ts_tablet_manager.cc:1434] T cc74e83833754f0ca19046031a24ebb9 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.148s user 0.002s sys 0.005s
I20251029 02:52:03.586905 1172 tablet_bootstrap.cc:492] T 56d05e4a500043c995bb51a204cb295d P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:03.587852 1665 raft_consensus.cc:493] T da2a21c3236a4bd99872b89b12a36508 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:03.588002 1665 raft_consensus.cc:515] T da2a21c3236a4bd99872b89b12a36508 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:03.588662 1665 leader_election.cc:290] T da2a21c3236a4bd99872b89b12a36508 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:03.588099 1175 rpcz_store.cc:276] 1029 02:52:01.443062 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:01.443419 (+ 357us) service_pool.cc:225] Handling call
1029 02:52:03.406848 (+1963429us) ts_tablet_manager.cc:1151] Deleting tablet 2e003e140a244d46b15a2c3e84f68936
1029 02:52:03.406860 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:03.406960 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1963166}
I20251029 02:52:03.589579 526 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_cb5488bb7e104676aad8f3db9be87802" } new_table_name: "default.Table_90a0cc2204ac4322861cf1f52d9f2a5b" modify_external_catalogs: true
I20251029 02:52:03.590323 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 569) took 2144 ms. Trace:
I20251029 02:52:03.590648 1175 rpcz_store.cc:276] 1029 02:52:01.445935 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:01.445990 (+ 55us) service_pool.cc:225] Handling call
1029 02:52:03.589884 (+2143894us) ts_tablet_manager.cc:1151] Deleting tablet 5bd2163bc4be467fb695629bfbb5a6ca
1029 02:52:03.589942 (+ 58us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:03.590300 (+ 358us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2143634}
I20251029 02:52:03.592402 1058 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da2a21c3236a4bd99872b89b12a36508" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:03.592378 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 570) took 2145 ms. Trace:
I20251029 02:52:03.592792 1166 tablet_bootstrap.cc:492] T 24cb2a71ba3841d4a5e3e5c6fe99ae2f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:03.592922 1166 ts_tablet_manager.cc:1403] T 24cb2a71ba3841d4a5e3e5c6fe99ae2f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.029s user 0.006s sys 0.000s
W20251029 02:52:03.593648 847 leader_election.cc:343] T 3cded156380f45cc95893825c71ba0e9 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 3cded156380f45cc95893825c71ba0e9
I20251029 02:52:03.593729 1166 raft_consensus.cc:359] T 24cb2a71ba3841d4a5e3e5c6fe99ae2f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:03.593889 1166 raft_consensus.cc:385] T 24cb2a71ba3841d4a5e3e5c6fe99ae2f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:03.593940 1166 raft_consensus.cc:740] T 24cb2a71ba3841d4a5e3e5c6fe99ae2f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:03.594120 1166 consensus_queue.cc:260] T 24cb2a71ba3841d4a5e3e5c6fe99ae2f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:03.594475 1058 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "faa835de84ff45b19525d1d603678987" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:03.594648 1166 ts_tablet_manager.cc:1434] T 24cb2a71ba3841d4a5e3e5c6fe99ae2f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 02:52:03.594655 1058 raft_consensus.cc:2468] T faa835de84ff45b19525d1d603678987 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:03.594762 879 tablet_service.cc:1552] Processing DeleteTablet for tablet 1fc87e2d7b4240e08f2b4ac7277293ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:56 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:03.594931 1166 tablet_bootstrap.cc:492] T 1f5b40c0403d4a49a4bfde222d8c947a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:03.594985 1773 leader_election.cc:290] T faa835de84ff45b19525d1d603678987 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:03.595176 849 leader_election.cc:304] T faa835de84ff45b19525d1d603678987 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:03.596577 1166 tablet_bootstrap.cc:654] T 1f5b40c0403d4a49a4bfde222d8c947a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:03.592681 1175 rpcz_store.cc:276] 1029 02:52:01.446851 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:01.446910 (+ 59us) service_pool.cc:225] Handling call
1029 02:52:03.591939 (+2145029us) ts_tablet_manager.cc:1151] Deleting tablet 19609bc7f06645dc811b4bae8ce506c8
1029 02:52:03.592001 (+ 62us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:03.592305 (+ 304us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2144749}
I20251029 02:52:03.598673 517 catalog_manager.cc:6194] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: default.Table_da047830460443a29c5269c98fc859e9 [id=f733948e92d8470da9d8050bd26a06b1] alter complete (version 2)
I20251029 02:52:03.599606 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 585) took 1973 ms. Trace:
I20251029 02:52:03.599897 1175 rpcz_store.cc:276] 1029 02:52:01.626210 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:01.626266 (+ 56us) service_pool.cc:225] Handling call
1029 02:52:03.599071 (+1972805us) ts_tablet_manager.cc:1151] Deleting tablet f2d47238761e4ed49c88abdbb677b18a
1029 02:52:03.599129 (+ 58us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:03.599480 (+ 351us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1955872}
I20251029 02:52:03.601219 1166 tablet_bootstrap.cc:492] T 1f5b40c0403d4a49a4bfde222d8c947a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:03.601198 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 584) took 1975 ms. Trace:
I20251029 02:52:03.601464 1166 ts_tablet_manager.cc:1403] T 1f5b40c0403d4a49a4bfde222d8c947a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.001s
I20251029 02:52:03.601490 1175 rpcz_store.cc:276] 1029 02:52:01.625890 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:01.647120 (+ 21230us) service_pool.cc:225] Handling call
1029 02:52:03.600767 (+1953647us) ts_tablet_manager.cc:1151] Deleting tablet 7d1408dc544944669de519a1553d9bb7
1029 02:52:03.600832 (+ 65us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:03.601125 (+ 293us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1953346}
I20251029 02:52:03.602943 1038 tablet_service.cc:1552] Processing DeleteTablet for tablet 69597d48631d43b6a5b92abbf60d5ed3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:57 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:03.603154 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 6ca9bd37de9e4be388501b15af4bda31 with delete_type TABLET_DATA_DELETED (replaced by tablet 20a762344c5f46f88481c4ac7ad6e43b) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:03.603219 1031 tablet_service.cc:1552] Processing DeleteTablet for tablet 65fdec7cf5d143208d78d4c3806b1352 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:52 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:03.603367 1033 tablet_service.cc:1552] Processing DeleteTablet for tablet 2242f5b2c20346d9b753d7cd29242441 with delete_type TABLET_DATA_DELETED (replaced by tablet e1fa4980f2624321b617b84c68b90113) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:03.603147 1166 raft_consensus.cc:359] T 1f5b40c0403d4a49a4bfde222d8c947a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:03.603438 741 tablet_service.cc:1552] Processing DeleteTablet for tablet f81aecb20d6e498bb9b498698886d65e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:53 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:03.603492 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6dc30c614be64b8ba629d9884d450290. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:03.603489 1166 raft_consensus.cc:385] T 1f5b40c0403d4a49a4bfde222d8c947a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:03.603564 1030 tablet_service.cc:1552] Processing DeleteTablet for tablet a5c3a15510cb4321bbb80d61fabd0c0a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:57 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:03.603595 1773 raft_consensus.cc:2804] T faa835de84ff45b19525d1d603678987 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:03.603713 1773 raft_consensus.cc:493] T faa835de84ff45b19525d1d603678987 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:03.603715 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet 1fc87e2d7b4240e08f2b4ac7277293ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:56 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:03.603727 1166 raft_consensus.cc:740] T 1f5b40c0403d4a49a4bfde222d8c947a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:03.603901 1042 tablet_service.cc:1552] Processing DeleteTablet for tablet faa835de84ff45b19525d1d603678987 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:57 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:03.604022 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf67a67854d048a49a96cccccf500ab6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:03.604069 1027 tablet_service.cc:1552] Processing DeleteTablet for tablet 44784a3a760946e0be24743bc2fc8138 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:54 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:03.603164 1026 tablet_service.cc:1552] Processing DeleteTablet for tablet fd0f89180aba43eabb20f926abccbaa6 with delete_type TABLET_DATA_DELETED (replaced by tablet 9bb2166a590d4210b02dd779ef6654ad) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:03.604231 1166 consensus_queue.cc:260] T 1f5b40c0403d4a49a4bfde222d8c947a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:03.604946 741 tablet_service.cc:1552] Processing DeleteTablet for tablet c764062d3ba04d4ba8b2e367ad8ed433 with delete_type TABLET_DATA_DELETED (replaced by tablet 372d93914abc41b38a702742b7db0bc2) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:03.605235 1166 ts_tablet_manager.cc:1434] T 1f5b40c0403d4a49a4bfde222d8c947a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251029 02:52:03.605634 1665 raft_consensus.cc:493] T dd6ad641fbda472fab07aa35c5e4523a P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:03.605787 1166 tablet_bootstrap.cc:492] T dbc2092b63e0417997565c8fadb53ae0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:03.606217 754 tablet_service.cc:1552] Processing DeleteTablet for tablet daa05a34c0684c07a35966290fd7d44c with delete_type TABLET_DATA_DELETED (replaced by tablet fcb3db8399aa4cf188b07dda220f7f5b) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:03.606345 738 tablet_service.cc:1552] Processing DeleteTablet for tablet f246f6c202cf44d48c9f5e4f335202c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:55 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:03.606782 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 6bd3fac2502e479aa9f4ceaabbb39504 with delete_type TABLET_DATA_DELETED (replaced by tablet b125cc9aaf4248b8bfa48fb152b832a6) from {username='slave'} at 127.0.0.1:44580
W20251029 02:52:03.606947 712 leader_election.cc:343] T da2a21c3236a4bd99872b89b12a36508 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:03.608006 1060 raft_consensus.cc:2468] T 3cded156380f45cc95893825c71ba0e9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:03.609913 517 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_be5e479a6ff642dd8b01f195cfa37d8f" } modify_external_catalogs: true
I20251029 02:52:03.612571 750 tablet_service.cc:1505] Processing CreateTablet for tablet 1e28a07b1a6242cd919b6ab2cb9b8a92 (DEFAULT_TABLE table=default.Table_ca44b863eb7a446693602cd5053ab82b [id=9b6853a5b9fd470cabc9323228567e14]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:03.612975 754 tablet_service.cc:1552] Processing DeleteTablet for tablet 74873a05ef634867a30aa63b31fcfcc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:54 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:03.613157 738 tablet_service.cc:1552] Processing DeleteTablet for tablet 4aa78f4ac38e4919864c924e50f1aaf4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:57 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:03.613348 745 tablet_service.cc:1552] Processing DeleteTablet for tablet e1fa4980f2624321b617b84c68b90113 with delete_type TABLET_DATA_DELETED (replaced by tablet e6e3e6a82b014d2b8de4e109b379461e) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:03.613538 741 tablet_service.cc:1552] Processing DeleteTablet for tablet bfec87f690a640c39c58c64e16cc67e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:57 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:03.613852 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fc7d7c62ebc4ff29db9c1a925f7acfe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:03.614050 743 tablet_service.cc:1552] Processing DeleteTablet for tablet a27496be3431446da9cb83452a3893e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:03 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:03.614405 747 tablet_service.cc:1505] Processing CreateTablet for tablet 3c3ef62323d643dda28f0ca269274600 (DEFAULT_TABLE table=default.Table_f6666e856c244ff0b9b28f1ffd730bf9 [id=8868237aaef64e8b821a9088ca5dd4d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:03.614740 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c3ef62323d643dda28f0ca269274600. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:03.616290 1166 tablet_bootstrap.cc:654] T dbc2092b63e0417997565c8fadb53ae0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
W20251029 02:52:03.619168 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 2e003e140a244d46b15a2c3e84f68936 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2e003e140a244d46b15a2c3e84f68936 already in progress: creating tablet
W20251029 02:52:03.619390 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 5bd2163bc4be467fb695629bfbb5a6ca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5bd2163bc4be467fb695629bfbb5a6ca already in progress: creating tablet
W20251029 02:52:03.619571 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 19609bc7f06645dc811b4bae8ce506c8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 19609bc7f06645dc811b4bae8ce506c8 already in progress: creating tablet
W20251029 02:52:03.619729 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet f2d47238761e4ed49c88abdbb677b18a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f2d47238761e4ed49c88abdbb677b18a already in progress: creating tablet
W20251029 02:52:03.619937 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 7d1408dc544944669de519a1553d9bb7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7d1408dc544944669de519a1553d9bb7 already in progress: creating tablet
I20251029 02:52:03.623256 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e28a07b1a6242cd919b6ab2cb9b8a92. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:03.605760 1665 raft_consensus.cc:515] T dd6ad641fbda472fab07aa35c5e4523a P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:03.627060 1665 leader_election.cc:290] T dd6ad641fbda472fab07aa35c5e4523a P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:03.628103 755 tablet_service.cc:1552] Processing DeleteTablet for tablet f187dbd8a55b45b292ea61db43caf47d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:56 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:03.628131 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd6ad641fbda472fab07aa35c5e4523a" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:52:03.628837 712 leader_election.cc:343] T dd6ad641fbda472fab07aa35c5e4523a P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:03.630416 1172 tablet_bootstrap.cc:654] T 56d05e4a500043c995bb51a204cb295d P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:03.642410 521 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet ae5a6caeb17f4851b27a3c6a3c0db83c
W20251029 02:52:03.642670 520 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a04bc4d9605d4c049a151d985035bb63 consensus state has no leader
I20251029 02:52:03.643074 520 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_ba15e59ee62346c7bd87a0bf7b87c9fc" } new_table_name: "default.Table_739be6ccc20d4088ab4b9d6e11f0b4cd" modify_external_catalogs: true
I20251029 02:52:03.643349 517 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_be5e479a6ff642dd8b01f195cfa37d8f" } modify_external_catalogs: true
I20251029 02:52:03.643445 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_fe1e563f6397461fbe5255d6db0338b4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:03.644483 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fe1e563f6397461fbe5255d6db0338b4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:03.644790 519 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 70b008011d5547bbaf3f45685394139f
W20251029 02:52:03.644894 525 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 65e761f9f9fb490586d6415a5a8565dd consensus state has no leader
I20251029 02:52:03.645238 525 master_service.cc:954] ReplaceTablet: received request to replace tablet 845a69a8696e469a8548787a7645d41a from {username='slave'} at 127.0.0.1:35314
I20251029 02:52:03.645486 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_ef95e34ffbdf43f9bbe0074cee266ac1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:03.646073 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ef95e34ffbdf43f9bbe0074cee266ac1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:03.646209 524 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet ce98420e61904af78d8fa03724c74489
I20251029 02:52:03.647553 1026 tablet_service.cc:1552] Processing DeleteTablet for tablet 70b008011d5547bbaf3f45685394139f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:03 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:03.649348 1026 tablet_service.cc:1552] Processing DeleteTablet for tablet ce98420e61904af78d8fa03724c74489 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:03 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:03.649838 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet ae5a6caeb17f4851b27a3c6a3c0db83c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:03 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:03.650451 1027 tablet_service.cc:1552] Processing DeleteTablet for tablet 7309d3190cfa415281c5253a634fa7f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:52 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:03.650707 1042 tablet_service.cc:1552] Processing DeleteTablet for tablet 392a36a6b59a42f78e3ac7c8fb63214b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:54 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:03.650707 1030 tablet_service.cc:1552] Processing DeleteTablet for tablet 3cded156380f45cc95893825c71ba0e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:53 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:03.650805 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet c65469173762446e9725262ae998dbf3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:52 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:03.651113 1026 tablet_service.cc:1552] Processing DeleteTablet for tablet 5695b5a7b1854c3983e2d4ab684898b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:52 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:03.651291 1027 tablet_service.cc:1552] Processing DeleteTablet for tablet 1f5b40c0403d4a49a4bfde222d8c947a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:53 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:03.652420 1030 tablet_service.cc:1552] Processing DeleteTablet for tablet 24cb2a71ba3841d4a5e3e5c6fe99ae2f with delete_type TABLET_DATA_DELETED (replaced by tablet f60e7c24982d40319801869ad060864d) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:03.652575 1042 tablet_service.cc:1552] Processing DeleteTablet for tablet 3362e95063af41c8a24e935f547e3089 with delete_type TABLET_DATA_DELETED (replaced by tablet da2a21c3236a4bd99872b89b12a36508) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:03.652776 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet 6feec2c8e030447fae9c292a626dcd1d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:52 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:03.652942 1026 tablet_service.cc:1552] Processing DeleteTablet for tablet 6cd7009e06a646acad0802918c2b425e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:54 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:03.653100 1027 tablet_service.cc:1552] Processing DeleteTablet for tablet 8afc01d52d784f2fa559cab7ce07c8be with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:59 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:03.653424 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet 1bab90fe904944af948dab3db06b7e34 with delete_type TABLET_DATA_DELETED (replaced by tablet b6d4f7bf702f4834bc5f4ef2b50b92a7) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:03.656682 899 tablet_service.cc:1552] Processing DeleteTablet for tablet 65fdec7cf5d143208d78d4c3806b1352 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:52 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:03.658188 747 tablet_service.cc:1552] Processing DeleteTablet for tablet ae5a6caeb17f4851b27a3c6a3c0db83c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:03 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:03.658430 756 tablet_service.cc:1552] Processing DeleteTablet for tablet ce98420e61904af78d8fa03724c74489 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:03 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:03.663115 742 tablet_service.cc:1552] Processing DeleteTablet for tablet 70b008011d5547bbaf3f45685394139f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:03 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:03.665823 849 leader_election.cc:304] T 3cded156380f45cc95893825c71ba0e9 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters:
W20251029 02:52:03.666173 849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:45003, user_credentials={real_user=slave}} blocked reactor thread for 57657.8us
I20251029 02:52:03.666328 1774 raft_consensus.cc:493] T 3362e95063af41c8a24e935f547e3089 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:03.666813 918 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da2a21c3236a4bd99872b89b12a36508" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:03.667387 711 leader_election.cc:343] T da2a21c3236a4bd99872b89b12a36508 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:03.667512 711 leader_election.cc:304] T da2a21c3236a4bd99872b89b12a36508 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:03.668030 1839 raft_consensus.cc:493] T 6feec2c8e030447fae9c292a626dcd1d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:03.668170 1839 raft_consensus.cc:515] T 6feec2c8e030447fae9c292a626dcd1d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:03.668651 879 tablet_service.cc:1552] Processing DeleteTablet for tablet 5695b5a7b1854c3983e2d4ab684898b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:52 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:03.669023 1839 leader_election.cc:290] T 6feec2c8e030447fae9c292a626dcd1d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:03.669454 1665 raft_consensus.cc:2749] T da2a21c3236a4bd99872b89b12a36508 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 02:52:03.672006 989 leader_election.cc:343] T 6feec2c8e030447fae9c292a626dcd1d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 6feec2c8e030447fae9c292a626dcd1d
I20251029 02:52:03.674700 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd6ad641fbda472fab07aa35c5e4523a" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:03.675380 1773 raft_consensus.cc:3060] T faa835de84ff45b19525d1d603678987 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
W20251029 02:52:03.675817 711 leader_election.cc:343] T dd6ad641fbda472fab07aa35c5e4523a P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:03.676015 711 leader_election.cc:304] T dd6ad641fbda472fab07aa35c5e4523a P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:03.676573 1665 raft_consensus.cc:2749] T dd6ad641fbda472fab07aa35c5e4523a P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:03.676924 881 tablet_service.cc:1552] Processing DeleteTablet for tablet 8afc01d52d784f2fa559cab7ce07c8be with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:59 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:03.677628 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e28a07b1a6242cd919b6ab2cb9b8a92. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:03.678128 884 tablet_service.cc:1552] Processing DeleteTablet for tablet 2242f5b2c20346d9b753d7cd29242441 with delete_type TABLET_DATA_DELETED (replaced by tablet e1fa4980f2624321b617b84c68b90113) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:03.678753 1774 raft_consensus.cc:515] T 3362e95063af41c8a24e935f547e3089 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:03.679529 1774 leader_election.cc:290] T 3362e95063af41c8a24e935f547e3089 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:03.680517 881 tablet_service.cc:1552] Processing DeleteTablet for tablet 392a36a6b59a42f78e3ac7c8fb63214b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:54 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:03.681564 891 tablet_service.cc:1552] Processing DeleteTablet for tablet 6cd7009e06a646acad0802918c2b425e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:54 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:03.681633 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_f26d0b6d64304372a0bc5120b663991e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:03.681779 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_a587f8486e2a4542a9e7cf80c0306f23"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:03.682253 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f26d0b6d64304372a0bc5120b663991e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:52:03.682329 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a587f8486e2a4542a9e7cf80c0306f23 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:03.683387 890 tablet_service.cc:1552] Processing DeleteTablet for tablet 69597d48631d43b6a5b92abbf60d5ed3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:57 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:03.683501 883 tablet_service.cc:1552] Processing DeleteTablet for tablet 6feec2c8e030447fae9c292a626dcd1d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:52 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:03.683597 893 tablet_service.cc:1552] Processing DeleteTablet for tablet ce98420e61904af78d8fa03724c74489 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:03 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:03.683919 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3362e95063af41c8a24e935f547e3089" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:03.684159 1060 raft_consensus.cc:2468] T 3362e95063af41c8a24e935f547e3089 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:03.685163 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_0c8e6b92d9a240fead676b6cefa506cb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:03.685653 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0c8e6b92d9a240fead676b6cefa506cb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:03.692616 1665 raft_consensus.cc:493] T 4aa78f4ac38e4919864c924e50f1aaf4 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:03.692822 1665 raft_consensus.cc:515] T 4aa78f4ac38e4919864c924e50f1aaf4 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:03.693571 1665 leader_election.cc:290] T 4aa78f4ac38e4919864c924e50f1aaf4 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:03.694250 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_4916f7d510ec49be9480cc88f4c7d3af"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:03.695204 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4916f7d510ec49be9480cc88f4c7d3af in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:03.696838 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4aa78f4ac38e4919864c924e50f1aaf4" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:52:03.697526 712 leader_election.cc:343] T 4aa78f4ac38e4919864c924e50f1aaf4 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
W20251029 02:52:03.697616 520 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dd16d1c715604774baba030853f6697c consensus state has no leader
I20251029 02:52:03.698968 517 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet bc16262db49e4fc69fe1a4f95dbde2cb
I20251029 02:52:03.699108 891 tablet_service.cc:1552] Processing DeleteTablet for tablet faa835de84ff45b19525d1d603678987 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:57 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:03.699327 898 tablet_service.cc:1552] Processing DeleteTablet for tablet c65469173762446e9725262ae998dbf3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:52 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:03.699465 885 tablet_service.cc:1552] Processing DeleteTablet for tablet ae5a6caeb17f4851b27a3c6a3c0db83c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:03 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:03.699606 899 tablet_service.cc:1552] Processing DeleteTablet for tablet 3cded156380f45cc95893825c71ba0e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:53 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:03.699723 879 tablet_service.cc:1552] Processing DeleteTablet for tablet fd0f89180aba43eabb20f926abccbaa6 with delete_type TABLET_DATA_DELETED (replaced by tablet 9bb2166a590d4210b02dd779ef6654ad) from {username='slave'} at 127.0.0.1:35702
W20251029 02:52:03.699805 526 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0415b52fadf948238c4e83dc09000f96 consensus state has no leader
I20251029 02:52:03.700385 526 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_ce73377094b942cc87faac169c0fbe4a" } new_table_name: "default.Table_0772f9db5d6142d988f6ffca2e96bdd2" modify_external_catalogs: true
I20251029 02:52:03.700687 1682 raft_consensus.cc:2804] T 3cded156380f45cc95893825c71ba0e9 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:03.700861 1682 raft_consensus.cc:697] T 3cded156380f45cc95893825c71ba0e9 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:52:03.701081 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4aa78f4ac38e4919864c924e50f1aaf4" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:03.701762 711 leader_election.cc:343] T 4aa78f4ac38e4919864c924e50f1aaf4 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:03.702008 1682 consensus_queue.cc:237] T 3cded156380f45cc95893825c71ba0e9 P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:03.702121 711 leader_election.cc:304] T 4aa78f4ac38e4919864c924e50f1aaf4 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:03.702859 1665 raft_consensus.cc:2749] T 4aa78f4ac38e4919864c924e50f1aaf4 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:03.703317 849 leader_election.cc:304] T 3362e95063af41c8a24e935f547e3089 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:03.703392 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet bc16262db49e4fc69fe1a4f95dbde2cb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:03 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:03.703553 918 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6feec2c8e030447fae9c292a626dcd1d" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:03.708626 1176 ts_tablet_manager.cc:1929] T 9964ad774890422f8e14f6b99f28cb55 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:03.708705 900 tablet_service.cc:1552] Processing DeleteTablet for tablet 70b008011d5547bbaf3f45685394139f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:03 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:03.708760 1176 log.cc:1199] T 9964ad774890422f8e14f6b99f28cb55 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/9964ad774890422f8e14f6b99f28cb55
I20251029 02:52:03.709060 1176 ts_tablet_manager.cc:1950] T 9964ad774890422f8e14f6b99f28cb55 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:03.709105 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c3ef62323d643dda28f0ca269274600. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:03.715740 751 tablet_service.cc:1552] Processing DeleteTablet for tablet bc16262db49e4fc69fe1a4f95dbde2cb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:03 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:03.720724 1176 tablet_replica.cc:333] T 69597d48631d43b6a5b92abbf60d5ed3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:03.720937 1176 raft_consensus.cc:2243] T 69597d48631d43b6a5b92abbf60d5ed3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:03.721089 1176 raft_consensus.cc:2272] T 69597d48631d43b6a5b92abbf60d5ed3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:03.722334 1176 ts_tablet_manager.cc:1916] T 69597d48631d43b6a5b92abbf60d5ed3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:03.723320 899 tablet_service.cc:1552] Processing DeleteTablet for tablet bc16262db49e4fc69fe1a4f95dbde2cb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:03 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:03.723645 918 raft_consensus.cc:2468] T 6feec2c8e030447fae9c292a626dcd1d P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 0.
I20251029 02:52:03.723899 1175 tablet_replica.cc:333] T 7e6c4f30be624769961c9ec17c5d5dcb P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:03.724087 1175 raft_consensus.cc:2243] T 7e6c4f30be624769961c9ec17c5d5dcb P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:03.724221 1175 raft_consensus.cc:2272] T 7e6c4f30be624769961c9ec17c5d5dcb P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:03.724689 1175 ts_tablet_manager.cc:1916] T 7e6c4f30be624769961c9ec17c5d5dcb P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:03.725940 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 9964ad774890422f8e14f6b99f28cb55 (table default.Table_da0be17b8bdc4e579b6c3faff71d9c98 [id=6e06ac70c6814e7d89a008bbaddccc7e]) successfully deleted
W20251029 02:52:03.726301 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet a27496be3431446da9cb83452a3893e3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a27496be3431446da9cb83452a3893e3 already in progress: creating tablet
I20251029 02:52:03.726289 990 leader_election.cc:304] T 6feec2c8e030447fae9c292a626dcd1d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters: b495a49349324f7d81908f2da700f0d9
I20251029 02:52:03.726691 1839 raft_consensus.cc:2804] T 6feec2c8e030447fae9c292a626dcd1d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:03.726775 1839 raft_consensus.cc:493] T 6feec2c8e030447fae9c292a626dcd1d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:03.726856 1839 raft_consensus.cc:3060] T 6feec2c8e030447fae9c292a626dcd1d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:03.732051 884 tablet_service.cc:1552] Processing DeleteTablet for tablet 44784a3a760946e0be24743bc2fc8138 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:54 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:03.732652 879 tablet_service.cc:1552] Processing DeleteTablet for tablet 3362e95063af41c8a24e935f547e3089 with delete_type TABLET_DATA_DELETED (replaced by tablet da2a21c3236a4bd99872b89b12a36508) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:03.734288 1854 consensus_queue.cc:1048] T 1fc87e2d7b4240e08f2b4ac7277293ff P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:03.735148 1774 raft_consensus.cc:2804] T 3362e95063af41c8a24e935f547e3089 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:03.735261 1774 raft_consensus.cc:493] T 3362e95063af41c8a24e935f547e3089 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:03.735867 1774 raft_consensus.cc:3060] T 3362e95063af41c8a24e935f547e3089 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:03.738258 881 tablet_service.cc:1552] Processing DeleteTablet for tablet a5c3a15510cb4321bbb80d61fabd0c0a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:57 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:03.738917 1854 raft_consensus.cc:493] T 7309d3190cfa415281c5253a634fa7f3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:03.739046 1854 raft_consensus.cc:515] T 7309d3190cfa415281c5253a634fa7f3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:03.739034 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_8b39f039e15540df86d5d1dd12a645a7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:03.739604 1854 leader_election.cc:290] T 7309d3190cfa415281c5253a634fa7f3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
W20251029 02:52:03.739614 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8b39f039e15540df86d5d1dd12a645a7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:52:03.740974 989 leader_election.cc:343] T 7309d3190cfa415281c5253a634fa7f3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 7309d3190cfa415281c5253a634fa7f3
I20251029 02:52:03.741590 1167 tablet_bootstrap.cc:492] T 1bab90fe904944af948dab3db06b7e34 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:03.741750 1167 ts_tablet_manager.cc:1403] T 1bab90fe904944af948dab3db06b7e34 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.361s user 0.003s sys 0.003s
I20251029 02:52:03.742385 519 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_ff796c36eb2c4bb1809f39e1cb68c223" } modify_external_catalogs: true
I20251029 02:52:03.742619 519 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_ff796c36eb2c4bb1809f39e1cb68c223" } modify_external_catalogs: true
I20251029 02:52:03.748610 1854 raft_consensus.cc:493] T c65469173762446e9725262ae998dbf3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:03.748811 1854 raft_consensus.cc:515] T c65469173762446e9725262ae998dbf3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:03.749564 1854 leader_election.cc:290] T c65469173762446e9725262ae998dbf3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
W20251029 02:52:03.749598 711 consensus_peers.cc:597] T 70e8cfe5eaa9492c85874dc3214a1e0b P b495a49349324f7d81908f2da700f0d9 -> Peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Couldn't send request to peer c756528642a543eeacd93cef3e335a7d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 70e8cfe5eaa9492c85874dc3214a1e0b. This is attempt 1: this message will repeat every 5th retry.
W20251029 02:52:03.751070 989 leader_election.cc:343] T c65469173762446e9725262ae998dbf3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: c65469173762446e9725262ae998dbf3
I20251029 02:52:03.757462 1167 raft_consensus.cc:359] T 1bab90fe904944af948dab3db06b7e34 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:03.757781 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7309d3190cfa415281c5253a634fa7f3" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:03.758289 1167 raft_consensus.cc:385] T 1bab90fe904944af948dab3db06b7e34 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:03.758432 1167 raft_consensus.cc:740] T 1bab90fe904944af948dab3db06b7e34 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:03.758860 1167 consensus_queue.cc:260] T 1bab90fe904944af948dab3db06b7e34 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
W20251029 02:52:03.759701 990 leader_election.cc:343] T 7309d3190cfa415281c5253a634fa7f3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:03.759826 990 leader_election.cc:304] T 7309d3190cfa415281c5253a634fa7f3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5; no voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:03.760385 1854 raft_consensus.cc:2749] T 7309d3190cfa415281c5253a634fa7f3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:03.759430 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c65469173762446e9725262ae998dbf3" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:03.760792 917 raft_consensus.cc:2468] T c65469173762446e9725262ae998dbf3 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 0.
I20251029 02:52:03.761325 990 leader_election.cc:304] T c65469173762446e9725262ae998dbf3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters: b495a49349324f7d81908f2da700f0d9
I20251029 02:52:03.761821 1854 raft_consensus.cc:2804] T c65469173762446e9725262ae998dbf3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:03.761929 1854 raft_consensus.cc:493] T c65469173762446e9725262ae998dbf3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:03.762004 1854 raft_consensus.cc:3060] T c65469173762446e9725262ae998dbf3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:03.765394 1903 raft_consensus.cc:493] T 392a36a6b59a42f78e3ac7c8fb63214b P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:03.765573 1903 raft_consensus.cc:515] T 392a36a6b59a42f78e3ac7c8fb63214b P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:03.767514 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "392a36a6b59a42f78e3ac7c8fb63214b" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:03.767776 1060 raft_consensus.cc:2468] T 392a36a6b59a42f78e3ac7c8fb63214b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:03.768330 849 leader_election.cc:304] T 392a36a6b59a42f78e3ac7c8fb63214b P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:03.772357 1682 raft_consensus.cc:2804] T 392a36a6b59a42f78e3ac7c8fb63214b P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:03.774106 1682 raft_consensus.cc:493] T 392a36a6b59a42f78e3ac7c8fb63214b P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:03.774355 1682 raft_consensus.cc:3060] T 392a36a6b59a42f78e3ac7c8fb63214b P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:03.772490 1903 leader_election.cc:290] T 392a36a6b59a42f78e3ac7c8fb63214b P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:03.790237 881 tablet_service.cc:1552] Processing DeleteTablet for tablet 1bab90fe904944af948dab3db06b7e34 with delete_type TABLET_DATA_DELETED (replaced by tablet b6d4f7bf702f4834bc5f4ef2b50b92a7) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:03.791067 520 master_service.cc:954] ReplaceTablet: received request to replace tablet a945d64287f94ca697447e540fab39ce from {username='slave'} at 127.0.0.1:35314
I20251029 02:52:03.797161 1906 raft_consensus.cc:493] T 6cd7009e06a646acad0802918c2b425e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:03.797947 1906 raft_consensus.cc:515] T 6cd7009e06a646acad0802918c2b425e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
W20251029 02:52:03.798254 847 leader_election.cc:343] T 3cded156380f45cc95893825c71ba0e9 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 3cded156380f45cc95893825c71ba0e9
W20251029 02:52:03.799041 847 leader_election.cc:343] T faa835de84ff45b19525d1d603678987 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: faa835de84ff45b19525d1d603678987
I20251029 02:52:03.800190 918 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6cd7009e06a646acad0802918c2b425e" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:03.800374 918 raft_consensus.cc:2468] T 6cd7009e06a646acad0802918c2b425e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 0.
W20251029 02:52:03.800618 847 leader_election.cc:343] T 3362e95063af41c8a24e935f547e3089 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 3362e95063af41c8a24e935f547e3089
I20251029 02:52:03.800920 990 leader_election.cc:304] T 6cd7009e06a646acad0802918c2b425e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:03.801209 1686 raft_consensus.cc:2804] T 6cd7009e06a646acad0802918c2b425e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:03.801295 1686 raft_consensus.cc:493] T 6cd7009e06a646acad0802918c2b425e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:03.801398 1686 raft_consensus.cc:3060] T 6cd7009e06a646acad0802918c2b425e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
W20251029 02:52:03.801378 847 leader_election.cc:343] T 392a36a6b59a42f78e3ac7c8fb63214b P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 392a36a6b59a42f78e3ac7c8fb63214b
I20251029 02:52:03.799247 1906 leader_election.cc:290] T 6cd7009e06a646acad0802918c2b425e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:03.802946 1174 ts_tablet_manager.cc:1929] T 9e07291d6b5645feb4d5890f58e1ef74 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:03.803097 1174 log.cc:1199] T 9e07291d6b5645feb4d5890f58e1ef74 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/9e07291d6b5645feb4d5890f58e1ef74
I20251029 02:52:03.803645 1665 raft_consensus.cc:493] T 519754f5d8f7484480f973c53708c5f3 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:03.803534 1174 ts_tablet_manager.cc:1950] T 9e07291d6b5645feb4d5890f58e1ef74 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:03.803799 1665 raft_consensus.cc:515] T 519754f5d8f7484480f973c53708c5f3 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:03.804533 1665 leader_election.cc:290] T 519754f5d8f7484480f973c53708c5f3 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:03.805248 918 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "519754f5d8f7484480f973c53708c5f3" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:03.805927 711 leader_election.cc:343] T 519754f5d8f7484480f973c53708c5f3 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:03.806241 1167 ts_tablet_manager.cc:1434] T 1bab90fe904944af948dab3db06b7e34 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.064s user 0.001s sys 0.008s
I20251029 02:52:03.806602 1167 tablet_bootstrap.cc:492] T 7309d3190cfa415281c5253a634fa7f3 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:03.808348 1167 tablet_bootstrap.cc:654] T 7309d3190cfa415281c5253a634fa7f3 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:03.809065 1175 ts_tablet_manager.cc:1929] T 7e6c4f30be624769961c9ec17c5d5dcb P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:03.809203 1175 log.cc:1199] T 7e6c4f30be624769961c9ec17c5d5dcb P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/7e6c4f30be624769961c9ec17c5d5dcb
I20251029 02:52:03.809569 1175 ts_tablet_manager.cc:1950] T 7e6c4f30be624769961c9ec17c5d5dcb P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:03.810151 1172 tablet_bootstrap.cc:492] T 56d05e4a500043c995bb51a204cb295d P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:03.810321 1172 ts_tablet_manager.cc:1403] T 56d05e4a500043c995bb51a204cb295d P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.223s user 0.011s sys 0.004s
I20251029 02:52:03.812011 1172 raft_consensus.cc:359] T 56d05e4a500043c995bb51a204cb295d P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:03.812219 1172 raft_consensus.cc:385] T 56d05e4a500043c995bb51a204cb295d P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:03.812312 1172 raft_consensus.cc:740] T 56d05e4a500043c995bb51a204cb295d P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:03.812570 1172 consensus_queue.cc:260] T 56d05e4a500043c995bb51a204cb295d P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:03.813086 1172 ts_tablet_manager.cc:1434] T 56d05e4a500043c995bb51a204cb295d P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20251029 02:52:03.814714 1839 raft_consensus.cc:515] T 6feec2c8e030447fae9c292a626dcd1d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:03.815749 1839 leader_election.cc:290] T 6feec2c8e030447fae9c292a626dcd1d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Requested vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:03.816032 918 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6feec2c8e030447fae9c292a626dcd1d" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
I20251029 02:52:03.816208 918 raft_consensus.cc:3060] T 6feec2c8e030447fae9c292a626dcd1d P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:03.819308 1774 raft_consensus.cc:515] T 3362e95063af41c8a24e935f547e3089 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:03.820354 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3362e95063af41c8a24e935f547e3089" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:52:03.820535 1060 raft_consensus.cc:3060] T 3362e95063af41c8a24e935f547e3089 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:03.822264 1854 raft_consensus.cc:515] T c65469173762446e9725262ae998dbf3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:03.822855 1854 leader_election.cc:290] T c65469173762446e9725262ae998dbf3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Requested vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:03.822973 1774 leader_election.cc:290] T 3362e95063af41c8a24e935f547e3089 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:03.823604 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 9e07291d6b5645feb4d5890f58e1ef74 (table default.Table_0edd4b07469b4d24b302ecdb6859d1d1 [id=e71aea48b9884bacb250412624ec4615]) successfully deleted
I20251029 02:52:03.823661 1174 tablet_replica.cc:333] T 1fc87e2d7b4240e08f2b4ac7277293ff P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:03.823865 1174 raft_consensus.cc:2243] T 1fc87e2d7b4240e08f2b4ac7277293ff P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:03.824018 1174 raft_consensus.cc:2272] T 1fc87e2d7b4240e08f2b4ac7277293ff P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:03.823992 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c65469173762446e9725262ae998dbf3" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
I20251029 02:52:03.824148 917 raft_consensus.cc:3060] T c65469173762446e9725262ae998dbf3 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:03.824257 1773 raft_consensus.cc:515] T faa835de84ff45b19525d1d603678987 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:03.825549 1773 leader_election.cc:290] T faa835de84ff45b19525d1d603678987 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:03.826092 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 583) took 2191 ms. Trace:
I20251029 02:52:03.826171 1175 rpcz_store.cc:276] 1029 02:52:01.619309 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:01.710295 (+ 90986us) service_pool.cc:225] Handling call
1029 02:52:03.602574 (+1892279us) ts_tablet_manager.cc:1151] Deleting tablet 7e6c4f30be624769961c9ec17c5d5dcb
1029 02:52:03.723852 (+121278us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:03.804624 (+ 80772us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:03.808949 (+ 4325us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:03.810497 (+ 1548us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1891737}
I20251029 02:52:03.826601 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 7e6c4f30be624769961c9ec17c5d5dcb (table default.Table_dea0766540d24b5e9bd6cc8b6338b47c [id=1dfd5b42973c4d5d91e0d9014e34269f]) successfully deleted
I20251029 02:52:03.824059 1060 raft_consensus.cc:2468] T 3362e95063af41c8a24e935f547e3089 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:03.825177 1058 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "faa835de84ff45b19525d1d603678987" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:52:03.826010 1176 ts_tablet_manager.cc:1929] T 69597d48631d43b6a5b92abbf60d5ed3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:03.826748 1686 raft_consensus.cc:515] T 6cd7009e06a646acad0802918c2b425e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:03.827189 1058 raft_consensus.cc:3060] T faa835de84ff45b19525d1d603678987 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
W20251029 02:52:03.827482 512 catalog_manager.cc:4952] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet b5d5b737c86844d58c51b2388c0b1705 because the tablet was not found. No further retry: Not found: Tablet not found: b5d5b737c86844d58c51b2388c0b1705
I20251029 02:52:03.827617 1686 leader_election.cc:290] T 6cd7009e06a646acad0802918c2b425e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Requested vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
W20251029 02:52:03.827759 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet a27496be3431446da9cb83452a3893e3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a27496be3431446da9cb83452a3893e3 already in progress: creating tablet
I20251029 02:52:03.827934 1176 log.cc:1199] T 69597d48631d43b6a5b92abbf60d5ed3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/69597d48631d43b6a5b92abbf60d5ed3
I20251029 02:52:03.828166 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6cd7009e06a646acad0802918c2b425e" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
I20251029 02:52:03.828275 1176 ts_tablet_manager.cc:1950] T 69597d48631d43b6a5b92abbf60d5ed3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:03.828301 920 raft_consensus.cc:3060] T 6cd7009e06a646acad0802918c2b425e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:03.828641 918 raft_consensus.cc:2468] T 6feec2c8e030447fae9c292a626dcd1d P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 1.
I20251029 02:52:03.828755 1176 tablet_replica.cc:333] T 65fdec7cf5d143208d78d4c3806b1352 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:03.828949 1176 raft_consensus.cc:2243] T 65fdec7cf5d143208d78d4c3806b1352 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:03.829119 1176 raft_consensus.cc:2272] T 65fdec7cf5d143208d78d4c3806b1352 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:03.829622 1176 ts_tablet_manager.cc:1916] T 65fdec7cf5d143208d78d4c3806b1352 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:03.829994 1058 raft_consensus.cc:2468] T faa835de84ff45b19525d1d603678987 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:03.830492 849 leader_election.cc:304] T 3362e95063af41c8a24e935f547e3089 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:03.830763 1773 raft_consensus.cc:2804] T 3362e95063af41c8a24e935f547e3089 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:03.830827 849 leader_election.cc:304] T faa835de84ff45b19525d1d603678987 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:03.830884 1773 raft_consensus.cc:697] T 3362e95063af41c8a24e935f547e3089 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:52:03.831125 1773 consensus_queue.cc:237] T 3362e95063af41c8a24e935f547e3089 P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:03.831972 990 leader_election.cc:304] T 6feec2c8e030447fae9c292a626dcd1d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:03.832414 1774 raft_consensus.cc:2804] T faa835de84ff45b19525d1d603678987 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:03.832531 1774 raft_consensus.cc:697] T faa835de84ff45b19525d1d603678987 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:52:03.832753 1774 consensus_queue.cc:237] T faa835de84ff45b19525d1d603678987 P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:03.834821 1682 raft_consensus.cc:515] T 392a36a6b59a42f78e3ac7c8fb63214b P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:03.835758 1058 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "392a36a6b59a42f78e3ac7c8fb63214b" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:52:03.835949 1058 raft_consensus.cc:3060] T 392a36a6b59a42f78e3ac7c8fb63214b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:03.836019 1174 ts_tablet_manager.cc:1916] T 1fc87e2d7b4240e08f2b4ac7277293ff P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:03.838661 1058 raft_consensus.cc:2468] T 392a36a6b59a42f78e3ac7c8fb63214b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:03.838898 917 raft_consensus.cc:2468] T c65469173762446e9725262ae998dbf3 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 1.
I20251029 02:52:03.839196 849 leader_election.cc:304] T 392a36a6b59a42f78e3ac7c8fb63214b P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:03.839242 1907 raft_consensus.cc:2804] T 6feec2c8e030447fae9c292a626dcd1d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:03.839493 1774 raft_consensus.cc:2804] T 392a36a6b59a42f78e3ac7c8fb63214b P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:03.839645 1774 raft_consensus.cc:697] T 392a36a6b59a42f78e3ac7c8fb63214b P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:52:03.840042 1774 consensus_queue.cc:237] T 392a36a6b59a42f78e3ac7c8fb63214b P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:03.841464 920 raft_consensus.cc:2468] T 6cd7009e06a646acad0802918c2b425e P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 1.
I20251029 02:52:03.842101 990 leader_election.cc:304] T c65469173762446e9725262ae998dbf3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:03.842427 990 leader_election.cc:304] T 6cd7009e06a646acad0802918c2b425e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:03.842743 1686 raft_consensus.cc:2804] T c65469173762446e9725262ae998dbf3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:03.842902 1686 raft_consensus.cc:697] T c65469173762446e9725262ae998dbf3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Becoming Leader. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Running, Role: LEADER
I20251029 02:52:03.842954 1906 raft_consensus.cc:2804] T 6cd7009e06a646acad0802918c2b425e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:03.843115 1906 raft_consensus.cc:697] T 6cd7009e06a646acad0802918c2b425e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Becoming Leader. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Running, Role: LEADER
I20251029 02:52:03.843178 1686 consensus_queue.cc:237] T c65469173762446e9725262ae998dbf3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:03.843344 1906 consensus_queue.cc:237] T 6cd7009e06a646acad0802918c2b425e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:03.848800 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 69597d48631d43b6a5b92abbf60d5ed3 (table default.Table_cec9500b890447609a5feb7346bcf7a0 [id=ffce392867b14b6ea9db44f4bdee57c3]) successfully deleted
I20251029 02:52:03.848883 1174 ts_tablet_manager.cc:1929] T 1fc87e2d7b4240e08f2b4ac7277293ff P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:03.848985 1174 log.cc:1199] T 1fc87e2d7b4240e08f2b4ac7277293ff P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/1fc87e2d7b4240e08f2b4ac7277293ff
I20251029 02:52:03.849311 1174 ts_tablet_manager.cc:1950] T 1fc87e2d7b4240e08f2b4ac7277293ff P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:03.850353 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 1fc87e2d7b4240e08f2b4ac7277293ff (table default.Table_9306136142dd41cca83965ab5a92c5a0 [id=130ea33221324d7388a014abd549b2ff]) successfully deleted
I20251029 02:52:03.850769 1174 tablet_replica.cc:333] T 65fdec7cf5d143208d78d4c3806b1352 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:03.850966 1174 raft_consensus.cc:2243] T 65fdec7cf5d143208d78d4c3806b1352 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:03.851186 1174 pending_rounds.cc:70] T 65fdec7cf5d143208d78d4c3806b1352 P c756528642a543eeacd93cef3e335a7d: Trying to abort 1 pending ops.
I20251029 02:52:03.851228 1174 pending_rounds.cc:77] T 65fdec7cf5d143208d78d4c3806b1352 P c756528642a543eeacd93cef3e335a7d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215949098515709952 op_type: NO_OP noop_request { }
I20251029 02:52:03.851305 1174 raft_consensus.cc:2889] T 65fdec7cf5d143208d78d4c3806b1352 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 02:52:03.851374 1174 raft_consensus.cc:2272] T 65fdec7cf5d143208d78d4c3806b1352 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:03.851679 1682 leader_election.cc:290] T 392a36a6b59a42f78e3ac7c8fb63214b P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:03.852689 1916 raft_consensus.cc:493] T 8afc01d52d784f2fa559cab7ce07c8be P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:03.852941 1916 raft_consensus.cc:515] T 8afc01d52d784f2fa559cab7ce07c8be P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:03.853830 1058 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "519754f5d8f7484480f973c53708c5f3" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:03.853981 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8afc01d52d784f2fa559cab7ce07c8be" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:03.854140 920 raft_consensus.cc:2468] T 8afc01d52d784f2fa559cab7ce07c8be P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 0.
I20251029 02:52:03.854233 1916 leader_election.cc:290] T 8afc01d52d784f2fa559cab7ce07c8be P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:03.854648 990 leader_election.cc:304] T 8afc01d52d784f2fa559cab7ce07c8be P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters:
W20251029 02:52:03.854883 712 leader_election.cc:343] T 519754f5d8f7484480f973c53708c5f3 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:03.855012 712 leader_election.cc:304] T 519754f5d8f7484480f973c53708c5f3 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:03.855250 1665 raft_consensus.cc:2749] T 519754f5d8f7484480f973c53708c5f3 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:03.857151 1854 raft_consensus.cc:2804] T 8afc01d52d784f2fa559cab7ce07c8be P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:03.857256 1854 raft_consensus.cc:493] T 8afc01d52d784f2fa559cab7ce07c8be P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:03.857352 1854 raft_consensus.cc:3060] T 8afc01d52d784f2fa559cab7ce07c8be P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:03.839395 1907 raft_consensus.cc:697] T 6feec2c8e030447fae9c292a626dcd1d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Becoming Leader. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Running, Role: LEADER
I20251029 02:52:03.857982 1907 consensus_queue.cc:237] T 6feec2c8e030447fae9c292a626dcd1d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:03.860215 1854 raft_consensus.cc:515] T 8afc01d52d784f2fa559cab7ce07c8be P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:03.860798 1854 leader_election.cc:290] T 8afc01d52d784f2fa559cab7ce07c8be P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Requested vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:03.864005 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_714c5abdd40047d1bf204ee53c29a06e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:03.864261 524 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet cc74e83833754f0ca19046031a24ebb9 (table default.Table_755825b17834457cb94b05163ae3e26d [id=57471769276f44299eae50b9d854b50c]). Expected version 1 got 0
W20251029 02:52:03.864626 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_714c5abdd40047d1bf204ee53c29a06e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:03.866104 751 tablet_service.cc:1552] Processing DeleteTablet for tablet dd6ad641fbda472fab07aa35c5e4523a with delete_type TABLET_DATA_DELETED (replaced by tablet 4f24ae08e2c34375abccb6fd187adbd4) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:03.866346 749 tablet_service.cc:1552] Processing DeleteTablet for tablet 519754f5d8f7484480f973c53708c5f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:53 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:03.867791 751 tablet_service.cc:1552] Processing DeleteTablet for tablet 0cc1957a845f4e9cb703a553b27d1347 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:53 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:03.868435 892 tablet_service.cc:1552] Processing DeleteTablet for tablet 1fc87e2d7b4240e08f2b4ac7277293ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:56 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:03.868773 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8afc01d52d784f2fa559cab7ce07c8be" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
I20251029 02:52:03.868984 920 raft_consensus.cc:3060] T 8afc01d52d784f2fa559cab7ce07c8be P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:03.869343 1176 ts_tablet_manager.cc:1929] T 65fdec7cf5d143208d78d4c3806b1352 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:03.869485 1176 log.cc:1199] T 65fdec7cf5d143208d78d4c3806b1352 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/65fdec7cf5d143208d78d4c3806b1352
I20251029 02:52:03.869854 1176 ts_tablet_manager.cc:1950] T 65fdec7cf5d143208d78d4c3806b1352 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:03.870357 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 65fdec7cf5d143208d78d4c3806b1352 (table default.Table_d1a375b674704cddb1c3bb13c566729e [id=909ecde47c7245b2b84dcd665e779b1e]) successfully deleted
I20251029 02:52:03.870973 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_494874895389492d9a687de4db8785b5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:03.871626 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_494874895389492d9a687de4db8785b5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:03.874673 920 raft_consensus.cc:2468] T 8afc01d52d784f2fa559cab7ce07c8be P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 1.
I20251029 02:52:03.875170 990 leader_election.cc:304] T 8afc01d52d784f2fa559cab7ce07c8be P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:03.875501 1839 raft_consensus.cc:2804] T 8afc01d52d784f2fa559cab7ce07c8be P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:03.875640 1839 raft_consensus.cc:697] T 8afc01d52d784f2fa559cab7ce07c8be P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Becoming Leader. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Running, Role: LEADER
I20251029 02:52:03.875922 1839 consensus_queue.cc:237] T 8afc01d52d784f2fa559cab7ce07c8be P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:03.886036 1176 tablet_replica.cc:333] T 2242f5b2c20346d9b753d7cd29242441 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:03.886431 1176 raft_consensus.cc:2243] T 2242f5b2c20346d9b753d7cd29242441 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:03.888274 1176 pending_rounds.cc:70] T 2242f5b2c20346d9b753d7cd29242441 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Trying to abort 1 pending ops.
I20251029 02:52:03.888235 522 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_151d1210897f41ceb7e07cca8a1e478c" } new_table_name: "default.Table_2f446f2f3f1d4c84a5a2dedc15d8300e" modify_external_catalogs: true
W20251029 02:52:03.889187 989 leader_election.cc:343] T 6cd7009e06a646acad0802918c2b425e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 6cd7009e06a646acad0802918c2b425e
I20251029 02:52:03.889703 1172 tablet_bootstrap.cc:492] T 71068739b4384badbf305313d902f0f5 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
W20251029 02:52:03.890000 989 leader_election.cc:343] T 6feec2c8e030447fae9c292a626dcd1d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 6feec2c8e030447fae9c292a626dcd1d
W20251029 02:52:03.890902 989 leader_election.cc:343] T 6cd7009e06a646acad0802918c2b425e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 6cd7009e06a646acad0802918c2b425e
I20251029 02:52:03.891563 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 590) took 2150 ms. Trace:
I20251029 02:52:03.891665 1175 rpcz_store.cc:276] 1029 02:52:01.741062 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:01.741109 (+ 47us) service_pool.cc:225] Handling call
1029 02:52:03.826880 (+2085771us) ts_tablet_manager.cc:1151] Deleting tablet 6142018e44af4d32818adefa2ee43d02
1029 02:52:03.891427 (+ 64547us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:03.891542 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2071190}
I20251029 02:52:03.892375 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 591) took 2145 ms. Trace:
I20251029 02:52:03.892472 1175 rpcz_store.cc:276] 1029 02:52:01.746926 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:01.747539 (+ 613us) service_pool.cc:225] Handling call
1029 02:52:03.891971 (+2144432us) ts_tablet_manager.cc:1151] Deleting tablet b3de6670357c4ae59eb77469b2fb434c
1029 02:52:03.892259 (+ 288us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:03.892357 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2135682}
I20251029 02:52:03.892830 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 592) took 2145 ms. Trace:
I20251029 02:52:03.892903 1175 rpcz_store.cc:276] 1029 02:52:01.747065 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:01.747794 (+ 729us) service_pool.cc:225] Handling call
1029 02:52:03.892716 (+2144922us) ts_tablet_manager.cc:1151] Deleting tablet 217144957930495aa3e9dfcfde268e71
1029 02:52:03.892726 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:03.892811 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2136352}
I20251029 02:52:03.893222 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 593) took 2143 ms. Trace:
I20251029 02:52:03.893287 1175 rpcz_store.cc:276] 1029 02:52:01.749667 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:01.749717 (+ 50us) service_pool.cc:225] Handling call
1029 02:52:03.893129 (+2143412us) ts_tablet_manager.cc:1151] Deleting tablet d041e87ceb11441983d0e38b7929c3a4
1029 02:52:03.893138 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:03.893209 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2136720}
I20251029 02:52:03.893577 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 601) took 1967 ms. Trace:
I20251029 02:52:03.893644 1175 rpcz_store.cc:276] 1029 02:52:01.925894 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:01.928908 (+ 3014us) service_pool.cc:225] Handling call
1029 02:52:03.893486 (+1964578us) ts_tablet_manager.cc:1151] Deleting tablet c64d6761ef924ccaa9c2579f8ba47e33
1029 02:52:03.893496 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:03.893564 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1954803}
I20251029 02:52:03.893975 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 603) took 1896 ms. Trace:
I20251029 02:52:03.894047 1175 rpcz_store.cc:276] 1029 02:52:01.997252 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:01.997307 (+ 55us) service_pool.cc:225] Handling call
1029 02:52:03.893871 (+1896564us) ts_tablet_manager.cc:1151] Deleting tablet 8f1d5847265b4bb0b29b5e513c35b9bb
1029 02:52:03.893883 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:03.893958 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1896346}
I20251029 02:52:03.894344 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 604) took 1890 ms. Trace:
I20251029 02:52:03.894918 1172 tablet_bootstrap.cc:654] T 71068739b4384badbf305313d902f0f5 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
W20251029 02:52:03.895653 847 leader_election.cc:343] T 3362e95063af41c8a24e935f547e3089 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 3362e95063af41c8a24e935f547e3089
I20251029 02:52:03.894414 1175 rpcz_store.cc:276] 1029 02:52:02.003784 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:02.003860 (+ 76us) service_pool.cc:225] Handling call
1029 02:52:03.894257 (+1890397us) ts_tablet_manager.cc:1151] Deleting tablet 5ef87f7a3dff4f00a0e7273eaec98dff
1029 02:52:03.894266 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:03.894332 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1890170}
I20251029 02:52:03.896006 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 605) took 1885 ms. Trace:
W20251029 02:52:03.896057 847 leader_election.cc:343] T faa835de84ff45b19525d1d603678987 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: faa835de84ff45b19525d1d603678987
I20251029 02:52:03.896087 1175 rpcz_store.cc:276] 1029 02:52:02.010475 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:02.010590 (+ 115us) service_pool.cc:225] Handling call
1029 02:52:03.895894 (+1885304us) ts_tablet_manager.cc:1151] Deleting tablet e65eaaae43ac435ebd665546de9d68dd
1029 02:52:03.895905 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:03.895990 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1885066}
W20251029 02:52:03.896358 847 leader_election.cc:343] T 392a36a6b59a42f78e3ac7c8fb63214b P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 392a36a6b59a42f78e3ac7c8fb63214b
I20251029 02:52:03.896395 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 606) took 1880 ms. Trace:
I20251029 02:52:03.896459 1175 rpcz_store.cc:276] 1029 02:52:02.015644 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:02.015692 (+ 48us) service_pool.cc:225] Handling call
1029 02:52:03.896300 (+1880608us) ts_tablet_manager.cc:1151] Deleting tablet e3000183230e4350b47f5085952f2db6
1029 02:52:03.896310 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:03.896382 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1880404}
I20251029 02:52:03.896749 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 607) took 1840 ms. Trace:
I20251029 02:52:03.896827 1175 rpcz_store.cc:276] 1029 02:52:02.056257 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:02.056328 (+ 71us) service_pool.cc:225] Handling call
1029 02:52:03.896655 (+1840327us) ts_tablet_manager.cc:1151] Deleting tablet 98bbaec01186435b8bcc7257a1390185
1029 02:52:03.896664 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:03.896736 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1840073}
I20251029 02:52:03.897106 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 608) took 1840 ms. Trace:
I20251029 02:52:03.897176 1175 rpcz_store.cc:276] 1029 02:52:02.056408 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:02.056621 (+ 213us) service_pool.cc:225] Handling call
1029 02:52:03.897016 (+1840395us) ts_tablet_manager.cc:1151] Deleting tablet c7b78fd458e4414b9fc3aa064a80b206
1029 02:52:03.897026 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:03.897092 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1840235}
I20251029 02:52:03.897522 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 630) took 1011 ms. Trace:
I20251029 02:52:03.897594 1175 rpcz_store.cc:276] 1029 02:52:02.886165 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:02.886245 (+ 80us) service_pool.cc:225] Handling call
1029 02:52:03.897416 (+1011171us) ts_tablet_manager.cc:1151] Deleting tablet 628a7d0419cb434f8171faf500d585d8
1029 02:52:03.897426 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:03.897509 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1010923}
I20251029 02:52:03.897852 1175 tablet_replica.cc:333] T b2b2d64dcf964da8be4b544dbbe31977 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:03.898072 1175 raft_consensus.cc:2243] T b2b2d64dcf964da8be4b544dbbe31977 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:03.898240 1175 raft_consensus.cc:2272] T b2b2d64dcf964da8be4b544dbbe31977 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:03.898758 1174 ts_tablet_manager.cc:1916] T 65fdec7cf5d143208d78d4c3806b1352 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 02:52:03.891371 989 leader_election.cc:343] T 8afc01d52d784f2fa559cab7ce07c8be P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 8afc01d52d784f2fa559cab7ce07c8be
W20251029 02:52:03.899462 989 consensus_peers.cc:597] T 1fc87e2d7b4240e08f2b4ac7277293ff P 0cfd4bee16f84b7c8b0ff3ce14db20d5 -> Peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Couldn't send request to peer b495a49349324f7d81908f2da700f0d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1fc87e2d7b4240e08f2b4ac7277293ff. This is attempt 1: this message will repeat every 5th retry.
W20251029 02:52:03.899747 989 leader_election.cc:343] T c65469173762446e9725262ae998dbf3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: c65469173762446e9725262ae998dbf3
I20251029 02:52:03.901230 1839 raft_consensus.cc:493] T 5695b5a7b1854c3983e2d4ab684898b0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:03.901432 1839 raft_consensus.cc:515] T 5695b5a7b1854c3983e2d4ab684898b0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:03.902578 1839 leader_election.cc:290] T 5695b5a7b1854c3983e2d4ab684898b0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:03.902874 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5695b5a7b1854c3983e2d4ab684898b0" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:03.903093 920 raft_consensus.cc:2468] T 5695b5a7b1854c3983e2d4ab684898b0 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 0.
I20251029 02:52:03.903470 990 leader_election.cc:304] T 5695b5a7b1854c3983e2d4ab684898b0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:03.903926 1839 raft_consensus.cc:2804] T 5695b5a7b1854c3983e2d4ab684898b0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:03.904078 1839 raft_consensus.cc:493] T 5695b5a7b1854c3983e2d4ab684898b0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:03.904222 1839 raft_consensus.cc:3060] T 5695b5a7b1854c3983e2d4ab684898b0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:03.906677 1175 ts_tablet_manager.cc:1916] T b2b2d64dcf964da8be4b544dbbe31977 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 02:52:03.907352 989 leader_election.cc:343] T 8afc01d52d784f2fa559cab7ce07c8be P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 8afc01d52d784f2fa559cab7ce07c8be
W20251029 02:52:03.908300 989 leader_election.cc:343] T 5695b5a7b1854c3983e2d4ab684898b0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 5695b5a7b1854c3983e2d4ab684898b0
I20251029 02:52:03.909037 1839 raft_consensus.cc:515] T 5695b5a7b1854c3983e2d4ab684898b0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:03.909912 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5695b5a7b1854c3983e2d4ab684898b0" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
I20251029 02:52:03.910113 920 raft_consensus.cc:3060] T 5695b5a7b1854c3983e2d4ab684898b0 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:03.912761 1176 pending_rounds.cc:77] T 2242f5b2c20346d9b753d7cd29242441 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215949100364963840 op_type: NO_OP noop_request { }
I20251029 02:52:03.914297 1176 raft_consensus.cc:2889] T 2242f5b2c20346d9b753d7cd29242441 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 02:52:03.914516 1176 raft_consensus.cc:2272] T 2242f5b2c20346d9b753d7cd29242441 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:03.915031 920 raft_consensus.cc:2468] T 5695b5a7b1854c3983e2d4ab684898b0 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 1.
I20251029 02:52:03.915421 990 leader_election.cc:304] T 5695b5a7b1854c3983e2d4ab684898b0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:03.915704 1906 raft_consensus.cc:2804] T 5695b5a7b1854c3983e2d4ab684898b0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:03.915863 1906 raft_consensus.cc:697] T 5695b5a7b1854c3983e2d4ab684898b0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Becoming Leader. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Running, Role: LEADER
I20251029 02:52:03.916173 1906 consensus_queue.cc:237] T 5695b5a7b1854c3983e2d4ab684898b0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:03.918171 1176 ts_tablet_manager.cc:1916] T 2242f5b2c20346d9b753d7cd29242441 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:03.919715 1167 tablet_bootstrap.cc:492] T 7309d3190cfa415281c5253a634fa7f3 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:03.919862 1167 ts_tablet_manager.cc:1403] T 7309d3190cfa415281c5253a634fa7f3 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.113s user 0.003s sys 0.000s
I20251029 02:52:03.920527 1167 raft_consensus.cc:359] T 7309d3190cfa415281c5253a634fa7f3 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:03.920684 1167 raft_consensus.cc:385] T 7309d3190cfa415281c5253a634fa7f3 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:03.920742 1167 raft_consensus.cc:740] T 7309d3190cfa415281c5253a634fa7f3 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:03.920939 1167 consensus_queue.cc:260] T 7309d3190cfa415281c5253a634fa7f3 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:03.921407 1167 ts_tablet_manager.cc:1434] T 7309d3190cfa415281c5253a634fa7f3 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251029 02:52:03.921650 1167 tablet_bootstrap.cc:492] T dbc2092b63e0417997565c8fadb53ae0 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
W20251029 02:52:03.892241 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 6142018e44af4d32818adefa2ee43d02 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6142018e44af4d32818adefa2ee43d02 already in progress: creating tablet
W20251029 02:52:03.934988 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet b3de6670357c4ae59eb77469b2fb434c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b3de6670357c4ae59eb77469b2fb434c already in progress: creating tablet
W20251029 02:52:03.935228 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 217144957930495aa3e9dfcfde268e71 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 217144957930495aa3e9dfcfde268e71 already in progress: creating tablet
W20251029 02:52:03.935367 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet d041e87ceb11441983d0e38b7929c3a4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d041e87ceb11441983d0e38b7929c3a4 already in progress: creating tablet
W20251029 02:52:03.935513 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet c64d6761ef924ccaa9c2579f8ba47e33 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c64d6761ef924ccaa9c2579f8ba47e33 already in progress: creating tablet
W20251029 02:52:03.935648 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 8f1d5847265b4bb0b29b5e513c35b9bb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8f1d5847265b4bb0b29b5e513c35b9bb already in progress: creating tablet
W20251029 02:52:03.935803 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 5ef87f7a3dff4f00a0e7273eaec98dff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5ef87f7a3dff4f00a0e7273eaec98dff already in progress: creating tablet
W20251029 02:52:03.935992 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet e65eaaae43ac435ebd665546de9d68dd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e65eaaae43ac435ebd665546de9d68dd already in progress: creating tablet
W20251029 02:52:03.936131 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet e3000183230e4350b47f5085952f2db6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e3000183230e4350b47f5085952f2db6 already in progress: creating tablet
W20251029 02:52:03.936254 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 98bbaec01186435b8bcc7257a1390185 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 98bbaec01186435b8bcc7257a1390185 already in progress: creating tablet
W20251029 02:52:03.936388 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet c7b78fd458e4414b9fc3aa064a80b206 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c7b78fd458e4414b9fc3aa064a80b206 already in progress: creating tablet
W20251029 02:52:03.936514 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 628a7d0419cb434f8171faf500d585d8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 628a7d0419cb434f8171faf500d585d8 already in progress: creating tablet
I20251029 02:52:03.923367 518 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet cc74e83833754f0ca19046031a24ebb9 (table default.Table_755825b17834457cb94b05163ae3e26d [id=57471769276f44299eae50b9d854b50c]). Expected version 1 got 0
W20251029 02:52:03.927541 526 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 02e4588b03144ea2850e13a1f8f29f06 consensus state has no leader
I20251029 02:52:03.939198 751 tablet_service.cc:1552] Processing DeleteTablet for tablet 56d05e4a500043c995bb51a204cb295d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:53 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:03.940933 1839 leader_election.cc:290] T 5695b5a7b1854c3983e2d4ab684898b0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Requested vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
W20251029 02:52:03.941377 1412 leader_election.cc:336] T 00000000000000000000000000000000 P e5e5222b26fe4145948620e51edf53fb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e9672114a6a742e2864c16ff60ba246e (127.0.104.62:42343): Timed out: RequestConsensusVote RPC to 127.0.104.62:42343 timed out after 0.643s (SENT)
W20251029 02:52:03.941488 989 consensus_peers.cc:597] T a5c3a15510cb4321bbb80d61fabd0c0a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 -> Peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Couldn't send request to peer b495a49349324f7d81908f2da700f0d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a5c3a15510cb4321bbb80d61fabd0c0a. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:03.941579 1412 leader_election.cc:304] T 00000000000000000000000000000000 P e5e5222b26fe4145948620e51edf53fb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5e5222b26fe4145948620e51edf53fb; no voters: e9261eb85ea443d5a3beb36bd0c549f0, e9672114a6a742e2864c16ff60ba246e
W20251029 02:52:03.942643 989 leader_election.cc:343] T 5695b5a7b1854c3983e2d4ab684898b0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 5695b5a7b1854c3983e2d4ab684898b0
I20251029 02:52:03.945255 1175 ts_tablet_manager.cc:1929] T b2b2d64dcf964da8be4b544dbbe31977 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:03.945391 1175 log.cc:1199] T b2b2d64dcf964da8be4b544dbbe31977 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/b2b2d64dcf964da8be4b544dbbe31977
I20251029 02:52:03.945705 1175 ts_tablet_manager.cc:1950] T b2b2d64dcf964da8be4b544dbbe31977 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:03.946789 1172 tablet_bootstrap.cc:492] T 71068739b4384badbf305313d902f0f5 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:03.946942 1172 ts_tablet_manager.cc:1403] T 71068739b4384badbf305313d902f0f5 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.057s user 0.003s sys 0.000s
I20251029 02:52:03.947881 1172 raft_consensus.cc:359] T 71068739b4384badbf305313d902f0f5 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:03.948100 1172 raft_consensus.cc:385] T 71068739b4384badbf305313d902f0f5 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:03.948156 1172 raft_consensus.cc:740] T 71068739b4384badbf305313d902f0f5 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:03.948350 1172 consensus_queue.cc:260] T 71068739b4384badbf305313d902f0f5 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:03.948992 1172 ts_tablet_manager.cc:1434] T 71068739b4384badbf305313d902f0f5 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 02:52:03.950479 1176 ts_tablet_manager.cc:1929] T 2242f5b2c20346d9b753d7cd29242441 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:03.951311 1176 log.cc:1199] T 2242f5b2c20346d9b753d7cd29242441 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/2242f5b2c20346d9b753d7cd29242441
I20251029 02:52:03.951639 1167 tablet_bootstrap.cc:654] T dbc2092b63e0417997565c8fadb53ae0 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:03.952158 1176 ts_tablet_manager.cc:1950] T 2242f5b2c20346d9b753d7cd29242441 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:03.953183 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 2242f5b2c20346d9b753d7cd29242441 (table default.Table_8a5ae35851664e0f85324ec1ca672031 [id=a0e161a5d9d94a3b936669e6dfe6fb4f]) successfully deleted
I20251029 02:52:03.953281 526 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_88c311b8b4534d5190945358ee18fe6a" } new_table_name: "default.Table_91ea489c97794537b7210ff0ce101dbf" modify_external_catalogs: true
I20251029 02:52:03.953941 1176 tablet_replica.cc:333] T a5c3a15510cb4321bbb80d61fabd0c0a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:03.954442 1176 raft_consensus.cc:2243] T a5c3a15510cb4321bbb80d61fabd0c0a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:03.955076 1176 pending_rounds.cc:70] T a5c3a15510cb4321bbb80d61fabd0c0a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Trying to abort 1 pending ops.
I20251029 02:52:03.955513 1922 raft_consensus.cc:2749] T 00000000000000000000000000000000 P e5e5222b26fe4145948620e51edf53fb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20251029 02:52:03.955552 1174 ts_tablet_manager.cc:1929] T 65fdec7cf5d143208d78d4c3806b1352 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:03.955751 1174 log.cc:1199] T 65fdec7cf5d143208d78d4c3806b1352 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/65fdec7cf5d143208d78d4c3806b1352
I20251029 02:52:03.955950 1176 pending_rounds.cc:77] T a5c3a15510cb4321bbb80d61fabd0c0a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215949099729084416 op_type: NO_OP noop_request { }
I20251029 02:52:03.956175 1174 ts_tablet_manager.cc:1950] T 65fdec7cf5d143208d78d4c3806b1352 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:03.956243 1176 raft_consensus.cc:2889] T a5c3a15510cb4321bbb80d61fabd0c0a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 02:52:03.956576 1176 raft_consensus.cc:2272] T a5c3a15510cb4321bbb80d61fabd0c0a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:03.957158 525 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_ca44b863eb7a446693602cd5053ab82b" } modify_external_catalogs: true
I20251029 02:52:03.957353 525 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_ca44b863eb7a446693602cd5053ab82b" } modify_external_catalogs: true
I20251029 02:52:03.958124 523 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_163ca13775fd4ae28ec7ead8297929c8" } modify_external_catalogs: true
I20251029 02:52:03.958885 1176 ts_tablet_manager.cc:1916] T a5c3a15510cb4321bbb80d61fabd0c0a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:03.959287 1174 tablet_replica.cc:333] T 5695b5a7b1854c3983e2d4ab684898b0 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:03.961010 523 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_163ca13775fd4ae28ec7ead8297929c8" } modify_external_catalogs: true
I20251029 02:52:03.961333 1174 raft_consensus.cc:2243] T 5695b5a7b1854c3983e2d4ab684898b0 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:03.961494 1174 raft_consensus.cc:2272] T 5695b5a7b1854c3983e2d4ab684898b0 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:03.961984 1174 ts_tablet_manager.cc:1916] T 5695b5a7b1854c3983e2d4ab684898b0 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:03.960853 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 65fdec7cf5d143208d78d4c3806b1352 (table default.Table_d1a375b674704cddb1c3bb13c566729e [id=909ecde47c7245b2b84dcd665e779b1e]) successfully deleted
W20251029 02:52:03.962992 511 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet a04bc4d9605d4c049a151d985035bb63 (table default.Table_f54f62cc94f34996970dcaddc2d91d6f [id=bb15ae87ebcd4a39a99982967fd215b7], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet a04bc4d9605d4c049a151d985035bb63 consensus state has no leader
I20251029 02:52:03.966100 1166 tablet_bootstrap.cc:492] T dbc2092b63e0417997565c8fadb53ae0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:03.966252 1166 ts_tablet_manager.cc:1403] T dbc2092b63e0417997565c8fadb53ae0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.361s user 0.037s sys 0.080s
I20251029 02:52:03.967003 1166 raft_consensus.cc:359] T dbc2092b63e0417997565c8fadb53ae0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:03.967218 1166 raft_consensus.cc:385] T dbc2092b63e0417997565c8fadb53ae0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:03.967296 1166 raft_consensus.cc:740] T dbc2092b63e0417997565c8fadb53ae0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:03.967393 518 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_92b1e1bb0c8e445aab37bb66b9dba064" } modify_external_catalogs: true
I20251029 02:52:03.967527 1166 consensus_queue.cc:260] T dbc2092b63e0417997565c8fadb53ae0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:03.967581 518 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_92b1e1bb0c8e445aab37bb66b9dba064" } modify_external_catalogs: true
I20251029 02:52:03.968030 1166 ts_tablet_manager.cc:1434] T dbc2092b63e0417997565c8fadb53ae0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 02:52:03.968309 1166 tablet_bootstrap.cc:492] T 51effc9324034ae79e2b25cb57a996c1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:03.968600 890 tablet_service.cc:1552] Processing DeleteTablet for tablet 7309d3190cfa415281c5253a634fa7f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:52 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:03.970088 1774 raft_consensus.cc:493] T 1bab90fe904944af948dab3db06b7e34 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:03.970219 1774 raft_consensus.cc:515] T 1bab90fe904944af948dab3db06b7e34 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:03.946478 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 631) took 1046 ms. Trace:
I20251029 02:52:03.970882 1774 leader_election.cc:290] T 1bab90fe904944af948dab3db06b7e34 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:03.970945 1175 rpcz_store.cc:276] 1029 02:52:02.899735 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:02.899801 (+ 66us) service_pool.cc:225] Handling call
1029 02:52:03.897807 (+998006us) ts_tablet_manager.cc:1151] Deleting tablet b2b2d64dcf964da8be4b544dbbe31977
1029 02:52:03.897816 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:03.910980 (+ 13164us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:03.945129 (+ 34149us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:03.946458 (+ 1329us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":997793}
I20251029 02:52:03.971346 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet b2b2d64dcf964da8be4b544dbbe31977 (table default.Table_e0e6e47b18d749e8ae80f822af1e23f0 [id=125ae5c7b5074378bcecb546e70f0c13]) successfully deleted
I20251029 02:52:03.973193 1058 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1bab90fe904944af948dab3db06b7e34" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:03.973399 1058 raft_consensus.cc:2468] T 1bab90fe904944af948dab3db06b7e34 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:03.974073 849 leader_election.cc:304] T 1bab90fe904944af948dab3db06b7e34 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:03.974787 1172 tablet_bootstrap.cc:492] T f60e7c24982d40319801869ad060864d P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:03.975019 517 master_service.cc:954] ReplaceTablet: received request to replace tablet caef1c93e33a4390bcada59a41f43465 from {username='slave'} at 127.0.0.1:35314
I20251029 02:52:03.975172 1774 raft_consensus.cc:2804] T 1bab90fe904944af948dab3db06b7e34 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:03.975244 1774 raft_consensus.cc:493] T 1bab90fe904944af948dab3db06b7e34 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:03.975301 1774 raft_consensus.cc:3060] T 1bab90fe904944af948dab3db06b7e34 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:03.976447 1172 tablet_bootstrap.cc:654] T f60e7c24982d40319801869ad060864d P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
W20251029 02:52:03.978042 847 leader_election.cc:343] T 1bab90fe904944af948dab3db06b7e34 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 1bab90fe904944af948dab3db06b7e34
I20251029 02:52:03.978618 1774 raft_consensus.cc:515] T 1bab90fe904944af948dab3db06b7e34 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:03.981155 1176 ts_tablet_manager.cc:1929] T a5c3a15510cb4321bbb80d61fabd0c0a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:03.981423 1176 log.cc:1199] T a5c3a15510cb4321bbb80d61fabd0c0a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/a5c3a15510cb4321bbb80d61fabd0c0a
I20251029 02:52:03.981876 1176 ts_tablet_manager.cc:1950] T a5c3a15510cb4321bbb80d61fabd0c0a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:03.982494 1176 tablet_replica.cc:333] T 1fc87e2d7b4240e08f2b4ac7277293ff P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
W20251029 02:52:03.982986 847 leader_election.cc:343] T 1bab90fe904944af948dab3db06b7e34 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 1bab90fe904944af948dab3db06b7e34
I20251029 02:52:03.983952 1774 leader_election.cc:290] T 1bab90fe904944af948dab3db06b7e34 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:03.982479 520 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet a945d64287f94ca697447e540fab39ce
I20251029 02:52:03.985731 520 catalog_manager.cc:6719] ReplaceTablet: tablet a945d64287f94ca697447e540fab39ce deleted and replaced by tablet 6fcf3e12810d41ddb96d9f2cf52f2c5d
I20251029 02:52:03.987448 1176 raft_consensus.cc:2243] T 1fc87e2d7b4240e08f2b4ac7277293ff P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:03.987792 1176 raft_consensus.cc:2272] T 1fc87e2d7b4240e08f2b4ac7277293ff P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:03.983815 1058 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1bab90fe904944af948dab3db06b7e34" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:52:03.988443 1058 raft_consensus.cc:3060] T 1bab90fe904944af948dab3db06b7e34 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:03.989662 1175 tablet_replica.cc:333] T 70e8cfe5eaa9492c85874dc3214a1e0b P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:03.989851 1175 raft_consensus.cc:2243] T 70e8cfe5eaa9492c85874dc3214a1e0b P b495a49349324f7d81908f2da700f0d9 [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:03.990142 1175 raft_consensus.cc:2272] T 70e8cfe5eaa9492c85874dc3214a1e0b P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:03.982703 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet a5c3a15510cb4321bbb80d61fabd0c0a (table default.Table_8cf076425649436289d327edc92d4642 [id=c124cfa700724f6db0f1a12fe495c2af]) successfully deleted
I20251029 02:52:03.990657 1175 ts_tablet_manager.cc:1916] T 70e8cfe5eaa9492c85874dc3214a1e0b P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:03.991183 1041 tablet_service.cc:1552] Processing DeleteTablet for tablet a945d64287f94ca697447e540fab39ce with delete_type TABLET_DATA_DELETED (replaced by tablet 6fcf3e12810d41ddb96d9f2cf52f2c5d) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:03.991236 1167 tablet_bootstrap.cc:492] T dbc2092b63e0417997565c8fadb53ae0 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:03.991349 1167 ts_tablet_manager.cc:1403] T dbc2092b63e0417997565c8fadb53ae0 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.070s user 0.003s sys 0.000s
I20251029 02:52:03.992079 1167 raft_consensus.cc:359] T dbc2092b63e0417997565c8fadb53ae0 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:03.992225 1167 raft_consensus.cc:385] T dbc2092b63e0417997565c8fadb53ae0 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:03.992265 1167 raft_consensus.cc:740] T dbc2092b63e0417997565c8fadb53ae0 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:03.992475 1167 consensus_queue.cc:260] T dbc2092b63e0417997565c8fadb53ae0 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:03.993186 1176 ts_tablet_manager.cc:1916] T 1fc87e2d7b4240e08f2b4ac7277293ff P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 02:52:03.983071 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 1fa8cc0c60cd43cdaa7a7646f70cfb97 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1fa8cc0c60cd43cdaa7a7646f70cfb97 already in progress: creating tablet
W20251029 02:52:03.995134 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 699ee74f39b342639ab84d0d65e53a70 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 699ee74f39b342639ab84d0d65e53a70 already in progress: creating tablet
W20251029 02:52:03.995357 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet a47821980eb4413b8af40eb5415b5d72 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a47821980eb4413b8af40eb5415b5d72 already in progress: creating tablet
W20251029 02:52:03.995568 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet ca7f7e8323f94ef1b106e14e6d87a602 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ca7f7e8323f94ef1b106e14e6d87a602 already in progress: creating tablet
I20251029 02:52:03.995716 1166 tablet_bootstrap.cc:654] T 51effc9324034ae79e2b25cb57a996c1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
W20251029 02:52:03.995774 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet fe24183d24e34579b96e6fb6bc0a914e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fe24183d24e34579b96e6fb6bc0a914e already in progress: creating tablet
I20251029 02:52:03.995764 751 tablet_service.cc:1552] Processing DeleteTablet for tablet a945d64287f94ca697447e540fab39ce with delete_type TABLET_DATA_DELETED (replaced by tablet 6fcf3e12810d41ddb96d9f2cf52f2c5d) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:03.999717 890 tablet_service.cc:1552] Processing DeleteTablet for tablet a945d64287f94ca697447e540fab39ce with delete_type TABLET_DATA_DELETED (replaced by tablet 6fcf3e12810d41ddb96d9f2cf52f2c5d) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:04.000164 1058 raft_consensus.cc:2468] T 1bab90fe904944af948dab3db06b7e34 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:04.001051 849 leader_election.cc:304] T 1bab90fe904944af948dab3db06b7e34 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters: b495a49349324f7d81908f2da700f0d9
I20251029 02:52:04.002635 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_e0978085f4ba4e798435ad96267706ac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:04.003290 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e0978085f4ba4e798435ad96267706ac in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:04.003456 521 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_73eec04729fd4870aca2779a5738d772" } new_table_name: "default.Table_cb4697efc8e049be929f4e7a004879d1" modify_external_catalogs: true
I20251029 02:52:04.003870 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_ac619c8d8e8a4b8bba9247c4b0b1c65e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:04.004182 890 tablet_service.cc:1552] Processing DeleteTablet for tablet dbc2092b63e0417997565c8fadb53ae0 with delete_type TABLET_DATA_DELETED (replaced by tablet 290d9d46914b4f328c7f2d9afe901889) from {username='slave'} at 127.0.0.1:35702
W20251029 02:52:04.004316 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ac619c8d8e8a4b8bba9247c4b0b1c65e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:52:04.005465 526 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8c972c88f64e486bb2b75306549bb23f consensus state has no leader
I20251029 02:52:04.005795 526 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_de3500a0d5f64dd788359ae260ffd539" } modify_external_catalogs: true
I20251029 02:52:04.005892 1032 tablet_service.cc:1552] Processing DeleteTablet for tablet dbc2092b63e0417997565c8fadb53ae0 with delete_type TABLET_DATA_DELETED (replaced by tablet 290d9d46914b4f328c7f2d9afe901889) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:04.005965 751 tablet_service.cc:1552] Processing DeleteTablet for tablet 71068739b4384badbf305313d902f0f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:56 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:04.005975 526 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_de3500a0d5f64dd788359ae260ffd539" } modify_external_catalogs: true
I20251029 02:52:04.006619 1774 raft_consensus.cc:2804] T 1bab90fe904944af948dab3db06b7e34 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:04.007088 1774 raft_consensus.cc:697] T 1bab90fe904944af948dab3db06b7e34 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:52:04.007328 1774 consensus_queue.cc:237] T 1bab90fe904944af948dab3db06b7e34 P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
W20251029 02:52:04.008358 522 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 67c3b526a5cd4da7800b7b72b0ebe29d consensus state has no leader
I20251029 02:52:04.012786 1176 ts_tablet_manager.cc:1929] T 1fc87e2d7b4240e08f2b4ac7277293ff P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:04.013212 522 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_d1050ce4c18f47c89dab970ce5edc755" } new_table_name: "default.Table_c32c91e69cf94a6fbc1a6de2109788e8" modify_external_catalogs: true
I20251029 02:52:04.018388 1167 ts_tablet_manager.cc:1434] T dbc2092b63e0417997565c8fadb53ae0 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.027s user 0.001s sys 0.000s
I20251029 02:52:04.018615 1176 log.cc:1199] T 1fc87e2d7b4240e08f2b4ac7277293ff P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/1fc87e2d7b4240e08f2b4ac7277293ff
I20251029 02:52:04.019064 1176 ts_tablet_manager.cc:1950] T 1fc87e2d7b4240e08f2b4ac7277293ff P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:04.019726 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 1fc87e2d7b4240e08f2b4ac7277293ff (table default.Table_9306136142dd41cca83965ab5a92c5a0 [id=130ea33221324d7388a014abd549b2ff]) successfully deleted
W20251029 02:52:04.019932 521 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 35d9b8dfd61e4d8db441427a49fb9dca consensus state has no leader
I20251029 02:52:04.019945 1906 raft_consensus.cc:493] T 24cb2a71ba3841d4a5e3e5c6fe99ae2f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:04.020073 1906 raft_consensus.cc:515] T 24cb2a71ba3841d4a5e3e5c6fe99ae2f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:04.020215 521 master_service.cc:954] ReplaceTablet: received request to replace tablet 2f198c3259864f7ab3625cb7bfae20b0 from {username='slave'} at 127.0.0.1:35314
I20251029 02:52:04.020489 1176 tablet_replica.cc:333] T faa835de84ff45b19525d1d603678987 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:04.020686 1906 leader_election.cc:290] T 24cb2a71ba3841d4a5e3e5c6fe99ae2f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:04.021039 1167 tablet_bootstrap.cc:492] T 1f5b40c0403d4a49a4bfde222d8c947a P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:04.020987 1176 raft_consensus.cc:2243] T faa835de84ff45b19525d1d603678987 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:04.021513 1176 raft_consensus.cc:2272] T faa835de84ff45b19525d1d603678987 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:04.021733 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24cb2a71ba3841d4a5e3e5c6fe99ae2f" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:04.021891 989 leader_election.cc:343] T 24cb2a71ba3841d4a5e3e5c6fe99ae2f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 24cb2a71ba3841d4a5e3e5c6fe99ae2f
I20251029 02:52:04.022207 1176 ts_tablet_manager.cc:1916] T faa835de84ff45b19525d1d603678987 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 02:52:04.022270 990 leader_election.cc:343] T 24cb2a71ba3841d4a5e3e5c6fe99ae2f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:04.022372 990 leader_election.cc:304] T 24cb2a71ba3841d4a5e3e5c6fe99ae2f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5; no voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:04.022801 1906 raft_consensus.cc:2749] T 24cb2a71ba3841d4a5e3e5c6fe99ae2f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:04.023454 1167 tablet_bootstrap.cc:654] T 1f5b40c0403d4a49a4bfde222d8c947a P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
W20251029 02:52:04.024551 522 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 643e20d9974447daa78885b3401f3b77 consensus state has no leader
I20251029 02:52:04.027647 1175 ts_tablet_manager.cc:1929] T 70e8cfe5eaa9492c85874dc3214a1e0b P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20251029 02:52:04.027781 1175 log.cc:1199] T 70e8cfe5eaa9492c85874dc3214a1e0b P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/70e8cfe5eaa9492c85874dc3214a1e0b
I20251029 02:52:04.028221 1175 ts_tablet_manager.cc:1950] T 70e8cfe5eaa9492c85874dc3214a1e0b P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:04.031353 1172 tablet_bootstrap.cc:492] T f60e7c24982d40319801869ad060864d P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:04.031490 1172 ts_tablet_manager.cc:1403] T f60e7c24982d40319801869ad060864d P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.057s user 0.001s sys 0.001s
I20251029 02:52:04.032228 1172 raft_consensus.cc:359] T f60e7c24982d40319801869ad060864d P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:04.032380 1172 raft_consensus.cc:385] T f60e7c24982d40319801869ad060864d P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:04.032430 1172 raft_consensus.cc:740] T f60e7c24982d40319801869ad060864d P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:04.039615 522 master_service.cc:954] ReplaceTablet: received request to replace tablet 72e91c64a9b54cbda52ca9dfcee2ce22 from {username='slave'} at 127.0.0.1:35314
I20251029 02:52:04.040252 1172 consensus_queue.cc:260] T f60e7c24982d40319801869ad060864d P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:04.041942 1172 ts_tablet_manager.cc:1434] T f60e7c24982d40319801869ad060864d P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20251029 02:52:04.042939 1167 tablet_bootstrap.cc:492] T 1f5b40c0403d4a49a4bfde222d8c947a P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:04.043079 1167 ts_tablet_manager.cc:1403] T 1f5b40c0403d4a49a4bfde222d8c947a P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.001s
I20251029 02:52:04.043859 1167 raft_consensus.cc:359] T 1f5b40c0403d4a49a4bfde222d8c947a P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:04.044031 1167 raft_consensus.cc:385] T 1f5b40c0403d4a49a4bfde222d8c947a P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:04.044090 1167 raft_consensus.cc:740] T 1f5b40c0403d4a49a4bfde222d8c947a P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:04.043987 1176 ts_tablet_manager.cc:1929] T faa835de84ff45b19525d1d603678987 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:04.044272 1176 log.cc:1199] T faa835de84ff45b19525d1d603678987 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/faa835de84ff45b19525d1d603678987
I20251029 02:52:04.044675 1176 ts_tablet_manager.cc:1950] T faa835de84ff45b19525d1d603678987 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
W20251029 02:52:04.045051 1412 connection.cc:537] client connection to 127.0.104.62:42343 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251029 02:52:04.045177 1176 tablet_replica.cc:333] T 44784a3a760946e0be24743bc2fc8138 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:04.045326 1176 raft_consensus.cc:2243] T 44784a3a760946e0be24743bc2fc8138 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:04.045313 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet faa835de84ff45b19525d1d603678987 (table default.Table_d9e164af67334c98a5d0821f2d80a35e [id=a184f139921d4864b1627057a7359810]) successfully deleted
I20251029 02:52:04.045555 1176 pending_rounds.cc:70] T 44784a3a760946e0be24743bc2fc8138 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Trying to abort 1 pending ops.
I20251029 02:52:04.045655 1176 pending_rounds.cc:77] T 44784a3a760946e0be24743bc2fc8138 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215949098968678400 op_type: NO_OP noop_request { }
I20251029 02:52:04.045833 1176 raft_consensus.cc:2889] T 44784a3a760946e0be24743bc2fc8138 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 02:52:04.045976 1176 raft_consensus.cc:2272] T 44784a3a760946e0be24743bc2fc8138 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251029 02:52:04.045727 988 connection.cc:537] client connection to 127.0.104.62:42343 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251029 02:52:04.046960 1176 ts_tablet_manager.cc:1916] T 44784a3a760946e0be24743bc2fc8138 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:04.053397 1174 ts_tablet_manager.cc:1929] T 5695b5a7b1854c3983e2d4ab684898b0 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:04.054765 1665 raft_consensus.cc:493] T 56d05e4a500043c995bb51a204cb295d P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:04.054910 1665 raft_consensus.cc:515] T 56d05e4a500043c995bb51a204cb295d P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:04.055521 1665 leader_election.cc:290] T 56d05e4a500043c995bb51a204cb295d P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:04.056740 1166 tablet_bootstrap.cc:492] T 51effc9324034ae79e2b25cb57a996c1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:04.056871 1166 ts_tablet_manager.cc:1403] T 51effc9324034ae79e2b25cb57a996c1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.089s user 0.003s sys 0.000s
I20251029 02:52:04.056919 1665 raft_consensus.cc:493] T cc74e83833754f0ca19046031a24ebb9 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:04.057024 1665 raft_consensus.cc:515] T cc74e83833754f0ca19046031a24ebb9 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:04.057499 1665 leader_election.cc:290] T cc74e83833754f0ca19046031a24ebb9 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:04.057739 1166 raft_consensus.cc:359] T 51effc9324034ae79e2b25cb57a996c1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:04.057890 1166 raft_consensus.cc:385] T 51effc9324034ae79e2b25cb57a996c1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:04.057950 1166 raft_consensus.cc:740] T 51effc9324034ae79e2b25cb57a996c1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:04.058109 1166 consensus_queue.cc:260] T 51effc9324034ae79e2b25cb57a996c1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:04.058552 1166 ts_tablet_manager.cc:1434] T 51effc9324034ae79e2b25cb57a996c1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 02:52:04.058907 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cc74e83833754f0ca19046031a24ebb9" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:52:04.059433 712 leader_election.cc:343] T cc74e83833754f0ca19046031a24ebb9 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:04.056885 1058 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "56d05e4a500043c995bb51a204cb295d" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:04.061547 1166 tablet_bootstrap.cc:492] T 7e6c4f30be624769961c9ec17c5d5dcb P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
W20251029 02:52:04.061549 712 leader_election.cc:343] T 56d05e4a500043c995bb51a204cb295d P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
W20251029 02:52:04.063525 709 connection.cc:537] client connection to 127.0.104.62:42343 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251029 02:52:04.063709 510 connection.cc:537] client connection to 127.0.104.62:42343 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20251029 02:52:04.063868 510 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107) [suppressed 8 similar messages]
I20251029 02:52:04.063927 1665 raft_consensus.cc:493] T 0cc1957a845f4e9cb703a553b27d1347 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:04.064028 1665 raft_consensus.cc:515] T 0cc1957a845f4e9cb703a553b27d1347 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:04.064211 492 master_runner.cc:186] Error getting master registration for 127.0.104.62:42343: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107),
I20251029 02:52:04.064333 492 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251029 02:52:04.064733 1665 leader_election.cc:290] T 0cc1957a845f4e9cb703a553b27d1347 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:04.067377 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_b1b81bdd58e14acd835564ba6c9f33da"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:04.068187 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b1b81bdd58e14acd835564ba6c9f33da in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:04.068975 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_a7906004ed1742f39fec333e17ab26de"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:04.069463 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a7906004ed1742f39fec333e17ab26de in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:04.070421 525 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 1e28a07b1a6242cd919b6ab2cb9b8a92
I20251029 02:52:04.070569 1058 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0cc1957a845f4e9cb703a553b27d1347" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:52:04.071069 712 leader_election.cc:343] T 0cc1957a845f4e9cb703a553b27d1347 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:04.071458 751 tablet_service.cc:1552] Processing DeleteTablet for tablet 1e28a07b1a6242cd919b6ab2cb9b8a92 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:03 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:04.071794 1032 tablet_service.cc:1552] Processing DeleteTablet for tablet 51effc9324034ae79e2b25cb57a996c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:56 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:04.057385 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "56d05e4a500043c995bb51a204cb295d" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:04.074743 1174 log.cc:1199] T 5695b5a7b1854c3983e2d4ab684898b0 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/5695b5a7b1854c3983e2d4ab684898b0
I20251029 02:52:04.057900 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cc74e83833754f0ca19046031a24ebb9" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:04.062731 846 connection.cc:537] client connection to 127.0.104.62:42343 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251029 02:52:04.075131 1041 tablet_service.cc:1552] Processing DeleteTablet for tablet 1e28a07b1a6242cd919b6ab2cb9b8a92 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:03 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:04.075367 1174 ts_tablet_manager.cc:1950] T 5695b5a7b1854c3983e2d4ab684898b0 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:04.072286 890 tablet_service.cc:1552] Processing DeleteTablet for tablet 1e28a07b1a6242cd919b6ab2cb9b8a92 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:03 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:04.076314 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0cc1957a845f4e9cb703a553b27d1347" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:04.076475 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 5695b5a7b1854c3983e2d4ab684898b0 (table default.Table_e43a2189b18a43f9beb9f3ce25848374 [id=d151d895aac64712b2eeee989d19b878]) successfully deleted
W20251029 02:52:04.076649 711 leader_election.cc:343] T 56d05e4a500043c995bb51a204cb295d P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:04.076756 711 leader_election.cc:304] T 56d05e4a500043c995bb51a204cb295d P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
W20251029 02:52:04.077196 711 leader_election.cc:343] T cc74e83833754f0ca19046031a24ebb9 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:04.077283 711 leader_election.cc:304] T cc74e83833754f0ca19046031a24ebb9 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:04.077538 1896 raft_consensus.cc:2749] T 56d05e4a500043c995bb51a204cb295d P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:04.077754 1896 raft_consensus.cc:2749] T cc74e83833754f0ca19046031a24ebb9 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:04.078712 1174 tablet_replica.cc:333] T 8afc01d52d784f2fa559cab7ce07c8be P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:04.079026 1174 raft_consensus.cc:2243] T 8afc01d52d784f2fa559cab7ce07c8be P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:04.079284 1174 raft_consensus.cc:2272] T 8afc01d52d784f2fa559cab7ce07c8be P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:04.079998 1174 ts_tablet_manager.cc:1916] T 8afc01d52d784f2fa559cab7ce07c8be P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:04.084802 1665 raft_consensus.cc:493] T 71068739b4384badbf305313d902f0f5 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:04.084966 1665 raft_consensus.cc:515] T 71068739b4384badbf305313d902f0f5 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:04.085649 1665 leader_election.cc:290] T 71068739b4384badbf305313d902f0f5 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:04.087078 1058 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "71068739b4384badbf305313d902f0f5" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:52:04.088761 712 leader_election.cc:343] T 71068739b4384badbf305313d902f0f5 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:04.089891 1172 tablet_bootstrap.cc:492] T 02e4588b03144ea2850e13a1f8f29f06 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:04.090070 1175 tablet_replica.cc:333] T 6ca9bd37de9e4be388501b15af4bda31 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:04.090246 1175 raft_consensus.cc:2243] T 6ca9bd37de9e4be388501b15af4bda31 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:04.090413 1175 raft_consensus.cc:2272] T 6ca9bd37de9e4be388501b15af4bda31 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:04.090950 1175 ts_tablet_manager.cc:1916] T 6ca9bd37de9e4be388501b15af4bda31 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:04.091701 1172 tablet_bootstrap.cc:654] T 02e4588b03144ea2850e13a1f8f29f06 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:04.093168 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 70e8cfe5eaa9492c85874dc3214a1e0b (table default.Table_9bf4e25d441f4d4dbd6651386eec512e [id=82df85af001340efb570040e047fa136]) successfully deleted
I20251029 02:52:04.094702 1166 tablet_bootstrap.cc:654] T 7e6c4f30be624769961c9ec17c5d5dcb P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
W20251029 02:52:04.097922 1930 negotiation.cc:337] Failed RPC negotiation. Trace:
1029 02:52:04.066632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:42343 (local address 127.0.104.1:46887)
1029 02:52:04.092306 (+ 25674us) negotiation.cc:107] Waiting for socket to connect
1029 02:52:04.092326 (+ 20us) client_negotiation.cc:174] Beginning negotiation
1029 02:52:04.092455 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1029 02:52:04.097702 (+ 5247us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:42343: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":25543,"threads_started":1}
W20251029 02:52:04.098125 1928 negotiation.cc:337] Failed RPC negotiation. Trace:
1029 02:52:04.047831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:42343 (local address 127.0.104.3:38237)
1029 02:52:04.067560 (+ 19729us) negotiation.cc:107] Waiting for socket to connect
1029 02:52:04.067581 (+ 21us) client_negotiation.cc:174] Beginning negotiation
1029 02:52:04.067703 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1029 02:52:04.097924 (+ 30221us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:42343: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":19598,"thread_start_us":32216,"threads_started":1}
I20251029 02:52:04.100564 1174 ts_tablet_manager.cc:1929] T 8afc01d52d784f2fa559cab7ce07c8be P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:04.100615 751 tablet_service.cc:1552] Processing DeleteTablet for tablet f60e7c24982d40319801869ad060864d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:53 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:04.100713 1174 log.cc:1199] T 8afc01d52d784f2fa559cab7ce07c8be P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/8afc01d52d784f2fa559cab7ce07c8be
I20251029 02:52:04.101009 1174 ts_tablet_manager.cc:1950] T 8afc01d52d784f2fa559cab7ce07c8be P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:04.104799 526 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 4929513bed104050bd6065f025303e1a
I20251029 02:52:04.106000 890 tablet_service.cc:1552] Processing DeleteTablet for tablet 4929513bed104050bd6065f025303e1a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:03 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:04.106215 1041 tablet_service.cc:1552] Processing DeleteTablet for tablet 4929513bed104050bd6065f025303e1a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:03 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:04.106431 751 tablet_service.cc:1552] Processing DeleteTablet for tablet 4929513bed104050bd6065f025303e1a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:03 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:04.107093 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 8afc01d52d784f2fa559cab7ce07c8be (table default.Table_2a10c9dfc91d4974845dbcd500d1d8be [id=ed7295f3f21848b79af75785d3983b67]) successfully deleted
I20251029 02:52:04.107462 1174 tablet_replica.cc:333] T 2242f5b2c20346d9b753d7cd29242441 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
W20251029 02:52:04.107585 1932 negotiation.cc:337] Failed RPC negotiation. Trace:
1029 02:52:04.092005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:42343 (local address 127.0.0.1:44274)
1029 02:52:04.107107 (+ 15102us) negotiation.cc:107] Waiting for socket to connect
1029 02:52:04.107266 (+ 159us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:42343: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":14956,"threads_started":1}
I20251029 02:52:04.107894 1174 raft_consensus.cc:2243] T 2242f5b2c20346d9b753d7cd29242441 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:04.108103 1174 raft_consensus.cc:2272] T 2242f5b2c20346d9b753d7cd29242441 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:04.108639 1174 ts_tablet_manager.cc:1916] T 2242f5b2c20346d9b753d7cd29242441 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 02:52:04.108794 510 consensus_peers.cc:597] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0 -> Peer e9672114a6a742e2864c16ff60ba246e (127.0.104.62:42343): Couldn't send request to peer e9672114a6a742e2864c16ff60ba246e. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.62:42343: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:04.109800 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_a53e7fa11e1f41af89d9421d2c107641"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:04.110450 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a53e7fa11e1f41af89d9421d2c107641 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:04.111801 1167 consensus_queue.cc:260] T 1f5b40c0403d4a49a4bfde222d8c947a P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:04.114289 680 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 4929513bed104050bd6065f025303e1a
I20251029 02:52:04.115335 1167 ts_tablet_manager.cc:1434] T 1f5b40c0403d4a49a4bfde222d8c947a P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.072s user 0.000s sys 0.002s
I20251029 02:52:04.115613 1167 tablet_bootstrap.cc:492] T 51effc9324034ae79e2b25cb57a996c1 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:04.115940 890 tablet_service.cc:1552] Processing DeleteTablet for tablet 4929513bed104050bd6065f025303e1a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:03 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:04.116498 1176 ts_tablet_manager.cc:1929] T 44784a3a760946e0be24743bc2fc8138 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:04.116611 1176 log.cc:1199] T 44784a3a760946e0be24743bc2fc8138 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/44784a3a760946e0be24743bc2fc8138
I20251029 02:52:04.116933 1176 ts_tablet_manager.cc:1950] T 44784a3a760946e0be24743bc2fc8138 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:04.117246 1167 tablet_bootstrap.cc:654] T 51effc9324034ae79e2b25cb57a996c1 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
W20251029 02:52:04.118285 1931 negotiation.cc:337] Failed RPC negotiation. Trace:
1029 02:52:04.077947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:42343 (local address 127.0.104.2:51289)
1029 02:52:04.117940 (+ 39993us) negotiation.cc:107] Waiting for socket to connect
1029 02:52:04.118021 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:42343: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":39873,"threads_started":1}
I20251029 02:52:04.119354 1172 tablet_bootstrap.cc:492] T 02e4588b03144ea2850e13a1f8f29f06 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:04.119542 1172 ts_tablet_manager.cc:1403] T 02e4588b03144ea2850e13a1f8f29f06 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.030s user 0.008s sys 0.000s
I20251029 02:52:04.120298 1172 raft_consensus.cc:359] T 02e4588b03144ea2850e13a1f8f29f06 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
W20251029 02:52:04.120357 711 leader_election.cc:343] T 0cc1957a845f4e9cb703a553b27d1347 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:04.120540 711 leader_election.cc:304] T 0cc1957a845f4e9cb703a553b27d1347 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:04.120975 1665 raft_consensus.cc:2749] T 0cc1957a845f4e9cb703a553b27d1347 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:04.121177 1172 raft_consensus.cc:385] T 02e4588b03144ea2850e13a1f8f29f06 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:04.121299 1172 raft_consensus.cc:740] T 02e4588b03144ea2850e13a1f8f29f06 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:04.121542 1172 consensus_queue.cc:260] T 02e4588b03144ea2850e13a1f8f29f06 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:04.125517 518 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 4e82c574dfa14ffbabf162a17355f841
I20251029 02:52:04.125936 1175 ts_tablet_manager.cc:1929] T 6ca9bd37de9e4be388501b15af4bda31 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:04.126009 1175 log.cc:1199] T 6ca9bd37de9e4be388501b15af4bda31 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/6ca9bd37de9e4be388501b15af4bda31
I20251029 02:52:04.126260 1175 ts_tablet_manager.cc:1950] T 6ca9bd37de9e4be388501b15af4bda31 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:04.126425 751 tablet_service.cc:1552] Processing DeleteTablet for tablet 4e82c574dfa14ffbabf162a17355f841 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:03 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:04.128108 1172 ts_tablet_manager.cc:1434] T 02e4588b03144ea2850e13a1f8f29f06 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20251029 02:52:04.129983 1172 tablet_bootstrap.cc:492] T 86fc22bb84794149b598b97df41eb881 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:04.129856 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_5b80ec00d8f74e97a7a8fa038920b55b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:04.130540 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5b80ec00d8f74e97a7a8fa038920b55b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:04.131034 521 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet 02e4588b03144ea2850e13a1f8f29f06 (table default.Table_0772f9db5d6142d988f6ffca2e96bdd2 [id=fd6d31d379394857a35422211af00642]). Expected version 3 got 0
I20251029 02:52:04.131685 1172 tablet_bootstrap.cc:654] T 86fc22bb84794149b598b97df41eb881 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:04.132028 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_592b4d977e464e6d8f257f3fa08797f3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:04.120775 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "71068739b4384badbf305313d902f0f5" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:04.132514 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_592b4d977e464e6d8f257f3fa08797f3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:04.127013 890 tablet_service.cc:1552] Processing DeleteTablet for tablet 4e82c574dfa14ffbabf162a17355f841 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:03 UTC) from {username='slave'} at 127.0.0.1:35702
W20251029 02:52:04.133033 711 leader_election.cc:343] T 71068739b4384badbf305313d902f0f5 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:04.133147 711 leader_election.cc:304] T 71068739b4384badbf305313d902f0f5 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:04.133567 1665 raft_consensus.cc:2749] T 71068739b4384badbf305313d902f0f5 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:04.128703 1174 ts_tablet_manager.cc:1929] T 2242f5b2c20346d9b753d7cd29242441 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:04.134172 1174 log.cc:1199] T 2242f5b2c20346d9b753d7cd29242441 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/2242f5b2c20346d9b753d7cd29242441
I20251029 02:52:04.134519 1174 ts_tablet_manager.cc:1950] T 2242f5b2c20346d9b753d7cd29242441 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:04.137142 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 2242f5b2c20346d9b753d7cd29242441 (table default.Table_8a5ae35851664e0f85324ec1ca672031 [id=a0e161a5d9d94a3b936669e6dfe6fb4f]) successfully deleted
I20251029 02:52:04.138679 1174 tablet_replica.cc:333] T 392a36a6b59a42f78e3ac7c8fb63214b P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:04.138893 1174 raft_consensus.cc:2243] T 392a36a6b59a42f78e3ac7c8fb63214b P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:04.139034 751 tablet_service.cc:1552] Processing DeleteTablet for tablet 4929513bed104050bd6065f025303e1a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:03 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:04.126835 1041 tablet_service.cc:1552] Processing DeleteTablet for tablet 4e82c574dfa14ffbabf162a17355f841 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:03 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:04.134945 1032 tablet_service.cc:1552] Processing DeleteTablet for tablet 4929513bed104050bd6065f025303e1a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:03 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:04.139776 890 tablet_service.cc:1505] Processing CreateTablet for tablet 50ab70b30aaa4f6bbae15855431159c8 (DEFAULT_TABLE table=default.Table_99f14277f09b47438fb0522c896dd4b4 [id=32f7ed9ceb894c44b2dfd0c92c2acce5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:04.140043 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 44784a3a760946e0be24743bc2fc8138 (table default.Table_3e28d37b4b6146bf95edf245d3f9d0cf [id=915b76276e0940618b7180ebc06cd464]) successfully deleted
I20251029 02:52:04.140153 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50ab70b30aaa4f6bbae15855431159c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:04.140236 751 tablet_service.cc:1505] Processing CreateTablet for tablet 50ab70b30aaa4f6bbae15855431159c8 (DEFAULT_TABLE table=default.Table_99f14277f09b47438fb0522c896dd4b4 [id=32f7ed9ceb894c44b2dfd0c92c2acce5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:04.140385 520 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_c32c91e69cf94a6fbc1a6de2109788e8" } new_table_name: "default.Table_7c1cb64e05be4f6e8ac28286c982a3e7" modify_external_catalogs: true
I20251029 02:52:04.140556 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50ab70b30aaa4f6bbae15855431159c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:04.143309 1041 tablet_service.cc:1505] Processing CreateTablet for tablet 1e17142dc6634db0a5288add9c199ffc (DEFAULT_TABLE table=default.Table_8015f6519aff4d269aa0f16e8cdd96dd [id=6a87697f4e1543ada5aa7689ef2473ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:04.143673 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e17142dc6634db0a5288add9c199ffc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:04.144942 1039 tablet_service.cc:1505] Processing CreateTablet for tablet 4660fd0534df49e0af43c36fb1122d84 (DEFAULT_TABLE table=default.Table_dd632ee443af4a59905aee8a89d0a4bb [id=c5b09bb0414a4e019de7b27873a2eda0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:04.145263 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4660fd0534df49e0af43c36fb1122d84. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:04.146592 1032 tablet_service.cc:1505] Processing CreateTablet for tablet 50ab70b30aaa4f6bbae15855431159c8 (DEFAULT_TABLE table=default.Table_99f14277f09b47438fb0522c896dd4b4 [id=32f7ed9ceb894c44b2dfd0c92c2acce5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:04.146883 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 50ab70b30aaa4f6bbae15855431159c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:04.147074 892 tablet_service.cc:1505] Processing CreateTablet for tablet 1e17142dc6634db0a5288add9c199ffc (DEFAULT_TABLE table=default.Table_8015f6519aff4d269aa0f16e8cdd96dd [id=6a87697f4e1543ada5aa7689ef2473ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:04.147373 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e17142dc6634db0a5288add9c199ffc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:04.150794 895 tablet_service.cc:1505] Processing CreateTablet for tablet 4660fd0534df49e0af43c36fb1122d84 (DEFAULT_TABLE table=default.Table_dd632ee443af4a59905aee8a89d0a4bb [id=c5b09bb0414a4e019de7b27873a2eda0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:04.151124 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4660fd0534df49e0af43c36fb1122d84. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:04.152766 1167 tablet_bootstrap.cc:492] T 51effc9324034ae79e2b25cb57a996c1 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:04.152874 1167 ts_tablet_manager.cc:1403] T 51effc9324034ae79e2b25cb57a996c1 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.037s user 0.001s sys 0.002s
I20251029 02:52:04.153153 751 tablet_service.cc:1505] Processing CreateTablet for tablet 1e17142dc6634db0a5288add9c199ffc (DEFAULT_TABLE table=default.Table_8015f6519aff4d269aa0f16e8cdd96dd [id=6a87697f4e1543ada5aa7689ef2473ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:04.153470 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e17142dc6634db0a5288add9c199ffc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:04.153621 1167 raft_consensus.cc:359] T 51effc9324034ae79e2b25cb57a996c1 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:04.153786 1167 raft_consensus.cc:385] T 51effc9324034ae79e2b25cb57a996c1 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:04.153846 1167 raft_consensus.cc:740] T 51effc9324034ae79e2b25cb57a996c1 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:04.154040 1167 consensus_queue.cc:260] T 51effc9324034ae79e2b25cb57a996c1 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:04.155133 749 tablet_service.cc:1505] Processing CreateTablet for tablet 4660fd0534df49e0af43c36fb1122d84 (DEFAULT_TABLE table=default.Table_dd632ee443af4a59905aee8a89d0a4bb [id=c5b09bb0414a4e019de7b27873a2eda0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:04.155478 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4660fd0534df49e0af43c36fb1122d84. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:04.155511 1166 tablet_bootstrap.cc:492] T 7e6c4f30be624769961c9ec17c5d5dcb P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:04.155611 1166 ts_tablet_manager.cc:1403] T 7e6c4f30be624769961c9ec17c5d5dcb P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.094s user 0.002s sys 0.000s
I20251029 02:52:04.156322 1166 raft_consensus.cc:359] T 7e6c4f30be624769961c9ec17c5d5dcb P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:04.156461 1166 raft_consensus.cc:385] T 7e6c4f30be624769961c9ec17c5d5dcb P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:04.156504 1166 raft_consensus.cc:740] T 7e6c4f30be624769961c9ec17c5d5dcb P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:04.156674 1166 consensus_queue.cc:260] T 7e6c4f30be624769961c9ec17c5d5dcb P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:04.158456 1166 ts_tablet_manager.cc:1434] T 7e6c4f30be624769961c9ec17c5d5dcb P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251029 02:52:04.158733 1166 tablet_bootstrap.cc:492] T c9dff4dff82f49e7ad49caad5e404f09 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:04.160018 1166 tablet_bootstrap.cc:654] T c9dff4dff82f49e7ad49caad5e404f09 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:04.162010 521 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet 02e4588b03144ea2850e13a1f8f29f06 (table default.Table_0772f9db5d6142d988f6ffca2e96bdd2 [id=fd6d31d379394857a35422211af00642]). Expected version 3 got 0
I20251029 02:52:04.162621 519 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 67651306f3b44e7eb14dcee9a2007a81
I20251029 02:52:04.163440 749 tablet_service.cc:1552] Processing DeleteTablet for tablet 67651306f3b44e7eb14dcee9a2007a81 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:03 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:04.163650 1032 tablet_service.cc:1552] Processing DeleteTablet for tablet 67651306f3b44e7eb14dcee9a2007a81 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:03 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:04.164439 881 tablet_service.cc:1552] Processing DeleteTablet for tablet 67651306f3b44e7eb14dcee9a2007a81 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:03 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:04.167145 1176 tablet_replica.cc:333] T fd0f89180aba43eabb20f926abccbaa6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:04.167496 1176 raft_consensus.cc:2243] T fd0f89180aba43eabb20f926abccbaa6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:04.167963 1176 raft_consensus.cc:2272] T fd0f89180aba43eabb20f926abccbaa6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:04.168875 522 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 72e91c64a9b54cbda52ca9dfcee2ce22
I20251029 02:52:04.169332 522 catalog_manager.cc:6719] ReplaceTablet: tablet 72e91c64a9b54cbda52ca9dfcee2ce22 deleted and replaced by tablet 10e0ad0693a1432091e1a590fb76a720
I20251029 02:52:04.169482 523 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 7fb5b908db77453da5cfb32e2c955f41
I20251029 02:52:04.170199 517 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet caef1c93e33a4390bcada59a41f43465
I20251029 02:52:04.170670 517 catalog_manager.cc:6719] ReplaceTablet: tablet caef1c93e33a4390bcada59a41f43465 deleted and replaced by tablet 49492d91b1e94e9ca086533dbd0440f6
I20251029 02:52:04.170791 750 tablet_service.cc:1552] Processing DeleteTablet for tablet 7fb5b908db77453da5cfb32e2c955f41 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:03 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:04.171512 750 tablet_service.cc:1552] Processing DeleteTablet for tablet caef1c93e33a4390bcada59a41f43465 with delete_type TABLET_DATA_DELETED (replaced by tablet 49492d91b1e94e9ca086533dbd0440f6) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:04.172104 749 tablet_service.cc:1552] Processing DeleteTablet for tablet 72e91c64a9b54cbda52ca9dfcee2ce22 with delete_type TABLET_DATA_DELETED (replaced by tablet 10e0ad0693a1432091e1a590fb76a720) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:04.170230 1041 tablet_service.cc:1552] Processing DeleteTablet for tablet 7fb5b908db77453da5cfb32e2c955f41 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:03 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:04.173364 1176 ts_tablet_manager.cc:1916] T fd0f89180aba43eabb20f926abccbaa6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:04.174981 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 6ca9bd37de9e4be388501b15af4bda31 (table default.Table_3ca4a6512b174cb897b142dfe1259549 [id=a862192912a04942a8f682fc9825b014]) successfully deleted
I20251029 02:52:04.175042 1175 tablet_replica.cc:333] T f81aecb20d6e498bb9b498698886d65e P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:04.175212 1175 raft_consensus.cc:2243] T f81aecb20d6e498bb9b498698886d65e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:04.175359 1175 raft_consensus.cc:2272] T f81aecb20d6e498bb9b498698886d65e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:04.175683 881 tablet_service.cc:1552] Processing DeleteTablet for tablet 72e91c64a9b54cbda52ca9dfcee2ce22 with delete_type TABLET_DATA_DELETED (replaced by tablet 10e0ad0693a1432091e1a590fb76a720) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:04.175851 1175 ts_tablet_manager.cc:1916] T f81aecb20d6e498bb9b498698886d65e P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:04.175918 879 tablet_service.cc:1552] Processing DeleteTablet for tablet 7fb5b908db77453da5cfb32e2c955f41 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:03 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:04.176154 1172 tablet_bootstrap.cc:492] T 86fc22bb84794149b598b97df41eb881 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:04.176162 884 tablet_service.cc:1552] Processing DeleteTablet for tablet caef1c93e33a4390bcada59a41f43465 with delete_type TABLET_DATA_DELETED (replaced by tablet 49492d91b1e94e9ca086533dbd0440f6) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:04.176275 1172 ts_tablet_manager.cc:1403] T 86fc22bb84794149b598b97df41eb881 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.046s user 0.000s sys 0.003s
I20251029 02:52:04.176716 892 tablet_service.cc:1505] Processing CreateTablet for tablet e4cad991d48341cbafb5f1d9b50f4e5b (DEFAULT_TABLE table=default.Table_aa561368dec842f2a73780eef725fd8d [id=5b2b639e8f2d421c862f2d650f5e3dbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:04.177103 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4cad991d48341cbafb5f1d9b50f4e5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:04.177096 1172 raft_consensus.cc:359] T 86fc22bb84794149b598b97df41eb881 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:04.177286 1172 raft_consensus.cc:385] T 86fc22bb84794149b598b97df41eb881 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:04.177345 1172 raft_consensus.cc:740] T 86fc22bb84794149b598b97df41eb881 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:04.184289 890 tablet_service.cc:1552] Processing DeleteTablet for tablet 1f5b40c0403d4a49a4bfde222d8c947a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:53 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:04.139156 1174 pending_rounds.cc:70] T 392a36a6b59a42f78e3ac7c8fb63214b P c756528642a543eeacd93cef3e335a7d: Trying to abort 1 pending ops.
I20251029 02:52:04.184989 1174 pending_rounds.cc:77] T 392a36a6b59a42f78e3ac7c8fb63214b P c756528642a543eeacd93cef3e335a7d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215949102453579776 op_type: NO_OP noop_request { }
I20251029 02:52:04.185166 1174 raft_consensus.cc:2889] T 392a36a6b59a42f78e3ac7c8fb63214b P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 02:52:04.185328 1174 raft_consensus.cc:2272] T 392a36a6b59a42f78e3ac7c8fb63214b P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:04.171535 1039 tablet_service.cc:1552] Processing DeleteTablet for tablet 72e91c64a9b54cbda52ca9dfcee2ce22 with delete_type TABLET_DATA_DELETED (replaced by tablet 10e0ad0693a1432091e1a590fb76a720) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:04.171834 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet caef1c93e33a4390bcada59a41f43465 with delete_type TABLET_DATA_DELETED (replaced by tablet 49492d91b1e94e9ca086533dbd0440f6) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:04.171936 1032 tablet_service.cc:1505] Processing CreateTablet for tablet e4cad991d48341cbafb5f1d9b50f4e5b (DEFAULT_TABLE table=default.Table_aa561368dec842f2a73780eef725fd8d [id=5b2b639e8f2d421c862f2d650f5e3dbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:04.186267 1041 tablet_service.cc:1552] Processing DeleteTablet for tablet 7e6c4f30be624769961c9ec17c5d5dcb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:55 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:04.186419 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4cad991d48341cbafb5f1d9b50f4e5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:04.187768 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_592b1c4cbb204f75838bdecb7f308d2b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:04.188383 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_592b1c4cbb204f75838bdecb7f308d2b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:04.193795 1172 consensus_queue.cc:260] T 86fc22bb84794149b598b97df41eb881 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:04.200512 749 tablet_service.cc:1505] Processing CreateTablet for tablet e4cad991d48341cbafb5f1d9b50f4e5b (DEFAULT_TABLE table=default.Table_aa561368dec842f2a73780eef725fd8d [id=5b2b639e8f2d421c862f2d650f5e3dbb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:04.201109 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4cad991d48341cbafb5f1d9b50f4e5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:04.201668 895 tablet_service.cc:1505] Processing CreateTablet for tablet 7766725381b04b12bef4ee4c7d70e91f (DEFAULT_TABLE table=default.Table_e24d2641727d40a79dd72860f4d9d272 [id=268a24fb2d344546a139deacdd1c08c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:04.202080 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7766725381b04b12bef4ee4c7d70e91f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:04.202915 518 master_service.cc:954] ReplaceTablet: received request to replace tablet 78662eeae70b4148a4ad0d47be5c566d from {username='slave'} at 127.0.0.1:35314
I20251029 02:52:04.204197 890 tablet_service.cc:1505] Processing CreateTablet for tablet f66a7af499b446c7a33965c9e37ce460 (DEFAULT_TABLE table=default.Table_18581b0529cb451d9bda8bf2c84a8144 [id=3310b73691284c798da8eb1fd623fa7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:04.204609 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f66a7af499b446c7a33965c9e37ce460. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:04.206633 750 tablet_service.cc:1505] Processing CreateTablet for tablet f66a7af499b446c7a33965c9e37ce460 (DEFAULT_TABLE table=default.Table_18581b0529cb451d9bda8bf2c84a8144 [id=3310b73691284c798da8eb1fd623fa7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:04.206972 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f66a7af499b446c7a33965c9e37ce460. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:04.207355 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_654ba01aef8c4aacb0f0aae654401c0a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:04.207873 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_654ba01aef8c4aacb0f0aae654401c0a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:04.210764 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_61d957e80df24f53a6dcac6f74783f9f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:04.211259 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_61d957e80df24f53a6dcac6f74783f9f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:04.213289 522 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_220c535de1944595a5c5c101efaed2f3" } new_table_name: "default.Table_e99e1cb4ad41429aa0160f749671630a" modify_external_catalogs: true
I20251029 02:52:04.214524 523 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_5b80ec00d8f74e97a7a8fa038920b55b" } new_table_name: "default.Table_97b75f1637e14d6a84e80e933e6aa059" modify_external_catalogs: true
I20251029 02:52:04.219471 1172 ts_tablet_manager.cc:1434] T 86fc22bb84794149b598b97df41eb881 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.043s user 0.000s sys 0.001s
I20251029 02:52:04.219879 1032 tablet_service.cc:1505] Processing CreateTablet for tablet f66a7af499b446c7a33965c9e37ce460 (DEFAULT_TABLE table=default.Table_18581b0529cb451d9bda8bf2c84a8144 [id=3310b73691284c798da8eb1fd623fa7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:04.227257 1172 tablet_bootstrap.cc:492] T c9bd3f2deb5d4af09b7918269a98509f P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:04.228199 519 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_8b39f039e15540df86d5d1dd12a645a7" } new_table_name: "default.Table_34ad95c52c7645a8a6c2db64e9dff633" modify_external_catalogs: true
I20251029 02:52:04.229676 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_905a7a1ab3ef4da9838ac7883ffa4cb1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:04.230095 1682 raft_consensus.cc:493] T 7309d3190cfa415281c5253a634fa7f3 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251029 02:52:04.230187 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_905a7a1ab3ef4da9838ac7883ffa4cb1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:04.230245 1682 raft_consensus.cc:515] T 7309d3190cfa415281c5253a634fa7f3 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:04.230818 1682 leader_election.cc:290] T 7309d3190cfa415281c5253a634fa7f3 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:04.232093 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7309d3190cfa415281c5253a634fa7f3" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:04.232270 1060 raft_consensus.cc:2468] T 7309d3190cfa415281c5253a634fa7f3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:04.232491 1172 tablet_bootstrap.cc:654] T c9bd3f2deb5d4af09b7918269a98509f P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
W20251029 02:52:04.233119 849 consensus_peers.cc:597] T fd0f89180aba43eabb20f926abccbaa6 P c756528642a543eeacd93cef3e335a7d -> Peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Couldn't send request to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:04.233359 849 leader_election.cc:304] T 7309d3190cfa415281c5253a634fa7f3 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:04.233412 749 tablet_service.cc:1552] Processing DeleteTablet for tablet 86fc22bb84794149b598b97df41eb881 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:58 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:04.233742 1903 raft_consensus.cc:2804] T 7309d3190cfa415281c5253a634fa7f3 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:04.233804 1903 raft_consensus.cc:493] T 7309d3190cfa415281c5253a634fa7f3 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:04.233855 1903 raft_consensus.cc:3060] T 7309d3190cfa415281c5253a634fa7f3 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
W20251029 02:52:04.236682 520 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 643e20d9974447daa78885b3401f3b77 consensus state has no leader
I20251029 02:52:04.236619 1903 raft_consensus.cc:515] T 7309d3190cfa415281c5253a634fa7f3 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:04.237152 1903 leader_election.cc:290] T 7309d3190cfa415281c5253a634fa7f3 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:04.239694 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7309d3190cfa415281c5253a634fa7f3" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:52:04.239866 1060 raft_consensus.cc:3060] T 7309d3190cfa415281c5253a634fa7f3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:04.239854 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_4fbd78edfb5a451f91ddcdf41de962a3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:04.240422 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4fbd78edfb5a451f91ddcdf41de962a3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:04.240556 1176 ts_tablet_manager.cc:1929] T fd0f89180aba43eabb20f926abccbaa6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:04.240694 1176 log.cc:1199] T fd0f89180aba43eabb20f926abccbaa6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/fd0f89180aba43eabb20f926abccbaa6
I20251029 02:52:04.241082 1176 ts_tablet_manager.cc:1950] T fd0f89180aba43eabb20f926abccbaa6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:04.241988 1175 ts_tablet_manager.cc:1929] T f81aecb20d6e498bb9b498698886d65e P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:04.242092 1175 log.cc:1199] T f81aecb20d6e498bb9b498698886d65e P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/f81aecb20d6e498bb9b498698886d65e
I20251029 02:52:04.242226 1176 tablet_replica.cc:333] T 7309d3190cfa415281c5253a634fa7f3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:04.242411 1175 ts_tablet_manager.cc:1950] T f81aecb20d6e498bb9b498698886d65e P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:04.243295 1175 tablet_replica.cc:333] T c764062d3ba04d4ba8b2e367ad8ed433 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:04.243469 1175 raft_consensus.cc:2243] T c764062d3ba04d4ba8b2e367ad8ed433 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:04.243479 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet f81aecb20d6e498bb9b498698886d65e (table default.Table_98034c2933664cefab94c26cb8349dc4 [id=efb8f381fea74cf48842af6b1a4d5a81]) successfully deleted
I20251029 02:52:04.243615 1175 raft_consensus.cc:2272] T c764062d3ba04d4ba8b2e367ad8ed433 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:04.244068 1175 ts_tablet_manager.cc:1916] T c764062d3ba04d4ba8b2e367ad8ed433 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:04.246579 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet fd0f89180aba43eabb20f926abccbaa6 (table default.Table_50252b3b3cc4403da5e259f9f91be915 [id=ff64a323de354792997beb99bae4a52c]) successfully deleted
W20251029 02:52:04.246908 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 70b008011d5547bbaf3f45685394139f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 70b008011d5547bbaf3f45685394139f already in progress: creating tablet
W20251029 02:52:04.247128 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet ce98420e61904af78d8fa03724c74489 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ce98420e61904af78d8fa03724c74489 already in progress: creating tablet
W20251029 02:52:04.247357 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet ae5a6caeb17f4851b27a3c6a3c0db83c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ae5a6caeb17f4851b27a3c6a3c0db83c already in progress: creating tablet
I20251029 02:52:04.250236 1060 raft_consensus.cc:2468] T 7309d3190cfa415281c5253a634fa7f3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:04.250774 849 leader_election.cc:304] T 7309d3190cfa415281c5253a634fa7f3 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:04.251116 1774 raft_consensus.cc:2804] T 7309d3190cfa415281c5253a634fa7f3 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:04.251230 1774 raft_consensus.cc:697] T 7309d3190cfa415281c5253a634fa7f3 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:52:04.251502 1774 consensus_queue.cc:237] T 7309d3190cfa415281c5253a634fa7f3 P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:04.251690 521 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_5263ec7f27d74f73892b0ca252e9fa83" } new_table_name: "default.Table_b594044b034c485b886a94f8699bb935" modify_external_catalogs: true
W20251029 02:52:04.259220 847 consensus_peers.cc:597] T fd0f89180aba43eabb20f926abccbaa6 P c756528642a543eeacd93cef3e335a7d -> Peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Couldn't send request to peer b495a49349324f7d81908f2da700f0d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fd0f89180aba43eabb20f926abccbaa6. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:04.259754 518 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 78662eeae70b4148a4ad0d47be5c566d
W20251029 02:52:04.260190 847 leader_election.cc:343] T 7309d3190cfa415281c5253a634fa7f3 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 7309d3190cfa415281c5253a634fa7f3
I20251029 02:52:04.260265 518 catalog_manager.cc:6719] ReplaceTablet: tablet 78662eeae70b4148a4ad0d47be5c566d deleted and replaced by tablet add6e431cdfe433b9fa9e0c377e3f8e8
W20251029 02:52:04.260527 847 leader_election.cc:343] T 7309d3190cfa415281c5253a634fa7f3 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 7309d3190cfa415281c5253a634fa7f3
I20251029 02:52:04.260855 749 tablet_service.cc:1552] Processing DeleteTablet for tablet 78662eeae70b4148a4ad0d47be5c566d with delete_type TABLET_DATA_DELETED (replaced by tablet add6e431cdfe433b9fa9e0c377e3f8e8) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:04.261221 1041 tablet_service.cc:1552] Processing DeleteTablet for tablet 78662eeae70b4148a4ad0d47be5c566d with delete_type TABLET_DATA_DELETED (replaced by tablet add6e431cdfe433b9fa9e0c377e3f8e8) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:04.267153 1174 ts_tablet_manager.cc:1916] T 392a36a6b59a42f78e3ac7c8fb63214b P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:04.271420 892 tablet_service.cc:1552] Processing DeleteTablet for tablet 78662eeae70b4148a4ad0d47be5c566d with delete_type TABLET_DATA_DELETED (replaced by tablet add6e431cdfe433b9fa9e0c377e3f8e8) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:04.272714 1906 raft_consensus.cc:493] T 51effc9324034ae79e2b25cb57a996c1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:04.272861 1906 raft_consensus.cc:515] T 51effc9324034ae79e2b25cb57a996c1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:04.273494 1906 leader_election.cc:290] T 51effc9324034ae79e2b25cb57a996c1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:52:04.274127 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51effc9324034ae79e2b25cb57a996c1" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:04.274246 989 leader_election.cc:343] T 51effc9324034ae79e2b25cb57a996c1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 51effc9324034ae79e2b25cb57a996c1
I20251029 02:52:04.274302 917 raft_consensus.cc:2468] T 51effc9324034ae79e2b25cb57a996c1 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 0.
I20251029 02:52:04.274662 990 leader_election.cc:304] T 51effc9324034ae79e2b25cb57a996c1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters: b495a49349324f7d81908f2da700f0d9
I20251029 02:52:04.275094 1906 raft_consensus.cc:2804] T 51effc9324034ae79e2b25cb57a996c1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:04.275166 1906 raft_consensus.cc:493] T 51effc9324034ae79e2b25cb57a996c1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:04.275236 1906 raft_consensus.cc:3060] T 51effc9324034ae79e2b25cb57a996c1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:04.276430 526 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_ace94370997c4944968f3e7633615ae2" } modify_external_catalogs: true
I20251029 02:52:04.276634 526 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_ace94370997c4944968f3e7633615ae2" } modify_external_catalogs: true
I20251029 02:52:04.278337 1906 raft_consensus.cc:515] T 51effc9324034ae79e2b25cb57a996c1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:04.279798 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51effc9324034ae79e2b25cb57a996c1" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
I20251029 02:52:04.279959 917 raft_consensus.cc:3060] T 51effc9324034ae79e2b25cb57a996c1 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
W20251029 02:52:04.281950 522 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7ff2399833e546448b85b9caeb5200fb consensus state has no leader
I20251029 02:52:04.282644 1906 leader_election.cc:290] T 51effc9324034ae79e2b25cb57a996c1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Requested vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:52:04.283416 917 raft_consensus.cc:2468] T 51effc9324034ae79e2b25cb57a996c1 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 1.
I20251029 02:52:04.283927 990 leader_election.cc:304] T 51effc9324034ae79e2b25cb57a996c1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:04.284629 1906 raft_consensus.cc:2804] T 51effc9324034ae79e2b25cb57a996c1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:04.284940 1906 raft_consensus.cc:697] T 51effc9324034ae79e2b25cb57a996c1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Becoming Leader. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Running, Role: LEADER
I20251029 02:52:04.285547 1906 consensus_queue.cc:237] T 51effc9324034ae79e2b25cb57a996c1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
W20251029 02:52:04.286705 989 leader_election.cc:343] T 51effc9324034ae79e2b25cb57a996c1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 51effc9324034ae79e2b25cb57a996c1
W20251029 02:52:04.289265 523 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 74eafdebb4eb4ba7a0c05d99e2e93fcb has no consensus state
I20251029 02:52:04.292249 1176 raft_consensus.cc:2243] T 7309d3190cfa415281c5253a634fa7f3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:04.292991 1176 raft_consensus.cc:2272] T 7309d3190cfa415281c5253a634fa7f3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:04.294059 1176 ts_tablet_manager.cc:1916] T 7309d3190cfa415281c5253a634fa7f3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:04.296531 1167 ts_tablet_manager.cc:1434] T 51effc9324034ae79e2b25cb57a996c1 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.144s user 0.001s sys 0.001s
I20251029 02:52:04.297194 1167 tablet_bootstrap.cc:492] T 24cb2a71ba3841d4a5e3e5c6fe99ae2f P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:04.297132 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_b8fe14e0ff054e67b0594e8b97483c71"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:04.297338 519 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 050186302c5d4c7da393f28f4b9e4262 has no consensus state
W20251029 02:52:04.297914 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b8fe14e0ff054e67b0594e8b97483c71 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:04.298430 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_b3877f1183ba42e9b368d74299f13f55"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:04.298923 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b3877f1183ba42e9b368d74299f13f55 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:04.306865 524 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_8015f6519aff4d269aa0f16e8cdd96dd" } modify_external_catalogs: true
I20251029 02:52:04.307060 524 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_8015f6519aff4d269aa0f16e8cdd96dd" } modify_external_catalogs: true
I20251029 02:52:04.308388 1175 ts_tablet_manager.cc:1929] T c764062d3ba04d4ba8b2e367ad8ed433 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:04.308516 1175 log.cc:1199] T c764062d3ba04d4ba8b2e367ad8ed433 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/c764062d3ba04d4ba8b2e367ad8ed433
I20251029 02:52:04.308897 1175 ts_tablet_manager.cc:1950] T c764062d3ba04d4ba8b2e367ad8ed433 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:04.309883 1175 tablet_replica.cc:333] T daa05a34c0684c07a35966290fd7d44c P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:04.309944 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet c764062d3ba04d4ba8b2e367ad8ed433 (table default.Table_bce83bae13a94cd5bf6733b53e6a1e9e [id=761ebdd0095c414488aa221cbc99d3bc]) successfully deleted
I20251029 02:52:04.310070 1175 raft_consensus.cc:2243] T daa05a34c0684c07a35966290fd7d44c P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:04.310209 1175 raft_consensus.cc:2272] T daa05a34c0684c07a35966290fd7d44c P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:04.311533 1041 tablet_service.cc:1505] Processing CreateTablet for tablet 7766725381b04b12bef4ee4c7d70e91f (DEFAULT_TABLE table=default.Table_e24d2641727d40a79dd72860f4d9d272 [id=268a24fb2d344546a139deacdd1c08c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:04.313596 749 tablet_service.cc:1505] Processing CreateTablet for tablet 7766725381b04b12bef4ee4c7d70e91f (DEFAULT_TABLE table=default.Table_e24d2641727d40a79dd72860f4d9d272 [id=268a24fb2d344546a139deacdd1c08c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:04.313980 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7766725381b04b12bef4ee4c7d70e91f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:04.315951 1166 tablet_bootstrap.cc:492] T c9dff4dff82f49e7ad49caad5e404f09 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:04.316232 1166 ts_tablet_manager.cc:1403] T c9dff4dff82f49e7ad49caad5e404f09 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.158s user 0.004s sys 0.000s
I20251029 02:52:04.317518 1166 raft_consensus.cc:359] T c9dff4dff82f49e7ad49caad5e404f09 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
W20251029 02:52:04.317822 521 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fd1bf6b0f45349d0981f9ddd89269489 consensus state has no leader
I20251029 02:52:04.317965 1166 raft_consensus.cc:385] T c9dff4dff82f49e7ad49caad5e404f09 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:04.318130 1166 raft_consensus.cc:740] T c9dff4dff82f49e7ad49caad5e404f09 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:04.324179 526 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet de7fac8751a04c19b68a52db7f167214
I20251029 02:52:04.325790 749 tablet_service.cc:1552] Processing DeleteTablet for tablet de7fac8751a04c19b68a52db7f167214 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:04 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:04.328225 526 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_dc5f0ee620604d15a617a3c7ed345c37" } modify_external_catalogs: true
I20251029 02:52:04.328424 526 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_dc5f0ee620604d15a617a3c7ed345c37" } modify_external_catalogs: true
I20251029 02:52:04.330222 749 tablet_service.cc:1505] Processing CreateTablet for tablet 0b71e81e74804dd0940e8c51af4bf0da (DEFAULT_TABLE table=default.Table_51d1934e7b1e438ebea90a3ff5da2ed5 [id=93663800bf204ac4a0a2ddd9ae9a9fdf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:04.330693 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b71e81e74804dd0940e8c51af4bf0da. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:04.332387 1175 ts_tablet_manager.cc:1916] T daa05a34c0684c07a35966290fd7d44c P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:04.318557 1166 consensus_queue.cc:260] T c9dff4dff82f49e7ad49caad5e404f09 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:04.325270 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet de7fac8751a04c19b68a52db7f167214 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:04 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:04.326409 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f66a7af499b446c7a33965c9e37ce460. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:04.336200 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7766725381b04b12bef4ee4c7d70e91f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:04.328748 1039 tablet_service.cc:1505] Processing CreateTablet for tablet 0b71e81e74804dd0940e8c51af4bf0da (DEFAULT_TABLE table=default.Table_51d1934e7b1e438ebea90a3ff5da2ed5 [id=93663800bf204ac4a0a2ddd9ae9a9fdf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:04.331532 1037 tablet_service.cc:1505] Processing CreateTablet for tablet 8a463b3889de47df8e93841cca4186bb (DEFAULT_TABLE table=default.Table_021cde575a7a44f8a24392dcf558996a [id=5aa8ffcf27d943ddbe5a113d2a2c3a29]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:04.336582 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b71e81e74804dd0940e8c51af4bf0da. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:04.337222 890 tablet_service.cc:1505] Processing CreateTablet for tablet 8a463b3889de47df8e93841cca4186bb (DEFAULT_TABLE table=default.Table_021cde575a7a44f8a24392dcf558996a [id=5aa8ffcf27d943ddbe5a113d2a2c3a29]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:04.337548 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a463b3889de47df8e93841cca4186bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:04.342643 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a463b3889de47df8e93841cca4186bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:04.344287 524 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 1e17142dc6634db0a5288add9c199ffc
I20251029 02:52:04.345455 751 tablet_service.cc:1552] Processing DeleteTablet for tablet 1e17142dc6634db0a5288add9c199ffc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:04 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:04.346284 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet 1e17142dc6634db0a5288add9c199ffc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:04 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:04.350860 892 tablet_service.cc:1505] Processing CreateTablet for tablet 0b71e81e74804dd0940e8c51af4bf0da (DEFAULT_TABLE table=default.Table_51d1934e7b1e438ebea90a3ff5da2ed5 [id=93663800bf204ac4a0a2ddd9ae9a9fdf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:04.351322 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0b71e81e74804dd0940e8c51af4bf0da. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:04.352236 1172 tablet_bootstrap.cc:492] T c9bd3f2deb5d4af09b7918269a98509f P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:04.352385 1172 ts_tablet_manager.cc:1403] T c9bd3f2deb5d4af09b7918269a98509f P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.125s user 0.005s sys 0.016s
I20251029 02:52:04.352890 526 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet c4b1ef53dbaa47c39361cc4ef3be71d8
I20251029 02:52:04.353159 1172 raft_consensus.cc:359] T c9bd3f2deb5d4af09b7918269a98509f P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:04.353324 1172 raft_consensus.cc:385] T c9bd3f2deb5d4af09b7918269a98509f P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:04.353396 1172 raft_consensus.cc:740] T c9bd3f2deb5d4af09b7918269a98509f P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:04.353834 1172 consensus_queue.cc:260] T c9bd3f2deb5d4af09b7918269a98509f P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:04.356985 1172 ts_tablet_manager.cc:1434] T c9bd3f2deb5d4af09b7918269a98509f P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251029 02:52:04.358059 1172 tablet_bootstrap.cc:492] T 4add552cf4d940d2a0a36d6b58379a34 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:04.358642 749 tablet_service.cc:1552] Processing DeleteTablet for tablet c4b1ef53dbaa47c39361cc4ef3be71d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:04 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:04.358675 1037 tablet_service.cc:1552] Processing DeleteTablet for tablet c4b1ef53dbaa47c39361cc4ef3be71d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:04 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:04.361585 1175 ts_tablet_manager.cc:1929] T daa05a34c0684c07a35966290fd7d44c P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:04.361699 1175 log.cc:1199] T daa05a34c0684c07a35966290fd7d44c P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/daa05a34c0684c07a35966290fd7d44c
I20251029 02:52:04.363827 1172 tablet_bootstrap.cc:654] T 4add552cf4d940d2a0a36d6b58379a34 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:04.365005 524 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_61d957e80df24f53a6dcac6f74783f9f" } modify_external_catalogs: true
I20251029 02:52:04.365294 524 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_61d957e80df24f53a6dcac6f74783f9f" } modify_external_catalogs: true
I20251029 02:52:04.367619 749 tablet_service.cc:1552] Processing DeleteTablet for tablet c9bd3f2deb5d4af09b7918269a98509f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:58 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:04.374356 1175 ts_tablet_manager.cc:1950] T daa05a34c0684c07a35966290fd7d44c P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:04.375298 1175 tablet_replica.cc:333] T f246f6c202cf44d48c9f5e4f335202c5 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:04.375471 1175 raft_consensus.cc:2243] T f246f6c202cf44d48c9f5e4f335202c5 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:04.375591 1175 raft_consensus.cc:2272] T f246f6c202cf44d48c9f5e4f335202c5 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:04.376041 1175 ts_tablet_manager.cc:1916] T f246f6c202cf44d48c9f5e4f335202c5 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:04.377719 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_e3cc9c53ee9e4fafb7f9684f6c518714"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:04.378322 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e3cc9c53ee9e4fafb7f9684f6c518714 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:04.381250 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet daa05a34c0684c07a35966290fd7d44c (table default.Table_33eacbf00740465185d70b46a3782893 [id=22e43c4483ae488b81fb94a7d221bdce]) successfully deleted
I20251029 02:52:04.386624 879 tablet_service.cc:1552] Processing DeleteTablet for tablet c4b1ef53dbaa47c39361cc4ef3be71d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:04 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:04.386847 884 tablet_service.cc:1552] Processing DeleteTablet for tablet 1e17142dc6634db0a5288add9c199ffc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:04 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:04.389112 524 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet a748355e52fc411abca51e542bbf2cc3
I20251029 02:52:04.391224 895 tablet_service.cc:1552] Processing DeleteTablet for tablet de7fac8751a04c19b68a52db7f167214 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:04 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:04.394517 526 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_592b1c4cbb204f75838bdecb7f308d2b" } modify_external_catalogs: true
I20251029 02:52:04.394706 526 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_592b1c4cbb204f75838bdecb7f308d2b" } modify_external_catalogs: true
I20251029 02:52:04.395854 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_dc31c9a0d89947c1aeeedd96b7678134"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:04.396409 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dc31c9a0d89947c1aeeedd96b7678134 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:04.404112 1906 raft_consensus.cc:493] T 1f5b40c0403d4a49a4bfde222d8c947a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:04.404264 1906 raft_consensus.cc:515] T 1f5b40c0403d4a49a4bfde222d8c947a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:04.404933 1906 leader_election.cc:290] T 1f5b40c0403d4a49a4bfde222d8c947a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
W20251029 02:52:04.406304 989 leader_election.cc:343] T 1f5b40c0403d4a49a4bfde222d8c947a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 1f5b40c0403d4a49a4bfde222d8c947a
I20251029 02:52:04.406668 1166 ts_tablet_manager.cc:1434] T c9dff4dff82f49e7ad49caad5e404f09 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.090s user 0.002s sys 0.000s
I20251029 02:52:04.407181 1166 tablet_bootstrap.cc:492] T 576c4e6e70e24b0399ac9597a30f0569 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:04.409224 1166 tablet_bootstrap.cc:654] T 576c4e6e70e24b0399ac9597a30f0569 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:04.411989 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1f5b40c0403d4a49a4bfde222d8c947a" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:04.412195 917 raft_consensus.cc:2468] T 1f5b40c0403d4a49a4bfde222d8c947a P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 0.
I20251029 02:52:04.412652 990 leader_election.cc:304] T 1f5b40c0403d4a49a4bfde222d8c947a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters: b495a49349324f7d81908f2da700f0d9
I20251029 02:52:04.413025 1906 raft_consensus.cc:2804] T 1f5b40c0403d4a49a4bfde222d8c947a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:04.413101 1906 raft_consensus.cc:493] T 1f5b40c0403d4a49a4bfde222d8c947a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:04.413127 1167 tablet_bootstrap.cc:654] T 24cb2a71ba3841d4a5e3e5c6fe99ae2f P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:04.413164 1906 raft_consensus.cc:3060] T 1f5b40c0403d4a49a4bfde222d8c947a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:04.416014 1906 raft_consensus.cc:515] T 1f5b40c0403d4a49a4bfde222d8c947a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:04.417243 1906 leader_election.cc:290] T 1f5b40c0403d4a49a4bfde222d8c947a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Requested vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:04.417258 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1f5b40c0403d4a49a4bfde222d8c947a" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
I20251029 02:52:04.417496 917 raft_consensus.cc:3060] T 1f5b40c0403d4a49a4bfde222d8c947a P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
W20251029 02:52:04.418208 989 leader_election.cc:343] T 1f5b40c0403d4a49a4bfde222d8c947a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 1f5b40c0403d4a49a4bfde222d8c947a
I20251029 02:52:04.422302 1175 ts_tablet_manager.cc:1929] T f246f6c202cf44d48c9f5e4f335202c5 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:04.422412 1175 log.cc:1199] T f246f6c202cf44d48c9f5e4f335202c5 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/f246f6c202cf44d48c9f5e4f335202c5
I20251029 02:52:04.429894 1175 ts_tablet_manager.cc:1950] T f246f6c202cf44d48c9f5e4f335202c5 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:04.432480 526 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 44fee8f0a3564eee8f292b1a309cdbba
I20251029 02:52:04.432803 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet f246f6c202cf44d48c9f5e4f335202c5 (table default.Table_e4de02c7c1f04704bc39e1f0efd2c9d9 [id=40738627473f4a55adc5ed250af29d6c]) successfully deleted
I20251029 02:52:04.434203 1175 tablet_replica.cc:333] T 6bd3fac2502e479aa9f4ceaabbb39504 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:04.434397 1175 raft_consensus.cc:2243] T 6bd3fac2502e479aa9f4ceaabbb39504 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:04.435561 1175 raft_consensus.cc:2272] T 6bd3fac2502e479aa9f4ceaabbb39504 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:04.437044 1175 ts_tablet_manager.cc:1916] T 6bd3fac2502e479aa9f4ceaabbb39504 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:04.442911 917 raft_consensus.cc:2468] T 1f5b40c0403d4a49a4bfde222d8c947a P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 1.
I20251029 02:52:04.443581 990 leader_election.cc:304] T 1f5b40c0403d4a49a4bfde222d8c947a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters: b495a49349324f7d81908f2da700f0d9
I20251029 02:52:04.444110 1906 raft_consensus.cc:2804] T 1f5b40c0403d4a49a4bfde222d8c947a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:04.444221 1906 raft_consensus.cc:697] T 1f5b40c0403d4a49a4bfde222d8c947a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Becoming Leader. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Running, Role: LEADER
I20251029 02:52:04.444447 1906 consensus_queue.cc:237] T 1f5b40c0403d4a49a4bfde222d8c947a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:04.468925 749 tablet_service.cc:1505] Processing CreateTablet for tablet 8a463b3889de47df8e93841cca4186bb (DEFAULT_TABLE table=default.Table_021cde575a7a44f8a24392dcf558996a [id=5aa8ffcf27d943ddbe5a113d2a2c3a29]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:04.469403 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a463b3889de47df8e93841cca4186bb. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:04.472060 849 consensus_peers.cc:597] T 69597d48631d43b6a5b92abbf60d5ed3 P c756528642a543eeacd93cef3e335a7d -> Peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Couldn't send request to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 69597d48631d43b6a5b92abbf60d5ed3. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:04.472846 1175 ts_tablet_manager.cc:1929] T 6bd3fac2502e479aa9f4ceaabbb39504 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:04.472944 1175 log.cc:1199] T 6bd3fac2502e479aa9f4ceaabbb39504 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/6bd3fac2502e479aa9f4ceaabbb39504
I20251029 02:52:04.473210 1175 ts_tablet_manager.cc:1950] T 6bd3fac2502e479aa9f4ceaabbb39504 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:04.475261 751 tablet_service.cc:1505] Processing CreateTablet for tablet 0d2077a74c5342eb813accde2d667017 (DEFAULT_TABLE table=default.Table_90cf8deabd5c47c1b162ce513d4c1fe4 [id=7be7a8527c3d4ec9985e8941894f305b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:04.476061 1167 tablet_bootstrap.cc:492] T 24cb2a71ba3841d4a5e3e5c6fe99ae2f P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:04.478266 1167 ts_tablet_manager.cc:1403] T 24cb2a71ba3841d4a5e3e5c6fe99ae2f P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.181s user 0.003s sys 0.003s
I20251029 02:52:04.479204 1167 raft_consensus.cc:359] T 24cb2a71ba3841d4a5e3e5c6fe99ae2f P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:04.481549 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d2077a74c5342eb813accde2d667017. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:04.484766 1041 tablet_service.cc:1505] Processing CreateTablet for tablet 0d2077a74c5342eb813accde2d667017 (DEFAULT_TABLE table=default.Table_90cf8deabd5c47c1b162ce513d4c1fe4 [id=7be7a8527c3d4ec9985e8941894f305b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:04.489241 1823 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "e9261eb85ea443d5a3beb36bd0c549f0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9672114a6a742e2864c16ff60ba246e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42343 } } peers { permanent_uuid: "e9261eb85ea443d5a3beb36bd0c549f0" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 34239 } } peers { permanent_uuid: "e5e5222b26fe4145948620e51edf53fb" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 41233 } } }
I20251029 02:52:04.489512 1823 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0 [sys.catalog]: This master's current role is: LEADER
I20251029 02:52:04.489750 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 6bd3fac2502e479aa9f4ceaabbb39504 (table default.Table_fa37ddda630a405d85ae341c82b4f204 [id=3513c69f6b3941849ac5b8ae0a5846d3]) successfully deleted
I20251029 02:52:04.489826 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_2156469f772942ffa8a3bdfc9614409a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:04.490345 519 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_fa37ddda630a405d85ae341c82b4f204" } new_table_name: "default.Table_110e1b64737c4040ada22360d5a373cb" modify_external_catalogs: true
W20251029 02:52:04.490567 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2156469f772942ffa8a3bdfc9614409a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:04.481324 890 tablet_service.cc:1505] Processing CreateTablet for tablet 4929513bed104050bd6065f025303e1a (DEFAULT_TABLE table=default.Table_de3500a0d5f64dd788359ae260ffd539 [id=7f60d0330c9745309bb6efc923f71c73]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:04.491645 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4929513bed104050bd6065f025303e1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:04.491715 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_71446f2a5631421ab14116c669e1d5ba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:04.483765 895 tablet_service.cc:1505] Processing CreateTablet for tablet 0d2077a74c5342eb813accde2d667017 (DEFAULT_TABLE table=default.Table_90cf8deabd5c47c1b162ce513d4c1fe4 [id=7be7a8527c3d4ec9985e8941894f305b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:04.492117 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d2077a74c5342eb813accde2d667017. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:04.492226 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_71446f2a5631421ab14116c669e1d5ba in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:04.493809 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_3ccbb335620141bca4af176e8a8b8c8f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:04.494408 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3ccbb335620141bca4af176e8a8b8c8f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:04.495019 517 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_455fef4acf064ba799a55df3d87a94ad" } new_table_name: "default.Table_9dce7f70d51d4a3c8d7f98a2435cbce3" modify_external_catalogs: true
I20251029 02:52:04.497568 750 tablet_service.cc:1505] Processing CreateTablet for tablet a2d08c252a934cf9a45d2f60e62f76aa (DEFAULT_TABLE table=default.Table_9bf4e25d441f4d4dbd6651386eec512e [id=82df85af001340efb570040e047fa136]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:04.497944 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2d08c252a934cf9a45d2f60e62f76aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:04.498855 1166 tablet_bootstrap.cc:492] T 576c4e6e70e24b0399ac9597a30f0569 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:04.499074 1166 ts_tablet_manager.cc:1403] T 576c4e6e70e24b0399ac9597a30f0569 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.092s user 0.004s sys 0.000s
I20251029 02:52:04.499568 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_b99166d4afe04f44a80ecd035fa90d72"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:04.500092 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b99166d4afe04f44a80ecd035fa90d72 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:04.500061 1166 raft_consensus.cc:359] T 576c4e6e70e24b0399ac9597a30f0569 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:04.500263 1166 raft_consensus.cc:385] T 576c4e6e70e24b0399ac9597a30f0569 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:04.500322 1166 raft_consensus.cc:740] T 576c4e6e70e24b0399ac9597a30f0569 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:04.500738 1166 consensus_queue.cc:260] T 576c4e6e70e24b0399ac9597a30f0569 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:04.501608 1166 ts_tablet_manager.cc:1434] T 576c4e6e70e24b0399ac9597a30f0569 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20251029 02:52:04.501933 749 tablet_service.cc:1505] Processing CreateTablet for tablet c7a3e6d8e3fb45898d8a5e094dbf0fd7 (DEFAULT_TABLE table=default.Table_4d82d8b113274cf1841f2141772815e0 [id=78d1000dc2e344b1a1fa497b9abe3062]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:04.502157 1166 tablet_bootstrap.cc:492] T 42cfdf88472b44979d86836561005ed1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:04.502286 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7a3e6d8e3fb45898d8a5e094dbf0fd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:04.503414 742 tablet_service.cc:1505] Processing CreateTablet for tablet 932f69a7ac8c4a0eb2a436c6b3e6e83f (DEFAULT_TABLE table=default.Table_240f7d6a65f04099abf3e204b2f47fb6 [id=a0e3209e45574a8e8efb821e4a0f56dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:04.503777 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 932f69a7ac8c4a0eb2a436c6b3e6e83f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:04.511353 1039 tablet_service.cc:1505] Processing CreateTablet for tablet c7a3e6d8e3fb45898d8a5e094dbf0fd7 (DEFAULT_TABLE table=default.Table_4d82d8b113274cf1841f2141772815e0 [id=78d1000dc2e344b1a1fa497b9abe3062]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:04.511919 1032 tablet_service.cc:1505] Processing CreateTablet for tablet a2d08c252a934cf9a45d2f60e62f76aa (DEFAULT_TABLE table=default.Table_9bf4e25d441f4d4dbd6651386eec512e [id=82df85af001340efb570040e047fa136]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:04.513352 884 tablet_service.cc:1505] Processing CreateTablet for tablet a2d08c252a934cf9a45d2f60e62f76aa (DEFAULT_TABLE table=default.Table_9bf4e25d441f4d4dbd6651386eec512e [id=82df85af001340efb570040e047fa136]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:04.513736 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2d08c252a934cf9a45d2f60e62f76aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:04.517979 879 tablet_service.cc:1552] Processing DeleteTablet for tablet 51effc9324034ae79e2b25cb57a996c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:56 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:04.517998 751 tablet_service.cc:1505] Processing CreateTablet for tablet 4929513bed104050bd6065f025303e1a (DEFAULT_TABLE table=default.Table_de3500a0d5f64dd788359ae260ffd539 [id=7f60d0330c9745309bb6efc923f71c73]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:04.518359 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4929513bed104050bd6065f025303e1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:04.518362 881 tablet_service.cc:1505] Processing CreateTablet for tablet 932f69a7ac8c4a0eb2a436c6b3e6e83f (DEFAULT_TABLE table=default.Table_240f7d6a65f04099abf3e204b2f47fb6 [id=a0e3209e45574a8e8efb821e4a0f56dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:04.520925 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 932f69a7ac8c4a0eb2a436c6b3e6e83f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:04.522620 1167 raft_consensus.cc:385] T 24cb2a71ba3841d4a5e3e5c6fe99ae2f P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:04.522691 1167 raft_consensus.cc:740] T 24cb2a71ba3841d4a5e3e5c6fe99ae2f P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:04.522781 1176 ts_tablet_manager.cc:1929] T 7309d3190cfa415281c5253a634fa7f3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:04.522951 1176 log.cc:1199] T 7309d3190cfa415281c5253a634fa7f3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/7309d3190cfa415281c5253a634fa7f3
I20251029 02:52:04.522957 1167 consensus_queue.cc:260] T 24cb2a71ba3841d4a5e3e5c6fe99ae2f P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:04.523427 1175 tablet_replica.cc:333] T 74873a05ef634867a30aa63b31fcfcc7 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:04.523594 1175 raft_consensus.cc:2243] T 74873a05ef634867a30aa63b31fcfcc7 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:04.523722 1175 raft_consensus.cc:2272] T 74873a05ef634867a30aa63b31fcfcc7 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:04.524207 1175 ts_tablet_manager.cc:1916] T 74873a05ef634867a30aa63b31fcfcc7 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:04.524401 1176 ts_tablet_manager.cc:1950] T 7309d3190cfa415281c5253a634fa7f3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:04.525519 1176 tablet_replica.cc:333] T 392a36a6b59a42f78e3ac7c8fb63214b P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:04.526023 1176 raft_consensus.cc:2243] T 392a36a6b59a42f78e3ac7c8fb63214b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:04.526381 1176 raft_consensus.cc:2272] T 392a36a6b59a42f78e3ac7c8fb63214b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:04.528167 1167 ts_tablet_manager.cc:1434] T 24cb2a71ba3841d4a5e3e5c6fe99ae2f P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.050s user 0.000s sys 0.002s
I20251029 02:52:04.528452 1167 tablet_bootstrap.cc:492] T c9dff4dff82f49e7ad49caad5e404f09 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:04.527477 1176 ts_tablet_manager.cc:1916] T 392a36a6b59a42f78e3ac7c8fb63214b P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:04.535272 1028 tablet_service.cc:1505] Processing CreateTablet for tablet 932f69a7ac8c4a0eb2a436c6b3e6e83f (DEFAULT_TABLE table=default.Table_240f7d6a65f04099abf3e204b2f47fb6 [id=a0e3209e45574a8e8efb821e4a0f56dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:04.536851 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d2077a74c5342eb813accde2d667017. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:04.539856 1167 tablet_bootstrap.cc:654] T c9dff4dff82f49e7ad49caad5e404f09 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:04.542325 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2d08c252a934cf9a45d2f60e62f76aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:04.545868 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 932f69a7ac8c4a0eb2a436c6b3e6e83f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:04.549142 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7a3e6d8e3fb45898d8a5e094dbf0fd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:04.561929 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 7309d3190cfa415281c5253a634fa7f3 (table default.Table_ef563eb489f947d4a4772678806a0777 [id=7de93d6d48ab4d58b5aed9319d7299a4]) successfully deleted
I20251029 02:52:04.562914 1037 tablet_service.cc:1505] Processing CreateTablet for tablet 4929513bed104050bd6065f025303e1a (DEFAULT_TABLE table=default.Table_de3500a0d5f64dd788359ae260ffd539 [id=7f60d0330c9745309bb6efc923f71c73]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:04.563319 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4929513bed104050bd6065f025303e1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:04.567982 520 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_34ad95c52c7645a8a6c2db64e9dff633" } new_table_name: "default.Table_a1811c44f5854d2b94b455330885520a" modify_external_catalogs: true
I20251029 02:52:04.572721 1166 tablet_bootstrap.cc:654] T 42cfdf88472b44979d86836561005ed1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:04.590299 1175 ts_tablet_manager.cc:1929] T 74873a05ef634867a30aa63b31fcfcc7 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:04.590617 1175 log.cc:1199] T 74873a05ef634867a30aa63b31fcfcc7 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/74873a05ef634867a30aa63b31fcfcc7
I20251029 02:52:04.591082 1175 ts_tablet_manager.cc:1950] T 74873a05ef634867a30aa63b31fcfcc7 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:04.592279 1175 tablet_replica.cc:333] T 4aa78f4ac38e4919864c924e50f1aaf4 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:04.592422 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 74873a05ef634867a30aa63b31fcfcc7 (table default.Table_4e97f5e298434c16913bf7ca93ca30f5 [id=48dfcbe8b3654d72b51b77473dd918f6]) successfully deleted
W20251029 02:52:04.592546 847 consensus_peers.cc:597] T 3362e95063af41c8a24e935f547e3089 P c756528642a543eeacd93cef3e335a7d -> Peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Couldn't send request to peer b495a49349324f7d81908f2da700f0d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3362e95063af41c8a24e935f547e3089. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:04.602488 892 tablet_service.cc:1505] Processing CreateTablet for tablet c7a3e6d8e3fb45898d8a5e094dbf0fd7 (DEFAULT_TABLE table=default.Table_4d82d8b113274cf1841f2141772815e0 [id=78d1000dc2e344b1a1fa497b9abe3062]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:04.603053 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7a3e6d8e3fb45898d8a5e094dbf0fd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:04.626920 1175 raft_consensus.cc:2243] T 4aa78f4ac38e4919864c924e50f1aaf4 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:04.627302 1175 raft_consensus.cc:2272] T 4aa78f4ac38e4919864c924e50f1aaf4 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:04.628139 1175 ts_tablet_manager.cc:1916] T 4aa78f4ac38e4919864c924e50f1aaf4 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:04.636163 1854 consensus_queue.cc:1048] T c65469173762446e9725262ae998dbf3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251029 02:52:04.639600 849 consensus_peers.cc:597] T faa835de84ff45b19525d1d603678987 P c756528642a543eeacd93cef3e335a7d -> Peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Couldn't send request to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: faa835de84ff45b19525d1d603678987. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:04.647348 1166 tablet_bootstrap.cc:492] T 42cfdf88472b44979d86836561005ed1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:04.647500 1166 ts_tablet_manager.cc:1403] T 42cfdf88472b44979d86836561005ed1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.146s user 0.002s sys 0.002s
I20251029 02:52:04.666921 1166 raft_consensus.cc:359] T 42cfdf88472b44979d86836561005ed1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:04.667152 1166 raft_consensus.cc:385] T 42cfdf88472b44979d86836561005ed1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:04.667223 1166 raft_consensus.cc:740] T 42cfdf88472b44979d86836561005ed1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:04.667428 1166 consensus_queue.cc:260] T 42cfdf88472b44979d86836561005ed1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:04.667912 1166 ts_tablet_manager.cc:1434] T 42cfdf88472b44979d86836561005ed1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.020s user 0.002s sys 0.000s
I20251029 02:52:04.668169 1166 tablet_bootstrap.cc:492] T cedfb53078d94c81b5e294ce7438325e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:04.669327 1176 ts_tablet_manager.cc:1929] T 392a36a6b59a42f78e3ac7c8fb63214b P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:04.670024 1166 tablet_bootstrap.cc:654] T cedfb53078d94c81b5e294ce7438325e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:04.680023 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_dbd031b128dd49d4acf13158a6352320"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:04.680941 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dbd031b128dd49d4acf13158a6352320 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:04.683300 521 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_592b4d977e464e6d8f257f3fa08797f3" } new_table_name: "default.Table_bb737a143cdb478d9a2b74b68bfacdeb" modify_external_catalogs: true
W20251029 02:52:04.688807 519 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b125cc9aaf4248b8bfa48fb152b832a6 consensus state has no leader
I20251029 02:52:04.689313 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_22f2a73359784f37b6869125fe10909e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:04.692183 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_22f2a73359784f37b6869125fe10909e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:04.690104 524 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_97b75f1637e14d6a84e80e933e6aa059" } new_table_name: "default.Table_6d69db05345d4ecdb52c0f4b4ac5be13" modify_external_catalogs: true
I20251029 02:52:04.701205 1175 ts_tablet_manager.cc:1929] T 4aa78f4ac38e4919864c924e50f1aaf4 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:04.701375 1175 log.cc:1199] T 4aa78f4ac38e4919864c924e50f1aaf4 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/4aa78f4ac38e4919864c924e50f1aaf4
W20251029 02:52:04.701383 524 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 74eafdebb4eb4ba7a0c05d99e2e93fcb has no consensus state
I20251029 02:52:04.701723 1175 ts_tablet_manager.cc:1950] T 4aa78f4ac38e4919864c924e50f1aaf4 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
W20251029 02:52:04.701845 517 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a9e0fcc759a64679b3f59a6c964877b6 consensus state has no leader
I20251029 02:52:04.701825 524 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_fd6ca61b22e442fabd966ea848f4389e" } modify_external_catalogs: true
I20251029 02:52:04.702183 524 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_fd6ca61b22e442fabd966ea848f4389e" } modify_external_catalogs: true
I20251029 02:52:04.702540 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_d844249d525b48428ebf40e1d6cb6c51"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:04.703001 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 658) took 1091 ms. Trace:
I20251029 02:52:04.703084 1175 rpcz_store.cc:276] 1029 02:52:03.611187 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:03.613128 (+ 1941us) service_pool.cc:225] Handling call
1029 02:52:04.592212 (+979084us) ts_tablet_manager.cc:1151] Deleting tablet 4aa78f4ac38e4919864c924e50f1aaf4
1029 02:52:04.592226 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:04.669539 (+ 77313us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:04.701069 (+ 31530us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:04.702978 (+ 1909us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":978969}
I20251029 02:52:04.703625 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 4aa78f4ac38e4919864c924e50f1aaf4 (table default.Table_e168a30d3a1e48ed8053847b304036e9 [id=a93c34a9cd344503954bfae8752a69a6]) successfully deleted
W20251029 02:52:04.706425 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d844249d525b48428ebf40e1d6cb6c51 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:04.712798 524 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet add6e431cdfe433b9fa9e0c377e3f8e8
I20251029 02:52:04.713165 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_2ea4713d35ab4b4fb84ffa9ae3ca4bf6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:04.713706 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2ea4713d35ab4b4fb84ffa9ae3ca4bf6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:04.714044 523 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_84e9fe6549cc42ed94c0f4c0b489f726" } modify_external_catalogs: true
I20251029 02:52:04.714236 523 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_84e9fe6549cc42ed94c0f4c0b489f726" } modify_external_catalogs: true
I20251029 02:52:04.715472 1175 tablet_replica.cc:333] T e1fa4980f2624321b617b84c68b90113 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:04.715786 1175 raft_consensus.cc:2243] T e1fa4980f2624321b617b84c68b90113 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:04.716046 1175 raft_consensus.cc:2272] T e1fa4980f2624321b617b84c68b90113 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:04.716809 1175 ts_tablet_manager.cc:1916] T e1fa4980f2624321b617b84c68b90113 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:04.717299 1854 consensus_queue.cc:1048] T 6feec2c8e030447fae9c292a626dcd1d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:04.717221 1665 raft_consensus.cc:493] T 86fc22bb84794149b598b97df41eb881 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:04.722677 1665 raft_consensus.cc:515] T 86fc22bb84794149b598b97df41eb881 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:04.723546 1665 leader_election.cc:290] T 86fc22bb84794149b598b97df41eb881 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:04.723713 1665 raft_consensus.cc:493] T 02e4588b03144ea2850e13a1f8f29f06 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:04.723843 1665 raft_consensus.cc:515] T 02e4588b03144ea2850e13a1f8f29f06 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:04.724134 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "86fc22bb84794149b598b97df41eb881" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:04.724478 1665 leader_election.cc:290] T 02e4588b03144ea2850e13a1f8f29f06 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
W20251029 02:52:04.724905 712 leader_election.cc:343] T 86fc22bb84794149b598b97df41eb881 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:04.725487 918 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "86fc22bb84794149b598b97df41eb881" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:04.725651 919 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "02e4588b03144ea2850e13a1f8f29f06" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:04.726130 711 leader_election.cc:343] T 86fc22bb84794149b598b97df41eb881 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:04.726239 711 leader_election.cc:304] T 86fc22bb84794149b598b97df41eb881 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
W20251029 02:52:04.726692 711 leader_election.cc:343] T 02e4588b03144ea2850e13a1f8f29f06 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:04.726982 1665 raft_consensus.cc:2749] T 86fc22bb84794149b598b97df41eb881 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:04.739711 522 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_fe1e563f6397461fbe5255d6db0338b4" } modify_external_catalogs: true
I20251029 02:52:04.740015 522 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_fe1e563f6397461fbe5255d6db0338b4" } modify_external_catalogs: true
I20251029 02:52:04.742188 525 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_dbf2e434d7964eac80709c205f61191e" } modify_external_catalogs: true
I20251029 02:52:04.742444 525 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_dbf2e434d7964eac80709c205f61191e" } modify_external_catalogs: true
I20251029 02:52:04.743520 1904 raft_consensus.cc:493] T dbc2092b63e0417997565c8fadb53ae0 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:04.743724 1904 raft_consensus.cc:515] T dbc2092b63e0417997565c8fadb53ae0 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:04.744505 1904 leader_election.cc:290] T dbc2092b63e0417997565c8fadb53ae0 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:04.745007 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dbc2092b63e0417997565c8fadb53ae0" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:04.745198 1060 raft_consensus.cc:2468] T dbc2092b63e0417997565c8fadb53ae0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:04.745698 849 leader_election.cc:304] T dbc2092b63e0417997565c8fadb53ae0 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:04.746716 1904 raft_consensus.cc:2804] T dbc2092b63e0417997565c8fadb53ae0 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:04.746825 1904 raft_consensus.cc:493] T dbc2092b63e0417997565c8fadb53ae0 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:04.746902 1904 raft_consensus.cc:3060] T dbc2092b63e0417997565c8fadb53ae0 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:04.750957 1904 raft_consensus.cc:515] T dbc2092b63e0417997565c8fadb53ae0 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:04.751971 1904 leader_election.cc:290] T dbc2092b63e0417997565c8fadb53ae0 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:04.752280 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dbc2092b63e0417997565c8fadb53ae0" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:52:04.752470 1060 raft_consensus.cc:3060] T dbc2092b63e0417997565c8fadb53ae0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:04.754474 524 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_31d42031714e415fab63cc5b9933fe0c" } modify_external_catalogs: true
I20251029 02:52:04.754735 524 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_31d42031714e415fab63cc5b9933fe0c" } modify_external_catalogs: true
I20251029 02:52:04.757993 1175 ts_tablet_manager.cc:1929] T e1fa4980f2624321b617b84c68b90113 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:04.758138 1175 log.cc:1199] T e1fa4980f2624321b617b84c68b90113 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/e1fa4980f2624321b617b84c68b90113
I20251029 02:52:04.758595 1175 ts_tablet_manager.cc:1950] T e1fa4980f2624321b617b84c68b90113 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:04.768649 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 657) took 1157 ms. Trace:
I20251029 02:52:04.769146 1175 rpcz_store.cc:276] 1029 02:52:03.611075 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:03.613295 (+ 2220us) service_pool.cc:225] Handling call
1029 02:52:04.715398 (+1102103us) ts_tablet_manager.cc:1151] Deleting tablet e1fa4980f2624321b617b84c68b90113
1029 02:52:04.715414 (+ 16us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:04.730260 (+ 14846us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:04.757841 (+ 27581us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:04.768610 (+ 10769us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1101952}
I20251029 02:52:04.775027 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet e1fa4980f2624321b617b84c68b90113 (table default.Table_8a5ae35851664e0f85324ec1ca672031 [id=a0e161a5d9d94a3b936669e6dfe6fb4f]) successfully deleted
I20251029 02:52:04.775542 1175 tablet_replica.cc:333] T bfec87f690a640c39c58c64e16cc67e7 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:04.775997 1175 raft_consensus.cc:2243] T bfec87f690a640c39c58c64e16cc67e7 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:04.776255 1175 raft_consensus.cc:2272] T bfec87f690a640c39c58c64e16cc67e7 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:04.777137 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_1455335d0ba84ae08eff9eca1366a8f0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:04.777987 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1455335d0ba84ae08eff9eca1366a8f0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:52:04.779575 521 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 58db4e3bf468475c9e2f7f28931b126c has no consensus state
I20251029 02:52:04.779897 521 master_service.cc:954] ReplaceTablet: received request to replace tablet 6ecc552c1ef643f8bb84622d9d440ba2 from {username='slave'} at 127.0.0.1:35314
I20251029 02:52:04.782611 523 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet ba699862e6f646c6beaa39f080d0faba
I20251029 02:52:04.783650 751 tablet_service.cc:1552] Processing DeleteTablet for tablet ba699862e6f646c6beaa39f080d0faba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:04 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:04.798801 1176 log.cc:1199] T 392a36a6b59a42f78e3ac7c8fb63214b P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/392a36a6b59a42f78e3ac7c8fb63214b
I20251029 02:52:04.799270 1176 ts_tablet_manager.cc:1950] T 392a36a6b59a42f78e3ac7c8fb63214b P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:04.803598 1058 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "02e4588b03144ea2850e13a1f8f29f06" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:04.808436 1025 tablet_service.cc:1552] Processing DeleteTablet for tablet ba699862e6f646c6beaa39f080d0faba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:04 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:04.808385 1904 consensus_queue.cc:1048] T 3362e95063af41c8a24e935f547e3089 P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:04.809007 1060 raft_consensus.cc:2468] T dbc2092b63e0417997565c8fadb53ae0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
W20251029 02:52:04.817602 512 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.60:41233, user_credentials={real_user=slave}} blocked reactor thread for 43769.7us
W20251029 02:52:04.818230 712 leader_election.cc:343] T 02e4588b03144ea2850e13a1f8f29f06 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:04.818411 712 leader_election.cc:304] T 02e4588b03144ea2850e13a1f8f29f06 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:04.818768 1665 raft_consensus.cc:2749] T 02e4588b03144ea2850e13a1f8f29f06 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:04.819036 849 leader_election.cc:304] T dbc2092b63e0417997565c8fadb53ae0 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:04.819609 1774 raft_consensus.cc:2804] T dbc2092b63e0417997565c8fadb53ae0 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:04.819721 1774 raft_consensus.cc:697] T dbc2092b63e0417997565c8fadb53ae0 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:52:04.819953 1774 consensus_queue.cc:237] T dbc2092b63e0417997565c8fadb53ae0 P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:04.820765 1939 consensus_queue.cc:1048] T 3cded156380f45cc95893825c71ba0e9 P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:04.826613 522 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 82e398f632154c88a846efc042823fb8
W20251029 02:52:04.830847 989 consensus_peers.cc:597] T 6cd7009e06a646acad0802918c2b425e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 -> Peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Couldn't send request to peer b495a49349324f7d81908f2da700f0d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6cd7009e06a646acad0802918c2b425e. This is attempt 1: this message will repeat every 5th retry.
W20251029 02:52:04.831008 989 consensus_peers.cc:597] T c65469173762446e9725262ae998dbf3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 -> Peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Couldn't send request to peer b495a49349324f7d81908f2da700f0d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c65469173762446e9725262ae998dbf3. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:04.836396 524 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 738046ab57334f2dab9f4aab979c5b42
I20251029 02:52:04.838797 751 tablet_service.cc:1552] Processing DeleteTablet for tablet 738046ab57334f2dab9f4aab979c5b42 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:04 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:04.838989 879 tablet_service.cc:1552] Processing DeleteTablet for tablet 738046ab57334f2dab9f4aab979c5b42 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:04 UTC) from {username='slave'} at 127.0.0.1:35702
W20251029 02:52:04.844398 520 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 050186302c5d4c7da393f28f4b9e4262 has no consensus state
I20251029 02:52:04.850046 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 688) took 1199 ms. Trace:
I20251029 02:52:04.850191 1176 rpcz_store.cc:276] 1029 02:52:03.650420 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:03.650667 (+ 247us) service_pool.cc:225] Handling call
1029 02:52:04.525429 (+874762us) ts_tablet_manager.cc:1151] Deleting tablet 392a36a6b59a42f78e3ac7c8fb63214b
1029 02:52:04.525439 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:04.638681 (+113242us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:04.669211 (+ 30530us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:04.850020 (+180809us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":874541}
I20251029 02:52:04.851287 1176 tablet_replica.cc:333] T 3cded156380f45cc95893825c71ba0e9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:04.864584 525 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet c081d2eacfac4ca8a0048e2a1c4f8f83
I20251029 02:52:04.865829 751 tablet_service.cc:1552] Processing DeleteTablet for tablet c081d2eacfac4ca8a0048e2a1c4f8f83 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:04 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:04.865890 879 tablet_service.cc:1552] Processing DeleteTablet for tablet c081d2eacfac4ca8a0048e2a1c4f8f83 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:04 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:04.869951 895 tablet_service.cc:1552] Processing DeleteTablet for tablet 24cb2a71ba3841d4a5e3e5c6fe99ae2f with delete_type TABLET_DATA_DELETED (replaced by tablet f60e7c24982d40319801869ad060864d) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:04.870172 1916 consensus_queue.cc:1048] T 6cd7009e06a646acad0802918c2b425e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:04.873811 884 tablet_service.cc:1552] Processing DeleteTablet for tablet ba699862e6f646c6beaa39f080d0faba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:04 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:04.876695 1172 tablet_bootstrap.cc:492] T 4add552cf4d940d2a0a36d6b58379a34 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:04.876866 1172 ts_tablet_manager.cc:1403] T 4add552cf4d940d2a0a36d6b58379a34 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.519s user 0.036s sys 0.076s
I20251029 02:52:04.877553 1172 raft_consensus.cc:359] T 4add552cf4d940d2a0a36d6b58379a34 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:04.877718 1172 raft_consensus.cc:385] T 4add552cf4d940d2a0a36d6b58379a34 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:04.877760 1172 raft_consensus.cc:740] T 4add552cf4d940d2a0a36d6b58379a34 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:04.877966 1172 consensus_queue.cc:260] T 4add552cf4d940d2a0a36d6b58379a34 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:04.878420 1172 ts_tablet_manager.cc:1434] T 4add552cf4d940d2a0a36d6b58379a34 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20251029 02:52:04.879041 1172 tablet_bootstrap.cc:492] T b3de6670357c4ae59eb77469b2fb434c P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:04.883123 1172 tablet_bootstrap.cc:654] T b3de6670357c4ae59eb77469b2fb434c P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:04.899274 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskYVokEG/build/asan/bin/kudu
/tmp/dist-test-taskYVokEG/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:42343
--webserver_interface=127.0.104.62
--webserver_port=46851
--builtin_ntp_servers=127.0.104.20:44691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:42343,127.0.104.61:34239,127.0.104.60:41233
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20251029 02:52:04.487649 1174 ts_tablet_manager.cc:1929] T 392a36a6b59a42f78e3ac7c8fb63214b P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:04.903545 1174 log.cc:1199] T 392a36a6b59a42f78e3ac7c8fb63214b P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/392a36a6b59a42f78e3ac7c8fb63214b
I20251029 02:52:04.904101 1174 ts_tablet_manager.cc:1950] T 392a36a6b59a42f78e3ac7c8fb63214b P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:04.914271 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 660) took 1236 ms. Trace:
I20251029 02:52:04.914386 1174 rpcz_store.cc:276] 1029 02:52:03.677981 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:03.678095 (+ 114us) service_pool.cc:225] Handling call
1029 02:52:04.138624 (+460529us) ts_tablet_manager.cc:1151] Deleting tablet 392a36a6b59a42f78e3ac7c8fb63214b
1029 02:52:04.138634 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:04.392082 (+253448us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:04.487507 (+ 95425us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:04.914242 (+426735us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":80375,"tablet-delete.queue_time_us":457962}
W20251029 02:52:04.914728 989 consensus_peers.cc:597] T 8afc01d52d784f2fa559cab7ce07c8be P 0cfd4bee16f84b7c8b0ff3ce14db20d5 -> Peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Couldn't send request to peer b495a49349324f7d81908f2da700f0d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8afc01d52d784f2fa559cab7ce07c8be. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:04.914960 1032 tablet_service.cc:1552] Processing DeleteTablet for tablet 738046ab57334f2dab9f4aab979c5b42 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:04 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:04.914948 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 392a36a6b59a42f78e3ac7c8fb63214b (table default.Table_e523efa604564452a1f56dada6e21daf [id=2d40c37ccbd541b9a1ed21c7f6bfb6c2]) successfully deleted
I20251029 02:52:04.916342 1174 tablet_replica.cc:333] T 6cd7009e06a646acad0802918c2b425e P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:04.916769 1032 tablet_service.cc:1552] Processing DeleteTablet for tablet c081d2eacfac4ca8a0048e2a1c4f8f83 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:04 UTC) from {username='slave'} at 127.0.0.1:37626
W20251029 02:52:04.916950 847 leader_election.cc:343] T dbc2092b63e0417997565c8fadb53ae0 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: dbc2092b63e0417997565c8fadb53ae0
W20251029 02:52:04.917318 847 leader_election.cc:343] T dbc2092b63e0417997565c8fadb53ae0 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: dbc2092b63e0417997565c8fadb53ae0
I20251029 02:52:04.918845 1904 consensus_queue.cc:1048] T 1bab90fe904944af948dab3db06b7e34 P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:04.925482 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_73edd4fe45f94ebab19f2a834a265b63"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:04.926250 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_73edd4fe45f94ebab19f2a834a265b63 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:04.929494 1176 raft_consensus.cc:2243] T 3cded156380f45cc95893825c71ba0e9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:04.929783 1176 pending_rounds.cc:70] T 3cded156380f45cc95893825c71ba0e9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Trying to abort 1 pending ops.
I20251029 02:52:04.929836 1176 pending_rounds.cc:77] T 3cded156380f45cc95893825c71ba0e9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215949101886033920 op_type: NO_OP noop_request { }
I20251029 02:52:04.929922 1176 raft_consensus.cc:2889] T 3cded156380f45cc95893825c71ba0e9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20251029 02:52:04.930020 1176 raft_consensus.cc:2272] T 3cded156380f45cc95893825c71ba0e9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:04.932008 1176 ts_tablet_manager.cc:1916] T 3cded156380f45cc95893825c71ba0e9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:04.932380 1916 raft_consensus.cc:493] T 7e6c4f30be624769961c9ec17c5d5dcb P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:04.932492 1916 raft_consensus.cc:515] T 7e6c4f30be624769961c9ec17c5d5dcb P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:04.933079 1916 leader_election.cc:290] T 7e6c4f30be624769961c9ec17c5d5dcb P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
W20251029 02:52:04.937064 989 leader_election.cc:343] T 7e6c4f30be624769961c9ec17c5d5dcb P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 7e6c4f30be624769961c9ec17c5d5dcb
W20251029 02:52:04.918135 847 consensus_peers.cc:597] T 3cded156380f45cc95893825c71ba0e9 P c756528642a543eeacd93cef3e335a7d -> Peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Couldn't send request to peer b495a49349324f7d81908f2da700f0d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3cded156380f45cc95893825c71ba0e9. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:04.937978 522 master_service.cc:954] ReplaceTablet: received request to replace tablet d041e87ceb11441983d0e38b7929c3a4 from {username='slave'} at 127.0.0.1:35314
I20251029 02:52:04.938074 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_b717a166304f42e4886196617960e361"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:04.938839 522 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_51aa8265801a4f8d9c0330cbf2bc72a4" } new_table_name: "default.Table_deb0585c92d04be9824a6ed88c13edbc" modify_external_catalogs: true
I20251029 02:52:04.939302 1176 ts_tablet_manager.cc:1929] T 3cded156380f45cc95893825c71ba0e9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:04.939473 1176 log.cc:1199] T 3cded156380f45cc95893825c71ba0e9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/3cded156380f45cc95893825c71ba0e9
I20251029 02:52:04.939868 1176 ts_tablet_manager.cc:1950] T 3cded156380f45cc95893825c71ba0e9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
W20251029 02:52:04.938895 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b717a166304f42e4886196617960e361 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:04.940785 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 690) took 1290 ms. Trace:
I20251029 02:52:04.940922 1176 rpcz_store.cc:276] 1029 02:52:03.650624 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:03.650677 (+ 53us) service_pool.cc:225] Handling call
1029 02:52:04.851227 (+1200550us) ts_tablet_manager.cc:1151] Deleting tablet 3cded156380f45cc95893825c71ba0e9
1029 02:52:04.851240 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:04.929705 (+ 78465us) spinlock_profiling.cc:243] Waited 78 ms on lock 0x616000125254. stack: 00007f34717829f6 00007f34717827b0 00007f347075e3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f347d64d100 00007f347e99aae3 00007f348155b287 00007f348158e2d4 00007f348156ed70 00007f348156eafd 0000000000a8867c 00007f34717c8d7f 00007f34717cc04b 00007f34717cbec8
1029 02:52:04.935566 (+ 5861us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:04.939213 (+ 3647us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:04.940765 (+ 1552us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":179293568,"tablet-delete.queue_time_us":1200354}
I20251029 02:52:04.941401 1176 tablet_replica.cc:333] T c65469173762446e9725262ae998dbf3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:04.941624 1176 raft_consensus.cc:2243] T c65469173762446e9725262ae998dbf3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:04.941953 1176 raft_consensus.cc:2272] T c65469173762446e9725262ae998dbf3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:04.942965 918 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7e6c4f30be624769961c9ec17c5d5dcb" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:04.943549 990 leader_election.cc:343] T 7e6c4f30be624769961c9ec17c5d5dcb P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:04.943697 990 leader_election.cc:304] T 7e6c4f30be624769961c9ec17c5d5dcb P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5; no voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:04.946497 1176 ts_tablet_manager.cc:1916] T c65469173762446e9725262ae998dbf3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:04.949402 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_ca4bf5512ff048ddb5feb708b19e3f05"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:04.949997 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ca4bf5512ff048ddb5feb708b19e3f05 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:04.951884 1906 raft_consensus.cc:2749] T 7e6c4f30be624769961c9ec17c5d5dcb P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 02:52:04.943041 849 consensus_peers.cc:597] T 3cded156380f45cc95893825c71ba0e9 P c756528642a543eeacd93cef3e335a7d -> Peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Couldn't send request to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3cded156380f45cc95893825c71ba0e9. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:04.954661 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 392a36a6b59a42f78e3ac7c8fb63214b (table default.Table_e523efa604564452a1f56dada6e21daf [id=2d40c37ccbd541b9a1ed21c7f6bfb6c2]) successfully deleted
I20251029 02:52:04.955135 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 3cded156380f45cc95893825c71ba0e9 (table default.Table_6254452e8df24c21ab7d8ae32a06bdfb [id=46182aaaeecc4a4d9ec60cefb55ff50d]) successfully deleted
I20251029 02:52:04.959777 1174 raft_consensus.cc:2243] T 6cd7009e06a646acad0802918c2b425e P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:04.962730 1174 pending_rounds.cc:70] T 6cd7009e06a646acad0802918c2b425e P c756528642a543eeacd93cef3e335a7d: Trying to abort 1 pending ops.
I20251029 02:52:04.962805 1174 pending_rounds.cc:77] T 6cd7009e06a646acad0802918c2b425e P c756528642a543eeacd93cef3e335a7d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215949102464626688 op_type: NO_OP noop_request { }
I20251029 02:52:04.963379 1174 raft_consensus.cc:2889] T 6cd7009e06a646acad0802918c2b425e P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20251029 02:52:04.963487 1174 raft_consensus.cc:2272] T 6cd7009e06a646acad0802918c2b425e P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:04.964785 1916 raft_consensus.cc:493] T c9dff4dff82f49e7ad49caad5e404f09 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:04.964944 1916 raft_consensus.cc:515] T c9dff4dff82f49e7ad49caad5e404f09 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:04.965651 1916 leader_election.cc:290] T c9dff4dff82f49e7ad49caad5e404f09 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:04.967046 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9dff4dff82f49e7ad49caad5e404f09" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:04.967106 1174 ts_tablet_manager.cc:1916] T 6cd7009e06a646acad0802918c2b425e P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:04.967245 768 raft_consensus.cc:2468] T c9dff4dff82f49e7ad49caad5e404f09 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 0.
I20251029 02:52:04.967679 989 leader_election.cc:304] T c9dff4dff82f49e7ad49caad5e404f09 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters:
W20251029 02:52:04.968006 511 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 74eafdebb4eb4ba7a0c05d99e2e93fcb (table default.Table_6d69db05345d4ecdb52c0f4b4ac5be13 [id=bfa24332683e42059d6bc82fb7909765], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 74eafdebb4eb4ba7a0c05d99e2e93fcb has no consensus state
I20251029 02:52:04.968312 1906 raft_consensus.cc:2804] T c9dff4dff82f49e7ad49caad5e404f09 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:04.968463 1906 raft_consensus.cc:493] T c9dff4dff82f49e7ad49caad5e404f09 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:04.968645 1906 raft_consensus.cc:3060] T c9dff4dff82f49e7ad49caad5e404f09 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
W20251029 02:52:04.971685 522 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet eb78efbb100946a781056fab1d8bfaca consensus state has no leader
I20251029 02:52:04.972448 1906 raft_consensus.cc:515] T c9dff4dff82f49e7ad49caad5e404f09 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:04.973520 919 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9dff4dff82f49e7ad49caad5e404f09" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:04.975035 990 leader_election.cc:343] T c9dff4dff82f49e7ad49caad5e404f09 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
W20251029 02:52:04.975427 990 consensus_peers.cc:597] T 6cd7009e06a646acad0802918c2b425e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 -> Peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Couldn't send request to peer c756528642a543eeacd93cef3e335a7d. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:04.977628 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9dff4dff82f49e7ad49caad5e404f09" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
I20251029 02:52:04.979202 1176 ts_tablet_manager.cc:1929] T c65469173762446e9725262ae998dbf3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:04.986663 525 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_a1811c44f5854d2b94b455330885520a" } new_table_name: "default.Table_224e79c0036b470c863299755de8e9e4" modify_external_catalogs: true
I20251029 02:52:04.989812 1174 ts_tablet_manager.cc:1929] T 6cd7009e06a646acad0802918c2b425e P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:04.990031 1174 log.cc:1199] T 6cd7009e06a646acad0802918c2b425e P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/6cd7009e06a646acad0802918c2b425e
I20251029 02:52:04.990535 1174 ts_tablet_manager.cc:1950] T 6cd7009e06a646acad0802918c2b425e P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
W20251029 02:52:04.982000 990 leader_election.cc:343] T c9dff4dff82f49e7ad49caad5e404f09 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:04.991246 1176 log.cc:1199] T c65469173762446e9725262ae998dbf3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/c65469173762446e9725262ae998dbf3
I20251029 02:52:04.991643 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 658) took 1325 ms. Trace:
I20251029 02:52:04.991771 1174 rpcz_store.cc:276] 1029 02:52:03.666179 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:03.666649 (+ 470us) service_pool.cc:225] Handling call
1029 02:52:04.916282 (+1249633us) ts_tablet_manager.cc:1151] Deleting tablet 6cd7009e06a646acad0802918c2b425e
1029 02:52:04.916294 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:04.962619 (+ 46325us) spinlock_profiling.cc:243] Waited 42.8 ms on lock 0x616000067354. stack: 00007f1e09cad9f6 00007f1e09cad7b0 00007f1e08c893ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f1e15b78100 00007f1e16ec5ae3 00007f1e19a86287 00007f1e19ab92d4 00007f1e19a99d70 00007f1e19a99afd 0000000000a8867c 00007f1e09cf3d7f 00007f1e09cf704b 00007f1e09cf6ec8
1029 02:52:04.983973 (+ 21354us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:04.989682 (+ 5709us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:04.991622 (+ 1940us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":98414848,"tablet-delete.queue_time_us":1234598}
I20251029 02:52:04.992040 1166 tablet_bootstrap.cc:492] T cedfb53078d94c81b5e294ce7438325e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:04.992177 1166 ts_tablet_manager.cc:1403] T cedfb53078d94c81b5e294ce7438325e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.324s user 0.005s sys 0.015s
I20251029 02:52:04.992672 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 6cd7009e06a646acad0802918c2b425e (table default.Table_198644e9ee1f45ecb6ad95a759b54607 [id=7810f54ddc4849a8b13bb2928eb761a7]) successfully deleted
I20251029 02:52:04.993076 1166 raft_consensus.cc:359] T cedfb53078d94c81b5e294ce7438325e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:04.993256 1166 raft_consensus.cc:385] T cedfb53078d94c81b5e294ce7438325e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:04.993187 1174 tablet_replica.cc:333] T 6feec2c8e030447fae9c292a626dcd1d P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:04.993321 1166 raft_consensus.cc:740] T cedfb53078d94c81b5e294ce7438325e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
W20251029 02:52:04.985931 989 consensus_peers.cc:597] T 5695b5a7b1854c3983e2d4ab684898b0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 -> Peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Couldn't send request to peer b495a49349324f7d81908f2da700f0d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5695b5a7b1854c3983e2d4ab684898b0. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:04.993587 522 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_ab351955aaf84fb4b49b3a63325f94ac" } modify_external_catalogs: true
I20251029 02:52:04.993626 1174 raft_consensus.cc:2243] T 6feec2c8e030447fae9c292a626dcd1d P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:04.993765 522 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_ab351955aaf84fb4b49b3a63325f94ac" } modify_external_catalogs: true
I20251029 02:52:04.993867 1174 pending_rounds.cc:70] T 6feec2c8e030447fae9c292a626dcd1d P c756528642a543eeacd93cef3e335a7d: Trying to abort 1 pending ops.
I20251029 02:52:04.993959 1176 ts_tablet_manager.cc:1950] T c65469173762446e9725262ae998dbf3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:04.993918 1174 pending_rounds.cc:77] T 6feec2c8e030447fae9c292a626dcd1d P c756528642a543eeacd93cef3e335a7d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215949102525214720 op_type: NO_OP noop_request { }
I20251029 02:52:04.994025 1174 raft_consensus.cc:2889] T 6feec2c8e030447fae9c292a626dcd1d P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20251029 02:52:04.993546 1166 consensus_queue.cc:260] T cedfb53078d94c81b5e294ce7438325e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:04.994118 1174 raft_consensus.cc:2272] T 6feec2c8e030447fae9c292a626dcd1d P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:04.994318 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 689) took 1343 ms. Trace:
I20251029 02:52:04.995476 1172 tablet_bootstrap.cc:492] T b3de6670357c4ae59eb77469b2fb434c P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:04.995710 1172 ts_tablet_manager.cc:1403] T b3de6670357c4ae59eb77469b2fb434c P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.117s user 0.004s sys 0.005s
I20251029 02:52:04.996435 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_90cad35ff5a14d0b87b2785627781b23"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:04.997205 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_90cad35ff5a14d0b87b2785627781b23 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:04.994406 1176 rpcz_store.cc:276] 1029 02:52:03.650520 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:03.650760 (+ 240us) service_pool.cc:225] Handling call
1029 02:52:04.941354 (+1290594us) ts_tablet_manager.cc:1151] Deleting tablet c65469173762446e9725262ae998dbf3
1029 02:52:04.941364 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:04.951335 (+ 9971us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:04.979086 (+ 27751us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:04.994299 (+ 15213us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1290401}
I20251029 02:52:04.998787 1176 tablet_replica.cc:333] T 5695b5a7b1854c3983e2d4ab684898b0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:04.999686 1174 ts_tablet_manager.cc:1916] T 6feec2c8e030447fae9c292a626dcd1d P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:05.000754 1166 ts_tablet_manager.cc:1434] T cedfb53078d94c81b5e294ce7438325e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.008s user 0.003s sys 0.001s
I20251029 02:52:05.001953 1176 raft_consensus.cc:2243] T 5695b5a7b1854c3983e2d4ab684898b0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:05.002337 1176 pending_rounds.cc:70] T 5695b5a7b1854c3983e2d4ab684898b0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Trying to abort 1 pending ops.
I20251029 02:52:05.005686 1167 tablet_bootstrap.cc:492] T c9dff4dff82f49e7ad49caad5e404f09 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:05.005865 1167 ts_tablet_manager.cc:1403] T c9dff4dff82f49e7ad49caad5e404f09 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.477s user 0.053s sys 0.137s
I20251029 02:52:05.003209 1166 tablet_bootstrap.cc:492] T 8df97119d0654d7d92bb5bd17c5d8562 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:05.006956 1176 pending_rounds.cc:77] T 5695b5a7b1854c3983e2d4ab684898b0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215949102763307008 op_type: NO_OP noop_request { }
I20251029 02:52:05.007007 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_bdf9abc4a0cd42f0ba2c936612afde6b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:05.007232 1176 raft_consensus.cc:2889] T 5695b5a7b1854c3983e2d4ab684898b0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 02:52:05.007319 1176 raft_consensus.cc:2272] T 5695b5a7b1854c3983e2d4ab684898b0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251029 02:52:05.007619 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bdf9abc4a0cd42f0ba2c936612afde6b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:05.007874 1176 ts_tablet_manager.cc:1916] T 5695b5a7b1854c3983e2d4ab684898b0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:05.009205 1166 tablet_bootstrap.cc:654] T 8df97119d0654d7d92bb5bd17c5d8562 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
W20251029 02:52:05.009316 989 consensus_peers.cc:597] T 6feec2c8e030447fae9c292a626dcd1d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 -> Peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Couldn't send request to peer b495a49349324f7d81908f2da700f0d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6feec2c8e030447fae9c292a626dcd1d. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:05.009517 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_2743cefd9b784474b3bad6e9d5f71f55"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:05.010083 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2743cefd9b784474b3bad6e9d5f71f55 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:05.010833 1167 raft_consensus.cc:359] T c9dff4dff82f49e7ad49caad5e404f09 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:05.011039 1167 raft_consensus.cc:385] T c9dff4dff82f49e7ad49caad5e404f09 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:05.011152 1167 raft_consensus.cc:740] T c9dff4dff82f49e7ad49caad5e404f09 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:05.014710 1172 raft_consensus.cc:359] T b3de6670357c4ae59eb77469b2fb434c P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:05.014969 1172 raft_consensus.cc:385] T b3de6670357c4ae59eb77469b2fb434c P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:05.015081 1172 raft_consensus.cc:740] T b3de6670357c4ae59eb77469b2fb434c P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:05.015507 1167 consensus_queue.cc:260] T c9dff4dff82f49e7ad49caad5e404f09 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:05.015841 1172 consensus_queue.cc:260] T b3de6670357c4ae59eb77469b2fb434c P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:05.016470 1172 ts_tablet_manager.cc:1434] T b3de6670357c4ae59eb77469b2fb434c P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.021s user 0.002s sys 0.000s
I20251029 02:52:05.017000 517 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_4b9147e899c043ad9023bf0d8d555a0c" } new_table_name: "default.Table_796f8f19849d4b01922da96435e6833c" modify_external_catalogs: true
I20251029 02:52:05.018487 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_6081d34b2c5742caa7681b275bb8e49f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:05.019085 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6081d34b2c5742caa7681b275bb8e49f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:05.019143 1167 ts_tablet_manager.cc:1434] T c9dff4dff82f49e7ad49caad5e404f09 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.013s user 0.003s sys 0.000s
I20251029 02:52:05.019459 1167 tablet_bootstrap.cc:492] T 7e6c4f30be624769961c9ec17c5d5dcb P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:05.021155 1167 tablet_bootstrap.cc:654] T 7e6c4f30be624769961c9ec17c5d5dcb P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:05.022315 1176 ts_tablet_manager.cc:1929] T 5695b5a7b1854c3983e2d4ab684898b0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:05.030265 1174 ts_tablet_manager.cc:1929] T 6feec2c8e030447fae9c292a626dcd1d P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20251029 02:52:05.023666 990 consensus_peers.cc:597] T 6feec2c8e030447fae9c292a626dcd1d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 -> Peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Couldn't send request to peer c756528642a543eeacd93cef3e335a7d. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:05.027351 1906 leader_election.cc:290] T c9dff4dff82f49e7ad49caad5e404f09 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Requested vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:05.036824 1176 log.cc:1199] T 5695b5a7b1854c3983e2d4ab684898b0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/5695b5a7b1854c3983e2d4ab684898b0
I20251029 02:52:05.037350 1174 log.cc:1199] T 6feec2c8e030447fae9c292a626dcd1d P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/6feec2c8e030447fae9c292a626dcd1d
I20251029 02:52:05.037873 1174 ts_tablet_manager.cc:1950] T 6feec2c8e030447fae9c292a626dcd1d P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:05.033154 1166 tablet_bootstrap.cc:492] T 8df97119d0654d7d92bb5bd17c5d8562 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:05.038801 1166 ts_tablet_manager.cc:1403] T 8df97119d0654d7d92bb5bd17c5d8562 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.036s user 0.004s sys 0.003s
I20251029 02:52:05.039599 1166 raft_consensus.cc:359] T 8df97119d0654d7d92bb5bd17c5d8562 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:05.039780 1166 raft_consensus.cc:385] T 8df97119d0654d7d92bb5bd17c5d8562 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:05.039835 1166 raft_consensus.cc:740] T 8df97119d0654d7d92bb5bd17c5d8562 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
W20251029 02:52:05.040208 847 consensus_peers.cc:597] T 1bab90fe904944af948dab3db06b7e34 P c756528642a543eeacd93cef3e335a7d -> Peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Couldn't send request to peer b495a49349324f7d81908f2da700f0d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1bab90fe904944af948dab3db06b7e34. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:05.040378 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 665) took 1358 ms. Trace:
I20251029 02:52:05.040488 1174 rpcz_store.cc:276] 1029 02:52:03.682186 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:03.683437 (+ 1251us) service_pool.cc:225] Handling call
1029 02:52:04.993115 (+1309678us) ts_tablet_manager.cc:1151] Deleting tablet 6feec2c8e030447fae9c292a626dcd1d
1029 02:52:04.993131 (+ 16us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:05.016608 (+ 23477us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:05.030095 (+ 13487us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:05.040348 (+ 10253us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1309517}
I20251029 02:52:05.040609 1166 consensus_queue.cc:260] T 8df97119d0654d7d92bb5bd17c5d8562 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:05.042129 1166 ts_tablet_manager.cc:1434] T 8df97119d0654d7d92bb5bd17c5d8562 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251029 02:52:05.026481 1957 raft_consensus.cc:493] T f60e7c24982d40319801869ad060864d P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:05.027369 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9dff4dff82f49e7ad49caad5e404f09" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:05.042629 768 raft_consensus.cc:3060] T c9dff4dff82f49e7ad49caad5e404f09 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:05.043310 1166 tablet_bootstrap.cc:492] T b2b2d64dcf964da8be4b544dbbe31977 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:05.044917 1166 tablet_bootstrap.cc:654] T b2b2d64dcf964da8be4b544dbbe31977 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:05.045600 1176 ts_tablet_manager.cc:1950] T 5695b5a7b1854c3983e2d4ab684898b0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:05.045778 768 raft_consensus.cc:2468] T c9dff4dff82f49e7ad49caad5e404f09 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 1.
I20251029 02:52:05.045851 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 669) took 1363 ms. Trace:
I20251029 02:52:05.046008 1174 rpcz_store.cc:276] 1029 02:52:03.682695 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:03.683475 (+ 780us) service_pool.cc:225] Handling call
1029 02:52:05.045654 (+1362179us) ts_tablet_manager.cc:1151] Deleting tablet ce98420e61904af78d8fa03724c74489
1029 02:52:05.045667 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:05.045791 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1361931}
I20251029 02:52:05.042419 1957 raft_consensus.cc:515] T f60e7c24982d40319801869ad060864d P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:05.046254 989 leader_election.cc:304] T c9dff4dff82f49e7ad49caad5e404f09 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters: c756528642a543eeacd93cef3e335a7d
I20251029 02:52:05.046823 1957 leader_election.cc:290] T f60e7c24982d40319801869ad060864d P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:05.046847 1906 raft_consensus.cc:2804] T c9dff4dff82f49e7ad49caad5e404f09 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:05.046967 1906 raft_consensus.cc:697] T c9dff4dff82f49e7ad49caad5e404f09 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Becoming Leader. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Running, Role: LEADER
I20251029 02:52:05.047171 1174 tablet_replica.cc:333] T faa835de84ff45b19525d1d603678987 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:05.047216 1906 consensus_queue.cc:237] T c9dff4dff82f49e7ad49caad5e404f09 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:05.047379 1174 raft_consensus.cc:2243] T faa835de84ff45b19525d1d603678987 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:05.047449 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f60e7c24982d40319801869ad060864d" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:05.048220 1174 pending_rounds.cc:70] T faa835de84ff45b19525d1d603678987 P c756528642a543eeacd93cef3e335a7d: Trying to abort 1 pending ops.
W20251029 02:52:05.048254 711 leader_election.cc:343] T f60e7c24982d40319801869ad060864d P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:05.048298 1174 pending_rounds.cc:77] T faa835de84ff45b19525d1d603678987 P c756528642a543eeacd93cef3e335a7d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215949102421344256 op_type: NO_OP noop_request { }
I20251029 02:52:05.048398 1174 raft_consensus.cc:2889] T faa835de84ff45b19525d1d603678987 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 02:52:05.048514 1174 raft_consensus.cc:2272] T faa835de84ff45b19525d1d603678987 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:05.049443 1058 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f60e7c24982d40319801869ad060864d" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:52:05.049968 712 leader_election.cc:343] T f60e7c24982d40319801869ad060864d P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:05.050067 712 leader_election.cc:304] T f60e7c24982d40319801869ad060864d P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:05.050464 1957 raft_consensus.cc:2749] T f60e7c24982d40319801869ad060864d P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:05.051054 1167 tablet_bootstrap.cc:492] T 7e6c4f30be624769961c9ec17c5d5dcb P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:05.051234 1167 ts_tablet_manager.cc:1403] T 7e6c4f30be624769961c9ec17c5d5dcb P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.032s user 0.002s sys 0.002s
I20251029 02:52:05.052148 1172 tablet_bootstrap.cc:492] T 1bc45078a6a942c38ff3350c15885eff P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:05.052325 1167 raft_consensus.cc:359] T 7e6c4f30be624769961c9ec17c5d5dcb P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:05.052559 1167 raft_consensus.cc:385] T 7e6c4f30be624769961c9ec17c5d5dcb P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:05.052657 1167 raft_consensus.cc:740] T 7e6c4f30be624769961c9ec17c5d5dcb P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:05.052978 1167 consensus_queue.cc:260] T 7e6c4f30be624769961c9ec17c5d5dcb P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:05.054013 1167 ts_tablet_manager.cc:1434] T 7e6c4f30be624769961c9ec17c5d5dcb P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20251029 02:52:05.046347 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 686) took 1396 ms. Trace:
I20251029 02:52:05.054693 1176 rpcz_store.cc:276] 1029 02:52:03.650239 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:03.651078 (+ 839us) service_pool.cc:225] Handling call
1029 02:52:04.998738 (+1347660us) ts_tablet_manager.cc:1151] Deleting tablet 5695b5a7b1854c3983e2d4ab684898b0
1029 02:52:04.998750 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:05.019038 (+ 20288us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:05.022213 (+ 3175us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:05.046321 (+ 24108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1347506}
I20251029 02:52:05.054875 1167 tablet_bootstrap.cc:492] T 576c4e6e70e24b0399ac9597a30f0569 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:05.055145 1174 ts_tablet_manager.cc:1916] T faa835de84ff45b19525d1d603678987 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:05.055555 1176 tablet_replica.cc:333] T 1f5b40c0403d4a49a4bfde222d8c947a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:05.055716 1172 tablet_bootstrap.cc:654] T 1bc45078a6a942c38ff3350c15885eff P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:05.056053 1176 raft_consensus.cc:2243] T 1f5b40c0403d4a49a4bfde222d8c947a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:05.056689 1176 pending_rounds.cc:70] T 1f5b40c0403d4a49a4bfde222d8c947a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Trying to abort 1 pending ops.
I20251029 02:52:05.057165 1176 pending_rounds.cc:77] T 1f5b40c0403d4a49a4bfde222d8c947a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215949104927199232 op_type: NO_OP noop_request { }
I20251029 02:52:05.057458 1176 raft_consensus.cc:2889] T 1f5b40c0403d4a49a4bfde222d8c947a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 02:52:05.057718 1176 raft_consensus.cc:2272] T 1f5b40c0403d4a49a4bfde222d8c947a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:05.058837 1176 ts_tablet_manager.cc:1916] T 1f5b40c0403d4a49a4bfde222d8c947a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:05.060719 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 6feec2c8e030447fae9c292a626dcd1d (table default.Table_cb3d27722cc74fa18ddc8740590639a9 [id=37499c7664774cb9bd0a80c7c5bef158]) successfully deleted
I20251029 02:52:05.061060 1167 tablet_bootstrap.cc:654] T 576c4e6e70e24b0399ac9597a30f0569 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
W20251029 02:52:05.061319 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet ce98420e61904af78d8fa03724c74489 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ce98420e61904af78d8fa03724c74489 already in progress: creating tablet
I20251029 02:52:05.065135 523 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_e99e1cb4ad41429aa0160f749671630a" } new_table_name: "default.Table_cfb1df2e2beb438baa234ac0e7f3b36b" modify_external_catalogs: true
I20251029 02:52:05.072806 1176 ts_tablet_manager.cc:1929] T 1f5b40c0403d4a49a4bfde222d8c947a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:05.075673 1906 raft_consensus.cc:493] T 42cfdf88472b44979d86836561005ed1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:05.076495 1906 raft_consensus.cc:515] T 42cfdf88472b44979d86836561005ed1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:05.077577 1906 leader_election.cc:290] T 42cfdf88472b44979d86836561005ed1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:05.077616 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42cfdf88472b44979d86836561005ed1" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:05.077800 1176 log.cc:1199] T 1f5b40c0403d4a49a4bfde222d8c947a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/1f5b40c0403d4a49a4bfde222d8c947a
I20251029 02:52:05.078118 1176 ts_tablet_manager.cc:1950] T 1f5b40c0403d4a49a4bfde222d8c947a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:05.078166 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42cfdf88472b44979d86836561005ed1" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:05.078353 768 raft_consensus.cc:2468] T 42cfdf88472b44979d86836561005ed1 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 0.
I20251029 02:52:05.079128 1167 tablet_bootstrap.cc:492] T 576c4e6e70e24b0399ac9597a30f0569 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
W20251029 02:52:05.078168 990 leader_election.cc:343] T 42cfdf88472b44979d86836561005ed1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:05.079277 1167 ts_tablet_manager.cc:1403] T 576c4e6e70e24b0399ac9597a30f0569 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.002s
I20251029 02:52:05.079355 989 leader_election.cc:304] T 42cfdf88472b44979d86836561005ed1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters: c756528642a543eeacd93cef3e335a7d
I20251029 02:52:05.079782 1906 raft_consensus.cc:2804] T 42cfdf88472b44979d86836561005ed1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:05.079854 1906 raft_consensus.cc:493] T 42cfdf88472b44979d86836561005ed1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:05.079910 1906 raft_consensus.cc:3060] T 42cfdf88472b44979d86836561005ed1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:05.079960 1167 raft_consensus.cc:359] T 576c4e6e70e24b0399ac9597a30f0569 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:05.080099 1167 raft_consensus.cc:385] T 576c4e6e70e24b0399ac9597a30f0569 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:05.080067 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet c65469173762446e9725262ae998dbf3 (table default.Table_06e7c4a46c50490196ec56306b13a2bb [id=859e6643e304437c8dc9c14973b21b88]) successfully deleted
I20251029 02:52:05.080148 1167 raft_consensus.cc:740] T 576c4e6e70e24b0399ac9597a30f0569 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:05.080364 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 5695b5a7b1854c3983e2d4ab684898b0 (table default.Table_e43a2189b18a43f9beb9f3ce25848374 [id=d151d895aac64712b2eeee989d19b878]) successfully deleted
I20251029 02:52:05.080376 1167 consensus_queue.cc:260] T 576c4e6e70e24b0399ac9597a30f0569 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:05.082554 1167 ts_tablet_manager.cc:1434] T 576c4e6e70e24b0399ac9597a30f0569 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251029 02:52:05.082577 1906 raft_consensus.cc:515] T 42cfdf88472b44979d86836561005ed1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:05.082991 1167 tablet_bootstrap.cc:492] T 42cfdf88472b44979d86836561005ed1 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:05.083629 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42cfdf88472b44979d86836561005ed1" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
I20251029 02:52:05.084043 1906 leader_election.cc:290] T 42cfdf88472b44979d86836561005ed1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Requested vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
W20251029 02:52:05.084169 990 leader_election.cc:343] T 42cfdf88472b44979d86836561005ed1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:05.084358 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42cfdf88472b44979d86836561005ed1" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:05.084542 768 raft_consensus.cc:3060] T 42cfdf88472b44979d86836561005ed1 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
W20251029 02:52:05.084597 990 consensus_peers.cc:597] T 8afc01d52d784f2fa559cab7ce07c8be P 0cfd4bee16f84b7c8b0ff3ce14db20d5 -> Peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Couldn't send request to peer c756528642a543eeacd93cef3e335a7d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8afc01d52d784f2fa559cab7ce07c8be. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:05.084933 1167 tablet_bootstrap.cc:654] T 42cfdf88472b44979d86836561005ed1 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:05.084976 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 691) took 1434 ms. Trace:
I20251029 02:52:05.085040 1176 rpcz_store.cc:276] 1029 02:52:03.650709 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:03.651259 (+ 550us) service_pool.cc:225] Handling call
1029 02:52:05.055509 (+1404250us) ts_tablet_manager.cc:1151] Deleting tablet 1f5b40c0403d4a49a4bfde222d8c947a
1029 02:52:05.055519 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:05.069008 (+ 13489us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:05.072707 (+ 3699us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:05.084958 (+ 12251us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1404116}
I20251029 02:52:05.085904 1166 tablet_bootstrap.cc:492] T b2b2d64dcf964da8be4b544dbbe31977 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:05.086006 1166 ts_tablet_manager.cc:1403] T b2b2d64dcf964da8be4b544dbbe31977 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.043s user 0.006s sys 0.002s
I20251029 02:52:05.086735 1166 raft_consensus.cc:359] T b2b2d64dcf964da8be4b544dbbe31977 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:05.086865 1166 raft_consensus.cc:385] T b2b2d64dcf964da8be4b544dbbe31977 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:05.086923 1166 raft_consensus.cc:740] T b2b2d64dcf964da8be4b544dbbe31977 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:05.087121 1166 consensus_queue.cc:260] T b2b2d64dcf964da8be4b544dbbe31977 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:05.088740 1166 ts_tablet_manager.cc:1434] T b2b2d64dcf964da8be4b544dbbe31977 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251029 02:52:05.089841 1166 tablet_bootstrap.cc:492] T c9e923724ca9402682f13b252dbc37aa P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:05.090718 1176 tablet_replica.cc:333] T 24cb2a71ba3841d4a5e3e5c6fe99ae2f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:05.091034 1176 raft_consensus.cc:2243] T 24cb2a71ba3841d4a5e3e5c6fe99ae2f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:05.092782 768 raft_consensus.cc:2468] T 42cfdf88472b44979d86836561005ed1 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 1.
I20251029 02:52:05.093292 989 leader_election.cc:304] T 42cfdf88472b44979d86836561005ed1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters: c756528642a543eeacd93cef3e335a7d
I20251029 02:52:05.093694 1906 raft_consensus.cc:2804] T 42cfdf88472b44979d86836561005ed1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:05.094208 1906 raft_consensus.cc:697] T 42cfdf88472b44979d86836561005ed1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Becoming Leader. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Running, Role: LEADER
I20251029 02:52:05.094439 1906 consensus_queue.cc:237] T 42cfdf88472b44979d86836561005ed1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:05.096230 1176 raft_consensus.cc:2272] T 24cb2a71ba3841d4a5e3e5c6fe99ae2f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251029 02:52:05.098042 525 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 050186302c5d4c7da393f28f4b9e4262 has no consensus state
I20251029 02:52:05.099160 1167 tablet_bootstrap.cc:492] T 42cfdf88472b44979d86836561005ed1 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:05.099344 1167 ts_tablet_manager.cc:1403] T 42cfdf88472b44979d86836561005ed1 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.000s
I20251029 02:52:05.100363 1167 raft_consensus.cc:359] T 42cfdf88472b44979d86836561005ed1 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:05.100560 1167 raft_consensus.cc:385] T 42cfdf88472b44979d86836561005ed1 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:05.100615 1167 raft_consensus.cc:740] T 42cfdf88472b44979d86836561005ed1 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:05.100875 1167 consensus_queue.cc:260] T 42cfdf88472b44979d86836561005ed1 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:05.101749 1167 ts_tablet_manager.cc:1434] T 42cfdf88472b44979d86836561005ed1 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20251029 02:52:05.101934 517 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5815626b05204b578cc4a110f246bf10 consensus state has no leader
I20251029 02:52:05.102124 1167 tablet_bootstrap.cc:492] T f81aecb20d6e498bb9b498698886d65e P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:05.102337 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_46188a55c5034e5d992c14c95303ab26"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:05.102969 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_46188a55c5034e5d992c14c95303ab26 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:05.103492 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_47a306977f08434993da629ff2ed490a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:05.104056 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_47a306977f08434993da629ff2ed490a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:05.104313 1167 tablet_bootstrap.cc:654] T f81aecb20d6e498bb9b498698886d65e P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:05.104791 749 tablet_service.cc:1552] Processing DeleteTablet for tablet 4add552cf4d940d2a0a36d6b58379a34 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:53 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:05.105515 526 master_service.cc:954] ReplaceTablet: received request to replace tablet 2089b058d0e94d99b40ad2718518353e from {username='slave'} at 127.0.0.1:35314
I20251029 02:52:05.106282 526 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_2e41802f8663436e83915f560cf8f94b" } modify_external_catalogs: true
I20251029 02:52:05.106482 526 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_2e41802f8663436e83915f560cf8f94b" } modify_external_catalogs: true
I20251029 02:52:05.107455 519 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_2ccb14f87f774da3b62823c798b31a7c" } modify_external_catalogs: true
I20251029 02:52:05.107650 519 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_2ccb14f87f774da3b62823c798b31a7c" } modify_external_catalogs: true
I20251029 02:52:05.110646 1176 ts_tablet_manager.cc:1916] T 24cb2a71ba3841d4a5e3e5c6fe99ae2f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:05.111671 1166 tablet_bootstrap.cc:654] T c9e923724ca9402682f13b252dbc37aa P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:05.112457 524 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_9aa86bf792a94cfeb454cf16575d139a" } new_table_name: "default.Table_045fe56124d445f987ef1b91282c7db5" modify_external_catalogs: true
I20251029 02:52:05.115535 525 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_ac4056c73c0f41cda4580d2736f4c6a5" } modify_external_catalogs: true
I20251029 02:52:05.115805 525 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_ac4056c73c0f41cda4580d2736f4c6a5" } modify_external_catalogs: true
I20251029 02:52:05.116896 892 tablet_service.cc:1552] Processing DeleteTablet for tablet 6feec2c8e030447fae9c292a626dcd1d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:52 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:05.130015 1176 ts_tablet_manager.cc:1929] T 24cb2a71ba3841d4a5e3e5c6fe99ae2f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:05.132855 1176 log.cc:1199] T 24cb2a71ba3841d4a5e3e5c6fe99ae2f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/24cb2a71ba3841d4a5e3e5c6fe99ae2f
I20251029 02:52:05.133158 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 1f5b40c0403d4a49a4bfde222d8c947a (table default.Table_72f8739f5f654172b10d619a9ac306bd [id=1565d03ae28649e48c2a614a8d067227]) successfully deleted
I20251029 02:52:05.133635 1176 ts_tablet_manager.cc:1950] T 24cb2a71ba3841d4a5e3e5c6fe99ae2f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:05.135427 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 695) took 1483 ms. Trace:
I20251029 02:52:05.136797 1174 ts_tablet_manager.cc:1929] T faa835de84ff45b19525d1d603678987 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:05.139367 1176 rpcz_store.cc:276] 1029 02:52:03.652316 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:03.652372 (+ 56us) service_pool.cc:225] Handling call
1029 02:52:05.088908 (+1436536us) ts_tablet_manager.cc:1151] Deleting tablet 24cb2a71ba3841d4a5e3e5c6fe99ae2f
1029 02:52:05.090662 (+ 1754us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:05.124103 (+ 33441us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:05.129878 (+ 5775us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:05.135344 (+ 5466us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1436333}
I20251029 02:52:05.143064 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 24cb2a71ba3841d4a5e3e5c6fe99ae2f (table default.Table_4350e92f14de4ba8a04afbe12ea7422a [id=226a2d412a7b40e59b6f5340b5a9b739]) successfully deleted
I20251029 02:52:05.143190 1174 log.cc:1199] T faa835de84ff45b19525d1d603678987 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/faa835de84ff45b19525d1d603678987
I20251029 02:52:05.143524 1176 tablet_replica.cc:333] T 3362e95063af41c8a24e935f547e3089 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:05.143790 1176 raft_consensus.cc:2243] T 3362e95063af41c8a24e935f547e3089 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:05.144022 1176 raft_consensus.cc:2272] T 3362e95063af41c8a24e935f547e3089 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:05.144081 1174 ts_tablet_manager.cc:1950] T faa835de84ff45b19525d1d603678987 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:05.145726 1176 ts_tablet_manager.cc:1916] T 3362e95063af41c8a24e935f547e3089 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:05.145704 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 662) took 1463 ms. Trace:
I20251029 02:52:05.146878 1174 rpcz_store.cc:276] 1029 02:52:03.681850 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:03.687307 (+ 5457us) service_pool.cc:225] Handling call
1029 02:52:05.047093 (+1359786us) ts_tablet_manager.cc:1151] Deleting tablet faa835de84ff45b19525d1d603678987
1029 02:52:05.047104 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:05.120131 (+ 73027us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:05.136679 (+ 16548us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:05.145681 (+ 9002us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1347847}
I20251029 02:52:05.147653 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet faa835de84ff45b19525d1d603678987 (table default.Table_d9e164af67334c98a5d0821f2d80a35e [id=a184f139921d4864b1627057a7359810]) successfully deleted
I20251029 02:52:05.148070 1174 tablet_replica.cc:333] T c65469173762446e9725262ae998dbf3 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:05.148463 1174 raft_consensus.cc:2243] T c65469173762446e9725262ae998dbf3 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:05.148939 1174 raft_consensus.cc:2272] T c65469173762446e9725262ae998dbf3 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:05.150048 1174 ts_tablet_manager.cc:1916] T c65469173762446e9725262ae998dbf3 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:05.152266 1172 tablet_bootstrap.cc:492] T 1bc45078a6a942c38ff3350c15885eff P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:05.152426 1172 ts_tablet_manager.cc:1403] T 1bc45078a6a942c38ff3350c15885eff P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.100s user 0.003s sys 0.000s
I20251029 02:52:05.152828 1166 tablet_bootstrap.cc:492] T c9e923724ca9402682f13b252dbc37aa P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:05.153033 1166 ts_tablet_manager.cc:1403] T c9e923724ca9402682f13b252dbc37aa P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.063s user 0.010s sys 0.000s
I20251029 02:52:05.153286 1172 raft_consensus.cc:359] T 1bc45078a6a942c38ff3350c15885eff P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:05.153468 1172 raft_consensus.cc:385] T 1bc45078a6a942c38ff3350c15885eff P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:05.153524 1172 raft_consensus.cc:740] T 1bc45078a6a942c38ff3350c15885eff P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:05.153759 1172 consensus_queue.cc:260] T 1bc45078a6a942c38ff3350c15885eff P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:05.153921 1166 raft_consensus.cc:359] T c9e923724ca9402682f13b252dbc37aa P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:05.154134 1166 raft_consensus.cc:385] T c9e923724ca9402682f13b252dbc37aa P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:05.154208 1166 raft_consensus.cc:740] T c9e923724ca9402682f13b252dbc37aa P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:05.154426 1166 consensus_queue.cc:260] T c9e923724ca9402682f13b252dbc37aa P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:05.154938 1166 ts_tablet_manager.cc:1434] T c9e923724ca9402682f13b252dbc37aa P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 02:52:05.157017 1166 tablet_bootstrap.cc:492] T f81aecb20d6e498bb9b498698886d65e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:05.158685 1166 tablet_bootstrap.cc:654] T f81aecb20d6e498bb9b498698886d65e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:05.166800 1176 ts_tablet_manager.cc:1929] T 3362e95063af41c8a24e935f547e3089 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:05.170956 520 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet 42cfdf88472b44979d86836561005ed1 (table default.Table_ad2ac3ba98784d68a94eb4eb03e530a4 [id=5e0c72957b9a491d960b33d3fffd8f70]). Expected version 2 got 0
I20251029 02:52:05.171119 749 tablet_service.cc:1505] Processing CreateTablet for tablet 25129594fe3b40f28d87c0c1b7c025af (DEFAULT_TABLE table=default.Table_e3cc9c53ee9e4fafb7f9684f6c518714 [id=7889c07084f341d091ab2081bc17c9ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:05.171583 1176 log.cc:1199] T 3362e95063af41c8a24e935f547e3089 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/3362e95063af41c8a24e935f547e3089
I20251029 02:52:05.171597 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25129594fe3b40f28d87c0c1b7c025af. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:05.172004 1176 ts_tablet_manager.cc:1950] T 3362e95063af41c8a24e935f547e3089 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:05.174304 1174 ts_tablet_manager.cc:1929] T c65469173762446e9725262ae998dbf3 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:05.174418 1174 log.cc:1199] T c65469173762446e9725262ae998dbf3 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/c65469173762446e9725262ae998dbf3
I20251029 02:52:05.176573 892 tablet_service.cc:1505] Processing CreateTablet for tablet 25129594fe3b40f28d87c0c1b7c025af (DEFAULT_TABLE table=default.Table_e3cc9c53ee9e4fafb7f9684f6c518714 [id=7889c07084f341d091ab2081bc17c9ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:05.176978 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25129594fe3b40f28d87c0c1b7c025af. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:05.177991 1174 ts_tablet_manager.cc:1950] T c65469173762446e9725262ae998dbf3 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:05.179625 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 696) took 1527 ms. Trace:
I20251029 02:52:05.179669 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 670) took 1496 ms. Trace:
I20251029 02:52:05.179737 1174 rpcz_store.cc:276] 1029 02:52:03.682793 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:03.683752 (+ 959us) service_pool.cc:225] Handling call
1029 02:52:05.148019 (+1464267us) ts_tablet_manager.cc:1151] Deleting tablet c65469173762446e9725262ae998dbf3
1029 02:52:05.148030 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:05.163059 (+ 15029us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:05.174188 (+ 11129us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:05.179650 (+ 5462us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1448592}
I20251029 02:52:05.180121 1176 rpcz_store.cc:276] 1029 02:52:03.652443 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:03.652501 (+ 58us) service_pool.cc:225] Handling call
1029 02:52:05.143424 (+1490923us) ts_tablet_manager.cc:1151] Deleting tablet 3362e95063af41c8a24e935f547e3089
1029 02:52:05.143437 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:05.149584 (+ 6147us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:05.166669 (+ 17085us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:05.179581 (+ 12912us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1490711}
I20251029 02:52:05.181022 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet c65469173762446e9725262ae998dbf3 (table default.Table_06e7c4a46c50490196ec56306b13a2bb [id=859e6643e304437c8dc9c14973b21b88]) successfully deleted
I20251029 02:52:05.181098 1176 tablet_replica.cc:333] T 6feec2c8e030447fae9c292a626dcd1d P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:05.181613 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 667) took 1499 ms. Trace:
I20251029 02:52:05.181700 1176 raft_consensus.cc:2243] T 6feec2c8e030447fae9c292a626dcd1d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:05.182171 1174 rpcz_store.cc:276] 1029 02:52:03.682374 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:03.683714 (+ 1340us) service_pool.cc:225] Handling call
1029 02:52:05.181451 (+1497737us) ts_tablet_manager.cc:1151] Deleting tablet ae5a6caeb17f4851b27a3c6a3c0db83c
1029 02:52:05.181465 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:05.181589 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1481883}
I20251029 02:52:05.182126 1176 raft_consensus.cc:2272] T 6feec2c8e030447fae9c292a626dcd1d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:05.182857 751 tablet_service.cc:1505] Processing CreateTablet for tablet 2d33bc6dafb64d948cd62feb50193b26 (DEFAULT_TABLE table=default.Table_b3877f1183ba42e9b368d74299f13f55 [id=a111929624254a918697d7ba42b83678]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:05.183282 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d33bc6dafb64d948cd62feb50193b26. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:05.183440 520 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet 42cfdf88472b44979d86836561005ed1 (table default.Table_ad2ac3ba98784d68a94eb4eb03e530a4 [id=5e0c72957b9a491d960b33d3fffd8f70]). Expected version 2 got 0
I20251029 02:52:05.183211 881 tablet_service.cc:1505] Processing CreateTablet for tablet 2d33bc6dafb64d948cd62feb50193b26 (DEFAULT_TABLE table=default.Table_b3877f1183ba42e9b368d74299f13f55 [id=a111929624254a918697d7ba42b83678]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:05.184386 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d33bc6dafb64d948cd62feb50193b26. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:05.184345 1176 ts_tablet_manager.cc:1916] T 6feec2c8e030447fae9c292a626dcd1d P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:05.186017 895 tablet_service.cc:1505] Processing CreateTablet for tablet 657cf290d68c45e98f0ffd05c7d65ad7 (DEFAULT_TABLE table=default.Table_4fbd78edfb5a451f91ddcdf41de962a3 [id=1685db596a8a493c9cdf209e22da22ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:05.186318 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 657cf290d68c45e98f0ffd05c7d65ad7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:05.186374 1032 tablet_service.cc:1505] Processing CreateTablet for tablet 25129594fe3b40f28d87c0c1b7c025af (DEFAULT_TABLE table=default.Table_e3cc9c53ee9e4fafb7f9684f6c518714 [id=7889c07084f341d091ab2081bc17c9ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:05.183501 884 tablet_service.cc:1505] Processing CreateTablet for tablet d8c66a892c90475c834f08c0734f41db (DEFAULT_TABLE table=default.Table_b8fe14e0ff054e67b0594e8b97483c71 [id=e27928a6cb8f4788b7e8c9c7a1c689d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:05.186756 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8c66a892c90475c834f08c0734f41db. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:05.186780 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 25129594fe3b40f28d87c0c1b7c025af. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:05.187141 1037 tablet_service.cc:1552] Processing DeleteTablet for tablet 3cded156380f45cc95893825c71ba0e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:53 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:05.187793 1037 tablet_service.cc:1505] Processing CreateTablet for tablet 2d33bc6dafb64d948cd62feb50193b26 (DEFAULT_TABLE table=default.Table_b3877f1183ba42e9b368d74299f13f55 [id=a111929624254a918697d7ba42b83678]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:05.188192 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d33bc6dafb64d948cd62feb50193b26. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:05.190740 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet 576c4e6e70e24b0399ac9597a30f0569 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:54 UTC) from {username='slave'} at 127.0.0.1:37626
W20251029 02:52:05.197131 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet ae5a6caeb17f4851b27a3c6a3c0db83c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ae5a6caeb17f4851b27a3c6a3c0db83c already in progress: creating tablet
I20251029 02:52:05.197587 1174 tablet_replica.cc:333] T 3cded156380f45cc95893825c71ba0e9 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:05.198040 1174 raft_consensus.cc:2243] T 3cded156380f45cc95893825c71ba0e9 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:05.198653 1174 raft_consensus.cc:2272] T 3cded156380f45cc95893825c71ba0e9 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:05.198933 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 3362e95063af41c8a24e935f547e3089 (table default.Table_2a63af6894694f16b45f4bf48f73a68f [id=fb42af08ec1f48498ee5e91cce62333e]) successfully deleted
I20251029 02:52:05.199786 1174 ts_tablet_manager.cc:1916] T 3cded156380f45cc95893825c71ba0e9 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 02:52:05.201476 524 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6f8cf09f23984b879b60ab6b27c7d52e consensus state has no leader
I20251029 02:52:05.201910 750 tablet_service.cc:1505] Processing CreateTablet for tablet 657cf290d68c45e98f0ffd05c7d65ad7 (DEFAULT_TABLE table=default.Table_4fbd78edfb5a451f91ddcdf41de962a3 [id=1685db596a8a493c9cdf209e22da22ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:05.202385 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 657cf290d68c45e98f0ffd05c7d65ad7. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:05.200712 523 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7ff2399833e546448b85b9caeb5200fb consensus state has no leader
I20251029 02:52:05.207646 1028 tablet_service.cc:1505] Processing CreateTablet for tablet 79b65cab235141f2916b10723c5a57cd (DEFAULT_TABLE table=default.Table_654ba01aef8c4aacb0f0aae654401c0a [id=67a7876d8e1a4b2b80123bc4162225ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:05.207863 1039 tablet_service.cc:1505] Processing CreateTablet for tablet 58db4e3bf468475c9e2f7f28931b126c (DEFAULT_TABLE table=default.Table_bb737a143cdb478d9a2b74b68bfacdeb [id=e8def477bd77468b81f72b05fc1eff48]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:05.208025 1041 tablet_service.cc:1505] Processing CreateTablet for tablet 5bcadc0b68e84ae290690edef392a45b (DEFAULT_TABLE table=default.Table_a53e7fa11e1f41af89d9421d2c107641 [id=f90f0e6c6567415b822145579e423d05]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:05.208112 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79b65cab235141f2916b10723c5a57cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:05.208554 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58db4e3bf468475c9e2f7f28931b126c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:05.208797 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5bcadc0b68e84ae290690edef392a45b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:05.209385 742 tablet_service.cc:1505] Processing CreateTablet for tablet 79b65cab235141f2916b10723c5a57cd (DEFAULT_TABLE table=default.Table_654ba01aef8c4aacb0f0aae654401c0a [id=67a7876d8e1a4b2b80123bc4162225ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:05.209738 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79b65cab235141f2916b10723c5a57cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:05.215261 756 tablet_service.cc:1505] Processing CreateTablet for tablet 58db4e3bf468475c9e2f7f28931b126c (DEFAULT_TABLE table=default.Table_bb737a143cdb478d9a2b74b68bfacdeb [id=e8def477bd77468b81f72b05fc1eff48]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:05.216390 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58db4e3bf468475c9e2f7f28931b126c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:05.219091 1037 tablet_service.cc:1505] Processing CreateTablet for tablet d8c66a892c90475c834f08c0734f41db (DEFAULT_TABLE table=default.Table_b8fe14e0ff054e67b0594e8b97483c71 [id=e27928a6cb8f4788b7e8c9c7a1c689d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:05.219475 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8c66a892c90475c834f08c0734f41db. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:05.225098 1039 tablet_service.cc:1505] Processing CreateTablet for tablet 417c8f5b2eb74088a44c7209a1fe986c (DEFAULT_TABLE table=default.Table_a7906004ed1742f39fec333e17ab26de [id=22512ce4e5e242e58682bbb86cf21723]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:05.225731 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 417c8f5b2eb74088a44c7209a1fe986c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:05.227412 747 tablet_service.cc:1505] Processing CreateTablet for tablet 5bcadc0b68e84ae290690edef392a45b (DEFAULT_TABLE table=default.Table_a53e7fa11e1f41af89d9421d2c107641 [id=f90f0e6c6567415b822145579e423d05]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:05.227753 1174 ts_tablet_manager.cc:1929] T 3cded156380f45cc95893825c71ba0e9 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:05.227809 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5bcadc0b68e84ae290690edef392a45b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:05.227880 1174 log.cc:1199] T 3cded156380f45cc95893825c71ba0e9 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/3cded156380f45cc95893825c71ba0e9
I20251029 02:52:05.228307 1174 ts_tablet_manager.cc:1950] T 3cded156380f45cc95893825c71ba0e9 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:05.229113 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 656) took 1572 ms. Trace:
I20251029 02:52:05.229293 1176 ts_tablet_manager.cc:1929] T 6feec2c8e030447fae9c292a626dcd1d P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:05.229197 1174 rpcz_store.cc:276] 1029 02:52:03.657001 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:03.657061 (+ 60us) service_pool.cc:225] Handling call
1029 02:52:05.197535 (+1540474us) ts_tablet_manager.cc:1151] Deleting tablet 3cded156380f45cc95893825c71ba0e9
1029 02:52:05.197544 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:05.221293 (+ 23749us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:05.227625 (+ 6332us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:05.229087 (+ 1462us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1497838}
I20251029 02:52:05.229581 1174 tablet_replica.cc:333] T fd0f89180aba43eabb20f926abccbaa6 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:05.229758 1174 raft_consensus.cc:2243] T fd0f89180aba43eabb20f926abccbaa6 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:05.230849 748 tablet_service.cc:1505] Processing CreateTablet for tablet 417c8f5b2eb74088a44c7209a1fe986c (DEFAULT_TABLE table=default.Table_a7906004ed1742f39fec333e17ab26de [id=22512ce4e5e242e58682bbb86cf21723]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:05.231185 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 417c8f5b2eb74088a44c7209a1fe986c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:05.232079 1025 tablet_service.cc:1505] Processing CreateTablet for tablet 1c24459f58a6414f94c62fbd48927c11 (DEFAULT_TABLE table=default.Table_494874895389492d9a687de4db8785b5 [id=1fa4959e78a743449e125c50669990f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:05.232468 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c24459f58a6414f94c62fbd48927c11. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:05.234822 755 tablet_service.cc:1505] Processing CreateTablet for tablet be23b3227a16415ab2fd4c6196dbcad4 (DEFAULT_TABLE table=default.Table_ac619c8d8e8a4b8bba9247c4b0b1c65e [id=cc75561297b24e76a34d665d9fe8735a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:05.238694 1174 pending_rounds.cc:70] T fd0f89180aba43eabb20f926abccbaa6 P c756528642a543eeacd93cef3e335a7d: Trying to abort 1 pending ops.
I20251029 02:52:05.238972 1174 pending_rounds.cc:77] T fd0f89180aba43eabb20f926abccbaa6 P c756528642a543eeacd93cef3e335a7d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215949099722649600 op_type: NO_OP noop_request { }
I20251029 02:52:05.239315 1174 raft_consensus.cc:2889] T fd0f89180aba43eabb20f926abccbaa6 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 02:52:05.239568 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be23b3227a16415ab2fd4c6196dbcad4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:05.239598 1174 raft_consensus.cc:2272] T fd0f89180aba43eabb20f926abccbaa6 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:05.239740 1903 raft_consensus.cc:493] T 24cb2a71ba3841d4a5e3e5c6fe99ae2f P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:05.239840 1903 raft_consensus.cc:515] T 24cb2a71ba3841d4a5e3e5c6fe99ae2f P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:05.240131 1174 ts_tablet_manager.cc:1916] T fd0f89180aba43eabb20f926abccbaa6 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:05.240361 1903 leader_election.cc:290] T 24cb2a71ba3841d4a5e3e5c6fe99ae2f P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:05.240996 1041 tablet_service.cc:1505] Processing CreateTablet for tablet be23b3227a16415ab2fd4c6196dbcad4 (DEFAULT_TABLE table=default.Table_ac619c8d8e8a4b8bba9247c4b0b1c65e [id=cc75561297b24e76a34d665d9fe8735a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:05.241350 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be23b3227a16415ab2fd4c6196dbcad4. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:05.241776 847 leader_election.cc:343] T 24cb2a71ba3841d4a5e3e5c6fe99ae2f P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 24cb2a71ba3841d4a5e3e5c6fe99ae2f
W20251029 02:52:05.242617 849 leader_election.cc:343] T 24cb2a71ba3841d4a5e3e5c6fe99ae2f P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Not found: Tablet not found: 24cb2a71ba3841d4a5e3e5c6fe99ae2f
I20251029 02:52:05.242713 849 leader_election.cc:304] T 24cb2a71ba3841d4a5e3e5c6fe99ae2f P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9
I20251029 02:52:05.243055 1903 raft_consensus.cc:2749] T 24cb2a71ba3841d4a5e3e5c6fe99ae2f P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:05.246964 1176 log.cc:1199] T 6feec2c8e030447fae9c292a626dcd1d P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/6feec2c8e030447fae9c292a626dcd1d
I20251029 02:52:05.247432 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 3cded156380f45cc95893825c71ba0e9 (table default.Table_6254452e8df24c21ab7d8ae32a06bdfb [id=46182aaaeecc4a4d9ec60cefb55ff50d]) successfully deleted
I20251029 02:52:05.247498 1176 ts_tablet_manager.cc:1950] T 6feec2c8e030447fae9c292a626dcd1d P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:05.248528 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 694) took 1596 ms. Trace:
I20251029 02:52:05.248646 879 tablet_service.cc:1552] Processing DeleteTablet for tablet 576c4e6e70e24b0399ac9597a30f0569 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:54 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:05.248692 1176 rpcz_store.cc:276] 1029 02:52:03.652224 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:03.652742 (+ 518us) service_pool.cc:225] Handling call
1029 02:52:05.181025 (+1528283us) ts_tablet_manager.cc:1151] Deleting tablet 6feec2c8e030447fae9c292a626dcd1d
1029 02:52:05.181037 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:05.216279 (+ 35242us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:05.229178 (+ 12899us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:05.248444 (+ 19266us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1528141}
I20251029 02:52:05.248929 895 tablet_service.cc:1505] Processing CreateTablet for tablet 79b65cab235141f2916b10723c5a57cd (DEFAULT_TABLE table=default.Table_654ba01aef8c4aacb0f0aae654401c0a [id=67a7876d8e1a4b2b80123bc4162225ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:05.249111 1176 tablet_replica.cc:333] T 6cd7009e06a646acad0802918c2b425e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:05.249315 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79b65cab235141f2916b10723c5a57cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:05.249372 1176 raft_consensus.cc:2243] T 6cd7009e06a646acad0802918c2b425e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:05.249558 898 tablet_service.cc:1505] Processing CreateTablet for tablet 1c24459f58a6414f94c62fbd48927c11 (DEFAULT_TABLE table=default.Table_494874895389492d9a687de4db8785b5 [id=1fa4959e78a743449e125c50669990f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:05.249835 1176 raft_consensus.cc:2272] T 6cd7009e06a646acad0802918c2b425e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:05.249917 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c24459f58a6414f94c62fbd48927c11. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:05.252350 884 tablet_service.cc:1552] Processing DeleteTablet for tablet 7e6c4f30be624769961c9ec17c5d5dcb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:55 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:05.252700 890 tablet_service.cc:1505] Processing CreateTablet for tablet 58db4e3bf468475c9e2f7f28931b126c (DEFAULT_TABLE table=default.Table_bb737a143cdb478d9a2b74b68bfacdeb [id=e8def477bd77468b81f72b05fc1eff48]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:05.252998 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 58db4e3bf468475c9e2f7f28931b126c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:05.253835 1176 ts_tablet_manager.cc:1916] T 6cd7009e06a646acad0802918c2b425e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:05.249259 885 tablet_service.cc:1505] Processing CreateTablet for tablet be23b3227a16415ab2fd4c6196dbcad4 (DEFAULT_TABLE table=default.Table_ac619c8d8e8a4b8bba9247c4b0b1c65e [id=cc75561297b24e76a34d665d9fe8735a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:05.256055 900 tablet_service.cc:1505] Processing CreateTablet for tablet 417c8f5b2eb74088a44c7209a1fe986c (DEFAULT_TABLE table=default.Table_a7906004ed1742f39fec333e17ab26de [id=22512ce4e5e242e58682bbb86cf21723]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:05.256321 881 tablet_service.cc:1552] Processing DeleteTablet for tablet c65469173762446e9725262ae998dbf3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:52 UTC) from {username='slave'} at 127.0.0.1:35702
W20251029 02:52:05.256434 989 consensus_peers.cc:597] T 51effc9324034ae79e2b25cb57a996c1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 -> Peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Couldn't send request to peer b495a49349324f7d81908f2da700f0d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 51effc9324034ae79e2b25cb57a996c1. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:05.248975 899 tablet_service.cc:1505] Processing CreateTablet for tablet 5bcadc0b68e84ae290690edef392a45b (DEFAULT_TABLE table=default.Table_a53e7fa11e1f41af89d9421d2c107641 [id=f90f0e6c6567415b822145579e423d05]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:05.258414 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be23b3227a16415ab2fd4c6196dbcad4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:05.258999 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5bcadc0b68e84ae290690edef392a45b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:05.261498 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 417c8f5b2eb74088a44c7209a1fe986c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:05.280027 1174 ts_tablet_manager.cc:1929] T fd0f89180aba43eabb20f926abccbaa6 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:05.280161 1174 log.cc:1199] T fd0f89180aba43eabb20f926abccbaa6 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/fd0f89180aba43eabb20f926abccbaa6
I20251029 02:52:05.280493 1174 ts_tablet_manager.cc:1950] T fd0f89180aba43eabb20f926abccbaa6 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:05.281279 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 664) took 1599 ms. Trace:
I20251029 02:52:05.281783 1176 ts_tablet_manager.cc:1929] T 6cd7009e06a646acad0802918c2b425e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:05.281383 1174 rpcz_store.cc:276] 1029 02:52:03.682098 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:03.684011 (+ 1913us) service_pool.cc:225] Handling call
1029 02:52:05.229533 (+1545522us) ts_tablet_manager.cc:1151] Deleting tablet fd0f89180aba43eabb20f926abccbaa6
1029 02:52:05.229544 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:05.270371 (+ 40827us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:05.279919 (+ 9548us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:05.281259 (+ 1340us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1529700}
I20251029 02:52:05.284619 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet fd0f89180aba43eabb20f926abccbaa6 (table default.Table_50252b3b3cc4403da5e259f9f91be915 [id=ff64a323de354792997beb99bae4a52c]) successfully deleted
I20251029 02:52:05.285189 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 668) took 1602 ms. Trace:
I20251029 02:52:05.285595 1174 rpcz_store.cc:276] 1029 02:52:03.682472 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:03.683798 (+ 1326us) service_pool.cc:225] Handling call
1029 02:52:05.285063 (+1601265us) ts_tablet_manager.cc:1151] Deleting tablet 70b008011d5547bbaf3f45685394139f
1029 02:52:05.285074 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:05.285174 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1576222}
W20251029 02:52:05.286458 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 70b008011d5547bbaf3f45685394139f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 70b008011d5547bbaf3f45685394139f already in progress: creating tablet
I20251029 02:52:05.286906 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 672) took 1579 ms. Trace:
I20251029 02:52:05.287639 1174 rpcz_store.cc:276] 1029 02:52:03.707524 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:03.707575 (+ 51us) service_pool.cc:225] Handling call
1029 02:52:05.286802 (+1579227us) ts_tablet_manager.cc:1151] Deleting tablet bc16262db49e4fc69fe1a4f95dbde2cb
1029 02:52:05.286812 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:05.286891 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1563339}
W20251029 02:52:05.288457 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet bc16262db49e4fc69fe1a4f95dbde2cb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bc16262db49e4fc69fe1a4f95dbde2cb already in progress: creating tablet
I20251029 02:52:05.289108 1174 tablet_replica.cc:333] T 44784a3a760946e0be24743bc2fc8138 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:05.289690 1174 raft_consensus.cc:2243] T 44784a3a760946e0be24743bc2fc8138 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:05.290455 1957 raft_consensus.cc:493] T c9bd3f2deb5d4af09b7918269a98509f P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:05.290542 1174 raft_consensus.cc:2272] T 44784a3a760946e0be24743bc2fc8138 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:05.290494 751 tablet_service.cc:1505] Processing CreateTablet for tablet d8c66a892c90475c834f08c0734f41db (DEFAULT_TABLE table=default.Table_b8fe14e0ff054e67b0594e8b97483c71 [id=e27928a6cb8f4788b7e8c9c7a1c689d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:05.290630 1957 raft_consensus.cc:515] T c9bd3f2deb5d4af09b7918269a98509f P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:05.290897 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8c66a892c90475c834f08c0734f41db. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:05.290908 1176 log.cc:1199] T 6cd7009e06a646acad0802918c2b425e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/6cd7009e06a646acad0802918c2b425e
I20251029 02:52:05.291178 1174 ts_tablet_manager.cc:1916] T 44784a3a760946e0be24743bc2fc8138 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:05.291219 1957 leader_election.cc:290] T c9bd3f2deb5d4af09b7918269a98509f P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:05.291326 1176 ts_tablet_manager.cc:1950] T 6cd7009e06a646acad0802918c2b425e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:05.291924 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9bd3f2deb5d4af09b7918269a98509f" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:05.292104 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 693) took 1639 ms. Trace:
I20251029 02:52:05.292212 1176 rpcz_store.cc:276] 1029 02:52:03.652141 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:03.652910 (+ 769us) service_pool.cc:225] Handling call
1029 02:52:05.249017 (+1596107us) ts_tablet_manager.cc:1151] Deleting tablet 6cd7009e06a646acad0802918c2b425e
1029 02:52:05.249031 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:05.257851 (+ 8820us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:05.281596 (+ 23745us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:05.292065 (+ 10469us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1595967}
W20251029 02:52:05.292858 711 leader_election.cc:343] T c9bd3f2deb5d4af09b7918269a98509f P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:05.292706 1058 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9bd3f2deb5d4af09b7918269a98509f" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:52:05.294401 712 leader_election.cc:343] T c9bd3f2deb5d4af09b7918269a98509f P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:05.294553 712 leader_election.cc:304] T c9bd3f2deb5d4af09b7918269a98509f P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:05.294973 1032 tablet_service.cc:1505] Processing CreateTablet for tablet 657cf290d68c45e98f0ffd05c7d65ad7 (DEFAULT_TABLE table=default.Table_4fbd78edfb5a451f91ddcdf41de962a3 [id=1685db596a8a493c9cdf209e22da22ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:05.295070 1957 raft_consensus.cc:2749] T c9bd3f2deb5d4af09b7918269a98509f P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:05.295526 1176 tablet_replica.cc:333] T 8afc01d52d784f2fa559cab7ce07c8be P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:05.297273 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 6feec2c8e030447fae9c292a626dcd1d (table default.Table_cb3d27722cc74fa18ddc8740590639a9 [id=37499c7664774cb9bd0a80c7c5bef158]) successfully deleted
I20251029 02:52:05.295807 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 657cf290d68c45e98f0ffd05c7d65ad7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:05.297719 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 6cd7009e06a646acad0802918c2b425e (table default.Table_198644e9ee1f45ecb6ad95a759b54607 [id=7810f54ddc4849a8b13bb2928eb761a7]) successfully deleted
I20251029 02:52:05.303058 1176 raft_consensus.cc:2243] T 8afc01d52d784f2fa559cab7ce07c8be P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:05.303928 1176 pending_rounds.cc:70] T 8afc01d52d784f2fa559cab7ce07c8be P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Trying to abort 1 pending ops.
I20251029 02:52:05.305506 1174 ts_tablet_manager.cc:1929] T 44784a3a760946e0be24743bc2fc8138 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:05.306098 1174 log.cc:1199] T 44784a3a760946e0be24743bc2fc8138 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/44784a3a760946e0be24743bc2fc8138
I20251029 02:52:05.306586 1174 ts_tablet_manager.cc:1950] T 44784a3a760946e0be24743bc2fc8138 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:05.307461 1167 tablet_bootstrap.cc:492] T f81aecb20d6e498bb9b498698886d65e P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:05.307605 1167 ts_tablet_manager.cc:1403] T f81aecb20d6e498bb9b498698886d65e P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.206s user 0.018s sys 0.031s
I20251029 02:52:05.308367 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 663) took 1626 ms. Trace:
I20251029 02:52:05.308444 1167 raft_consensus.cc:359] T f81aecb20d6e498bb9b498698886d65e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:05.308686 1176 pending_rounds.cc:77] T 8afc01d52d784f2fa559cab7ce07c8be P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215949102598299648 op_type: NO_OP noop_request { }
I20251029 02:52:05.308808 1176 raft_consensus.cc:2889] T 8afc01d52d784f2fa559cab7ce07c8be P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 02:52:05.308902 1167 raft_consensus.cc:385] T f81aecb20d6e498bb9b498698886d65e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:05.308974 1167 raft_consensus.cc:740] T f81aecb20d6e498bb9b498698886d65e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:05.308944 1176 raft_consensus.cc:2272] T 8afc01d52d784f2fa559cab7ce07c8be P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:05.309187 1167 consensus_queue.cc:260] T f81aecb20d6e498bb9b498698886d65e P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:05.309818 1167 ts_tablet_manager.cc:1434] T f81aecb20d6e498bb9b498698886d65e P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 02:52:05.310153 1167 tablet_bootstrap.cc:492] T cedfb53078d94c81b5e294ce7438325e P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:05.311280 1176 ts_tablet_manager.cc:1916] T 8afc01d52d784f2fa559cab7ce07c8be P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:05.311784 1167 tablet_bootstrap.cc:654] T cedfb53078d94c81b5e294ce7438325e P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:05.315135 1174 rpcz_store.cc:276] 1029 02:52:03.681994 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:03.687358 (+ 5364us) service_pool.cc:225] Handling call
1029 02:52:05.289060 (+1601702us) ts_tablet_manager.cc:1151] Deleting tablet 44784a3a760946e0be24743bc2fc8138
1029 02:52:05.289069 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:05.298334 (+ 9265us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:05.305411 (+ 7077us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:05.308343 (+ 2932us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1556855}
I20251029 02:52:05.316089 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 44784a3a760946e0be24743bc2fc8138 (table default.Table_3e28d37b4b6146bf95edf245d3f9d0cf [id=915b76276e0940618b7180ebc06cd464]) successfully deleted
I20251029 02:52:05.316658 1174 tablet_replica.cc:333] T 3362e95063af41c8a24e935f547e3089 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:05.317518 1174 raft_consensus.cc:2243] T 3362e95063af41c8a24e935f547e3089 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:05.317842 1174 raft_consensus.cc:2272] T 3362e95063af41c8a24e935f547e3089 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:05.321286 748 tablet_service.cc:1505] Processing CreateTablet for tablet 050186302c5d4c7da393f28f4b9e4262 (DEFAULT_TABLE table=default.Table_224e79c0036b470c863299755de8e9e4 [id=8c3b50bd1e434dfebe679141ceb18be8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:05.321791 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 050186302c5d4c7da393f28f4b9e4262. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:05.322770 750 tablet_service.cc:1505] Processing CreateTablet for tablet 1c24459f58a6414f94c62fbd48927c11 (DEFAULT_TABLE table=default.Table_494874895389492d9a687de4db8785b5 [id=1fa4959e78a743449e125c50669990f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:05.323110 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c24459f58a6414f94c62fbd48927c11. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:05.325740 900 tablet_service.cc:1505] Processing CreateTablet for tablet f72dae5a0e604eb384c1f5f06fe680bb (DEFAULT_TABLE table=default.Table_a587f8486e2a4542a9e7cf80c0306f23 [id=85e7a33dd3b34e82803e389b3cbbaca2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:05.326143 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f72dae5a0e604eb384c1f5f06fe680bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:05.325771 890 tablet_service.cc:1505] Processing CreateTablet for tablet e5c88e03d9324a918020688c884eb4f6 (DEFAULT_TABLE table=default.Table_f26d0b6d64304372a0bc5120b663991e [id=f0084bbedcfc4e17a7b3e5fb938b37da]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:05.327176 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5c88e03d9324a918020688c884eb4f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:05.327356 1025 tablet_service.cc:1505] Processing CreateTablet for tablet f72dae5a0e604eb384c1f5f06fe680bb (DEFAULT_TABLE table=default.Table_a587f8486e2a4542a9e7cf80c0306f23 [id=85e7a33dd3b34e82803e389b3cbbaca2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:05.327706 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f72dae5a0e604eb384c1f5f06fe680bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:05.329198 1028 tablet_service.cc:1505] Processing CreateTablet for tablet 050186302c5d4c7da393f28f4b9e4262 (DEFAULT_TABLE table=default.Table_224e79c0036b470c863299755de8e9e4 [id=8c3b50bd1e434dfebe679141ceb18be8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:05.329572 1903 raft_consensus.cc:493] T 7e6c4f30be624769961c9ec17c5d5dcb P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:05.329653 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 050186302c5d4c7da393f28f4b9e4262. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:05.329697 1903 raft_consensus.cc:515] T 7e6c4f30be624769961c9ec17c5d5dcb P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:05.330253 1903 leader_election.cc:290] T 7e6c4f30be624769961c9ec17c5d5dcb P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:05.331270 1058 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7e6c4f30be624769961c9ec17c5d5dcb" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:05.331444 1058 raft_consensus.cc:2468] T 7e6c4f30be624769961c9ec17c5d5dcb P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:05.331820 849 leader_election.cc:304] T 7e6c4f30be624769961c9ec17c5d5dcb P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters:
W20251029 02:52:05.332206 847 leader_election.cc:343] T 7e6c4f30be624769961c9ec17c5d5dcb P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 7e6c4f30be624769961c9ec17c5d5dcb
I20251029 02:52:05.332753 1904 raft_consensus.cc:2804] T 7e6c4f30be624769961c9ec17c5d5dcb P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:05.332818 1904 raft_consensus.cc:493] T 7e6c4f30be624769961c9ec17c5d5dcb P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:05.332870 1904 raft_consensus.cc:3060] T 7e6c4f30be624769961c9ec17c5d5dcb P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:05.334944 1174 ts_tablet_manager.cc:1916] T 3362e95063af41c8a24e935f547e3089 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:05.334946 1172 ts_tablet_manager.cc:1434] T 1bc45078a6a942c38ff3350c15885eff P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.182s user 0.001s sys 0.000s
I20251029 02:52:05.335264 1172 tablet_bootstrap.cc:492] T 1bdb67125493430398ff5c80b73de2ac P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:05.335400 1904 raft_consensus.cc:515] T 7e6c4f30be624769961c9ec17c5d5dcb P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:05.335901 1904 leader_election.cc:290] T 7e6c4f30be624769961c9ec17c5d5dcb P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:05.336961 1172 tablet_bootstrap.cc:654] T 1bdb67125493430398ff5c80b73de2ac P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:05.337958 1058 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7e6c4f30be624769961c9ec17c5d5dcb" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:52:05.338106 1058 raft_consensus.cc:3060] T 7e6c4f30be624769961c9ec17c5d5dcb P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:05.340552 1175 ts_tablet_manager.cc:1916] T bfec87f690a640c39c58c64e16cc67e7 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 02:52:05.342111 847 leader_election.cc:343] T 7e6c4f30be624769961c9ec17c5d5dcb P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 7e6c4f30be624769961c9ec17c5d5dcb
I20251029 02:52:05.342918 1176 ts_tablet_manager.cc:1929] T 8afc01d52d784f2fa559cab7ce07c8be P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20251029 02:52:05.343194 849 consensus_peers.cc:597] T 7309d3190cfa415281c5253a634fa7f3 P c756528642a543eeacd93cef3e335a7d -> Peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Couldn't send request to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7309d3190cfa415281c5253a634fa7f3. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:05.347776 756 tablet_service.cc:1505] Processing CreateTablet for tablet 415ff17c323e4ed88da125d823ad4af1 (DEFAULT_TABLE table=default.Table_4916f7d510ec49be9480cc88f4c7d3af [id=d10ee6be0b5c4f4c972a4bb04ad65755]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:05.348721 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 415ff17c323e4ed88da125d823ad4af1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:05.345688 1058 raft_consensus.cc:2468] T 7e6c4f30be624769961c9ec17c5d5dcb P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:05.349822 1176 log.cc:1199] T 8afc01d52d784f2fa559cab7ce07c8be P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/8afc01d52d784f2fa559cab7ce07c8be
I20251029 02:52:05.348269 749 tablet_service.cc:1505] Processing CreateTablet for tablet f72dae5a0e604eb384c1f5f06fe680bb (DEFAULT_TABLE table=default.Table_a587f8486e2a4542a9e7cf80c0306f23 [id=85e7a33dd3b34e82803e389b3cbbaca2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:05.350152 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f72dae5a0e604eb384c1f5f06fe680bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:05.350207 1176 ts_tablet_manager.cc:1950] T 8afc01d52d784f2fa559cab7ce07c8be P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:05.350260 849 leader_election.cc:304] T 7e6c4f30be624769961c9ec17c5d5dcb P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters: b495a49349324f7d81908f2da700f0d9
I20251029 02:52:05.350721 1903 raft_consensus.cc:2804] T 7e6c4f30be624769961c9ec17c5d5dcb P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:05.350847 1903 raft_consensus.cc:697] T 7e6c4f30be624769961c9ec17c5d5dcb P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:52:05.351060 1903 consensus_queue.cc:237] T 7e6c4f30be624769961c9ec17c5d5dcb P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:05.354749 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 692) took 1702 ms. Trace:
I20251029 02:52:05.355077 895 tablet_service.cc:1505] Processing CreateTablet for tablet 415ff17c323e4ed88da125d823ad4af1 (DEFAULT_TABLE table=default.Table_4916f7d510ec49be9480cc88f4c7d3af [id=d10ee6be0b5c4f4c972a4bb04ad65755]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:05.356137 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 415ff17c323e4ed88da125d823ad4af1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:05.358554 1176 rpcz_store.cc:276] 1029 02:52:03.652034 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:03.653068 (+ 1034us) service_pool.cc:225] Handling call
1029 02:52:05.295460 (+1642392us) ts_tablet_manager.cc:1151] Deleting tablet 8afc01d52d784f2fa559cab7ce07c8be
1029 02:52:05.295470 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:05.323840 (+ 28370us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:05.342812 (+ 18972us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:05.354721 (+ 11909us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1642260}
I20251029 02:52:05.359014 1176 tablet_replica.cc:333] T 1bab90fe904944af948dab3db06b7e34 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:05.359165 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 8afc01d52d784f2fa559cab7ce07c8be (table default.Table_2a10c9dfc91d4974845dbcd500d1d8be [id=ed7295f3f21848b79af75785d3983b67]) successfully deleted
I20251029 02:52:05.359239 1176 raft_consensus.cc:2243] T 1bab90fe904944af948dab3db06b7e34 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:05.355485 892 tablet_service.cc:1505] Processing CreateTablet for tablet 050186302c5d4c7da393f28f4b9e4262 (DEFAULT_TABLE table=default.Table_224e79c0036b470c863299755de8e9e4 [id=8c3b50bd1e434dfebe679141ceb18be8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:05.359400 1176 raft_consensus.cc:2272] T 1bab90fe904944af948dab3db06b7e34 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:05.360150 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 050186302c5d4c7da393f28f4b9e4262. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:05.361277 1174 ts_tablet_manager.cc:1929] T 3362e95063af41c8a24e935f547e3089 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:05.361953 1176 ts_tablet_manager.cc:1916] T 1bab90fe904944af948dab3db06b7e34 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:05.362567 1906 consensus_queue.cc:1048] T 51effc9324034ae79e2b25cb57a996c1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:05.366588 1174 log.cc:1199] T 3362e95063af41c8a24e935f547e3089 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/3362e95063af41c8a24e935f547e3089
I20251029 02:52:05.365072 1037 tablet_service.cc:1505] Processing CreateTablet for tablet 415ff17c323e4ed88da125d823ad4af1 (DEFAULT_TABLE table=default.Table_4916f7d510ec49be9480cc88f4c7d3af [id=d10ee6be0b5c4f4c972a4bb04ad65755]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:05.369261 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 415ff17c323e4ed88da125d823ad4af1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:05.370842 1174 ts_tablet_manager.cc:1950] T 3362e95063af41c8a24e935f547e3089 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:05.371616 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 671) took 1671 ms. Trace:
I20251029 02:52:05.376554 1174 rpcz_store.cc:276] 1029 02:52:03.700142 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:03.701190 (+ 1048us) service_pool.cc:225] Handling call
1029 02:52:05.316570 (+1615380us) ts_tablet_manager.cc:1151] Deleting tablet 3362e95063af41c8a24e935f547e3089
1029 02:52:05.316613 (+ 43us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:05.350912 (+ 34299us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:05.361191 (+ 10279us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:05.371598 (+ 10407us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1583794}
I20251029 02:52:05.377087 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 3362e95063af41c8a24e935f547e3089 (table default.Table_2a63af6894694f16b45f4bf48f73a68f [id=fb42af08ec1f48498ee5e91cce62333e]) successfully deleted
I20251029 02:52:05.377449 1174 tablet_replica.cc:333] T a5c3a15510cb4321bbb80d61fabd0c0a P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:05.377676 1174 raft_consensus.cc:2243] T a5c3a15510cb4321bbb80d61fabd0c0a P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:05.377902 1174 raft_consensus.cc:2272] T a5c3a15510cb4321bbb80d61fabd0c0a P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:05.378378 1174 ts_tablet_manager.cc:1916] T a5c3a15510cb4321bbb80d61fabd0c0a P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:05.379292 524 catalog_manager.cc:5649] T 42cfdf88472b44979d86836561005ed1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } health_report { overall_health: UNKNOWN } } }
I20251029 02:52:05.379647 1025 tablet_service.cc:1505] Processing CreateTablet for tablet 266aaea633bf4c059b9c02f092a474be (DEFAULT_TABLE table=default.Table_ab2b0e5b127542b6ab30655507c3e682 [id=0d485ab0bcc7444181770971b6d44302]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:05.380100 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 266aaea633bf4c059b9c02f092a474be. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:05.381911 750 tablet_service.cc:1552] Processing DeleteTablet for tablet b3de6670357c4ae59eb77469b2fb434c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:01 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:05.382220 524 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet 42cfdf88472b44979d86836561005ed1 (table default.Table_ad2ac3ba98784d68a94eb4eb03e530a4 [id=5e0c72957b9a491d960b33d3fffd8f70]). Expected version 2 got 0
I20251029 02:52:05.382897 524 catalog_manager.cc:5649] T c9dff4dff82f49e7ad49caad5e404f09 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } health_report { overall_health: UNKNOWN } } }
I20251029 02:52:05.383425 524 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet c9e923724ca9402682f13b252dbc37aa (table default.Table_443f33587b8f428484b99e07355d69ca [id=03a25420ce6a46fc93a9fb753246bc7b]). Expected version 3 got 0
I20251029 02:52:05.381960 523 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_8b6217e43e8d467693c104ef84e78911" } modify_external_catalogs: true
I20251029 02:52:05.384009 523 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_8b6217e43e8d467693c104ef84e78911" } modify_external_catalogs: true
I20251029 02:52:05.383773 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_ff037b99b812492d9f064ca5485e9162"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:05.384795 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ff037b99b812492d9f064ca5485e9162 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:05.385088 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_ba4b224d24404916b18cb61a05d08b75"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:05.379050 1167 tablet_bootstrap.cc:492] T cedfb53078d94c81b5e294ce7438325e P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:05.385576 1167 ts_tablet_manager.cc:1403] T cedfb53078d94c81b5e294ce7438325e P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.075s user 0.005s sys 0.000s
W20251029 02:52:05.385646 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ba4b224d24404916b18cb61a05d08b75 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:05.386246 524 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet cedfb53078d94c81b5e294ce7438325e (table default.Table_e75db5cf8f5f49ea837cb6d63f4b7ebf [id=86195563fb4a4d7abd746f4ac7cd241c]). Expected version 1 got 0
I20251029 02:52:05.381547 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_ddd7ee9b37df4bb4be7957aef7e82cb8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:05.387457 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ddd7ee9b37df4bb4be7957aef7e82cb8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:05.391866 1176 ts_tablet_manager.cc:1929] T 1bab90fe904944af948dab3db06b7e34 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:05.392023 1176 log.cc:1199] T 1bab90fe904944af948dab3db06b7e34 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/1bab90fe904944af948dab3db06b7e34
I20251029 02:52:05.386796 1167 raft_consensus.cc:359] T cedfb53078d94c81b5e294ce7438325e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:05.394089 1167 raft_consensus.cc:385] T cedfb53078d94c81b5e294ce7438325e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:05.396354 1167 raft_consensus.cc:740] T cedfb53078d94c81b5e294ce7438325e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:05.397054 750 tablet_service.cc:1505] Processing CreateTablet for tablet 266aaea633bf4c059b9c02f092a474be (DEFAULT_TABLE table=default.Table_ab2b0e5b127542b6ab30655507c3e682 [id=0d485ab0bcc7444181770971b6d44302]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:05.397409 1167 consensus_queue.cc:260] T cedfb53078d94c81b5e294ce7438325e P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:05.397563 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 266aaea633bf4c059b9c02f092a474be. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:05.398571 1903 raft_consensus.cc:493] T 42cfdf88472b44979d86836561005ed1 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:05.398952 1176 ts_tablet_manager.cc:1950] T 1bab90fe904944af948dab3db06b7e34 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:05.398881 1903 raft_consensus.cc:515] T 42cfdf88472b44979d86836561005ed1 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:05.399019 751 tablet_service.cc:1505] Processing CreateTablet for tablet e5c88e03d9324a918020688c884eb4f6 (DEFAULT_TABLE table=default.Table_f26d0b6d64304372a0bc5120b663991e [id=f0084bbedcfc4e17a7b3e5fb938b37da]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:05.399338 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5c88e03d9324a918020688c884eb4f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:05.399638 1903 leader_election.cc:290] T 42cfdf88472b44979d86836561005ed1 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:05.399645 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 685) took 1749 ms. Trace:
I20251029 02:52:05.399713 1176 rpcz_store.cc:276] 1029 02:52:03.650136 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:03.653375 (+ 3239us) service_pool.cc:225] Handling call
1029 02:52:05.358965 (+1705590us) ts_tablet_manager.cc:1151] Deleting tablet 1bab90fe904944af948dab3db06b7e34
1029 02:52:05.358977 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:05.368932 (+ 9955us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:05.391765 (+ 22833us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:05.399626 (+ 7861us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1705420}
I20251029 02:52:05.400187 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 1bab90fe904944af948dab3db06b7e34 (table default.Table_c70eb9f284cd49109fd09e430a1cea6c [id=be817eb6ec33406eb316d36575c134fb]) successfully deleted
I20251029 02:52:05.400454 1058 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42cfdf88472b44979d86836561005ed1" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:05.401439 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "42cfdf88472b44979d86836561005ed1" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:05.401692 768 raft_consensus.cc:2393] T 42cfdf88472b44979d86836561005ed1 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c756528642a543eeacd93cef3e335a7d in current term 1: Already voted for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in this term.
I20251029 02:52:05.402168 847 leader_election.cc:304] T 42cfdf88472b44979d86836561005ed1 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9
I20251029 02:52:05.402689 1903 raft_consensus.cc:3060] T 42cfdf88472b44979d86836561005ed1 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:05.394806 1041 tablet_service.cc:1505] Processing CreateTablet for tablet e5c88e03d9324a918020688c884eb4f6 (DEFAULT_TABLE table=default.Table_f26d0b6d64304372a0bc5120b663991e [id=f0084bbedcfc4e17a7b3e5fb938b37da]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:05.403612 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5c88e03d9324a918020688c884eb4f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:05.406026 1903 raft_consensus.cc:2749] T 42cfdf88472b44979d86836561005ed1 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:05.396277 1166 tablet_bootstrap.cc:492] T f81aecb20d6e498bb9b498698886d65e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:05.396662 1039 tablet_service.cc:1505] Processing CreateTablet for tablet 7bcb1cb3cb0949edbfd96cd7f6b11f1c (DEFAULT_TABLE table=default.Table_8746e5337ced40038d965fae5bcb4acd [id=1c6437b0862340eb8b38b2e0259d4dd4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:05.406373 1166 ts_tablet_manager.cc:1403] T f81aecb20d6e498bb9b498698886d65e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.249s user 0.022s sys 0.021s
I20251029 02:52:05.406708 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7bcb1cb3cb0949edbfd96cd7f6b11f1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:05.406828 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 697) took 1707 ms. Trace:
I20251029 02:52:05.407127 1166 raft_consensus.cc:359] T f81aecb20d6e498bb9b498698886d65e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:05.407397 1166 raft_consensus.cc:385] T f81aecb20d6e498bb9b498698886d65e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:05.407450 1166 raft_consensus.cc:740] T f81aecb20d6e498bb9b498698886d65e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:05.407718 1166 consensus_queue.cc:260] T f81aecb20d6e498bb9b498698886d65e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:05.408596 1174 ts_tablet_manager.cc:1929] T a5c3a15510cb4321bbb80d61fabd0c0a P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:05.409469 1166 ts_tablet_manager.cc:1434] T f81aecb20d6e498bb9b498698886d65e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20251029 02:52:05.409650 1174 log.cc:1199] T a5c3a15510cb4321bbb80d61fabd0c0a P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/a5c3a15510cb4321bbb80d61fabd0c0a
I20251029 02:52:05.409379 900 tablet_service.cc:1505] Processing CreateTablet for tablet 266aaea633bf4c059b9c02f092a474be (DEFAULT_TABLE table=default.Table_ab2b0e5b127542b6ab30655507c3e682 [id=0d485ab0bcc7444181770971b6d44302]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:05.410034 1174 ts_tablet_manager.cc:1950] T a5c3a15510cb4321bbb80d61fabd0c0a P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:05.410640 1166 tablet_bootstrap.cc:492] T 6bd3fac2502e479aa9f4ceaabbb39504 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:05.411448 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 661) took 1730 ms. Trace:
I20251029 02:52:05.411849 526 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 4a55f22b0d3b4ebe98ec2e054d05b095
I20251029 02:52:05.412103 1166 tablet_bootstrap.cc:654] T 6bd3fac2502e479aa9f4ceaabbb39504 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:05.406970 1176 rpcz_store.cc:276] 1029 02:52:03.699611 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:03.703252 (+ 3641us) service_pool.cc:225] Handling call
1029 02:52:05.406624 (+1703372us) ts_tablet_manager.cc:1151] Deleting tablet bc16262db49e4fc69fe1a4f95dbde2cb
1029 02:52:05.406634 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:05.406810 (+ 176us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1703002}
I20251029 02:52:05.413196 521 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 6ecc552c1ef643f8bb84622d9d440ba2
I20251029 02:52:05.413414 1176 tablet_replica.cc:333] T dbc2092b63e0417997565c8fadb53ae0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:05.413635 521 catalog_manager.cc:6719] ReplaceTablet: tablet 6ecc552c1ef643f8bb84622d9d440ba2 deleted and replaced by tablet 44c2ef5426704e03bd50775adcdc571b
I20251029 02:52:05.413678 1176 raft_consensus.cc:2243] T dbc2092b63e0417997565c8fadb53ae0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:05.413818 1176 raft_consensus.cc:2272] T dbc2092b63e0417997565c8fadb53ae0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:05.414417 1176 ts_tablet_manager.cc:1916] T dbc2092b63e0417997565c8fadb53ae0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:05.414577 755 tablet_service.cc:1552] Processing DeleteTablet for tablet 4a55f22b0d3b4ebe98ec2e054d05b095 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:05 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:05.414878 742 tablet_service.cc:1552] Processing DeleteTablet for tablet 6ecc552c1ef643f8bb84622d9d440ba2 with delete_type TABLET_DATA_DELETED (replaced by tablet 44c2ef5426704e03bd50775adcdc571b) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:05.409027 890 tablet_service.cc:1505] Processing CreateTablet for tablet 7bcb1cb3cb0949edbfd96cd7f6b11f1c (DEFAULT_TABLE table=default.Table_8746e5337ced40038d965fae5bcb4acd [id=1c6437b0862340eb8b38b2e0259d4dd4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:05.410079 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 266aaea633bf4c059b9c02f092a474be. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:05.415014 885 tablet_service.cc:1552] Processing DeleteTablet for tablet 6ecc552c1ef643f8bb84622d9d440ba2 with delete_type TABLET_DATA_DELETED (replaced by tablet 44c2ef5426704e03bd50775adcdc571b) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:05.414861 899 tablet_service.cc:1552] Processing DeleteTablet for tablet 4a55f22b0d3b4ebe98ec2e054d05b095 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:05 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:05.415434 1032 tablet_service.cc:1552] Processing DeleteTablet for tablet 4a55f22b0d3b4ebe98ec2e054d05b095 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:05 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:05.415674 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7bcb1cb3cb0949edbfd96cd7f6b11f1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:05.416389 1032 tablet_service.cc:1552] Processing DeleteTablet for tablet 6ecc552c1ef643f8bb84622d9d440ba2 with delete_type TABLET_DATA_DELETED (replaced by tablet 44c2ef5426704e03bd50775adcdc571b) from {username='slave'} at 127.0.0.1:37626
W20251029 02:52:05.416443 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet bc16262db49e4fc69fe1a4f95dbde2cb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bc16262db49e4fc69fe1a4f95dbde2cb already in progress: creating tablet
I20251029 02:52:05.419629 1167 ts_tablet_manager.cc:1434] T cedfb53078d94c81b5e294ce7438325e P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.034s user 0.002s sys 0.000s
I20251029 02:52:05.419965 1167 tablet_bootstrap.cc:492] T b2b2d64dcf964da8be4b544dbbe31977 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:05.420748 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_65e6cdfafc9f46e8ac0c16c6babfc7d8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:05.421339 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_65e6cdfafc9f46e8ac0c16c6babfc7d8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:05.421357 1175 ts_tablet_manager.cc:1929] T bfec87f690a640c39c58c64e16cc67e7 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:05.421445 1175 log.cc:1199] T bfec87f690a640c39c58c64e16cc67e7 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/bfec87f690a640c39c58c64e16cc67e7
I20251029 02:52:05.421522 1167 tablet_bootstrap.cc:654] T b2b2d64dcf964da8be4b544dbbe31977 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:05.414963 1174 rpcz_store.cc:276] 1029 02:52:03.680831 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:03.680954 (+ 123us) service_pool.cc:225] Handling call
1029 02:52:05.377395 (+1696441us) ts_tablet_manager.cc:1151] Deleting tablet a5c3a15510cb4321bbb80d61fabd0c0a
1029 02:52:05.377405 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:05.386156 (+ 8751us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:05.408468 (+ 22312us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:05.411428 (+ 2960us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1639016}
I20251029 02:52:05.421746 1175 ts_tablet_manager.cc:1950] T bfec87f690a640c39c58c64e16cc67e7 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:05.422117 742 tablet_service.cc:1552] Processing DeleteTablet for tablet 1bc45078a6a942c38ff3350c15885eff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:56 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:05.423311 521 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_35d17629e08e4b41bf6621b43343ed6b" } new_table_name: "default.Table_95133cb1a69941d0b8a6321091752822" modify_external_catalogs: true
I20251029 02:52:05.424851 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet a5c3a15510cb4321bbb80d61fabd0c0a (table default.Table_8cf076425649436289d327edc92d4642 [id=c124cfa700724f6db0f1a12fe495c2af]) successfully deleted
I20251029 02:52:05.425061 1174 tablet_replica.cc:333] T 1bab90fe904944af948dab3db06b7e34 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:05.425223 1174 raft_consensus.cc:2243] T 1bab90fe904944af948dab3db06b7e34 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:05.425558 1174 raft_consensus.cc:2272] T 1bab90fe904944af948dab3db06b7e34 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:05.425767 742 tablet_service.cc:1505] Processing CreateTablet for tablet 7bcb1cb3cb0949edbfd96cd7f6b11f1c (DEFAULT_TABLE table=default.Table_8746e5337ced40038d965fae5bcb4acd [id=1c6437b0862340eb8b38b2e0259d4dd4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:05.426198 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7bcb1cb3cb0949edbfd96cd7f6b11f1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:05.426826 1174 ts_tablet_manager.cc:1916] T 1bab90fe904944af948dab3db06b7e34 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:05.427282 522 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet ea7b977224f24bf49dadfb83ffa28310
I20251029 02:52:05.428079 525 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 81f936792aae4b699b3122457d7b631d
I20251029 02:52:05.429525 755 tablet_service.cc:1552] Processing DeleteTablet for tablet ea7b977224f24bf49dadfb83ffa28310 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:04 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:05.429805 747 tablet_service.cc:1552] Processing DeleteTablet for tablet 81f936792aae4b699b3122457d7b631d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:05 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:05.429975 892 tablet_service.cc:1552] Processing DeleteTablet for tablet 81f936792aae4b699b3122457d7b631d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:05 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:05.430025 895 tablet_service.cc:1552] Processing DeleteTablet for tablet ea7b977224f24bf49dadfb83ffa28310 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:04 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:05.430723 1167 tablet_bootstrap.cc:492] T b2b2d64dcf964da8be4b544dbbe31977 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:05.430879 1167 ts_tablet_manager.cc:1403] T b2b2d64dcf964da8be4b544dbbe31977 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20251029 02:52:05.431829 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_111cfdff9be446e09a352fd6f3265ec5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:05.432083 1025 tablet_service.cc:1552] Processing DeleteTablet for tablet 81f936792aae4b699b3122457d7b631d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:05 UTC) from {username='slave'} at 127.0.0.1:37626
W20251029 02:52:05.432389 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_111cfdff9be446e09a352fd6f3265ec5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:05.432442 1167 raft_consensus.cc:359] T b2b2d64dcf964da8be4b544dbbe31977 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:05.432591 1041 tablet_service.cc:1552] Processing DeleteTablet for tablet ea7b977224f24bf49dadfb83ffa28310 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:04 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:05.433763 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_3266aac3f5744a23954b5465d9af1b31"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:05.434082 1167 raft_consensus.cc:385] T b2b2d64dcf964da8be4b544dbbe31977 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251029 02:52:05.434250 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3266aac3f5744a23954b5465d9af1b31 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:05.434311 1167 raft_consensus.cc:740] T b2b2d64dcf964da8be4b544dbbe31977 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:05.434856 1167 consensus_queue.cc:260] T b2b2d64dcf964da8be4b544dbbe31977 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:05.436154 519 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 46cd75ff18b14212b556b9bcb3012dd9
I20251029 02:52:05.437162 1167 ts_tablet_manager.cc:1434] T b2b2d64dcf964da8be4b544dbbe31977 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20251029 02:52:05.437628 742 tablet_service.cc:1552] Processing DeleteTablet for tablet 46cd75ff18b14212b556b9bcb3012dd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:05 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:05.437642 1041 tablet_service.cc:1552] Processing DeleteTablet for tablet 46cd75ff18b14212b556b9bcb3012dd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:05 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:05.438270 895 tablet_service.cc:1552] Processing DeleteTablet for tablet 46cd75ff18b14212b556b9bcb3012dd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:05 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:05.440536 1176 ts_tablet_manager.cc:1929] T dbc2092b63e0417997565c8fadb53ae0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:05.440732 1167 tablet_bootstrap.cc:492] T 8df97119d0654d7d92bb5bd17c5d8562 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:05.441949 1174 ts_tablet_manager.cc:1929] T 1bab90fe904944af948dab3db06b7e34 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:05.444684 742 tablet_service.cc:1505] Processing CreateTablet for tablet 76648db142894fd9972b38b5ee485fed (DEFAULT_TABLE table=default.Table_714c5abdd40047d1bf204ee53c29a06e [id=4a8aee3131e34716b8bfed53f169b062]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:05.445132 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76648db142894fd9972b38b5ee485fed. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:05.443672 1166 tablet_bootstrap.cc:492] T 6bd3fac2502e479aa9f4ceaabbb39504 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:05.445386 1166 ts_tablet_manager.cc:1403] T 6bd3fac2502e479aa9f4ceaabbb39504 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.002s
I20251029 02:52:05.445106 1041 tablet_service.cc:1505] Processing CreateTablet for tablet 76648db142894fd9972b38b5ee485fed (DEFAULT_TABLE table=default.Table_714c5abdd40047d1bf204ee53c29a06e [id=4a8aee3131e34716b8bfed53f169b062]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:05.446929 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76648db142894fd9972b38b5ee485fed. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:05.446934 1166 raft_consensus.cc:359] T 6bd3fac2502e479aa9f4ceaabbb39504 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:05.447127 1166 raft_consensus.cc:385] T 6bd3fac2502e479aa9f4ceaabbb39504 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:05.447180 1166 raft_consensus.cc:740] T 6bd3fac2502e479aa9f4ceaabbb39504 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:05.447360 1166 consensus_queue.cc:260] T 6bd3fac2502e479aa9f4ceaabbb39504 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:05.448674 1174 log.cc:1199] T 1bab90fe904944af948dab3db06b7e34 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/1bab90fe904944af948dab3db06b7e34
I20251029 02:52:05.448853 1166 ts_tablet_manager.cc:1434] T 6bd3fac2502e479aa9f4ceaabbb39504 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251029 02:52:05.444306 895 tablet_service.cc:1505] Processing CreateTablet for tablet 31a157697e2c45ac9ba92c31c6936fa0 (DEFAULT_TABLE table=default.Table_df7d932b50b64537b4dbc68a846665fc [id=d7c62808bc9a4e74b85ba6a0ddb01495]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:05.449247 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31a157697e2c45ac9ba92c31c6936fa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:05.442390 1167 tablet_bootstrap.cc:654] T 8df97119d0654d7d92bb5bd17c5d8562 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:05.449923 1166 tablet_bootstrap.cc:492] T 71068739b4384badbf305313d902f0f5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:05.451324 1166 tablet_bootstrap.cc:654] T 71068739b4384badbf305313d902f0f5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:05.451762 519 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_3800864035b741c29cf00bfe2b35a32f" } new_table_name: "default.Table_c1c6933af1e046aa895e0ee915178f5c" modify_external_catalogs: true
I20251029 02:52:05.451987 1176 log.cc:1199] T dbc2092b63e0417997565c8fadb53ae0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/dbc2092b63e0417997565c8fadb53ae0
I20251029 02:52:05.452278 1176 ts_tablet_manager.cc:1950] T dbc2092b63e0417997565c8fadb53ae0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:05.452327 1174 ts_tablet_manager.cc:1950] T 1bab90fe904944af948dab3db06b7e34 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:05.452992 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 699) took 1447 ms. Trace:
I20251029 02:52:05.453090 1176 rpcz_store.cc:276] 1029 02:52:04.005787 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:04.005843 (+ 56us) service_pool.cc:225] Handling call
1029 02:52:05.413348 (+1407505us) ts_tablet_manager.cc:1151] Deleting tablet dbc2092b63e0417997565c8fadb53ae0
1029 02:52:05.413375 (+ 27us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:05.429248 (+ 15873us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:05.440450 (+ 11202us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:05.452974 (+ 12524us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1407333}
I20251029 02:52:05.453209 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 673) took 1663 ms. Trace:
I20251029 02:52:05.453281 1174 rpcz_store.cc:276] 1029 02:52:03.789965 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:03.790154 (+ 189us) service_pool.cc:225] Handling call
1029 02:52:05.425017 (+1634863us) ts_tablet_manager.cc:1151] Deleting tablet 1bab90fe904944af948dab3db06b7e34
1029 02:52:05.425026 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:05.438690 (+ 13664us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:05.441877 (+ 3187us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:05.453192 (+ 11315us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1634617}
I20251029 02:52:05.453661 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_11ffae1fbbae415189fd5c7bdb829875"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:05.454283 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_11ffae1fbbae415189fd5c7bdb829875 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:05.454336 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 1bab90fe904944af948dab3db06b7e34 (table default.Table_c70eb9f284cd49109fd09e430a1cea6c [id=be817eb6ec33406eb316d36575c134fb]) successfully deleted
I20251029 02:52:05.455031 518 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_78b81f8805aa49b8a07a940c8d886de5" } new_table_name: "default.Table_46e2f96551d3497c919e1cab3521d63b" modify_external_catalogs: true
I20251029 02:52:05.455091 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet dbc2092b63e0417997565c8fadb53ae0 (table default.Table_a873546f69334d12b0cbecce2cd489fb [id=1358e2055c254030a51e2de446eb742f]) successfully deleted
I20251029 02:52:05.455343 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 674) took 1587 ms. Trace:
I20251029 02:52:05.455478 1174 rpcz_store.cc:276] 1029 02:52:03.868310 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:03.868358 (+ 48us) service_pool.cc:225] Handling call
1029 02:52:05.454694 (+1586336us) ts_tablet_manager.cc:1151] Deleting tablet 1fc87e2d7b4240e08f2b4ac7277293ff
1029 02:52:05.455226 (+ 532us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:05.455324 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1586078}
I20251029 02:52:05.457226 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_abf02c5d46974d879e8c163933cb78ab"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:05.457913 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_abf02c5d46974d879e8c163933cb78ab in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:52:05.458920 512 catalog_manager.cc:4952] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 1fc87e2d7b4240e08f2b4ac7277293ff because the tablet was not found. No further retry: Not found: Tablet not found: 1fc87e2d7b4240e08f2b4ac7277293ff
I20251029 02:52:05.459069 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 698) took 1468 ms. Trace:
I20251029 02:52:05.459313 1176 rpcz_store.cc:276] 1029 02:52:03.991054 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:03.991107 (+ 53us) service_pool.cc:225] Handling call
1029 02:52:05.455308 (+1464201us) ts_tablet_manager.cc:1151] Deleting tablet a945d64287f94ca697447e540fab39ce
1029 02:52:05.456237 (+ 929us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:05.459051 (+ 2814us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1406385}
I20251029 02:52:05.459683 1176 tablet_replica.cc:333] T 51effc9324034ae79e2b25cb57a996c1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
W20251029 02:52:05.459843 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet a945d64287f94ca697447e540fab39ce because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a945d64287f94ca697447e540fab39ce already in progress: creating tablet
I20251029 02:52:05.459918 1176 raft_consensus.cc:2243] T 51effc9324034ae79e2b25cb57a996c1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:05.460178 1176 pending_rounds.cc:70] T 51effc9324034ae79e2b25cb57a996c1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Trying to abort 1 pending ops.
I20251029 02:52:05.460237 1176 pending_rounds.cc:77] T 51effc9324034ae79e2b25cb57a996c1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215949104276525056 op_type: NO_OP noop_request { }
I20251029 02:52:05.460333 1176 raft_consensus.cc:2889] T 51effc9324034ae79e2b25cb57a996c1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 02:52:05.460424 1176 raft_consensus.cc:2272] T 51effc9324034ae79e2b25cb57a996c1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:05.460952 1176 ts_tablet_manager.cc:1916] T 51effc9324034ae79e2b25cb57a996c1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:05.461980 1174 tablet_replica.cc:333] T 69597d48631d43b6a5b92abbf60d5ed3 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:05.462060 1906 raft_consensus.cc:493] T 576c4e6e70e24b0399ac9597a30f0569 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:05.462162 1174 raft_consensus.cc:2243] T 69597d48631d43b6a5b92abbf60d5ed3 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:05.462163 1906 raft_consensus.cc:515] T 576c4e6e70e24b0399ac9597a30f0569 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:05.462456 1174 raft_consensus.cc:2272] T 69597d48631d43b6a5b92abbf60d5ed3 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:05.462762 1906 leader_election.cc:290] T 576c4e6e70e24b0399ac9597a30f0569 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:05.463025 1174 ts_tablet_manager.cc:1916] T 69597d48631d43b6a5b92abbf60d5ed3 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:05.463459 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "576c4e6e70e24b0399ac9597a30f0569" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:05.463634 917 raft_consensus.cc:2468] T 576c4e6e70e24b0399ac9597a30f0569 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 0.
W20251029 02:52:05.464478 989 leader_election.cc:343] T 576c4e6e70e24b0399ac9597a30f0569 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 576c4e6e70e24b0399ac9597a30f0569
I20251029 02:52:05.464846 990 leader_election.cc:304] T 576c4e6e70e24b0399ac9597a30f0569 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters: b495a49349324f7d81908f2da700f0d9
I20251029 02:52:05.465294 1973 raft_consensus.cc:2804] T 576c4e6e70e24b0399ac9597a30f0569 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:05.465364 1973 raft_consensus.cc:493] T 576c4e6e70e24b0399ac9597a30f0569 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:05.465415 1973 raft_consensus.cc:3060] T 576c4e6e70e24b0399ac9597a30f0569 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:05.467991 890 tablet_service.cc:1505] Processing CreateTablet for tablet 76648db142894fd9972b38b5ee485fed (DEFAULT_TABLE table=default.Table_714c5abdd40047d1bf204ee53c29a06e [id=4a8aee3131e34716b8bfed53f169b062]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:05.468004 1973 raft_consensus.cc:515] T 576c4e6e70e24b0399ac9597a30f0569 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:05.468343 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76648db142894fd9972b38b5ee485fed. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:05.469343 1973 leader_election.cc:290] T 576c4e6e70e24b0399ac9597a30f0569 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Requested vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
W20251029 02:52:05.469517 989 leader_election.cc:343] T 576c4e6e70e24b0399ac9597a30f0569 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 576c4e6e70e24b0399ac9597a30f0569
I20251029 02:52:05.471158 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "576c4e6e70e24b0399ac9597a30f0569" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
I20251029 02:52:05.471362 920 raft_consensus.cc:3060] T 576c4e6e70e24b0399ac9597a30f0569 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:05.473676 1167 tablet_bootstrap.cc:492] T 8df97119d0654d7d92bb5bd17c5d8562 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:05.473805 1167 ts_tablet_manager.cc:1403] T 8df97119d0654d7d92bb5bd17c5d8562 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.036s user 0.001s sys 0.001s
I20251029 02:52:05.474707 1167 raft_consensus.cc:359] T 8df97119d0654d7d92bb5bd17c5d8562 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:05.474869 1167 raft_consensus.cc:385] T 8df97119d0654d7d92bb5bd17c5d8562 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:05.474942 1167 raft_consensus.cc:740] T 8df97119d0654d7d92bb5bd17c5d8562 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:05.475157 1167 consensus_queue.cc:260] T 8df97119d0654d7d92bb5bd17c5d8562 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:05.475592 1167 ts_tablet_manager.cc:1434] T 8df97119d0654d7d92bb5bd17c5d8562 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251029 02:52:05.476056 1172 tablet_bootstrap.cc:492] T 1bdb67125493430398ff5c80b73de2ac P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:05.476150 1167 tablet_bootstrap.cc:492] T c9e923724ca9402682f13b252dbc37aa P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:05.476229 1172 ts_tablet_manager.cc:1403] T 1bdb67125493430398ff5c80b73de2ac P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.141s user 0.002s sys 0.002s
I20251029 02:52:05.476931 1172 raft_consensus.cc:359] T 1bdb67125493430398ff5c80b73de2ac P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:05.477154 1172 raft_consensus.cc:385] T 1bdb67125493430398ff5c80b73de2ac P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:05.477244 1172 raft_consensus.cc:740] T 1bdb67125493430398ff5c80b73de2ac P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:05.477495 1172 consensus_queue.cc:260] T 1bdb67125493430398ff5c80b73de2ac P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:05.477741 1167 tablet_bootstrap.cc:654] T c9e923724ca9402682f13b252dbc37aa P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:05.479777 1174 ts_tablet_manager.cc:1929] T 69597d48631d43b6a5b92abbf60d5ed3 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:05.480636 1174 log.cc:1199] T 69597d48631d43b6a5b92abbf60d5ed3 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/69597d48631d43b6a5b92abbf60d5ed3
W20251029 02:52:05.481086 847 consensus_peers.cc:597] T 7309d3190cfa415281c5253a634fa7f3 P c756528642a543eeacd93cef3e335a7d -> Peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Couldn't send request to peer b495a49349324f7d81908f2da700f0d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7309d3190cfa415281c5253a634fa7f3. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:05.481135 1174 ts_tablet_manager.cc:1950] T 69597d48631d43b6a5b92abbf60d5ed3 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:05.483109 1172 ts_tablet_manager.cc:1434] T 1bdb67125493430398ff5c80b73de2ac P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20251029 02:52:05.482904 920 raft_consensus.cc:2468] T 576c4e6e70e24b0399ac9597a30f0569 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 1.
I20251029 02:52:05.486809 990 leader_election.cc:304] T 576c4e6e70e24b0399ac9597a30f0569 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters: b495a49349324f7d81908f2da700f0d9
I20251029 02:52:05.488088 1172 tablet_bootstrap.cc:492] T caef1c93e33a4390bcada59a41f43465 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:05.488971 1906 raft_consensus.cc:2804] T 576c4e6e70e24b0399ac9597a30f0569 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:05.489099 1906 raft_consensus.cc:697] T 576c4e6e70e24b0399ac9597a30f0569 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Becoming Leader. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Running, Role: LEADER
I20251029 02:52:05.489326 1906 consensus_queue.cc:237] T 576c4e6e70e24b0399ac9597a30f0569 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:05.490378 1172 tablet_bootstrap.cc:654] T caef1c93e33a4390bcada59a41f43465 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:05.495589 756 tablet_service.cc:1505] Processing CreateTablet for tablet 31a157697e2c45ac9ba92c31c6936fa0 (DEFAULT_TABLE table=default.Table_df7d932b50b64537b4dbc68a846665fc [id=d7c62808bc9a4e74b85ba6a0ddb01495]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:05.496093 900 tablet_service.cc:1505] Processing CreateTablet for tablet 59fd5608e4c04fa293ca04595a20d858 (DEFAULT_TABLE table=default.Table_e0978085f4ba4e798435ad96267706ac [id=e35688bf2ad64ce9a06bb8d95fe902c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:05.496562 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59fd5608e4c04fa293ca04595a20d858. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:05.496928 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31a157697e2c45ac9ba92c31c6936fa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:05.498162 1166 tablet_bootstrap.cc:492] T 71068739b4384badbf305313d902f0f5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:05.498296 1166 ts_tablet_manager.cc:1403] T 71068739b4384badbf305313d902f0f5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.048s user 0.005s sys 0.001s
I20251029 02:52:05.499289 748 tablet_service.cc:1505] Processing CreateTablet for tablet 10e0ad0693a1432091e1a590fb76a720 (DEFAULT_TABLE table=default.Table_877abcb0d221493c8875368e1c75382b [id=6ea46a01a86140b9ae63e33b90b03dd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:05.499516 895 tablet_service.cc:1505] Processing CreateTablet for tablet 10e0ad0693a1432091e1a590fb76a720 (DEFAULT_TABLE table=default.Table_877abcb0d221493c8875368e1c75382b [id=6ea46a01a86140b9ae63e33b90b03dd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:05.499586 1025 tablet_service.cc:1505] Processing CreateTablet for tablet 59fd5608e4c04fa293ca04595a20d858 (DEFAULT_TABLE table=default.Table_e0978085f4ba4e798435ad96267706ac [id=e35688bf2ad64ce9a06bb8d95fe902c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:05.499853 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10e0ad0693a1432091e1a590fb76a720. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:05.499948 749 tablet_service.cc:1505] Processing CreateTablet for tablet 59fd5608e4c04fa293ca04595a20d858 (DEFAULT_TABLE table=default.Table_e0978085f4ba4e798435ad96267706ac [id=e35688bf2ad64ce9a06bb8d95fe902c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:05.500430 1166 raft_consensus.cc:359] T 71068739b4384badbf305313d902f0f5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:05.500808 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59fd5608e4c04fa293ca04595a20d858. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:05.500845 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10e0ad0693a1432091e1a590fb76a720. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:05.506395 1166 raft_consensus.cc:385] T 71068739b4384badbf305313d902f0f5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:05.506520 1166 raft_consensus.cc:740] T 71068739b4384badbf305313d902f0f5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:05.506745 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59fd5608e4c04fa293ca04595a20d858. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:05.506803 1166 consensus_queue.cc:260] T 71068739b4384badbf305313d902f0f5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:05.508709 1166 ts_tablet_manager.cc:1434] T 71068739b4384badbf305313d902f0f5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20251029 02:52:05.484043 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 666) took 1801 ms. Trace:
I20251029 02:52:05.509400 1174 rpcz_store.cc:276] 1029 02:52:03.682267 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:03.683319 (+ 1052us) service_pool.cc:225] Handling call
1029 02:52:05.459259 (+1775940us) ts_tablet_manager.cc:1151] Deleting tablet 69597d48631d43b6a5b92abbf60d5ed3
1029 02:52:05.461936 (+ 2677us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:05.467911 (+ 5975us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:05.479690 (+ 11779us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:05.484024 (+ 4334us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1518824}
I20251029 02:52:05.510932 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 69597d48631d43b6a5b92abbf60d5ed3 (table default.Table_cec9500b890447609a5feb7346bcf7a0 [id=ffce392867b14b6ea9db44f4bdee57c3]) successfully deleted
I20251029 02:52:05.511408 1174 tablet_replica.cc:333] T 7309d3190cfa415281c5253a634fa7f3 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:05.512106 1041 tablet_service.cc:1505] Processing CreateTablet for tablet 31a157697e2c45ac9ba92c31c6936fa0 (DEFAULT_TABLE table=default.Table_df7d932b50b64537b4dbc68a846665fc [id=d7c62808bc9a4e74b85ba6a0ddb01495]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:05.512259 1174 raft_consensus.cc:2243] T 7309d3190cfa415281c5253a634fa7f3 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:05.512584 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31a157697e2c45ac9ba92c31c6936fa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:05.512729 1174 pending_rounds.cc:70] T 7309d3190cfa415281c5253a634fa7f3 P c756528642a543eeacd93cef3e335a7d: Trying to abort 1 pending ops.
I20251029 02:52:05.512243 1166 tablet_bootstrap.cc:492] T da2a21c3236a4bd99872b89b12a36508 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:05.513907 1174 pending_rounds.cc:77] T 7309d3190cfa415281c5253a634fa7f3 P c756528642a543eeacd93cef3e335a7d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215949104136466432 op_type: NO_OP noop_request { }
I20251029 02:52:05.514312 1174 raft_consensus.cc:2889] T 7309d3190cfa415281c5253a634fa7f3 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 02:52:05.514552 1174 raft_consensus.cc:2272] T 7309d3190cfa415281c5253a634fa7f3 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:05.514192 1167 tablet_bootstrap.cc:492] T c9e923724ca9402682f13b252dbc37aa P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:05.515158 1166 tablet_bootstrap.cc:654] T da2a21c3236a4bd99872b89b12a36508 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:05.515271 1167 ts_tablet_manager.cc:1403] T c9e923724ca9402682f13b252dbc37aa P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.039s user 0.005s sys 0.004s
I20251029 02:52:05.516112 1167 raft_consensus.cc:359] T c9e923724ca9402682f13b252dbc37aa P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:05.517983 1167 raft_consensus.cc:385] T c9e923724ca9402682f13b252dbc37aa P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:05.518225 1167 raft_consensus.cc:740] T c9e923724ca9402682f13b252dbc37aa P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:05.516914 1174 ts_tablet_manager.cc:1916] T 7309d3190cfa415281c5253a634fa7f3 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:05.520979 1167 consensus_queue.cc:260] T c9e923724ca9402682f13b252dbc37aa P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:05.526018 1166 tablet_bootstrap.cc:492] T da2a21c3236a4bd99872b89b12a36508 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:05.526165 1166 ts_tablet_manager.cc:1403] T da2a21c3236a4bd99872b89b12a36508 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.000s
I20251029 02:52:05.526315 1167 ts_tablet_manager.cc:1434] T c9e923724ca9402682f13b252dbc37aa P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.011s user 0.000s sys 0.002s
I20251029 02:52:05.527518 1166 raft_consensus.cc:359] T da2a21c3236a4bd99872b89b12a36508 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:05.527693 1166 raft_consensus.cc:385] T da2a21c3236a4bd99872b89b12a36508 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:05.527750 1166 raft_consensus.cc:740] T da2a21c3236a4bd99872b89b12a36508 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:05.528004 1166 consensus_queue.cc:260] T da2a21c3236a4bd99872b89b12a36508 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:05.528439 1166 ts_tablet_manager.cc:1434] T da2a21c3236a4bd99872b89b12a36508 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 02:52:05.530900 1166 tablet_bootstrap.cc:492] T 74873a05ef634867a30aa63b31fcfcc7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:05.532245 1166 tablet_bootstrap.cc:654] T 74873a05ef634867a30aa63b31fcfcc7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:05.540813 1176 ts_tablet_manager.cc:1929] T 51effc9324034ae79e2b25cb57a996c1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:05.548906 748 tablet_service.cc:1505] Processing CreateTablet for tablet 6fcf3e12810d41ddb96d9f2cf52f2c5d (DEFAULT_TABLE table=default.Table_d901c631b88743509930ea084b31e7a8 [id=74b92cc3eadb43ec96c77b178917a04d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:05.549418 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6fcf3e12810d41ddb96d9f2cf52f2c5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:05.552397 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 660) took 1941 ms. Trace:
I20251029 02:52:05.552560 1175 rpcz_store.cc:276] 1029 02:52:03.611368 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:03.613492 (+ 2124us) service_pool.cc:225] Handling call
1029 02:52:04.775463 (+1161971us) ts_tablet_manager.cc:1151] Deleting tablet bfec87f690a640c39c58c64e16cc67e7
1029 02:52:04.775481 (+ 18us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:05.418243 (+642762us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:05.421251 (+ 3008us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:05.552375 (+131124us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1161824}
I20251029 02:52:05.553146 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 662) took 1941 ms. Trace:
I20251029 02:52:05.553200 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet bfec87f690a640c39c58c64e16cc67e7 (table default.Table_11de4e6789214efca8b43cbcdae81a93 [id=6506de17cec5443b8a4628bd46a39b42]) successfully deleted
I20251029 02:52:05.553283 1175 rpcz_store.cc:276] 1029 02:52:03.611566 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:03.614009 (+ 2443us) service_pool.cc:225] Handling call
1029 02:52:05.553011 (+1939002us) ts_tablet_manager.cc:1151] Deleting tablet a27496be3431446da9cb83452a3893e3
1029 02:52:05.553021 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:05.553118 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1938864}
I20251029 02:52:05.553560 1175 tablet_replica.cc:333] T f187dbd8a55b45b292ea61db43caf47d P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:05.553740 1175 raft_consensus.cc:2243] T f187dbd8a55b45b292ea61db43caf47d P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
W20251029 02:52:05.553799 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet a27496be3431446da9cb83452a3893e3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a27496be3431446da9cb83452a3893e3 already in progress: creating tablet
I20251029 02:52:05.553921 1175 raft_consensus.cc:2272] T f187dbd8a55b45b292ea61db43caf47d P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:05.554592 756 tablet_service.cc:1505] Processing CreateTablet for tablet 49492d91b1e94e9ca086533dbd0440f6 (DEFAULT_TABLE table=default.Table_b9f8911906c3473da4de53896081cb2c [id=8cccaacc256b40deabe999de1d75032e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:05.555531 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49492d91b1e94e9ca086533dbd0440f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:05.554622 1175 ts_tablet_manager.cc:1916] T f187dbd8a55b45b292ea61db43caf47d P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:05.560460 1167 tablet_bootstrap.cc:492] T 74873a05ef634867a30aa63b31fcfcc7 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:05.560407 892 tablet_service.cc:1505] Processing CreateTablet for tablet 74eafdebb4eb4ba7a0c05d99e2e93fcb (DEFAULT_TABLE table=default.Table_6d69db05345d4ecdb52c0f4b4ac5be13 [id=bfa24332683e42059d6bc82fb7909765]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:05.560842 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74eafdebb4eb4ba7a0c05d99e2e93fcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:05.561982 895 tablet_service.cc:1505] Processing CreateTablet for tablet 6fcf3e12810d41ddb96d9f2cf52f2c5d (DEFAULT_TABLE table=default.Table_d901c631b88743509930ea084b31e7a8 [id=74b92cc3eadb43ec96c77b178917a04d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:05.562392 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6fcf3e12810d41ddb96d9f2cf52f2c5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:05.563616 1167 tablet_bootstrap.cc:654] T 74873a05ef634867a30aa63b31fcfcc7 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
W20251029 02:52:05.564371 521 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6dc30c614be64b8ba629d9884d450290 consensus state has no leader
I20251029 02:52:05.567422 1957 raft_consensus.cc:493] T 1bc45078a6a942c38ff3350c15885eff P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:05.567559 1957 raft_consensus.cc:515] T 1bc45078a6a942c38ff3350c15885eff P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:05.568439 1957 leader_election.cc:290] T 1bc45078a6a942c38ff3350c15885eff P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:05.569157 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1bc45078a6a942c38ff3350c15885eff" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:05.564085 1903 raft_consensus.cc:493] T b2b2d64dcf964da8be4b544dbbe31977 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:05.569733 1903 raft_consensus.cc:515] T b2b2d64dcf964da8be4b544dbbe31977 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:05.570385 1903 leader_election.cc:290] T b2b2d64dcf964da8be4b544dbbe31977 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
W20251029 02:52:05.571327 711 leader_election.cc:343] T 1bc45078a6a942c38ff3350c15885eff P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
W20251029 02:52:05.571971 847 leader_election.cc:343] T b2b2d64dcf964da8be4b544dbbe31977 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: b2b2d64dcf964da8be4b544dbbe31977
I20251029 02:52:05.567094 1174 ts_tablet_manager.cc:1929] T 7309d3190cfa415281c5253a634fa7f3 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:05.574708 1174 log.cc:1199] T 7309d3190cfa415281c5253a634fa7f3 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/7309d3190cfa415281c5253a634fa7f3
I20251029 02:52:05.575100 1174 ts_tablet_manager.cc:1950] T 7309d3190cfa415281c5253a634fa7f3 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:05.576267 522 master_service.cc:954] ReplaceTablet: received request to replace tablet 76287edc126a403dbbec4a3b41effe84 from {username='slave'} at 127.0.0.1:35314
I20251029 02:52:05.577749 525 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_e1ecca1682bb467789ae7beec4a3f11d" } new_table_name: "default.Table_14bdca3a0cc3493ca71d056763b69493" modify_external_catalogs: true
I20251029 02:52:05.579075 523 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet dd90fb9eb167478cbc3ec9363af86295
I20251029 02:52:05.579735 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_ced93e731339479fa9ca51ef65dea8a6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:05.580281 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ced93e731339479fa9ca51ef65dea8a6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:52:05.581473 519 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d7abaf84ba5d445aa17433b3be891548 consensus state has no leader
I20251029 02:52:05.581804 519 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_bce83bae13a94cd5bf6733b53e6a1e9e" } new_table_name: "default.Table_4e9d94609b4a4d1f94615dcc9830d717" modify_external_catalogs: true
I20251029 02:52:05.583148 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 675) took 1622 ms. Trace:
I20251029 02:52:05.583253 1174 rpcz_store.cc:276] 1029 02:52:03.960683 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:03.968506 (+ 7823us) service_pool.cc:225] Handling call
1029 02:52:05.511353 (+1542847us) ts_tablet_manager.cc:1151] Deleting tablet 7309d3190cfa415281c5253a634fa7f3
1029 02:52:05.511363 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:05.564116 (+ 52753us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:05.567019 (+ 2903us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:05.583118 (+ 16099us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1542622}
I20251029 02:52:05.583814 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_f908888068114a9083c998cf80a8441a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:05.583915 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 676) took 1584 ms. Trace:
I20251029 02:52:05.584012 1174 rpcz_store.cc:276] 1029 02:52:03.999477 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:03.999667 (+ 190us) service_pool.cc:225] Handling call
1029 02:52:05.583751 (+1584084us) ts_tablet_manager.cc:1151] Deleting tablet a945d64287f94ca697447e540fab39ce
1029 02:52:05.583767 (+ 16us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:05.583893 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1583830}
W20251029 02:52:05.584300 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f908888068114a9083c998cf80a8441a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:05.584473 1174 tablet_replica.cc:333] T dbc2092b63e0417997565c8fadb53ae0 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:05.585027 1174 raft_consensus.cc:2243] T dbc2092b63e0417997565c8fadb53ae0 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:05.585291 1174 pending_rounds.cc:70] T dbc2092b63e0417997565c8fadb53ae0 P c756528642a543eeacd93cef3e335a7d: Trying to abort 1 pending ops.
I20251029 02:52:05.585328 1174 pending_rounds.cc:77] T dbc2092b63e0417997565c8fadb53ae0 P c756528642a543eeacd93cef3e335a7d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215949106464501760 op_type: NO_OP noop_request { }
I20251029 02:52:05.585417 1174 raft_consensus.cc:2889] T dbc2092b63e0417997565c8fadb53ae0 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 02:52:05.585498 1174 raft_consensus.cc:2272] T dbc2092b63e0417997565c8fadb53ae0 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:05.592464 750 tablet_service.cc:1505] Processing CreateTablet for tablet ff139eb4f65343938ef0d3b854340a7d (DEFAULT_TABLE table=default.Table_905a7a1ab3ef4da9838ac7883ffa4cb1 [id=1a21f772083e4eda9283c8ac7db51d2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:05.593024 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff139eb4f65343938ef0d3b854340a7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:05.593819 526 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet 1bdb67125493430398ff5c80b73de2ac (table default.Table_2846af457a4b4e27a739af50c7f8c7f7 [id=8c3d3d88395e4712af0f62c4ad14edea]). Expected version 3 got 0
I20251029 02:52:05.594547 526 master_service.cc:954] ReplaceTablet: received request to replace tablet 785733da86ff425a90ccfab2933fc89e from {username='slave'} at 127.0.0.1:35314
I20251029 02:52:05.595439 751 tablet_service.cc:1505] Processing CreateTablet for tablet 74eafdebb4eb4ba7a0c05d99e2e93fcb (DEFAULT_TABLE table=default.Table_6d69db05345d4ecdb52c0f4b4ac5be13 [id=bfa24332683e42059d6bc82fb7909765]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:05.596378 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74eafdebb4eb4ba7a0c05d99e2e93fcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:05.596285 742 tablet_service.cc:1552] Processing DeleteTablet for tablet dd90fb9eb167478cbc3ec9363af86295 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:05 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:05.597788 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 7309d3190cfa415281c5253a634fa7f3 (table default.Table_ef563eb489f947d4a4772678806a0777 [id=7de93d6d48ab4d58b5aed9319d7299a4]) successfully deleted
W20251029 02:52:05.598033 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet a945d64287f94ca697447e540fab39ce because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a945d64287f94ca697447e540fab39ce already in progress: creating tablet
I20251029 02:52:05.598953 899 tablet_service.cc:1552] Processing DeleteTablet for tablet f81aecb20d6e498bb9b498698886d65e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:53 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:05.599344 892 tablet_service.cc:1505] Processing CreateTablet for tablet ff139eb4f65343938ef0d3b854340a7d (DEFAULT_TABLE table=default.Table_905a7a1ab3ef4da9838ac7883ffa4cb1 [id=1a21f772083e4eda9283c8ac7db51d2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:05.599781 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff139eb4f65343938ef0d3b854340a7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:05.602860 885 tablet_service.cc:1505] Processing CreateTablet for tablet 636e9fe4eeab45749368cd1eb9d48b03 (DEFAULT_TABLE table=default.Table_ef95e34ffbdf43f9bbe0074cee266ac1 [id=7ef14a66267546ca92eb58eedf69e3ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:05.603269 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 636e9fe4eeab45749368cd1eb9d48b03. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:05.607386 900 tablet_service.cc:1505] Processing CreateTablet for tablet 49492d91b1e94e9ca086533dbd0440f6 (DEFAULT_TABLE table=default.Table_b9f8911906c3473da4de53896081cb2c [id=8cccaacc256b40deabe999de1d75032e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:05.607810 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49492d91b1e94e9ca086533dbd0440f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:05.619444 890 tablet_service.cc:1552] Processing DeleteTablet for tablet dd90fb9eb167478cbc3ec9363af86295 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:05 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:05.547748 1025 tablet_service.cc:1505] Processing CreateTablet for tablet 10e0ad0693a1432091e1a590fb76a720 (DEFAULT_TABLE table=default.Table_877abcb0d221493c8875368e1c75382b [id=6ea46a01a86140b9ae63e33b90b03dd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:05.549432 1037 tablet_service.cc:1505] Processing CreateTablet for tablet 6fcf3e12810d41ddb96d9f2cf52f2c5d (DEFAULT_TABLE table=default.Table_d901c631b88743509930ea084b31e7a8 [id=74b92cc3eadb43ec96c77b178917a04d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:05.620007 1176 log.cc:1199] T 51effc9324034ae79e2b25cb57a996c1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/51effc9324034ae79e2b25cb57a996c1
I20251029 02:52:05.620119 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6fcf3e12810d41ddb96d9f2cf52f2c5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:05.620482 1176 ts_tablet_manager.cc:1950] T 51effc9324034ae79e2b25cb57a996c1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:05.620819 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 701) took 1549 ms. Trace:
I20251029 02:52:05.620905 1176 rpcz_store.cc:276] 1029 02:52:04.071674 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:04.071740 (+ 66us) service_pool.cc:225] Handling call
1029 02:52:05.459641 (+1387901us) ts_tablet_manager.cc:1151] Deleting tablet 51effc9324034ae79e2b25cb57a996c1
1029 02:52:05.459649 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:05.536729 (+ 77080us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:05.540724 (+ 3995us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:05.620803 (+ 80079us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1387685}
I20251029 02:52:05.621311 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 700) took 1549 ms. Trace:
I20251029 02:52:05.621381 1176 rpcz_store.cc:276] 1029 02:52:04.071536 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:04.075083 (+ 3547us) service_pool.cc:225] Handling call
1029 02:52:05.621185 (+1546102us) ts_tablet_manager.cc:1151] Deleting tablet 1e28a07b1a6242cd919b6ab2cb9b8a92
1029 02:52:05.621194 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:05.621294 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1545942}
I20251029 02:52:05.621707 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 702) took 1515 ms. Trace:
I20251029 02:52:05.621775 1176 rpcz_store.cc:276] 1029 02:52:04.106099 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:04.106161 (+ 62us) service_pool.cc:225] Handling call
1029 02:52:05.621613 (+1515452us) ts_tablet_manager.cc:1151] Deleting tablet 4929513bed104050bd6065f025303e1a
1029 02:52:05.621621 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:05.621693 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1515276}
I20251029 02:52:05.622071 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 703) took 1495 ms. Trace:
I20251029 02:52:05.622133 1176 rpcz_store.cc:276] 1029 02:52:04.126426 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:04.126784 (+ 358us) service_pool.cc:225] Handling call
1029 02:52:05.621975 (+1495191us) ts_tablet_manager.cc:1151] Deleting tablet 4e82c574dfa14ffbabf162a17355f841
1029 02:52:05.621985 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:05.622057 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1482668}
I20251029 02:52:05.622416 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 704) took 1487 ms. Trace:
I20251029 02:52:05.623291 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 51effc9324034ae79e2b25cb57a996c1 (table default.Table_ee95bb03ab1246e8826f28083411050d [id=09fbf31759f047be924a5ef1f0030765]) successfully deleted
W20251029 02:52:05.623533 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 1e28a07b1a6242cd919b6ab2cb9b8a92 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1e28a07b1a6242cd919b6ab2cb9b8a92 already in progress: creating tablet
W20251029 02:52:05.623749 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 4929513bed104050bd6065f025303e1a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4929513bed104050bd6065f025303e1a already in progress: creating tablet
W20251029 02:52:05.623934 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 4e82c574dfa14ffbabf162a17355f841 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e82c574dfa14ffbabf162a17355f841 already in progress: creating tablet
I20251029 02:52:05.624297 524 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_05a3479e89744e828535b90bbea0e8e0" } modify_external_catalogs: true
I20251029 02:52:05.550366 1166 tablet_bootstrap.cc:492] T 74873a05ef634867a30aa63b31fcfcc7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:05.626644 1166 ts_tablet_manager.cc:1403] T 74873a05ef634867a30aa63b31fcfcc7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.096s user 0.005s sys 0.001s
W20251029 02:52:05.626766 518 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dc33fd3404ec41c3935b02c84a12971b consensus state has no leader
I20251029 02:52:05.627269 522 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 76287edc126a403dbbec4a3b41effe84
I20251029 02:52:05.627389 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10e0ad0693a1432091e1a590fb76a720. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:05.627650 522 catalog_manager.cc:6719] ReplaceTablet: tablet 76287edc126a403dbbec4a3b41effe84 deleted and replaced by tablet 00f2e98739fa41eea23927494fe222f3
W20251029 02:52:05.628278 525 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 839f3f679c344fc5a3746da6df2cfe93 consensus state has no leader
I20251029 02:52:05.629356 1038 tablet_service.cc:1552] Processing DeleteTablet for tablet 76287edc126a403dbbec4a3b41effe84 with delete_type TABLET_DATA_DELETED (replaced by tablet 00f2e98739fa41eea23927494fe222f3) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:05.629611 524 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_05a3479e89744e828535b90bbea0e8e0" } modify_external_catalogs: true
I20251029 02:52:05.551237 1032 tablet_service.cc:1505] Processing CreateTablet for tablet 49492d91b1e94e9ca086533dbd0440f6 (DEFAULT_TABLE table=default.Table_b9f8911906c3473da4de53896081cb2c [id=8cccaacc256b40deabe999de1d75032e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:05.551570 1028 tablet_service.cc:1505] Processing CreateTablet for tablet 74eafdebb4eb4ba7a0c05d99e2e93fcb (DEFAULT_TABLE table=default.Table_6d69db05345d4ecdb52c0f4b4ac5be13 [id=bfa24332683e42059d6bc82fb7909765]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:05.570559 1058 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1bc45078a6a942c38ff3350c15885eff" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:05.571040 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b2b2d64dcf964da8be4b544dbbe31977" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:05.630931 1056 raft_consensus.cc:2468] T b2b2d64dcf964da8be4b544dbbe31977 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:05.631165 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49492d91b1e94e9ca086533dbd0440f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:05.584838 1906 raft_consensus.cc:493] T cedfb53078d94c81b5e294ce7438325e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:05.631335 1906 raft_consensus.cc:515] T cedfb53078d94c81b5e294ce7438325e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:05.631668 521 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet cedfb53078d94c81b5e294ce7438325e (table default.Table_e75db5cf8f5f49ea837cb6d63f4b7ebf [id=86195563fb4a4d7abd746f4ac7cd241c]). Expected version 1 got 0
I20251029 02:52:05.631866 521 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet c9e923724ca9402682f13b252dbc37aa (table default.Table_443f33587b8f428484b99e07355d69ca [id=03a25420ce6a46fc93a9fb753246bc7b]). Expected version 3 got 0
I20251029 02:52:05.632297 1906 leader_election.cc:290] T cedfb53078d94c81b5e294ce7438325e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:05.632875 1957 raft_consensus.cc:493] T b3de6670357c4ae59eb77469b2fb434c P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:05.633054 1957 raft_consensus.cc:515] T b3de6670357c4ae59eb77469b2fb434c P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:05.633579 899 tablet_service.cc:1552] Processing DeleteTablet for tablet b2b2d64dcf964da8be4b544dbbe31977 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:53 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:05.633671 1957 leader_election.cc:290] T b3de6670357c4ae59eb77469b2fb434c P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:05.633805 898 tablet_service.cc:1552] Processing DeleteTablet for tablet 8df97119d0654d7d92bb5bd17c5d8562 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:59 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:05.634392 849 leader_election.cc:304] T b2b2d64dcf964da8be4b544dbbe31977 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters: b495a49349324f7d81908f2da700f0d9
I20251029 02:52:05.634922 1903 raft_consensus.cc:2804] T b2b2d64dcf964da8be4b544dbbe31977 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:05.635033 1903 raft_consensus.cc:493] T b2b2d64dcf964da8be4b544dbbe31977 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:05.635160 1903 raft_consensus.cc:3060] T b2b2d64dcf964da8be4b544dbbe31977 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:05.635293 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74eafdebb4eb4ba7a0c05d99e2e93fcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:05.636615 526 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet 1bdb67125493430398ff5c80b73de2ac (table default.Table_2846af457a4b4e27a739af50c7f8c7f7 [id=8c3d3d88395e4712af0f62c4ad14edea]). Expected version 3 got 0
I20251029 02:52:05.637759 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cedfb53078d94c81b5e294ce7438325e" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:05.637998 920 raft_consensus.cc:2468] T cedfb53078d94c81b5e294ce7438325e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 0.
I20251029 02:52:05.638243 1903 raft_consensus.cc:515] T b2b2d64dcf964da8be4b544dbbe31977 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:05.638649 990 leader_election.cc:304] T cedfb53078d94c81b5e294ce7438325e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:05.640242 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b2b2d64dcf964da8be4b544dbbe31977" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:52:05.640453 1056 raft_consensus.cc:3060] T b2b2d64dcf964da8be4b544dbbe31977 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:05.640520 1906 raft_consensus.cc:2804] T cedfb53078d94c81b5e294ce7438325e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:05.640574 1906 raft_consensus.cc:493] T cedfb53078d94c81b5e294ce7438325e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:05.640631 1906 raft_consensus.cc:3060] T cedfb53078d94c81b5e294ce7438325e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
W20251029 02:52:05.640638 847 leader_election.cc:343] T b2b2d64dcf964da8be4b544dbbe31977 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: b2b2d64dcf964da8be4b544dbbe31977
W20251029 02:52:05.641788 712 leader_election.cc:343] T 1bc45078a6a942c38ff3350c15885eff P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:05.641922 712 leader_election.cc:304] T 1bc45078a6a942c38ff3350c15885eff P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:05.642423 1058 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3de6670357c4ae59eb77469b2fb434c" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:05.642684 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cedfb53078d94c81b5e294ce7438325e" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:05.642851 768 raft_consensus.cc:2468] T cedfb53078d94c81b5e294ce7438325e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 0.
I20251029 02:52:05.603088 1973 raft_consensus.cc:493] T da2a21c3236a4bd99872b89b12a36508 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:05.611665 1030 tablet_service.cc:1552] Processing DeleteTablet for tablet 1bab90fe904944af948dab3db06b7e34 with delete_type TABLET_DATA_DELETED (replaced by tablet b6d4f7bf702f4834bc5f4ef2b50b92a7) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:05.611716 1033 tablet_service.cc:1552] Processing DeleteTablet for tablet 6feec2c8e030447fae9c292a626dcd1d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:52 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:05.611869 1029 tablet_service.cc:1505] Processing CreateTablet for tablet 636e9fe4eeab45749368cd1eb9d48b03 (DEFAULT_TABLE table=default.Table_ef95e34ffbdf43f9bbe0074cee266ac1 [id=7ef14a66267546ca92eb58eedf69e3ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:05.643217 1957 raft_consensus.cc:2749] T 1bc45078a6a942c38ff3350c15885eff P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:05.612217 1027 tablet_service.cc:1552] Processing DeleteTablet for tablet b2b2d64dcf964da8be4b544dbbe31977 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:53 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:05.612332 1026 tablet_service.cc:1552] Processing DeleteTablet for tablet 8df97119d0654d7d92bb5bd17c5d8562 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:59 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:05.612421 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet 6cd7009e06a646acad0802918c2b425e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:54 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:05.612519 1042 tablet_service.cc:1552] Processing DeleteTablet for tablet 8afc01d52d784f2fa559cab7ce07c8be with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:59 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:05.643522 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 636e9fe4eeab45749368cd1eb9d48b03. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:05.643611 1174 ts_tablet_manager.cc:1916] T dbc2092b63e0417997565c8fadb53ae0 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 02:52:05.643653 712 leader_election.cc:343] T b3de6670357c4ae59eb77469b2fb434c P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:05.643826 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3de6670357c4ae59eb77469b2fb434c" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:05.644284 711 leader_election.cc:343] T b3de6670357c4ae59eb77469b2fb434c P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:05.644373 711 leader_election.cc:304] T b3de6670357c4ae59eb77469b2fb434c P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:05.644704 1957 raft_consensus.cc:2749] T b3de6670357c4ae59eb77469b2fb434c P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:05.645145 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_ee9b7cb33d8b4032b185338d44561a69"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:05.645696 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ee9b7cb33d8b4032b185338d44561a69 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:05.646006 1056 raft_consensus.cc:2468] T b2b2d64dcf964da8be4b544dbbe31977 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:05.646382 849 leader_election.cc:304] T b2b2d64dcf964da8be4b544dbbe31977 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters: b495a49349324f7d81908f2da700f0d9
I20251029 02:52:05.646684 1904 raft_consensus.cc:2804] T b2b2d64dcf964da8be4b544dbbe31977 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:05.646859 1904 raft_consensus.cc:697] T b2b2d64dcf964da8be4b544dbbe31977 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:52:05.647128 523 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_2ea4713d35ab4b4fb84ffa9ae3ca4bf6" } modify_external_catalogs: true
I20251029 02:52:05.647161 1904 consensus_queue.cc:237] T b2b2d64dcf964da8be4b544dbbe31977 P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:05.647303 523 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_2ea4713d35ab4b4fb84ffa9ae3ca4bf6" } modify_external_catalogs: true
I20251029 02:52:05.651304 1166 raft_consensus.cc:359] T 74873a05ef634867a30aa63b31fcfcc7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:05.651471 1166 raft_consensus.cc:385] T 74873a05ef634867a30aa63b31fcfcc7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:05.651518 1166 raft_consensus.cc:740] T 74873a05ef634867a30aa63b31fcfcc7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:05.651712 1166 consensus_queue.cc:260] T 74873a05ef634867a30aa63b31fcfcc7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:05.652216 1166 ts_tablet_manager.cc:1434] T 74873a05ef634867a30aa63b31fcfcc7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.025s user 0.001s sys 0.001s
I20251029 02:52:05.652482 1166 tablet_bootstrap.cc:492] T 6ca9bd37de9e4be388501b15af4bda31 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:05.652645 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_287452ecd9d54f83be8e5d9186551e3c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:05.653291 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_287452ecd9d54f83be8e5d9186551e3c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:05.653787 1166 tablet_bootstrap.cc:654] T 6ca9bd37de9e4be388501b15af4bda31 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:05.654631 526 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_8a277d5e05fb437096e36f29a6307d74" } modify_external_catalogs: true
I20251029 02:52:05.654886 526 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_8a277d5e05fb437096e36f29a6307d74" } modify_external_catalogs: true
I20251029 02:52:05.655870 1903 leader_election.cc:290] T b2b2d64dcf964da8be4b544dbbe31977 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:05.612681 1039 tablet_service.cc:1505] Processing CreateTablet for tablet ff139eb4f65343938ef0d3b854340a7d (DEFAULT_TABLE table=default.Table_905a7a1ab3ef4da9838ac7883ffa4cb1 [id=1a21f772083e4eda9283c8ac7db51d2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:05.656293 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff139eb4f65343938ef0d3b854340a7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:05.656549 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_b7096d022c484e079ae5c12c056da371"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:05.643153 1973 raft_consensus.cc:515] T da2a21c3236a4bd99872b89b12a36508 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
W20251029 02:52:05.657130 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b7096d022c484e079ae5c12c056da371 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:05.657188 1973 leader_election.cc:290] T da2a21c3236a4bd99872b89b12a36508 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:52:05.658378 521 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_13dfef4db9f244d8b98b5ff7a9c7463d" } modify_external_catalogs: true
I20251029 02:52:05.658633 521 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_13dfef4db9f244d8b98b5ff7a9c7463d" } modify_external_catalogs: true
I20251029 02:52:05.661650 742 tablet_service.cc:1552] Processing DeleteTablet for tablet 76287edc126a403dbbec4a3b41effe84 with delete_type TABLET_DATA_DELETED (replaced by tablet 00f2e98739fa41eea23927494fe222f3) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:05.643157 1906 raft_consensus.cc:515] T cedfb53078d94c81b5e294ce7438325e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:05.662428 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da2a21c3236a4bd99872b89b12a36508" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:05.662655 768 raft_consensus.cc:2468] T da2a21c3236a4bd99872b89b12a36508 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 0.
I20251029 02:52:05.662959 1906 leader_election.cc:290] T cedfb53078d94c81b5e294ce7438325e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Requested vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:05.664815 1174 ts_tablet_manager.cc:1929] T dbc2092b63e0417997565c8fadb53ae0 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:05.665000 1174 log.cc:1199] T dbc2092b63e0417997565c8fadb53ae0 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/dbc2092b63e0417997565c8fadb53ae0
I20251029 02:52:05.665380 1174 ts_tablet_manager.cc:1950] T dbc2092b63e0417997565c8fadb53ae0 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:05.665804 518 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet da2a21c3236a4bd99872b89b12a36508 (table default.Table_2a63af6894694f16b45f4bf48f73a68f [id=fb42af08ec1f48498ee5e91cce62333e]). Expected version 5 got 0
I20251029 02:52:05.666276 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 677) took 1662 ms. Trace:
I20251029 02:52:05.666390 1174 rpcz_store.cc:276] 1029 02:52:04.004077 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:04.004136 (+ 59us) service_pool.cc:225] Handling call
1029 02:52:05.584409 (+1580273us) ts_tablet_manager.cc:1151] Deleting tablet dbc2092b63e0417997565c8fadb53ae0
1029 02:52:05.584422 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:05.647511 (+ 63089us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:05.664693 (+ 17182us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:05.666206 (+ 1513us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1580108}
I20251029 02:52:05.667062 890 tablet_service.cc:1552] Processing DeleteTablet for tablet 76287edc126a403dbbec4a3b41effe84 with delete_type TABLET_DATA_DELETED (replaced by tablet 00f2e98739fa41eea23927494fe222f3) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:05.667169 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 678) took 1595 ms. Trace:
I20251029 02:52:05.667450 1174 rpcz_store.cc:276] 1029 02:52:04.072167 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:04.072235 (+ 68us) service_pool.cc:225] Handling call
1029 02:52:05.667000 (+1594765us) ts_tablet_manager.cc:1151] Deleting tablet 1e28a07b1a6242cd919b6ab2cb9b8a92
1029 02:52:05.667009 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:05.667105 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1591511}
I20251029 02:52:05.667989 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 679) took 1562 ms. Trace:
I20251029 02:52:05.668082 1174 rpcz_store.cc:276] 1029 02:52:04.105867 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:04.105942 (+ 75us) service_pool.cc:225] Handling call
1029 02:52:05.667840 (+1561898us) ts_tablet_manager.cc:1151] Deleting tablet 4929513bed104050bd6065f025303e1a
1029 02:52:05.667851 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:05.667935 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1561691}
I20251029 02:52:05.668495 1172 tablet_bootstrap.cc:492] T caef1c93e33a4390bcada59a41f43465 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:05.668537 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 680) took 1552 ms. Trace:
I20251029 02:52:05.668666 1174 rpcz_store.cc:276] 1029 02:52:04.115810 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:04.115892 (+ 82us) service_pool.cc:225] Handling call
1029 02:52:05.668408 (+1552516us) ts_tablet_manager.cc:1151] Deleting tablet 4929513bed104050bd6065f025303e1a
1029 02:52:05.668415 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:05.668485 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1552359}
I20251029 02:52:05.668779 1172 ts_tablet_manager.cc:1403] T caef1c93e33a4390bcada59a41f43465 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.181s user 0.001s sys 0.007s
I20251029 02:52:05.669130 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 681) took 1542 ms. Trace:
I20251029 02:52:05.669225 1174 rpcz_store.cc:276] 1029 02:52:04.126716 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:04.126958 (+ 242us) service_pool.cc:225] Handling call
1029 02:52:05.668991 (+1542033us) ts_tablet_manager.cc:1151] Deleting tablet 4e82c574dfa14ffbabf162a17355f841
1029 02:52:05.668999 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:05.669072 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1536141}
I20251029 02:52:05.669687 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 685) took 1505 ms. Trace:
I20251029 02:52:05.669790 1174 rpcz_store.cc:276] 1029 02:52:04.164300 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:04.164363 (+ 63us) service_pool.cc:225] Handling call
1029 02:52:05.669540 (+1505177us) ts_tablet_manager.cc:1151] Deleting tablet 67651306f3b44e7eb14dcee9a2007a81
1029 02:52:05.669549 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:05.669627 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1504987}
I20251029 02:52:05.669842 1172 raft_consensus.cc:359] T caef1c93e33a4390bcada59a41f43465 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:05.670116 1172 raft_consensus.cc:385] T caef1c93e33a4390bcada59a41f43465 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:05.670224 1172 raft_consensus.cc:740] T caef1c93e33a4390bcada59a41f43465 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:05.670372 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 687) took 1497 ms. Trace:
I20251029 02:52:05.670465 1174 rpcz_store.cc:276] 1029 02:52:04.173084 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:04.175623 (+ 2539us) service_pool.cc:225] Handling call
1029 02:52:05.670229 (+1494606us) ts_tablet_manager.cc:1151] Deleting tablet 72e91c64a9b54cbda52ca9dfcee2ce22
1029 02:52:05.670239 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:05.670316 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1494428}
I20251029 02:52:05.670605 1172 consensus_queue.cc:260] T caef1c93e33a4390bcada59a41f43465 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:05.670969 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 688) took 1497 ms. Trace:
I20251029 02:52:05.671253 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet dbc2092b63e0417997565c8fadb53ae0 (table default.Table_a873546f69334d12b0cbecce2cd489fb [id=1358e2055c254030a51e2de446eb742f]) successfully deleted
W20251029 02:52:05.671617 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 1e28a07b1a6242cd919b6ab2cb9b8a92 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1e28a07b1a6242cd919b6ab2cb9b8a92 already in progress: creating tablet
W20251029 02:52:05.671833 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 4929513bed104050bd6065f025303e1a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4929513bed104050bd6065f025303e1a already in progress: creating tablet
W20251029 02:52:05.672009 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 4929513bed104050bd6065f025303e1a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4929513bed104050bd6065f025303e1a already in progress: creating tablet
W20251029 02:52:05.672196 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 4e82c574dfa14ffbabf162a17355f841 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e82c574dfa14ffbabf162a17355f841 already in progress: creating tablet
W20251029 02:52:05.672353 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 67651306f3b44e7eb14dcee9a2007a81 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 67651306f3b44e7eb14dcee9a2007a81 already in progress: creating tablet
W20251029 02:52:05.672544 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 72e91c64a9b54cbda52ca9dfcee2ce22 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 72e91c64a9b54cbda52ca9dfcee2ce22 already in progress: creating tablet
I20251029 02:52:05.673956 1172 ts_tablet_manager.cc:1434] T caef1c93e33a4390bcada59a41f43465 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20251029 02:52:05.674328 1172 tablet_bootstrap.cc:492] T 6f35412a66b74051a475dce5dafa7792 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:05.675964 989 leader_election.cc:304] T da2a21c3236a4bd99872b89b12a36508 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters:
I20251029 02:52:05.676554 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cedfb53078d94c81b5e294ce7438325e" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:05.676659 1973 raft_consensus.cc:2804] T da2a21c3236a4bd99872b89b12a36508 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:05.676730 768 raft_consensus.cc:3060] T cedfb53078d94c81b5e294ce7438325e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:05.676731 1973 raft_consensus.cc:493] T da2a21c3236a4bd99872b89b12a36508 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:05.676796 1973 raft_consensus.cc:3060] T da2a21c3236a4bd99872b89b12a36508 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:05.679286 770 raft_consensus.cc:1275] T 42cfdf88472b44979d86836561005ed1 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Refusing update from remote peer 0cfd4bee16f84b7c8b0ff3ce14db20d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251029 02:52:05.679388 768 raft_consensus.cc:2468] T cedfb53078d94c81b5e294ce7438325e P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 1.
I20251029 02:52:05.680378 1172 tablet_bootstrap.cc:654] T 6f35412a66b74051a475dce5dafa7792 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:05.680527 1973 raft_consensus.cc:515] T da2a21c3236a4bd99872b89b12a36508 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:05.683614 742 tablet_service.cc:1505] Processing CreateTablet for tablet 636e9fe4eeab45749368cd1eb9d48b03 (DEFAULT_TABLE table=default.Table_ef95e34ffbdf43f9bbe0074cee266ac1 [id=7ef14a66267546ca92eb58eedf69e3ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:05.684080 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 636e9fe4eeab45749368cd1eb9d48b03. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:05.687347 747 tablet_service.cc:1505] Processing CreateTablet for tablet f9238d1f18a34a8588687dbb585864dc (DEFAULT_TABLE table=default.Table_0c8e6b92d9a240fead676b6cefa506cb [id=b965d60982334bf0a399bb0b089c90ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:05.687798 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9238d1f18a34a8588687dbb585864dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:05.696758 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet f81aecb20d6e498bb9b498698886d65e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:53 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:05.697288 1037 tablet_service.cc:1552] Processing DeleteTablet for tablet 6bd3fac2502e479aa9f4ceaabbb39504 with delete_type TABLET_DATA_DELETED (replaced by tablet b125cc9aaf4248b8bfa48fb152b832a6) from {username='slave'} at 127.0.0.1:37626
W20251029 02:52:05.700131 990 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:41701, user_credentials={real_user=slave}} blocked reactor thread for 61501.9us
I20251029 02:52:05.707122 1973 leader_election.cc:290] T da2a21c3236a4bd99872b89b12a36508 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Requested vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:52:05.708082 749 tablet_service.cc:1505] Processing CreateTablet for tablet 29c04a12d37b4eb49698c76fb6e124bf (DEFAULT_TABLE table=default.Table_b1b81bdd58e14acd835564ba6c9f33da [id=d5b7dda8c15843cba1d1ff34c4f54e1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:05.708706 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29c04a12d37b4eb49698c76fb6e124bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:05.622479 1176 rpcz_store.cc:276] 1029 02:52:04.134835 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:04.134899 (+ 64us) service_pool.cc:225] Handling call
1029 02:52:05.622320 (+1487421us) ts_tablet_manager.cc:1151] Deleting tablet 4929513bed104050bd6065f025303e1a
1029 02:52:05.622328 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:05.622400 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1482908}
I20251029 02:52:05.719074 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 708) took 1555 ms. Trace:
I20251029 02:52:05.719226 1176 rpcz_store.cc:276] 1029 02:52:04.163546 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:04.163592 (+ 46us) service_pool.cc:225] Handling call
1029 02:52:05.718917 (+1555325us) ts_tablet_manager.cc:1151] Deleting tablet 67651306f3b44e7eb14dcee9a2007a81
1029 02:52:05.718930 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:05.719049 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3238,"tablet-delete.queue_time_us":1555156}
I20251029 02:52:05.719689 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 711) took 1549 ms. Trace:
I20251029 02:52:05.719808 1176 rpcz_store.cc:276] 1029 02:52:04.170119 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:04.170180 (+ 61us) service_pool.cc:225] Handling call
1029 02:52:05.719576 (+1549396us) ts_tablet_manager.cc:1151] Deleting tablet 7fb5b908db77453da5cfb32e2c955f41
1029 02:52:05.719585 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:05.719674 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1546750}
I20251029 02:52:05.720170 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 710) took 1550 ms. Trace:
I20251029 02:52:05.720280 1176 rpcz_store.cc:276] 1029 02:52:04.169996 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:04.171474 (+ 1478us) service_pool.cc:225] Handling call
1029 02:52:05.720083 (+1548609us) ts_tablet_manager.cc:1151] Deleting tablet 72e91c64a9b54cbda52ca9dfcee2ce22
1029 02:52:05.720090 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:05.720156 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1533952}
I20251029 02:52:05.720628 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 712) took 1549 ms. Trace:
I20251029 02:52:05.720734 1176 rpcz_store.cc:276] 1029 02:52:04.171426 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:04.171783 (+ 357us) service_pool.cc:225] Handling call
1029 02:52:05.720529 (+1548746us) ts_tablet_manager.cc:1151] Deleting tablet caef1c93e33a4390bcada59a41f43465
1029 02:52:05.720537 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:05.720615 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1534329}
I20251029 02:52:05.721029 1176 tablet_replica.cc:333] T 7e6c4f30be624769961c9ec17c5d5dcb P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:05.721253 1176 raft_consensus.cc:2243] T 7e6c4f30be624769961c9ec17c5d5dcb P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:05.721586 1176 raft_consensus.cc:2272] T 7e6c4f30be624769961c9ec17c5d5dcb P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:05.722177 1175 ts_tablet_manager.cc:1929] T f187dbd8a55b45b292ea61db43caf47d P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:05.722208 1176 ts_tablet_manager.cc:1916] T 7e6c4f30be624769961c9ec17c5d5dcb P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:05.722301 1175 log.cc:1199] T f187dbd8a55b45b292ea61db43caf47d P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/f187dbd8a55b45b292ea61db43caf47d
I20251029 02:52:05.716385 1041 tablet_service.cc:1552] Processing DeleteTablet for tablet dd90fb9eb167478cbc3ec9363af86295 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:05 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:05.701761 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cedfb53078d94c81b5e294ce7438325e" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
I20251029 02:52:05.732694 1041 tablet_service.cc:1552] Processing DeleteTablet for tablet 71068739b4384badbf305313d902f0f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:56 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:05.733067 1031 tablet_service.cc:1505] Processing CreateTablet for tablet f9238d1f18a34a8588687dbb585864dc (DEFAULT_TABLE table=default.Table_0c8e6b92d9a240fead676b6cefa506cb [id=b965d60982334bf0a399bb0b089c90ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:05.733526 989 leader_election.cc:304] T cedfb53078d94c81b5e294ce7438325e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters:
I20251029 02:52:05.734403 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9238d1f18a34a8588687dbb585864dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:05.734606 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da2a21c3236a4bd99872b89b12a36508" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:05.734788 768 raft_consensus.cc:3060] T da2a21c3236a4bd99872b89b12a36508 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:05.737447 768 raft_consensus.cc:2468] T da2a21c3236a4bd99872b89b12a36508 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 1.
I20251029 02:52:05.738101 1172 tablet_bootstrap.cc:492] T 6f35412a66b74051a475dce5dafa7792 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:05.738252 1172 ts_tablet_manager.cc:1403] T 6f35412a66b74051a475dce5dafa7792 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.064s user 0.002s sys 0.000s
I20251029 02:52:05.738113 1906 consensus_queue.cc:1048] T 42cfdf88472b44979d86836561005ed1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:05.738835 1906 raft_consensus.cc:2804] T cedfb53078d94c81b5e294ce7438325e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:05.738957 1172 raft_consensus.cc:359] T 6f35412a66b74051a475dce5dafa7792 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:05.739112 1172 raft_consensus.cc:385] T 6f35412a66b74051a475dce5dafa7792 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:05.739156 1172 raft_consensus.cc:740] T 6f35412a66b74051a475dce5dafa7792 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:05.739244 1906 raft_consensus.cc:697] T cedfb53078d94c81b5e294ce7438325e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Becoming Leader. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Running, Role: LEADER
I20251029 02:52:05.739351 1172 consensus_queue.cc:260] T 6f35412a66b74051a475dce5dafa7792 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:05.739611 1906 consensus_queue.cc:237] T cedfb53078d94c81b5e294ce7438325e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:05.741818 1039 tablet_service.cc:1505] Processing CreateTablet for tablet 29c04a12d37b4eb49698c76fb6e124bf (DEFAULT_TABLE table=default.Table_b1b81bdd58e14acd835564ba6c9f33da [id=d5b7dda8c15843cba1d1ff34c4f54e1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:05.743508 989 leader_election.cc:304] T da2a21c3236a4bd99872b89b12a36508 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters:
I20251029 02:52:05.743981 1973 raft_consensus.cc:2804] T da2a21c3236a4bd99872b89b12a36508 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:05.744226 1973 raft_consensus.cc:697] T da2a21c3236a4bd99872b89b12a36508 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Becoming Leader. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Running, Role: LEADER
I20251029 02:52:05.744501 1973 consensus_queue.cc:237] T da2a21c3236a4bd99872b89b12a36508 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
W20251029 02:52:05.745925 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 4929513bed104050bd6065f025303e1a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4929513bed104050bd6065f025303e1a already in progress: creating tablet
W20251029 02:52:05.746248 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 67651306f3b44e7eb14dcee9a2007a81 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 67651306f3b44e7eb14dcee9a2007a81 already in progress: creating tablet
W20251029 02:52:05.746573 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 7fb5b908db77453da5cfb32e2c955f41 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7fb5b908db77453da5cfb32e2c955f41 already in progress: creating tablet
W20251029 02:52:05.746832 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 72e91c64a9b54cbda52ca9dfcee2ce22 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 72e91c64a9b54cbda52ca9dfcee2ce22 already in progress: creating tablet
W20251029 02:52:05.747117 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet caef1c93e33a4390bcada59a41f43465 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet caef1c93e33a4390bcada59a41f43465 already in progress: creating tablet
W20251029 02:52:05.754868 519 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 372d93914abc41b38a702742b7db0bc2 consensus state has no leader
I20251029 02:52:05.756907 519 master_service.cc:954] ReplaceTablet: received request to replace tablet 59d03a190b784ab18a3e63a572675891 from {username='slave'} at 127.0.0.1:35314
I20251029 02:52:05.757575 524 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 8f100a12d78145c2b8586a0713fc70c4
I20251029 02:52:05.758206 523 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 1478eb9c72a64d99aa273632926f7b99
I20251029 02:52:05.758373 1031 tablet_service.cc:1552] Processing DeleteTablet for tablet 74873a05ef634867a30aa63b31fcfcc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:54 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:05.758946 526 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 72f9d4408f384e7093316d85f092e1b2
I20251029 02:52:05.759054 1174 rpcz_store.cc:276] 1029 02:52:04.173168 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:04.175836 (+ 2668us) service_pool.cc:225] Handling call
1029 02:52:05.670836 (+1495000us) ts_tablet_manager.cc:1151] Deleting tablet 7fb5b908db77453da5cfb32e2c955f41
1029 02:52:05.670846 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:05.670916 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1494801}
I20251029 02:52:05.761075 917 raft_consensus.cc:3060] T cedfb53078d94c81b5e294ce7438325e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:05.761193 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da2a21c3236a4bd99872b89b12a36508" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:05.762786 1903 raft_consensus.cc:493] T c9dff4dff82f49e7ad49caad5e404f09 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251029 02:52:05.766495 990 leader_election.cc:343] T da2a21c3236a4bd99872b89b12a36508 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:05.766723 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet 72f9d4408f384e7093316d85f092e1b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:05 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:05.767109 1903 raft_consensus.cc:515] T c9dff4dff82f49e7ad49caad5e404f09 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
W20251029 02:52:05.767463 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 7fb5b908db77453da5cfb32e2c955f41 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7fb5b908db77453da5cfb32e2c955f41 already in progress: creating tablet
I20251029 02:52:05.767958 892 tablet_service.cc:1505] Processing CreateTablet for tablet f9238d1f18a34a8588687dbb585864dc (DEFAULT_TABLE table=default.Table_0c8e6b92d9a240fead676b6cefa506cb [id=b965d60982334bf0a399bb0b089c90ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:05.768379 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9dff4dff82f49e7ad49caad5e404f09" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:05.768790 770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9dff4dff82f49e7ad49caad5e404f09" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:05.769035 770 raft_consensus.cc:2393] T c9dff4dff82f49e7ad49caad5e404f09 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c756528642a543eeacd93cef3e335a7d in current term 1: Already voted for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in this term.
I20251029 02:52:05.770323 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29c04a12d37b4eb49698c76fb6e124bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:05.771278 917 raft_consensus.cc:2468] T cedfb53078d94c81b5e294ce7438325e P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 1.
I20251029 02:52:05.771384 918 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da2a21c3236a4bd99872b89b12a36508" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
I20251029 02:52:05.771692 1031 tablet_service.cc:1552] Processing DeleteTablet for tablet 8f100a12d78145c2b8586a0713fc70c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:05 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:05.771282 890 tablet_service.cc:1505] Processing CreateTablet for tablet 29c04a12d37b4eb49698c76fb6e124bf (DEFAULT_TABLE table=default.Table_b1b81bdd58e14acd835564ba6c9f33da [id=d5b7dda8c15843cba1d1ff34c4f54e1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:05.776068 847 leader_election.cc:304] T c9dff4dff82f49e7ad49caad5e404f09 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9
I20251029 02:52:05.776439 885 tablet_service.cc:1552] Processing DeleteTablet for tablet 8f100a12d78145c2b8586a0713fc70c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:05 UTC) from {username='slave'} at 127.0.0.1:35702
W20251029 02:52:05.777940 990 leader_election.cc:343] T da2a21c3236a4bd99872b89b12a36508 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:05.778911 1773 raft_consensus.cc:3060] T c9dff4dff82f49e7ad49caad5e404f09 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:05.781824 920 raft_consensus.cc:1275] T 42cfdf88472b44979d86836561005ed1 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Refusing update from remote peer 0cfd4bee16f84b7c8b0ff3ce14db20d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251029 02:52:05.782827 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9238d1f18a34a8588687dbb585864dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:05.783576 518 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_cfb1df2e2beb438baa234ac0e7f3b36b" } new_table_name: "default.Table_464b6de22be542b89e56489d27d2d521" modify_external_catalogs: true
I20251029 02:52:05.785504 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_5557b4fcc4f34c0cacd4af39e9d24e35"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:05.786157 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5557b4fcc4f34c0cacd4af39e9d24e35 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:05.786542 1903 leader_election.cc:290] T c9dff4dff82f49e7ad49caad5e404f09 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:05.789829 885 tablet_service.cc:1552] Processing DeleteTablet for tablet 72f9d4408f384e7093316d85f092e1b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:05 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:05.790092 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 689) took 1586 ms. Trace:
I20251029 02:52:05.790225 1174 rpcz_store.cc:276] 1029 02:52:04.173284 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:04.176072 (+ 2788us) service_pool.cc:225] Handling call
1029 02:52:05.759369 (+1583297us) ts_tablet_manager.cc:1151] Deleting tablet caef1c93e33a4390bcada59a41f43465
1029 02:52:05.759383 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:05.759505 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1583110}
I20251029 02:52:05.790696 1174 tablet_replica.cc:333] T 1f5b40c0403d4a49a4bfde222d8c947a P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:05.790987 1174 raft_consensus.cc:2243] T 1f5b40c0403d4a49a4bfde222d8c947a P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:05.794054 1166 tablet_bootstrap.cc:492] T 6ca9bd37de9e4be388501b15af4bda31 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:05.794239 1166 ts_tablet_manager.cc:1403] T 6ca9bd37de9e4be388501b15af4bda31 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.142s user 0.001s sys 0.001s
I20251029 02:52:05.795178 1166 raft_consensus.cc:359] T 6ca9bd37de9e4be388501b15af4bda31 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:05.795382 1166 raft_consensus.cc:385] T 6ca9bd37de9e4be388501b15af4bda31 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:05.795457 1166 raft_consensus.cc:740] T 6ca9bd37de9e4be388501b15af4bda31 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:05.793537 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29c04a12d37b4eb49698c76fb6e124bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:05.795845 1166 consensus_queue.cc:260] T 6ca9bd37de9e4be388501b15af4bda31 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:05.796343 1167 tablet_bootstrap.cc:492] T 74873a05ef634867a30aa63b31fcfcc7 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:05.796546 1166 ts_tablet_manager.cc:1434] T 6ca9bd37de9e4be388501b15af4bda31 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20251029 02:52:05.796864 1166 tablet_bootstrap.cc:492] T f246f6c202cf44d48c9f5e4f335202c5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:05.793547 1773 raft_consensus.cc:2749] T c9dff4dff82f49e7ad49caad5e404f09 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:05.791258 1174 raft_consensus.cc:2272] T 1f5b40c0403d4a49a4bfde222d8c947a P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:05.798835 1166 tablet_bootstrap.cc:654] T f246f6c202cf44d48c9f5e4f335202c5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:05.800426 522 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_a4fcd46aef6146e986d4b8d3599d66f4" } new_table_name: "default.Table_3c0a0c360c4545dab0a663809e2d0acc" modify_external_catalogs: true
I20251029 02:52:05.800494 1176 ts_tablet_manager.cc:1929] T 7e6c4f30be624769961c9ec17c5d5dcb P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:05.800624 1176 log.cc:1199] T 7e6c4f30be624769961c9ec17c5d5dcb P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/7e6c4f30be624769961c9ec17c5d5dcb
I20251029 02:52:05.800937 1176 ts_tablet_manager.cc:1950] T 7e6c4f30be624769961c9ec17c5d5dcb P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:05.801249 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 713) took 1618 ms. Trace:
I20251029 02:52:05.801324 1176 rpcz_store.cc:276] 1029 02:52:04.182750 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:04.186222 (+ 3472us) service_pool.cc:225] Handling call
1029 02:52:05.720976 (+1534754us) ts_tablet_manager.cc:1151] Deleting tablet 7e6c4f30be624769961c9ec17c5d5dcb
1029 02:52:05.720984 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:05.741051 (+ 20067us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:05.800379 (+ 59328us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:05.801231 (+ 852us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1534599}
I20251029 02:52:05.801791 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 715) took 1540 ms. Trace:
I20251029 02:52:05.801877 1176 rpcz_store.cc:276] 1029 02:52:04.261048 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:04.261164 (+ 116us) service_pool.cc:225] Handling call
1029 02:52:05.801646 (+1540482us) ts_tablet_manager.cc:1151] Deleting tablet 78662eeae70b4148a4ad0d47be5c566d
1029 02:52:05.801658 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:05.801773 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1540231}
I20251029 02:52:05.802218 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 717) took 1477 ms. Trace:
I20251029 02:52:05.802296 1176 rpcz_store.cc:276] 1029 02:52:04.325158 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:04.325210 (+ 52us) service_pool.cc:225] Handling call
1029 02:52:05.802107 (+1476897us) ts_tablet_manager.cc:1151] Deleting tablet de7fac8751a04c19b68a52db7f167214
1029 02:52:05.802117 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:05.802202 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1466058}
I20251029 02:52:05.802472 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_9037a876e81e47ec96334f1dc2022e2b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:05.803042 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9037a876e81e47ec96334f1dc2022e2b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:05.804737 525 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_982d01dccabe47bfaac7a582282c5d54" } modify_external_catalogs: true
I20251029 02:52:05.804680 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_3bd4c012655a48aa970875bb07006ac2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:05.804901 525 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_982d01dccabe47bfaac7a582282c5d54" } modify_external_catalogs: true
I20251029 02:52:05.804837 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_ec7e5a0085cd43e28618d7b9d3d30c65"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:05.805251 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3bd4c012655a48aa970875bb07006ac2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:52:05.805397 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ec7e5a0085cd43e28618d7b9d3d30c65 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:05.807809 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 7e6c4f30be624769961c9ec17c5d5dcb (table default.Table_dea0766540d24b5e9bd6cc8b6338b47c [id=1dfd5b42973c4d5d91e0d9014e34269f]) successfully deleted
I20251029 02:52:05.808444 1906 consensus_queue.cc:1048] T 42cfdf88472b44979d86836561005ed1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251029 02:52:05.810540 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet de7fac8751a04c19b68a52db7f167214 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet de7fac8751a04c19b68a52db7f167214 already in progress: creating tablet
I20251029 02:52:05.812822 1973 raft_consensus.cc:493] T c9e923724ca9402682f13b252dbc37aa P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:05.813035 1973 raft_consensus.cc:515] T c9e923724ca9402682f13b252dbc37aa P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:05.813836 1973 leader_election.cc:290] T c9e923724ca9402682f13b252dbc37aa P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:52:05.814872 1166 tablet_bootstrap.cc:492] T f246f6c202cf44d48c9f5e4f335202c5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:05.815047 1166 ts_tablet_manager.cc:1403] T f246f6c202cf44d48c9f5e4f335202c5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.000s
I20251029 02:52:05.816805 770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9e923724ca9402682f13b252dbc37aa" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:05.816991 770 raft_consensus.cc:2468] T c9e923724ca9402682f13b252dbc37aa P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 0.
I20251029 02:52:05.819355 1957 raft_consensus.cc:493] T 4add552cf4d940d2a0a36d6b58379a34 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:05.819473 1957 raft_consensus.cc:515] T 4add552cf4d940d2a0a36d6b58379a34 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:05.820019 1957 leader_election.cc:290] T 4add552cf4d940d2a0a36d6b58379a34 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:05.816052 1166 raft_consensus.cc:359] T f246f6c202cf44d48c9f5e4f335202c5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:05.822582 1166 raft_consensus.cc:385] T f246f6c202cf44d48c9f5e4f335202c5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:05.822669 1166 raft_consensus.cc:740] T f246f6c202cf44d48c9f5e4f335202c5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:05.823161 1166 consensus_queue.cc:260] T f246f6c202cf44d48c9f5e4f335202c5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:05.823746 1166 ts_tablet_manager.cc:1434] T f246f6c202cf44d48c9f5e4f335202c5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20251029 02:52:05.830013 1166 tablet_bootstrap.cc:492] T 0cc1957a845f4e9cb703a553b27d1347 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:05.831085 519 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_290338e0424d46cf9166ed27b1c765a1" } new_table_name: "default.Table_1852aaabf5044b05901236546f66f88c" modify_external_catalogs: true
I20251029 02:52:05.836884 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 720) took 1490 ms. Trace:
I20251029 02:52:05.836980 1176 rpcz_store.cc:276] 1029 02:52:04.346150 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:04.346216 (+ 66us) service_pool.cc:225] Handling call
1029 02:52:05.836725 (+1490509us) ts_tablet_manager.cc:1151] Deleting tablet 1e17142dc6634db0a5288add9c199ffc
1029 02:52:05.836749 (+ 24us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:05.836860 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1456115}
I20251029 02:52:05.838308 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 721) took 1484 ms. Trace:
I20251029 02:52:05.838408 1176 rpcz_store.cc:276] 1029 02:52:04.354282 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:04.358623 (+ 4341us) service_pool.cc:225] Handling call
1029 02:52:05.838156 (+1479533us) ts_tablet_manager.cc:1151] Deleting tablet c4b1ef53dbaa47c39361cc4ef3be71d8
1029 02:52:05.838171 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:05.838286 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1479336}
I20251029 02:52:05.838807 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 727) took 1034 ms. Trace:
I20251029 02:52:05.838896 1176 rpcz_store.cc:276] 1029 02:52:04.804009 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:04.804058 (+ 49us) service_pool.cc:225] Handling call
1029 02:52:05.838693 (+1034635us) ts_tablet_manager.cc:1151] Deleting tablet ba699862e6f646c6beaa39f080d0faba
1029 02:52:05.838704 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:05.838791 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1030082}
I20251029 02:52:05.839577 1176 tablet_replica.cc:333] T 576c4e6e70e24b0399ac9597a30f0569 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:05.839819 1176 raft_consensus.cc:2243] T 576c4e6e70e24b0399ac9597a30f0569 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Raft consensus shutting down.
W20251029 02:52:05.841053 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet caef1c93e33a4390bcada59a41f43465 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet caef1c93e33a4390bcada59a41f43465 already in progress: creating tablet
I20251029 02:52:05.843235 1176 pending_rounds.cc:70] T 576c4e6e70e24b0399ac9597a30f0569 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Trying to abort 1 pending ops.
I20251029 02:52:05.846235 749 tablet_service.cc:1552] Processing DeleteTablet for tablet 8f100a12d78145c2b8586a0713fc70c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:05 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:05.846622 1176 pending_rounds.cc:77] T 576c4e6e70e24b0399ac9597a30f0569 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215949109206577152 op_type: NO_OP noop_request { }
I20251029 02:52:05.846791 1176 raft_consensus.cc:2889] T 576c4e6e70e24b0399ac9597a30f0569 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 02:52:05.846951 1176 raft_consensus.cc:2272] T 576c4e6e70e24b0399ac9597a30f0569 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:05.849596 1172 ts_tablet_manager.cc:1434] T 6f35412a66b74051a475dce5dafa7792 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.111s user 0.002s sys 0.000s
I20251029 02:52:05.852048 1175 ts_tablet_manager.cc:1950] T f187dbd8a55b45b292ea61db43caf47d P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:05.852984 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 651) took 2260 ms. Trace:
I20251029 02:52:05.853374 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4add552cf4d940d2a0a36d6b58379a34" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:05.855363 1166 tablet_bootstrap.cc:654] T 0cc1957a845f4e9cb703a553b27d1347 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:05.855666 1176 ts_tablet_manager.cc:1916] T 576c4e6e70e24b0399ac9597a30f0569 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:05.867156 1175 rpcz_store.cc:276] 1029 02:52:03.592832 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:03.593005 (+ 173us) service_pool.cc:225] Handling call
1029 02:52:05.553514 (+1960509us) ts_tablet_manager.cc:1151] Deleting tablet f187dbd8a55b45b292ea61db43caf47d
1029 02:52:05.553524 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:05.719114 (+165590us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:05.722081 (+ 2967us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:05.852960 (+130879us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1925260}
I20251029 02:52:05.867941 517 catalog_manager.cc:5649] T cedfb53078d94c81b5e294ce7438325e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } health_report { overall_health: UNKNOWN } } }
I20251029 02:52:05.868499 517 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet cedfb53078d94c81b5e294ce7438325e (table default.Table_e75db5cf8f5f49ea837cb6d63f4b7ebf [id=86195563fb4a4d7abd746f4ac7cd241c]). Expected version 1 got 0
I20251029 02:52:05.869194 517 catalog_manager.cc:5649] T da2a21c3236a4bd99872b89b12a36508 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } health_report { overall_health: HEALTHY } } }
I20251029 02:52:05.869602 517 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet da2a21c3236a4bd99872b89b12a36508 (table default.Table_2a63af6894694f16b45f4bf48f73a68f [id=fb42af08ec1f48498ee5e91cce62333e]). Expected version 5 got 0
I20251029 02:52:05.853901 989 leader_election.cc:304] T c9e923724ca9402682f13b252dbc37aa P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters:
I20251029 02:52:05.875339 1166 tablet_bootstrap.cc:492] T 0cc1957a845f4e9cb703a553b27d1347 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:05.875367 521 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 143f4a42001b43e683bbc6226c0550a9
I20251029 02:52:05.875658 1166 ts_tablet_manager.cc:1403] T 0cc1957a845f4e9cb703a553b27d1347 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.052s user 0.004s sys 0.001s
I20251029 02:52:05.876204 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_6a444f431d374afa8f25c801ef2a92d7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:05.876830 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6a444f431d374afa8f25c801ef2a92d7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:05.877600 1166 raft_consensus.cc:359] T 0cc1957a845f4e9cb703a553b27d1347 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:05.877921 1166 raft_consensus.cc:385] T 0cc1957a845f4e9cb703a553b27d1347 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:05.878042 1166 raft_consensus.cc:740] T 0cc1957a845f4e9cb703a553b27d1347 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:05.878407 1166 consensus_queue.cc:260] T 0cc1957a845f4e9cb703a553b27d1347 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:05.880375 1166 ts_tablet_manager.cc:1434] T 0cc1957a845f4e9cb703a553b27d1347 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
W20251029 02:52:05.881014 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 1e17142dc6634db0a5288add9c199ffc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1e17142dc6634db0a5288add9c199ffc already in progress: creating tablet
W20251029 02:52:05.881672 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet c4b1ef53dbaa47c39361cc4ef3be71d8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c4b1ef53dbaa47c39361cc4ef3be71d8 already in progress: creating tablet
W20251029 02:52:05.882162 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet ba699862e6f646c6beaa39f080d0faba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ba699862e6f646c6beaa39f080d0faba already in progress: creating tablet
W20251029 02:52:05.882686 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 738046ab57334f2dab9f4aab979c5b42 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 738046ab57334f2dab9f4aab979c5b42 already in progress: creating tablet
W20251029 02:52:05.883164 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet c081d2eacfac4ca8a0048e2a1c4f8f83 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c081d2eacfac4ca8a0048e2a1c4f8f83 already in progress: creating tablet
W20251029 02:52:05.883672 513 catalog_manager.cc:4952] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 3cded156380f45cc95893825c71ba0e9 because the tablet was not found. No further retry: Not found: Tablet not found: 3cded156380f45cc95893825c71ba0e9
I20251029 02:52:05.854493 1172 tablet_bootstrap.cc:492] T dfa8b1b9a80e4e8689e0acb44e16346a P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:05.861663 2018 raft_consensus.cc:493] T 1bdb67125493430398ff5c80b73de2ac P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:05.885231 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet f187dbd8a55b45b292ea61db43caf47d (table default.Table_d6a9ff6b4cea442eb2394340301eac96 [id=3f2d90da0f88434592a21c0687450ded]) successfully deleted
I20251029 02:52:05.885224 2018 raft_consensus.cc:515] T 1bdb67125493430398ff5c80b73de2ac P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:05.885967 2018 leader_election.cc:290] T 1bdb67125493430398ff5c80b73de2ac P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:05.854096 748 tablet_service.cc:1552] Processing DeleteTablet for tablet 72f9d4408f384e7093316d85f092e1b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:05 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:05.876533 749 tablet_service.cc:1552] Processing DeleteTablet for tablet 143f4a42001b43e683bbc6226c0550a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:05 UTC) from {username='slave'} at 127.0.0.1:44580
W20251029 02:52:05.884454 712 leader_election.cc:343] T 4add552cf4d940d2a0a36d6b58379a34 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:05.884646 1039 tablet_service.cc:1552] Processing DeleteTablet for tablet 143f4a42001b43e683bbc6226c0550a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:05 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:05.884794 1906 raft_consensus.cc:2804] T c9e923724ca9402682f13b252dbc37aa P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:05.890566 1906 raft_consensus.cc:493] T c9e923724ca9402682f13b252dbc37aa P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:05.890715 1906 raft_consensus.cc:3060] T c9e923724ca9402682f13b252dbc37aa P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:05.894948 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 664) took 2250 ms. Trace:
I20251029 02:52:05.895068 1175 rpcz_store.cc:276] 1029 02:52:03.644660 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:03.644735 (+ 75us) service_pool.cc:225] Handling call
1029 02:52:05.894759 (+2250024us) ts_tablet_manager.cc:1151] Deleting tablet ae5a6caeb17f4851b27a3c6a3c0db83c
1029 02:52:05.894777 (+ 18us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:05.894919 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2236428}
W20251029 02:52:05.895699 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet ae5a6caeb17f4851b27a3c6a3c0db83c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ae5a6caeb17f4851b27a3c6a3c0db83c already in progress: creating tablet
I20251029 02:52:05.896351 1166 tablet_bootstrap.cc:492] T e1fa4980f2624321b617b84c68b90113 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:05.896391 1172 tablet_bootstrap.cc:654] T dfa8b1b9a80e4e8689e0acb44e16346a P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:05.903100 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1bdb67125493430398ff5c80b73de2ac" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:52:05.908946 712 leader_election.cc:343] T 1bdb67125493430398ff5c80b73de2ac P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:05.903445 1906 raft_consensus.cc:515] T c9e923724ca9402682f13b252dbc37aa P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:05.926281 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 666) took 2268 ms. Trace:
I20251029 02:52:05.926393 1175 rpcz_store.cc:276] 1029 02:52:03.657514 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:03.658372 (+ 858us) service_pool.cc:225] Handling call
1029 02:52:05.926128 (+2267756us) ts_tablet_manager.cc:1151] Deleting tablet ce98420e61904af78d8fa03724c74489
1029 02:52:05.926138 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:05.926253 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2267550}
W20251029 02:52:05.926813 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet ce98420e61904af78d8fa03724c74489 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ce98420e61904af78d8fa03724c74489 already in progress: creating tablet
I20251029 02:52:05.926898 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 665) took 2281 ms. Trace:
I20251029 02:52:05.926975 1175 rpcz_store.cc:276] 1029 02:52:03.645478 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:03.653309 (+ 7831us) service_pool.cc:225] Handling call
1029 02:52:05.926778 (+2273469us) ts_tablet_manager.cc:1151] Deleting tablet 70b008011d5547bbaf3f45685394139f
1029 02:52:05.926787 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:05.926879 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2263526}
W20251029 02:52:05.927328 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 70b008011d5547bbaf3f45685394139f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 70b008011d5547bbaf3f45685394139f already in progress: creating tablet
I20251029 02:52:05.927443 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 667) took 2225 ms. Trace:
I20251029 02:52:05.927507 1175 rpcz_store.cc:276] 1029 02:52:03.701645 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:03.715636 (+ 13991us) service_pool.cc:225] Handling call
1029 02:52:05.927341 (+2211705us) ts_tablet_manager.cc:1151] Deleting tablet bc16262db49e4fc69fe1a4f95dbde2cb
1029 02:52:05.927350 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:05.927428 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2211348}
W20251029 02:52:05.927814 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet bc16262db49e4fc69fe1a4f95dbde2cb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bc16262db49e4fc69fe1a4f95dbde2cb already in progress: creating tablet
I20251029 02:52:05.927845 1175 tablet_replica.cc:333] T dd6ad641fbda472fab07aa35c5e4523a P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:05.928079 1175 raft_consensus.cc:2243] T dd6ad641fbda472fab07aa35c5e4523a P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:05.928260 1175 raft_consensus.cc:2272] T dd6ad641fbda472fab07aa35c5e4523a P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:05.927851 1172 tablet_bootstrap.cc:492] T dfa8b1b9a80e4e8689e0acb44e16346a P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:05.928650 1172 ts_tablet_manager.cc:1403] T dfa8b1b9a80e4e8689e0acb44e16346a P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.074s user 0.003s sys 0.000s
W20251029 02:52:05.931483 518 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7ff2399833e546448b85b9caeb5200fb consensus state has no leader
I20251029 02:52:05.932830 900 tablet_service.cc:1552] Processing DeleteTablet for tablet 143f4a42001b43e683bbc6226c0550a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:05 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:05.933267 1175 ts_tablet_manager.cc:1916] T dd6ad641fbda472fab07aa35c5e4523a P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:05.933274 518 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet 42cfdf88472b44979d86836561005ed1 (table default.Table_ad2ac3ba98784d68a94eb4eb03e530a4 [id=5e0c72957b9a491d960b33d3fffd8f70]). Expected version 2 got 0
I20251029 02:52:05.934011 518 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_c5b59d221b994982a9867326266b7bcd" } new_table_name: "default.Table_dd6aaf5ed87548009b3025e430874d1e" modify_external_catalogs: true
I20251029 02:52:05.929610 1172 raft_consensus.cc:359] T dfa8b1b9a80e4e8689e0acb44e16346a P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:05.934367 1172 raft_consensus.cc:385] T dfa8b1b9a80e4e8689e0acb44e16346a P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:05.934469 1172 raft_consensus.cc:740] T dfa8b1b9a80e4e8689e0acb44e16346a P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:05.934486 1973 raft_consensus.cc:493] T 71068739b4384badbf305313d902f0f5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:05.934667 1973 raft_consensus.cc:515] T 71068739b4384badbf305313d902f0f5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:05.934782 1172 consensus_queue.cc:260] T dfa8b1b9a80e4e8689e0acb44e16346a P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:05.934846 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_27e1e56b7ff0486facdfdd318dc845fc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:05.935290 1973 leader_election.cc:290] T 71068739b4384badbf305313d902f0f5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
W20251029 02:52:05.935439 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_27e1e56b7ff0486facdfdd318dc845fc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:05.935704 1172 ts_tablet_manager.cc:1434] T dfa8b1b9a80e4e8689e0acb44e16346a P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20251029 02:52:05.935971 1172 tablet_bootstrap.cc:492] T 0f80b840ed954230bfbfe7e40127cd2f P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
W20251029 02:52:05.937127 522 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet da904606986f452cadd06187fff06785 consensus state has no leader
I20251029 02:52:05.937371 1172 tablet_bootstrap.cc:654] T 0f80b840ed954230bfbfe7e40127cd2f P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:05.937378 522 master_service.cc:954] ReplaceTablet: received request to replace tablet 59af906061d049c99ae2153945597007 from {username='slave'} at 127.0.0.1:35314
I20251029 02:52:05.939072 1973 raft_consensus.cc:493] T 8df97119d0654d7d92bb5bd17c5d8562 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:05.939803 770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "71068739b4384badbf305313d902f0f5" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:05.939815 1973 raft_consensus.cc:515] T 8df97119d0654d7d92bb5bd17c5d8562 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:05.939970 770 raft_consensus.cc:2468] T 71068739b4384badbf305313d902f0f5 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 0.
I20251029 02:52:05.940353 1973 leader_election.cc:290] T 8df97119d0654d7d92bb5bd17c5d8562 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:05.940339 1174 ts_tablet_manager.cc:1916] T 1f5b40c0403d4a49a4bfde222d8c947a P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:05.898648 1166 tablet_bootstrap.cc:654] T e1fa4980f2624321b617b84c68b90113 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:05.941831 1906 leader_election.cc:290] T c9e923724ca9402682f13b252dbc37aa P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Requested vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:52:05.945690 1176 ts_tablet_manager.cc:1929] T 576c4e6e70e24b0399ac9597a30f0569 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:05.945817 766 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9e923724ca9402682f13b252dbc37aa" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:05.946007 766 raft_consensus.cc:3060] T c9e923724ca9402682f13b252dbc37aa P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:05.946822 1175 ts_tablet_manager.cc:1929] T dd6ad641fbda472fab07aa35c5e4523a P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:05.946930 1175 log.cc:1199] T dd6ad641fbda472fab07aa35c5e4523a P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/dd6ad641fbda472fab07aa35c5e4523a
I20251029 02:52:05.947197 1175 ts_tablet_manager.cc:1950] T dd6ad641fbda472fab07aa35c5e4523a P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:05.948212 525 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet ad214c870ee649279ed7321c5b69cbee
I20251029 02:52:05.949397 749 tablet_service.cc:1552] Processing DeleteTablet for tablet ad214c870ee649279ed7321c5b69cbee with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:05 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:05.949486 766 raft_consensus.cc:2468] T c9e923724ca9402682f13b252dbc37aa P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 1.
I20251029 02:52:05.954790 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 668) took 2088 ms. Trace:
I20251029 02:52:05.954890 1175 rpcz_store.cc:276] 1029 02:52:03.865968 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:03.866046 (+ 78us) service_pool.cc:225] Handling call
1029 02:52:05.927805 (+2061759us) ts_tablet_manager.cc:1151] Deleting tablet dd6ad641fbda472fab07aa35c5e4523a
1029 02:52:05.927812 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:05.943670 (+ 15858us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:05.946739 (+ 3069us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:05.954769 (+ 8030us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2061563}
I20251029 02:52:05.955320 1175 tablet_replica.cc:333] T 519754f5d8f7484480f973c53708c5f3 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:05.955334 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet dd6ad641fbda472fab07aa35c5e4523a (table default.Table_29766c136dbc44e486d4256e7dbba851 [id=1d39d6caa9e24fe38b5cf9a20a6c66b9]) successfully deleted
I20251029 02:52:05.955497 1175 raft_consensus.cc:2243] T 519754f5d8f7484480f973c53708c5f3 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:05.955642 1175 raft_consensus.cc:2272] T 519754f5d8f7484480f973c53708c5f3 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:05.956190 1175 ts_tablet_manager.cc:1916] T 519754f5d8f7484480f973c53708c5f3 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:05.958030 1773 raft_consensus.cc:493] T c9e923724ca9402682f13b252dbc37aa P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:05.958237 1773 raft_consensus.cc:515] T c9e923724ca9402682f13b252dbc37aa P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:05.958853 1174 ts_tablet_manager.cc:1929] T 1f5b40c0403d4a49a4bfde222d8c947a P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:05.960016 766 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9e923724ca9402682f13b252dbc37aa" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:05.962335 524 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_23772dc0c0cf4ab08ba7f164661de878" } new_table_name: "default.Table_8e43b2bad80b45ceb71aada04cdbc4b8" modify_external_catalogs: true
I20251029 02:52:05.962639 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_c0fb3766a9c04aa9b2e3caca2e7b9b3a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:05.963281 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c0fb3766a9c04aa9b2e3caca2e7b9b3a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:05.963500 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_603240f584a74f1d965ee97ee1a2d2dc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:05.963987 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_603240f584a74f1d965ee97ee1a2d2dc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:05.964717 766 raft_consensus.cc:2393] T c9e923724ca9402682f13b252dbc37aa P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c756528642a543eeacd93cef3e335a7d in current term 1: Already voted for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in this term.
I20251029 02:52:05.959002 1773 leader_election.cc:290] T c9e923724ca9402682f13b252dbc37aa P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:05.959121 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9e923724ca9402682f13b252dbc37aa" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:05.960062 918 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4add552cf4d940d2a0a36d6b58379a34" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:05.960098 1903 raft_consensus.cc:493] T 8df97119d0654d7d92bb5bd17c5d8562 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:05.966164 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1bdb67125493430398ff5c80b73de2ac" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:05.966225 919 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "71068739b4384badbf305313d902f0f5" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:05.970357 1176 log.cc:1199] T 576c4e6e70e24b0399ac9597a30f0569 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/576c4e6e70e24b0399ac9597a30f0569
W20251029 02:52:05.970342 711 leader_election.cc:343] T 4add552cf4d940d2a0a36d6b58379a34 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:05.945941 989 leader_election.cc:304] T 71068739b4384badbf305313d902f0f5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters:
I20251029 02:52:05.970469 711 leader_election.cc:304] T 4add552cf4d940d2a0a36d6b58379a34 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
W20251029 02:52:05.970912 711 leader_election.cc:343] T 1bdb67125493430398ff5c80b73de2ac P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:05.971004 711 leader_election.cc:304] T 1bdb67125493430398ff5c80b73de2ac P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
W20251029 02:52:05.971129 989 leader_election.cc:343] T 8df97119d0654d7d92bb5bd17c5d8562 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 8df97119d0654d7d92bb5bd17c5d8562
I20251029 02:52:05.971141 1039 tablet_service.cc:1552] Processing DeleteTablet for tablet ad214c870ee649279ed7321c5b69cbee with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:05 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:05.971318 2018 raft_consensus.cc:2749] T 4add552cf4d940d2a0a36d6b58379a34 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:05.971336 989 leader_election.cc:304] T c9e923724ca9402682f13b252dbc37aa P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters:
I20251029 02:52:05.971475 2018 raft_consensus.cc:2749] T 1bdb67125493430398ff5c80b73de2ac P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:05.953810 1166 tablet_bootstrap.cc:492] T e1fa4980f2624321b617b84c68b90113 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:05.971706 1166 ts_tablet_manager.cc:1403] T e1fa4980f2624321b617b84c68b90113 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.075s user 0.003s sys 0.002s
I20251029 02:52:05.972630 1166 raft_consensus.cc:359] T e1fa4980f2624321b617b84c68b90113 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:05.972851 1166 raft_consensus.cc:385] T e1fa4980f2624321b617b84c68b90113 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:05.972913 1166 raft_consensus.cc:740] T e1fa4980f2624321b617b84c68b90113 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:05.973181 1166 consensus_queue.cc:260] T e1fa4980f2624321b617b84c68b90113 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:05.973941 1166 ts_tablet_manager.cc:1434] T e1fa4980f2624321b617b84c68b90113 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20251029 02:52:05.974982 510 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 7ff2399833e546448b85b9caeb5200fb (table default.Table_464b6de22be542b89e56489d27d2d521 [id=9c994e4d21f042b2a5c1b86b27abfbb4], current schema version=4) failed: Not found: failed to reset TS proxy: No leader found: Tablet 7ff2399833e546448b85b9caeb5200fb consensus state has no leader
I20251029 02:52:05.975396 1166 tablet_bootstrap.cc:492] T 56d05e4a500043c995bb51a204cb295d P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:05.976609 1175 ts_tablet_manager.cc:1929] T 519754f5d8f7484480f973c53708c5f3 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:05.977456 1166 tablet_bootstrap.cc:654] T 56d05e4a500043c995bb51a204cb295d P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:05.977926 1175 log.cc:1199] T 519754f5d8f7484480f973c53708c5f3 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/519754f5d8f7484480f973c53708c5f3
I20251029 02:52:05.978148 1973 raft_consensus.cc:2804] T 71068739b4384badbf305313d902f0f5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:05.978292 1973 raft_consensus.cc:493] T 71068739b4384badbf305313d902f0f5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:05.978437 1973 raft_consensus.cc:3060] T 71068739b4384badbf305313d902f0f5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:05.978680 1167 ts_tablet_manager.cc:1403] T 74873a05ef634867a30aa63b31fcfcc7 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.236s user 0.021s sys 0.031s
I20251029 02:52:05.978930 1175 ts_tablet_manager.cc:1950] T 519754f5d8f7484480f973c53708c5f3 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:05.978991 890 tablet_service.cc:1552] Processing DeleteTablet for tablet ad214c870ee649279ed7321c5b69cbee with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:05 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:05.980001 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 669) took 2113 ms. Trace:
I20251029 02:52:05.980091 1175 rpcz_store.cc:276] 1029 02:52:03.866202 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:03.866287 (+ 85us) service_pool.cc:225] Handling call
1029 02:52:05.955267 (+2088980us) ts_tablet_manager.cc:1151] Deleting tablet 519754f5d8f7484480f973c53708c5f3
1029 02:52:05.955279 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:05.965110 (+ 9831us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:05.976504 (+ 11394us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:05.979976 (+ 3472us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2088821}
I20251029 02:52:05.981546 1175 tablet_replica.cc:333] T 0cc1957a845f4e9cb703a553b27d1347 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:05.981674 1973 raft_consensus.cc:515] T 71068739b4384badbf305313d902f0f5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:05.982205 1973 leader_election.cc:290] T 71068739b4384badbf305313d902f0f5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Requested vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:05.982342 1973 raft_consensus.cc:2804] T c9e923724ca9402682f13b252dbc37aa P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:05.982419 1973 raft_consensus.cc:697] T c9e923724ca9402682f13b252dbc37aa P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Becoming Leader. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Running, Role: LEADER
I20251029 02:52:05.982646 1973 consensus_queue.cc:237] T c9e923724ca9402682f13b252dbc37aa P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:05.983043 1175 raft_consensus.cc:2243] T 0cc1957a845f4e9cb703a553b27d1347 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:05.983019 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 519754f5d8f7484480f973c53708c5f3 (table default.Table_96a1c9ac99714785a6bb853741027180 [id=d16cec879a3a45189e45088c4c0913ed]) successfully deleted
I20251029 02:52:05.983183 1175 raft_consensus.cc:2272] T 0cc1957a845f4e9cb703a553b27d1347 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:05.983655 1175 ts_tablet_manager.cc:1916] T 0cc1957a845f4e9cb703a553b27d1347 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 02:52:05.985098 990 leader_election.cc:343] T 71068739b4384badbf305313d902f0f5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:05.967028 1906 consensus_queue.cc:1048] T c9dff4dff82f49e7ad49caad5e404f09 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:05.986884 916 raft_consensus.cc:2468] T c9e923724ca9402682f13b252dbc37aa P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 0.
I20251029 02:52:05.987627 1176 ts_tablet_manager.cc:1950] T 576c4e6e70e24b0399ac9597a30f0569 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:05.987085 1903 raft_consensus.cc:515] T 8df97119d0654d7d92bb5bd17c5d8562 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:05.989219 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 576c4e6e70e24b0399ac9597a30f0569 (table default.Table_e8685346a99d41078ecd38cd4ce4fec5 [id=7774a12db22f414f9266236d92b3b511]) successfully deleted
I20251029 02:52:05.954461 2022 tablet.cc:1722] T 42cfdf88472b44979d86836561005ed1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20251029 02:52:05.989516 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 4a55f22b0d3b4ebe98ec2e054d05b095 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4a55f22b0d3b4ebe98ec2e054d05b095 already in progress: creating tablet
W20251029 02:52:05.989732 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 6ecc552c1ef643f8bb84622d9d440ba2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6ecc552c1ef643f8bb84622d9d440ba2 already in progress: creating tablet
I20251029 02:52:05.986596 914 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9e923724ca9402682f13b252dbc37aa" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
W20251029 02:52:05.989964 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 81f936792aae4b699b3122457d7b631d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 81f936792aae4b699b3122457d7b631d already in progress: creating tablet
W20251029 02:52:05.990149 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet ea7b977224f24bf49dadfb83ffa28310 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ea7b977224f24bf49dadfb83ffa28310 already in progress: creating tablet
W20251029 02:52:05.990371 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 46cd75ff18b14212b556b9bcb3012dd9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 46cd75ff18b14212b556b9bcb3012dd9 already in progress: creating tablet
W20251029 02:52:05.990607 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 76287edc126a403dbbec4a3b41effe84 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 76287edc126a403dbbec4a3b41effe84 already in progress: creating tablet
I20251029 02:52:05.990581 766 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "71068739b4384badbf305313d902f0f5" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:05.990744 766 raft_consensus.cc:3060] T 71068739b4384badbf305313d902f0f5 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
W20251029 02:52:05.990839 513 catalog_manager.cc:4952] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 1bab90fe904944af948dab3db06b7e34 because the tablet was not found. No further retry: Not found: Tablet not found: 1bab90fe904944af948dab3db06b7e34
I20251029 02:52:05.993705 766 raft_consensus.cc:2468] T 71068739b4384badbf305313d902f0f5 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 1.
I20251029 02:52:05.995129 1903 leader_election.cc:290] T 8df97119d0654d7d92bb5bd17c5d8562 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:05.995266 520 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_8dc0ccf505314b8db07c2b5c2ad231c0" } modify_external_catalogs: true
I20251029 02:52:05.995296 989 leader_election.cc:304] T 71068739b4384badbf305313d902f0f5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters:
I20251029 02:52:05.995469 520 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_8dc0ccf505314b8db07c2b5c2ad231c0" } modify_external_catalogs: true
I20251029 02:52:05.995548 1906 raft_consensus.cc:2804] T 71068739b4384badbf305313d902f0f5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:05.995692 1906 raft_consensus.cc:697] T 71068739b4384badbf305313d902f0f5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Becoming Leader. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Running, Role: LEADER
I20251029 02:52:05.994853 919 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "71068739b4384badbf305313d902f0f5" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
I20251029 02:52:05.995985 1906 consensus_queue.cc:237] T 71068739b4384badbf305313d902f0f5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:05.996450 1957 raft_consensus.cc:493] T caef1c93e33a4390bcada59a41f43465 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251029 02:52:05.996547 990 leader_election.cc:343] T 71068739b4384badbf305313d902f0f5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:05.996588 1957 raft_consensus.cc:515] T caef1c93e33a4390bcada59a41f43465 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:05.997195 1957 leader_election.cc:290] T caef1c93e33a4390bcada59a41f43465 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:05.997699 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "caef1c93e33a4390bcada59a41f43465" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:52:05.998144 712 leader_election.cc:343] T caef1c93e33a4390bcada59a41f43465 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
W20251029 02:52:05.998752 847 leader_election.cc:343] T 8df97119d0654d7d92bb5bd17c5d8562 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 8df97119d0654d7d92bb5bd17c5d8562
I20251029 02:52:05.999275 2024 tablet.cc:1722] T 42cfdf88472b44979d86836561005ed1 P b495a49349324f7d81908f2da700f0d9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 02:52:05.995007 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8df97119d0654d7d92bb5bd17c5d8562" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:05.999292 914 raft_consensus.cc:3060] T c9e923724ca9402682f13b252dbc37aa P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:06.002985 1176 tablet_replica.cc:333] T b2b2d64dcf964da8be4b544dbbe31977 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:06.003247 1176 raft_consensus.cc:2243] T b2b2d64dcf964da8be4b544dbbe31977 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:06.003441 1176 raft_consensus.cc:2272] T b2b2d64dcf964da8be4b544dbbe31977 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:06.000604 915 raft_consensus.cc:2468] T 8df97119d0654d7d92bb5bd17c5d8562 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 0.
I20251029 02:52:06.003686 919 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "caef1c93e33a4390bcada59a41f43465" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:06.004053 1176 ts_tablet_manager.cc:1916] T b2b2d64dcf964da8be4b544dbbe31977 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 02:52:06.004385 711 leader_election.cc:343] T caef1c93e33a4390bcada59a41f43465 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
W20251029 02:52:06.004405 513 catalog_manager.cc:4952] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 6feec2c8e030447fae9c292a626dcd1d because the tablet was not found. No further retry: Not found: Tablet not found: 6feec2c8e030447fae9c292a626dcd1d
I20251029 02:52:06.004503 711 leader_election.cc:304] T caef1c93e33a4390bcada59a41f43465 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:06.001730 1167 raft_consensus.cc:359] T 74873a05ef634867a30aa63b31fcfcc7 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.005155 1167 raft_consensus.cc:385] T 74873a05ef634867a30aa63b31fcfcc7 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:06.005239 1167 raft_consensus.cc:740] T 74873a05ef634867a30aa63b31fcfcc7 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:06.005481 1167 consensus_queue.cc:260] T 74873a05ef634867a30aa63b31fcfcc7 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.006047 1167 ts_tablet_manager.cc:1434] T 74873a05ef634867a30aa63b31fcfcc7 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.027s user 0.001s sys 0.001s
W20251029 02:52:06.006158 519 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 72ac7d5f409344319fac08797e0d26bd consensus state has no leader
I20251029 02:52:06.006390 1167 tablet_bootstrap.cc:492] T d5741dde463a43d6a776ab92dcfb04bf P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:06.006851 990 leader_election.cc:304] T 8df97119d0654d7d92bb5bd17c5d8562 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters: b495a49349324f7d81908f2da700f0d9
I20251029 02:52:06.007577 1973 raft_consensus.cc:2804] T 8df97119d0654d7d92bb5bd17c5d8562 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:06.007702 1973 raft_consensus.cc:493] T 8df97119d0654d7d92bb5bd17c5d8562 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:06.007798 1973 raft_consensus.cc:3060] T 8df97119d0654d7d92bb5bd17c5d8562 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:06.009258 1167 tablet_bootstrap.cc:654] T d5741dde463a43d6a776ab92dcfb04bf P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:06.010353 1957 raft_consensus.cc:2749] T caef1c93e33a4390bcada59a41f43465 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:06.010787 1973 raft_consensus.cc:515] T 8df97119d0654d7d92bb5bd17c5d8562 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:06.012923 914 raft_consensus.cc:2468] T c9e923724ca9402682f13b252dbc37aa P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 1.
I20251029 02:52:06.020612 1174 log.cc:1199] T 1f5b40c0403d4a49a4bfde222d8c947a P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/1f5b40c0403d4a49a4bfde222d8c947a
I20251029 02:52:06.021239 1174 ts_tablet_manager.cc:1950] T 1f5b40c0403d4a49a4bfde222d8c947a P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:06.023340 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 690) took 1839 ms. Trace:
I20251029 02:52:06.026831 1174 rpcz_store.cc:276] 1029 02:52:04.184180 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:04.184237 (+ 57us) service_pool.cc:225] Handling call
1029 02:52:05.790587 (+1606350us) ts_tablet_manager.cc:1151] Deleting tablet 1f5b40c0403d4a49a4bfde222d8c947a
1029 02:52:05.790602 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:05.943711 (+153109us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:05.958699 (+ 14988us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:06.023300 (+ 64601us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1606095}
I20251029 02:52:06.030248 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 1f5b40c0403d4a49a4bfde222d8c947a (table default.Table_72f8739f5f654172b10d619a9ac306bd [id=1565d03ae28649e48c2a614a8d067227]) successfully deleted
I20251029 02:52:06.030715 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 693) took 1769 ms. Trace:
I20251029 02:52:06.032804 1174 rpcz_store.cc:276] 1029 02:52:04.261636 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:04.271289 (+ 9653us) service_pool.cc:225] Handling call
1029 02:52:06.030324 (+1759035us) ts_tablet_manager.cc:1151] Deleting tablet 78662eeae70b4148a4ad0d47be5c566d
1029 02:52:06.030334 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:06.030677 (+ 343us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1758607}
I20251029 02:52:06.034384 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 698) took 1654 ms. Trace:
I20251029 02:52:06.039530 1174 rpcz_store.cc:276] 1029 02:52:04.379808 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:04.381028 (+ 1220us) service_pool.cc:225] Handling call
1029 02:52:06.034205 (+1653177us) ts_tablet_manager.cc:1151] Deleting tablet c4b1ef53dbaa47c39361cc4ef3be71d8
1029 02:52:06.034215 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:06.034321 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1647418}
I20251029 02:52:06.040669 1172 tablet_bootstrap.cc:492] T 0f80b840ed954230bfbfe7e40127cd2f P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:06.041204 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 697) took 1661 ms. Trace:
I20251029 02:52:06.043120 1172 ts_tablet_manager.cc:1403] T 0f80b840ed954230bfbfe7e40127cd2f P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.107s user 0.006s sys 0.012s
I20251029 02:52:06.042632 1174 rpcz_store.cc:276] 1029 02:52:04.379630 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:04.380997 (+ 1367us) service_pool.cc:225] Handling call
1029 02:52:06.041064 (+1660067us) ts_tablet_manager.cc:1151] Deleting tablet 1e17142dc6634db0a5288add9c199ffc
1029 02:52:06.041074 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:06.041169 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1654123}
I20251029 02:52:06.043743 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 694) took 1707 ms. Trace:
I20251029 02:52:06.044710 1172 raft_consensus.cc:359] T 0f80b840ed954230bfbfe7e40127cd2f P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.045008 1174 rpcz_store.cc:276] 1029 02:52:04.336693 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:04.380579 (+ 43886us) service_pool.cc:225] Handling call
1029 02:52:06.043590 (+1663011us) ts_tablet_manager.cc:1151] Deleting tablet de7fac8751a04c19b68a52db7f167214
1029 02:52:06.043601 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:06.043695 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1652093}
I20251029 02:52:06.046907 2035 tablet.cc:1722] T 42cfdf88472b44979d86836561005ed1 P c756528642a543eeacd93cef3e335a7d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 02:52:06.047187 1174 tablet_replica.cc:333] T 51effc9324034ae79e2b25cb57a996c1 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:06.047497 1174 raft_consensus.cc:2243] T 51effc9324034ae79e2b25cb57a996c1 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus shutting down.
W20251029 02:52:06.047837 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet c4b1ef53dbaa47c39361cc4ef3be71d8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c4b1ef53dbaa47c39361cc4ef3be71d8 already in progress: creating tablet
I20251029 02:52:06.046376 1957 raft_consensus.cc:493] T 9e6eb099648549dbbf9969e59865f51f P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251029 02:52:06.048080 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 1e17142dc6634db0a5288add9c199ffc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1e17142dc6634db0a5288add9c199ffc already in progress: creating tablet
I20251029 02:52:06.048091 1957 raft_consensus.cc:515] T 9e6eb099648549dbbf9969e59865f51f P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
W20251029 02:52:06.048305 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet de7fac8751a04c19b68a52db7f167214 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet de7fac8751a04c19b68a52db7f167214 already in progress: creating tablet
I20251029 02:52:06.048767 1957 leader_election.cc:290] T 9e6eb099648549dbbf9969e59865f51f P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:06.049546 1174 pending_rounds.cc:70] T 51effc9324034ae79e2b25cb57a996c1 P c756528642a543eeacd93cef3e335a7d: Trying to abort 1 pending ops.
I20251029 02:52:06.049665 1172 raft_consensus.cc:385] T 0f80b840ed954230bfbfe7e40127cd2f P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:06.053108 1172 raft_consensus.cc:740] T 0f80b840ed954230bfbfe7e40127cd2f P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:06.021461 1166 tablet_bootstrap.cc:492] T 56d05e4a500043c995bb51a204cb295d P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:06.025442 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c9e923724ca9402682f13b252dbc37aa" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:06.053474 1166 ts_tablet_manager.cc:1403] T 56d05e4a500043c995bb51a204cb295d P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.078s user 0.003s sys 0.003s
I20251029 02:52:06.025796 1058 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8df97119d0654d7d92bb5bd17c5d8562" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:06.054273 1058 raft_consensus.cc:2393] T 8df97119d0654d7d92bb5bd17c5d8562 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c756528642a543eeacd93cef3e335a7d in current term 1: Already voted for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in this term.
I20251029 02:52:06.054306 1166 raft_consensus.cc:359] T 56d05e4a500043c995bb51a204cb295d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.054437 1166 raft_consensus.cc:385] T 56d05e4a500043c995bb51a204cb295d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:06.054492 1166 raft_consensus.cc:740] T 56d05e4a500043c995bb51a204cb295d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:06.054737 1166 consensus_queue.cc:260] T 56d05e4a500043c995bb51a204cb295d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.055357 1166 ts_tablet_manager.cc:1434] T 56d05e4a500043c995bb51a204cb295d P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251029 02:52:06.055711 1166 tablet_bootstrap.cc:492] T dd6ad641fbda472fab07aa35c5e4523a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:06.049603 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e6eb099648549dbbf9969e59865f51f" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:06.057463 1174 pending_rounds.cc:77] T 51effc9324034ae79e2b25cb57a996c1 P c756528642a543eeacd93cef3e335a7d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215949104276525056 op_type: NO_OP noop_request { }
I20251029 02:52:06.054023 849 leader_election.cc:304] T c9e923724ca9402682f13b252dbc37aa P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9
W20251029 02:52:06.057889 711 leader_election.cc:343] T 9e6eb099648549dbbf9969e59865f51f P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:06.058256 1172 consensus_queue.cc:260] T 0f80b840ed954230bfbfe7e40127cd2f P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.058162 1904 raft_consensus.cc:2749] T c9e923724ca9402682f13b252dbc37aa P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:06.058352 849 leader_election.cc:304] T 8df97119d0654d7d92bb5bd17c5d8562 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9
I20251029 02:52:06.057610 1174 raft_consensus.cc:2889] T 51effc9324034ae79e2b25cb57a996c1 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20251029 02:52:06.058938 1904 raft_consensus.cc:3060] T 8df97119d0654d7d92bb5bd17c5d8562 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:06.039062 1906 consensus_queue.cc:1048] T c9dff4dff82f49e7ad49caad5e404f09 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:06.049302 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e6eb099648549dbbf9969e59865f51f" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:52:06.059439 712 leader_election.cc:343] T 9e6eb099648549dbbf9969e59865f51f P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:06.059538 712 leader_election.cc:304] T 9e6eb099648549dbbf9969e59865f51f P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:06.061287 1166 tablet_bootstrap.cc:654] T dd6ad641fbda472fab07aa35c5e4523a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:06.062150 1904 raft_consensus.cc:2749] T 8df97119d0654d7d92bb5bd17c5d8562 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:06.062366 1973 leader_election.cc:290] T 8df97119d0654d7d92bb5bd17c5d8562 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Requested vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:06.064988 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8df97119d0654d7d92bb5bd17c5d8562" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
I20251029 02:52:06.065725 1167 tablet_bootstrap.cc:492] T d5741dde463a43d6a776ab92dcfb04bf P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:06.065886 1167 ts_tablet_manager.cc:1403] T d5741dde463a43d6a776ab92dcfb04bf P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.060s user 0.003s sys 0.003s
I20251029 02:52:06.067108 1167 raft_consensus.cc:359] T d5741dde463a43d6a776ab92dcfb04bf P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.067282 1167 raft_consensus.cc:385] T d5741dde463a43d6a776ab92dcfb04bf P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:06.067380 1167 raft_consensus.cc:740] T d5741dde463a43d6a776ab92dcfb04bf P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:06.067643 1167 consensus_queue.cc:260] T d5741dde463a43d6a776ab92dcfb04bf P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.067945 916 raft_consensus.cc:2468] T 8df97119d0654d7d92bb5bd17c5d8562 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 1.
I20251029 02:52:06.070727 990 leader_election.cc:304] T 8df97119d0654d7d92bb5bd17c5d8562 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:06.071049 1973 raft_consensus.cc:2804] T 8df97119d0654d7d92bb5bd17c5d8562 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:06.071398 1973 raft_consensus.cc:697] T 8df97119d0654d7d92bb5bd17c5d8562 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Becoming Leader. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Running, Role: LEADER
I20251029 02:52:06.071413 1167 ts_tablet_manager.cc:1434] T d5741dde463a43d6a776ab92dcfb04bf P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251029 02:52:06.072362 1167 tablet_bootstrap.cc:492] T 519754f5d8f7484480f973c53708c5f3 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:06.072216 1973 consensus_queue.cc:237] T 8df97119d0654d7d92bb5bd17c5d8562 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:06.075023 1174 raft_consensus.cc:2272] T 51effc9324034ae79e2b25cb57a996c1 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:06.079449 1957 raft_consensus.cc:2749] T 9e6eb099648549dbbf9969e59865f51f P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 02:52:06.079866 989 leader_election.cc:343] T 8df97119d0654d7d92bb5bd17c5d8562 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 8df97119d0654d7d92bb5bd17c5d8562
I20251029 02:52:06.080353 1172 ts_tablet_manager.cc:1434] T 0f80b840ed954230bfbfe7e40127cd2f P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.036s user 0.001s sys 0.001s
I20251029 02:52:06.080703 1172 tablet_bootstrap.cc:492] T 785733da86ff425a90ccfab2933fc89e P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:06.082216 1172 tablet_bootstrap.cc:654] T 785733da86ff425a90ccfab2933fc89e P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:06.083696 1174 ts_tablet_manager.cc:1916] T 51effc9324034ae79e2b25cb57a996c1 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:06.091236 1167 tablet_bootstrap.cc:654] T 519754f5d8f7484480f973c53708c5f3 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:06.092193 1176 ts_tablet_manager.cc:1929] T b2b2d64dcf964da8be4b544dbbe31977 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:06.094866 1176 log.cc:1199] T b2b2d64dcf964da8be4b544dbbe31977 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/b2b2d64dcf964da8be4b544dbbe31977
I20251029 02:52:06.095337 1176 ts_tablet_manager.cc:1950] T b2b2d64dcf964da8be4b544dbbe31977 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:06.095849 1176 tablet_replica.cc:333] T 8df97119d0654d7d92bb5bd17c5d8562 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:06.096068 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet b2b2d64dcf964da8be4b544dbbe31977 (table default.Table_e0e6e47b18d749e8ae80f822af1e23f0 [id=125ae5c7b5074378bcecb546e70f0c13]) successfully deleted
I20251029 02:52:06.105149 1176 raft_consensus.cc:2243] T 8df97119d0654d7d92bb5bd17c5d8562 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:06.105715 1176 pending_rounds.cc:70] T 8df97119d0654d7d92bb5bd17c5d8562 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Trying to abort 1 pending ops.
I20251029 02:52:06.105821 1176 pending_rounds.cc:77] T 8df97119d0654d7d92bb5bd17c5d8562 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215949111595450368 op_type: NO_OP noop_request { }
I20251029 02:52:06.105978 1176 raft_consensus.cc:2889] T 8df97119d0654d7d92bb5bd17c5d8562 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 02:52:06.106120 1176 raft_consensus.cc:2272] T 8df97119d0654d7d92bb5bd17c5d8562 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:06.106137 1172 tablet_bootstrap.cc:492] T 785733da86ff425a90ccfab2933fc89e P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:06.106278 1172 ts_tablet_manager.cc:1403] T 785733da86ff425a90ccfab2933fc89e P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.026s user 0.000s sys 0.002s
I20251029 02:52:06.107110 1172 raft_consensus.cc:359] T 785733da86ff425a90ccfab2933fc89e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.107277 1172 raft_consensus.cc:385] T 785733da86ff425a90ccfab2933fc89e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:06.107326 1172 raft_consensus.cc:740] T 785733da86ff425a90ccfab2933fc89e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:06.107542 1172 consensus_queue.cc:260] T 785733da86ff425a90ccfab2933fc89e P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.108243 1172 ts_tablet_manager.cc:1434] T 785733da86ff425a90ccfab2933fc89e P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 02:52:06.108510 1172 tablet_bootstrap.cc:492] T 5db519edc75d4bd0985b32322825ffae P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:06.109803 1176 ts_tablet_manager.cc:1916] T 8df97119d0654d7d92bb5bd17c5d8562 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:06.111611 1175 ts_tablet_manager.cc:1929] T 0cc1957a845f4e9cb703a553b27d1347 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:06.111712 1175 log.cc:1199] T 0cc1957a845f4e9cb703a553b27d1347 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/0cc1957a845f4e9cb703a553b27d1347
I20251029 02:52:06.111706 1166 tablet_bootstrap.cc:492] T dd6ad641fbda472fab07aa35c5e4523a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:06.111848 1166 ts_tablet_manager.cc:1403] T dd6ad641fbda472fab07aa35c5e4523a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.056s user 0.004s sys 0.014s
I20251029 02:52:06.111984 1175 ts_tablet_manager.cc:1950] T 0cc1957a845f4e9cb703a553b27d1347 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:06.112645 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 670) took 2246 ms. Trace:
I20251029 02:52:06.112720 1166 raft_consensus.cc:359] T dd6ad641fbda472fab07aa35c5e4523a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.112725 1175 rpcz_store.cc:276] 1029 02:52:03.866302 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:03.867752 (+ 1450us) service_pool.cc:225] Handling call
1029 02:52:05.981500 (+2113748us) ts_tablet_manager.cc:1151] Deleting tablet 0cc1957a845f4e9cb703a553b27d1347
1029 02:52:05.981510 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:05.992160 (+ 10650us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:06.111516 (+119356us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:06.112626 (+ 1110us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1312,"tablet-delete.queue_time_us":2113598}
I20251029 02:52:06.112892 1166 raft_consensus.cc:385] T dd6ad641fbda472fab07aa35c5e4523a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:06.112972 1166 raft_consensus.cc:740] T dd6ad641fbda472fab07aa35c5e4523a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:06.113103 1175 tablet_replica.cc:333] T 56d05e4a500043c995bb51a204cb295d P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:06.113229 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 0cc1957a845f4e9cb703a553b27d1347 (table default.Table_262db98ea44648baab95fc2b3081dc4e [id=843f681b71c64427b49becbabcad9451]) successfully deleted
I20251029 02:52:06.113276 1175 raft_consensus.cc:2243] T 56d05e4a500043c995bb51a204cb295d P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:06.113257 1166 consensus_queue.cc:260] T dd6ad641fbda472fab07aa35c5e4523a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.113400 1175 raft_consensus.cc:2272] T 56d05e4a500043c995bb51a204cb295d P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:06.113898 1166 ts_tablet_manager.cc:1434] T dd6ad641fbda472fab07aa35c5e4523a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 02:52:06.114352 526 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_ff3d06085614492a97e31873f16f4fdc" } modify_external_catalogs: true
I20251029 02:52:06.114452 1166 tablet_bootstrap.cc:492] T daa05a34c0684c07a35966290fd7d44c P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:06.114578 526 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_ff3d06085614492a97e31873f16f4fdc" } modify_external_catalogs: true
I20251029 02:52:06.116473 1166 tablet_bootstrap.cc:654] T daa05a34c0684c07a35966290fd7d44c P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:06.116788 1167 tablet_bootstrap.cc:492] T 519754f5d8f7484480f973c53708c5f3 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:06.116986 1167 ts_tablet_manager.cc:1403] T 519754f5d8f7484480f973c53708c5f3 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.045s user 0.003s sys 0.000s
I20251029 02:52:06.117758 1167 raft_consensus.cc:359] T 519754f5d8f7484480f973c53708c5f3 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.117980 1167 raft_consensus.cc:385] T 519754f5d8f7484480f973c53708c5f3 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:06.118065 1167 raft_consensus.cc:740] T 519754f5d8f7484480f973c53708c5f3 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:06.120211 522 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 59af906061d049c99ae2153945597007
I20251029 02:52:06.120872 522 catalog_manager.cc:6719] ReplaceTablet: tablet 59af906061d049c99ae2153945597007 deleted and replaced by tablet bc3cfa2d7fe8411b954ccb3e16d643a9
I20251029 02:52:06.121083 890 tablet_service.cc:1552] Processing DeleteTablet for tablet 59af906061d049c99ae2153945597007 with delete_type TABLET_DATA_DELETED (replaced by tablet bc3cfa2d7fe8411b954ccb3e16d643a9) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:06.121477 1039 tablet_service.cc:1552] Processing DeleteTablet for tablet 59af906061d049c99ae2153945597007 with delete_type TABLET_DATA_DELETED (replaced by tablet bc3cfa2d7fe8411b954ccb3e16d643a9) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:06.121654 749 tablet_service.cc:1552] Processing DeleteTablet for tablet 59af906061d049c99ae2153945597007 with delete_type TABLET_DATA_DELETED (replaced by tablet bc3cfa2d7fe8411b954ccb3e16d643a9) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:06.121948 1167 consensus_queue.cc:260] T 519754f5d8f7484480f973c53708c5f3 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.124781 1167 ts_tablet_manager.cc:1434] T 519754f5d8f7484480f973c53708c5f3 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.008s user 0.002s sys 0.001s
I20251029 02:52:06.125507 1166 tablet_bootstrap.cc:492] T daa05a34c0684c07a35966290fd7d44c P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:06.125701 1166 ts_tablet_manager.cc:1403] T daa05a34c0684c07a35966290fd7d44c P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.002s
I20251029 02:52:06.125973 1167 tablet_bootstrap.cc:492] T f60e7c24982d40319801869ad060864d P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:06.126283 1176 ts_tablet_manager.cc:1929] T 8df97119d0654d7d92bb5bd17c5d8562 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:06.127492 1903 raft_consensus.cc:493] T f81aecb20d6e498bb9b498698886d65e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:06.127642 1903 raft_consensus.cc:515] T f81aecb20d6e498bb9b498698886d65e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:06.128221 1903 leader_election.cc:290] T f81aecb20d6e498bb9b498698886d65e P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:06.128567 1176 log.cc:1199] T 8df97119d0654d7d92bb5bd17c5d8562 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/8df97119d0654d7d92bb5bd17c5d8562
I20251029 02:52:06.128661 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f81aecb20d6e498bb9b498698886d65e" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:06.128844 1060 raft_consensus.cc:2468] T f81aecb20d6e498bb9b498698886d65e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:06.129038 1176 ts_tablet_manager.cc:1950] T 8df97119d0654d7d92bb5bd17c5d8562 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:06.129231 522 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_494874895389492d9a687de4db8785b5" } modify_external_catalogs: true
I20251029 02:52:06.129451 522 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_494874895389492d9a687de4db8785b5" } modify_external_catalogs: true
W20251029 02:52:06.129499 847 leader_election.cc:343] T f81aecb20d6e498bb9b498698886d65e P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: f81aecb20d6e498bb9b498698886d65e
I20251029 02:52:06.129725 1176 tablet_replica.cc:333] T f81aecb20d6e498bb9b498698886d65e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:06.129959 1176 raft_consensus.cc:2243] T f81aecb20d6e498bb9b498698886d65e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:06.129942 849 leader_election.cc:304] T f81aecb20d6e498bb9b498698886d65e P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters: b495a49349324f7d81908f2da700f0d9
I20251029 02:52:06.130120 1176 raft_consensus.cc:2272] T f81aecb20d6e498bb9b498698886d65e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:06.130424 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 8df97119d0654d7d92bb5bd17c5d8562 (table default.Table_99592a0735024ba7a0d8d6145b61f2e4 [id=ced78d857c314dc7802545a7909bdf46]) successfully deleted
W20251029 02:52:06.130756 513 catalog_manager.cc:4952] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 6cd7009e06a646acad0802918c2b425e because the tablet was not found. No further retry: Not found: Tablet not found: 6cd7009e06a646acad0802918c2b425e
W20251029 02:52:06.131008 513 catalog_manager.cc:4952] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 8afc01d52d784f2fa559cab7ce07c8be because the tablet was not found. No further retry: Not found: Tablet not found: 8afc01d52d784f2fa559cab7ce07c8be
I20251029 02:52:06.131338 1903 raft_consensus.cc:2804] T f81aecb20d6e498bb9b498698886d65e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:06.131430 1903 raft_consensus.cc:493] T f81aecb20d6e498bb9b498698886d65e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:06.131523 1903 raft_consensus.cc:3060] T f81aecb20d6e498bb9b498698886d65e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:06.131559 1175 ts_tablet_manager.cc:1916] T 56d05e4a500043c995bb51a204cb295d P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:06.134693 1172 tablet_bootstrap.cc:654] T 5db519edc75d4bd0985b32322825ffae P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:06.136886 1903 raft_consensus.cc:515] T f81aecb20d6e498bb9b498698886d65e P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:06.138365 1176 ts_tablet_manager.cc:1916] T f81aecb20d6e498bb9b498698886d65e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:06.138340 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f81aecb20d6e498bb9b498698886d65e" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
W20251029 02:52:06.138935 849 leader_election.cc:343] T f81aecb20d6e498bb9b498698886d65e P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
W20251029 02:52:06.139946 847 leader_election.cc:343] T f81aecb20d6e498bb9b498698886d65e P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: f81aecb20d6e498bb9b498698886d65e
I20251029 02:52:06.140086 847 leader_election.cc:304] T f81aecb20d6e498bb9b498698886d65e P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9
I20251029 02:52:06.140537 1904 raft_consensus.cc:2749] T f81aecb20d6e498bb9b498698886d65e P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251029 02:52:06.140745 1903 leader_election.cc:290] T f81aecb20d6e498bb9b498698886d65e P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:06.126696 1166 raft_consensus.cc:359] T daa05a34c0684c07a35966290fd7d44c P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.141197 1166 raft_consensus.cc:385] T daa05a34c0684c07a35966290fd7d44c P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:06.141295 1166 raft_consensus.cc:740] T daa05a34c0684c07a35966290fd7d44c P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:06.141571 1166 consensus_queue.cc:260] T daa05a34c0684c07a35966290fd7d44c P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.142263 1166 ts_tablet_manager.cc:1434] T daa05a34c0684c07a35966290fd7d44c P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
W20251029 02:52:06.145313 518 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e3ac6597d52441569a7aeef051087ad2 consensus state has no leader
I20251029 02:52:06.145875 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_5c2d435abf6a4195bf1009f4e1aa725d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:06.146183 1167 tablet_bootstrap.cc:654] T f60e7c24982d40319801869ad060864d P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
W20251029 02:52:06.146449 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5c2d435abf6a4195bf1009f4e1aa725d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:06.149703 1973 raft_consensus.cc:493] T dd6ad641fbda472fab07aa35c5e4523a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:06.149868 1973 raft_consensus.cc:515] T dd6ad641fbda472fab07aa35c5e4523a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.150485 1973 leader_election.cc:290] T dd6ad641fbda472fab07aa35c5e4523a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:06.150632 1176 ts_tablet_manager.cc:1929] T f81aecb20d6e498bb9b498698886d65e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:06.150666 1174 ts_tablet_manager.cc:1929] T 51effc9324034ae79e2b25cb57a996c1 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:06.150759 1176 log.cc:1199] T f81aecb20d6e498bb9b498698886d65e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/f81aecb20d6e498bb9b498698886d65e
I20251029 02:52:06.150794 1174 log.cc:1199] T 51effc9324034ae79e2b25cb57a996c1 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/51effc9324034ae79e2b25cb57a996c1
I20251029 02:52:06.151162 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dd6ad641fbda472fab07aa35c5e4523a" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:06.151710 990 leader_election.cc:343] T dd6ad641fbda472fab07aa35c5e4523a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
W20251029 02:52:06.152344 989 leader_election.cc:343] T dd6ad641fbda472fab07aa35c5e4523a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: dd6ad641fbda472fab07aa35c5e4523a
I20251029 02:52:06.152474 989 leader_election.cc:304] T dd6ad641fbda472fab07aa35c5e4523a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5; no voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:06.152727 1176 ts_tablet_manager.cc:1950] T f81aecb20d6e498bb9b498698886d65e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:06.152751 2018 raft_consensus.cc:493] T 6f35412a66b74051a475dce5dafa7792 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:06.152920 1973 raft_consensus.cc:2749] T dd6ad641fbda472fab07aa35c5e4523a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:06.152880 2018 raft_consensus.cc:515] T 6f35412a66b74051a475dce5dafa7792 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.153409 2018 leader_election.cc:290] T 6f35412a66b74051a475dce5dafa7792 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:06.153841 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f35412a66b74051a475dce5dafa7792" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:06.154331 711 leader_election.cc:343] T 6f35412a66b74051a475dce5dafa7792 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:06.154721 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f35412a66b74051a475dce5dafa7792" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:52:06.155226 712 leader_election.cc:343] T 6f35412a66b74051a475dce5dafa7792 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:06.155337 712 leader_election.cc:304] T 6f35412a66b74051a475dce5dafa7792 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:06.155767 2018 raft_consensus.cc:2749] T 6f35412a66b74051a475dce5dafa7792 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:06.156193 1172 tablet_bootstrap.cc:492] T 5db519edc75d4bd0985b32322825ffae P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:06.156343 1172 ts_tablet_manager.cc:1403] T 5db519edc75d4bd0985b32322825ffae P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.048s user 0.000s sys 0.003s
I20251029 02:52:06.157258 1172 raft_consensus.cc:359] T 5db519edc75d4bd0985b32322825ffae P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.157492 1172 raft_consensus.cc:385] T 5db519edc75d4bd0985b32322825ffae P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:06.157588 1172 raft_consensus.cc:740] T 5db519edc75d4bd0985b32322825ffae P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:06.157876 1172 consensus_queue.cc:260] T 5db519edc75d4bd0985b32322825ffae P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.158573 1172 ts_tablet_manager.cc:1434] T 5db519edc75d4bd0985b32322825ffae P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20251029 02:52:06.158934 1172 tablet_bootstrap.cc:492] T ccdca31db2834de4b6ed0f2d67a10535 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:06.161432 1172 tablet_bootstrap.cc:654] T ccdca31db2834de4b6ed0f2d67a10535 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:06.161919 1175 ts_tablet_manager.cc:1929] T 56d05e4a500043c995bb51a204cb295d P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:06.162056 1175 log.cc:1199] T 56d05e4a500043c995bb51a204cb295d P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/56d05e4a500043c995bb51a204cb295d
I20251029 02:52:06.162400 1175 ts_tablet_manager.cc:1950] T 56d05e4a500043c995bb51a204cb295d P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:06.169608 525 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_570f159e617548ec974b41a654c1d800" } modify_external_catalogs: true
I20251029 02:52:06.169842 525 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_570f159e617548ec974b41a654c1d800" } modify_external_catalogs: true
I20251029 02:52:06.173331 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_f6c47489fbba4d65bbab23c587dddb63"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:06.173974 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f6c47489fbba4d65bbab23c587dddb63 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:52:06.175891 524 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9269622ca92645d5aa8e73341201ccbb consensus state has no leader
I20251029 02:52:06.176177 524 master_service.cc:954] ReplaceTablet: received request to replace tablet b0fec06ad37d4c12a7045d424b88cc5e from {username='slave'} at 127.0.0.1:35314
I20251029 02:52:06.176901 1167 tablet_bootstrap.cc:492] T f60e7c24982d40319801869ad060864d P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:06.177050 1167 ts_tablet_manager.cc:1403] T f60e7c24982d40319801869ad060864d P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.051s user 0.001s sys 0.002s
I20251029 02:52:06.177832 1167 raft_consensus.cc:359] T f60e7c24982d40319801869ad060864d P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.178012 1167 raft_consensus.cc:385] T f60e7c24982d40319801869ad060864d P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:06.178061 1167 raft_consensus.cc:740] T f60e7c24982d40319801869ad060864d P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:06.178282 1167 consensus_queue.cc:260] T f60e7c24982d40319801869ad060864d P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.179977 1172 tablet_bootstrap.cc:492] T ccdca31db2834de4b6ed0f2d67a10535 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:06.180162 1172 ts_tablet_manager.cc:1403] T ccdca31db2834de4b6ed0f2d67a10535 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.021s user 0.014s sys 0.006s
I20251029 02:52:06.180709 1167 ts_tablet_manager.cc:1434] T f60e7c24982d40319801869ad060864d P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20251029 02:52:06.181166 1167 tablet_bootstrap.cc:492] T 6ca9bd37de9e4be388501b15af4bda31 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:06.181396 1172 raft_consensus.cc:359] T ccdca31db2834de4b6ed0f2d67a10535 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.181922 1172 raft_consensus.cc:385] T ccdca31db2834de4b6ed0f2d67a10535 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:06.182047 1172 raft_consensus.cc:740] T ccdca31db2834de4b6ed0f2d67a10535 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:06.182338 1172 consensus_queue.cc:260] T ccdca31db2834de4b6ed0f2d67a10535 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.183099 1172 ts_tablet_manager.cc:1434] T ccdca31db2834de4b6ed0f2d67a10535 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20251029 02:52:06.183413 1172 tablet_bootstrap.cc:492] T 95f342d1cfb84c4eb31bd3d15ea3bdc2 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:06.187057 1167 tablet_bootstrap.cc:654] T 6ca9bd37de9e4be388501b15af4bda31 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:06.187460 1172 tablet_bootstrap.cc:654] T 95f342d1cfb84c4eb31bd3d15ea3bdc2 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:06.189972 518 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_e8e977ce21ca40d4a22d0434d73ef70b" } new_table_name: "default.Table_d7cbd7b4c4bf4886a4ccf4c89da3cbce" modify_external_catalogs: true
I20251029 02:52:06.195652 1172 tablet_bootstrap.cc:492] T 95f342d1cfb84c4eb31bd3d15ea3bdc2 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:06.195845 1172 ts_tablet_manager.cc:1403] T 95f342d1cfb84c4eb31bd3d15ea3bdc2 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.005s
I20251029 02:52:06.196787 1172 raft_consensus.cc:359] T 95f342d1cfb84c4eb31bd3d15ea3bdc2 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.197036 1172 raft_consensus.cc:385] T 95f342d1cfb84c4eb31bd3d15ea3bdc2 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:06.197111 1172 raft_consensus.cc:740] T 95f342d1cfb84c4eb31bd3d15ea3bdc2 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:06.197429 1172 consensus_queue.cc:260] T 95f342d1cfb84c4eb31bd3d15ea3bdc2 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.198559 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 671) took 2255 ms. Trace:
I20251029 02:52:06.198576 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_f8a743f1d5174309921ff1ff7f6b10af"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:06.198699 1175 rpcz_store.cc:276] 1029 02:52:03.939072 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:03.939137 (+ 65us) service_pool.cc:225] Handling call
1029 02:52:06.113056 (+2173919us) ts_tablet_manager.cc:1151] Deleting tablet 56d05e4a500043c995bb51a204cb295d
1029 02:52:06.113066 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:06.143618 (+ 30552us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:06.161762 (+ 18144us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:06.194356 (+ 32594us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2173690}
I20251029 02:52:06.199110 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 56d05e4a500043c995bb51a204cb295d (table default.Table_0c1681c9cd0347f2ba2eab73b8cf11a9 [id=fdb365438df14f12a382aa046aef41a4]) successfully deleted
W20251029 02:52:06.199165 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f8a743f1d5174309921ff1ff7f6b10af in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:06.199242 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 672) took 2203 ms. Trace:
I20251029 02:52:06.201754 1172 ts_tablet_manager.cc:1434] T 95f342d1cfb84c4eb31bd3d15ea3bdc2 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251029 02:52:06.202080 1172 tablet_bootstrap.cc:492] T 49d9ce32e4f04f11be2ecefc5d523a8b P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:06.203783 1172 tablet_bootstrap.cc:654] T 49d9ce32e4f04f11be2ecefc5d523a8b P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
W20251029 02:52:06.208958 518 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 506f8c112a3b43e1a3d6bbafb8cb81e8 consensus state has no leader
I20251029 02:52:06.209401 518 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_57a0f7b13bf146d890469978e60ac4be" } new_table_name: "default.Table_973ef856041949598a23da3dab8b40db" modify_external_catalogs: true
I20251029 02:52:06.212208 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_e3aae8a6f0314d75b52739cd81d573d9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:06.201243 1175 rpcz_store.cc:276] 1029 02:52:03.995657 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:03.995713 (+ 56us) service_pool.cc:225] Handling call
1029 02:52:06.199101 (+2203388us) ts_tablet_manager.cc:1151] Deleting tablet a945d64287f94ca697447e540fab39ce
1029 02:52:06.199110 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:06.199210 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2203211}
W20251029 02:52:06.212775 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e3aae8a6f0314d75b52739cd81d573d9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:52:06.213145 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet a945d64287f94ca697447e540fab39ce because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a945d64287f94ca697447e540fab39ce already in progress: creating tablet
I20251029 02:52:06.213896 1175 tablet_replica.cc:333] T 71068739b4384badbf305313d902f0f5 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:06.214145 1175 raft_consensus.cc:2243] T 71068739b4384badbf305313d902f0f5 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:06.214334 1175 raft_consensus.cc:2272] T 71068739b4384badbf305313d902f0f5 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:06.215281 1166 tablet_bootstrap.cc:492] T bfec87f690a640c39c58c64e16cc67e7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:06.216545 1176 tablet_replica.cc:333] T 6bd3fac2502e479aa9f4ceaabbb39504 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:06.216758 1176 raft_consensus.cc:2243] T 6bd3fac2502e479aa9f4ceaabbb39504 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:06.216986 1176 raft_consensus.cc:2272] T 6bd3fac2502e479aa9f4ceaabbb39504 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:06.217542 1166 tablet_bootstrap.cc:654] T bfec87f690a640c39c58c64e16cc67e7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:06.217983 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet f81aecb20d6e498bb9b498698886d65e (table default.Table_98034c2933664cefab94c26cb8349dc4 [id=efb8f381fea74cf48842af6b1a4d5a81]) successfully deleted
I20251029 02:52:06.217545 1176 ts_tablet_manager.cc:1916] T 6bd3fac2502e479aa9f4ceaabbb39504 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:06.223522 1167 tablet_bootstrap.cc:492] T 6ca9bd37de9e4be388501b15af4bda31 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:06.223677 1167 ts_tablet_manager.cc:1403] T 6ca9bd37de9e4be388501b15af4bda31 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.043s user 0.003s sys 0.000s
I20251029 02:52:06.224443 1167 raft_consensus.cc:359] T 6ca9bd37de9e4be388501b15af4bda31 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.224606 1167 raft_consensus.cc:385] T 6ca9bd37de9e4be388501b15af4bda31 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:06.224655 1167 raft_consensus.cc:740] T 6ca9bd37de9e4be388501b15af4bda31 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:06.224886 1167 consensus_queue.cc:260] T 6ca9bd37de9e4be388501b15af4bda31 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.227685 1176 ts_tablet_manager.cc:1929] T 6bd3fac2502e479aa9f4ceaabbb39504 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:06.227816 1176 log.cc:1199] T 6bd3fac2502e479aa9f4ceaabbb39504 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/6bd3fac2502e479aa9f4ceaabbb39504
I20251029 02:52:06.228003 1167 ts_tablet_manager.cc:1434] T 6ca9bd37de9e4be388501b15af4bda31 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251029 02:52:06.228145 1176 ts_tablet_manager.cc:1950] T 6bd3fac2502e479aa9f4ceaabbb39504 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:06.228924 1176 tablet_replica.cc:333] T dd90fb9eb167478cbc3ec9363af86295 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:06.228945 1166 tablet_bootstrap.cc:492] T bfec87f690a640c39c58c64e16cc67e7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:06.229094 1166 ts_tablet_manager.cc:1403] T bfec87f690a640c39c58c64e16cc67e7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.002s
I20251029 02:52:06.229053 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 6bd3fac2502e479aa9f4ceaabbb39504 (table default.Table_110e1b64737c4040ada22360d5a373cb [id=3513c69f6b3941849ac5b8ae0a5846d3]) successfully deleted
I20251029 02:52:06.231523 1172 tablet_bootstrap.cc:492] T 49d9ce32e4f04f11be2ecefc5d523a8b P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:06.231714 1172 ts_tablet_manager.cc:1403] T 49d9ce32e4f04f11be2ecefc5d523a8b P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.030s user 0.002s sys 0.006s
I20251029 02:52:06.232213 1167 tablet_bootstrap.cc:492] T 6bd3fac2502e479aa9f4ceaabbb39504 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:06.232836 1172 raft_consensus.cc:359] T 49d9ce32e4f04f11be2ecefc5d523a8b P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.233047 1172 raft_consensus.cc:385] T 49d9ce32e4f04f11be2ecefc5d523a8b P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:06.233140 1172 raft_consensus.cc:740] T 49d9ce32e4f04f11be2ecefc5d523a8b P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:06.233364 1172 consensus_queue.cc:260] T 49d9ce32e4f04f11be2ecefc5d523a8b P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.233592 1167 tablet_bootstrap.cc:654] T 6bd3fac2502e479aa9f4ceaabbb39504 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:06.234835 1172 ts_tablet_manager.cc:1434] T 49d9ce32e4f04f11be2ecefc5d523a8b P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251029 02:52:06.234753 1166 raft_consensus.cc:359] T bfec87f690a640c39c58c64e16cc67e7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.234997 1166 raft_consensus.cc:385] T bfec87f690a640c39c58c64e16cc67e7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:06.235098 1166 raft_consensus.cc:740] T bfec87f690a640c39c58c64e16cc67e7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:06.235354 1166 consensus_queue.cc:260] T bfec87f690a640c39c58c64e16cc67e7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.236363 1166 ts_tablet_manager.cc:1434] T bfec87f690a640c39c58c64e16cc67e7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20251029 02:52:06.236687 1166 tablet_bootstrap.cc:492] T 4aa78f4ac38e4919864c924e50f1aaf4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:06.238130 1175 ts_tablet_manager.cc:1916] T 71068739b4384badbf305313d902f0f5 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:06.239147 1166 tablet_bootstrap.cc:654] T 4aa78f4ac38e4919864c924e50f1aaf4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:06.240018 1176 raft_consensus.cc:2243] T dd90fb9eb167478cbc3ec9363af86295 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Raft consensus shutting down.
W20251029 02:52:06.240090 849 consensus_peers.cc:597] T 7e6c4f30be624769961c9ec17c5d5dcb P c756528642a543eeacd93cef3e335a7d -> Peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Couldn't send request to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7e6c4f30be624769961c9ec17c5d5dcb. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:06.240345 1176 raft_consensus.cc:2272] T dd90fb9eb167478cbc3ec9363af86295 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:06.240908 1176 ts_tablet_manager.cc:1916] T dd90fb9eb167478cbc3ec9363af86295 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:06.243911 1174 ts_tablet_manager.cc:1950] T 51effc9324034ae79e2b25cb57a996c1 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:06.245945 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 703) took 1732 ms. Trace:
I20251029 02:52:06.247635 1174 rpcz_store.cc:276] 1029 02:52:04.513015 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:04.517914 (+ 4899us) service_pool.cc:225] Handling call
1029 02:52:06.045456 (+1527542us) ts_tablet_manager.cc:1151] Deleting tablet 51effc9324034ae79e2b25cb57a996c1
1029 02:52:06.045466 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:06.102800 (+ 57334us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:06.139558 (+ 36758us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:06.245922 (+106364us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1527337}
I20251029 02:52:06.247541 1167 tablet_bootstrap.cc:492] T 6bd3fac2502e479aa9f4ceaabbb39504 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:06.248051 1167 ts_tablet_manager.cc:1403] T 6bd3fac2502e479aa9f4ceaabbb39504 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.000s
I20251029 02:52:06.248800 1167 raft_consensus.cc:359] T 6bd3fac2502e479aa9f4ceaabbb39504 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.249042 1167 raft_consensus.cc:385] T 6bd3fac2502e479aa9f4ceaabbb39504 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:06.249145 1167 raft_consensus.cc:740] T 6bd3fac2502e479aa9f4ceaabbb39504 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:06.249436 1167 consensus_queue.cc:260] T 6bd3fac2502e479aa9f4ceaabbb39504 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
W20251029 02:52:06.254752 518 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f4729ecd182042c2a10f67ca25224101 consensus state has no leader
I20251029 02:52:06.256686 1167 ts_tablet_manager.cc:1434] T 6bd3fac2502e479aa9f4ceaabbb39504 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20251029 02:52:06.258353 1176 ts_tablet_manager.cc:1929] T dd90fb9eb167478cbc3ec9363af86295 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20251029 02:52:06.259359 1167 tablet_bootstrap.cc:492] T 4aa78f4ac38e4919864c924e50f1aaf4 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:06.260383 1176 log.cc:1199] T dd90fb9eb167478cbc3ec9363af86295 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/dd90fb9eb167478cbc3ec9363af86295
I20251029 02:52:06.261058 518 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_f2a72498cf4540949baff93eba1f5ef0" } modify_external_catalogs: true
I20251029 02:52:06.261283 518 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_f2a72498cf4540949baff93eba1f5ef0" } modify_external_catalogs: true
I20251029 02:52:06.261827 1176 ts_tablet_manager.cc:1950] T dd90fb9eb167478cbc3ec9363af86295 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:06.262818 1167 tablet_bootstrap.cc:654] T 4aa78f4ac38e4919864c924e50f1aaf4 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:06.263083 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet dd90fb9eb167478cbc3ec9363af86295 (table default.Table_8b6217e43e8d467693c104ef84e78911 [id=052abf3dcc904bdcb869d5b885723dab]) successfully deleted
I20251029 02:52:06.263784 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 51effc9324034ae79e2b25cb57a996c1 (table default.Table_ee95bb03ab1246e8826f28083411050d [id=09fbf31759f047be924a5ef1f0030765]) successfully deleted
I20251029 02:52:06.262826 1176 tablet_replica.cc:333] T 71068739b4384badbf305313d902f0f5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:06.265542 1176 raft_consensus.cc:2243] T 71068739b4384badbf305313d902f0f5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:06.266021 1172 tablet_bootstrap.cc:492] T f529e17e97074dca91ec2f48954cfd32 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:06.266323 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 707) took 1427 ms. Trace:
I20251029 02:52:06.266422 1174 rpcz_store.cc:276] 1029 02:52:04.838863 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:04.838925 (+ 62us) service_pool.cc:225] Handling call
1029 02:52:06.266184 (+1427259us) ts_tablet_manager.cc:1151] Deleting tablet 738046ab57334f2dab9f4aab979c5b42
1029 02:52:06.266195 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:06.266301 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1427031}
I20251029 02:52:06.266728 1176 pending_rounds.cc:70] T 71068739b4384badbf305313d902f0f5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Trying to abort 1 pending ops.
I20251029 02:52:06.266920 1176 pending_rounds.cc:77] T 71068739b4384badbf305313d902f0f5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215949111282204672 op_type: NO_OP noop_request { }
I20251029 02:52:06.268116 1172 tablet_bootstrap.cc:654] T f529e17e97074dca91ec2f48954cfd32 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:06.268117 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 708) took 1402 ms. Trace:
I20251029 02:52:06.268146 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_786b080d63354a4cb479483f172ba0af"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:06.268199 1174 rpcz_store.cc:276] 1029 02:52:04.865769 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:04.865837 (+ 68us) service_pool.cc:225] Handling call
1029 02:52:06.268001 (+1402164us) ts_tablet_manager.cc:1151] Deleting tablet c081d2eacfac4ca8a0048e2a1c4f8f83
1029 02:52:06.268011 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:06.268098 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1401992}
I20251029 02:52:06.268445 1176 raft_consensus.cc:2889] T 71068739b4384badbf305313d902f0f5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 02:52:06.268560 1176 raft_consensus.cc:2272] T 71068739b4384badbf305313d902f0f5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:06.268574 1174 tablet_replica.cc:333] T 24cb2a71ba3841d4a5e3e5c6fe99ae2f P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
W20251029 02:52:06.268788 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_786b080d63354a4cb479483f172ba0af in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:06.268810 1174 raft_consensus.cc:2243] T 24cb2a71ba3841d4a5e3e5c6fe99ae2f P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:06.268959 1174 raft_consensus.cc:2272] T 24cb2a71ba3841d4a5e3e5c6fe99ae2f P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:06.269459 1174 ts_tablet_manager.cc:1916] T 24cb2a71ba3841d4a5e3e5c6fe99ae2f P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 02:52:06.270745 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 738046ab57334f2dab9f4aab979c5b42 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 738046ab57334f2dab9f4aab979c5b42 already in progress: creating tablet
W20251029 02:52:06.270964 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet c081d2eacfac4ca8a0048e2a1c4f8f83 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c081d2eacfac4ca8a0048e2a1c4f8f83 already in progress: creating tablet
I20251029 02:52:06.277004 1176 ts_tablet_manager.cc:1916] T 71068739b4384badbf305313d902f0f5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:06.279692 1903 raft_consensus.cc:493] T d5741dde463a43d6a776ab92dcfb04bf P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:06.279879 1903 raft_consensus.cc:515] T d5741dde463a43d6a776ab92dcfb04bf P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.280532 1903 leader_election.cc:290] T d5741dde463a43d6a776ab92dcfb04bf P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:06.281064 519 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet 42cfdf88472b44979d86836561005ed1 (table default.Table_ad2ac3ba98784d68a94eb4eb03e530a4 [id=5e0c72957b9a491d960b33d3fffd8f70]). Expected version 2 got 0
I20251029 02:52:06.281579 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d5741dde463a43d6a776ab92dcfb04bf" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:06.281818 768 raft_consensus.cc:2468] T d5741dde463a43d6a776ab92dcfb04bf P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:06.282123 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d5741dde463a43d6a776ab92dcfb04bf" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:06.282380 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_e72155685e4942c5916842ac8547cf57"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:06.282673 849 leader_election.cc:343] T d5741dde463a43d6a776ab92dcfb04bf P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
W20251029 02:52:06.282994 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e72155685e4942c5916842ac8547cf57 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:06.283063 847 leader_election.cc:304] T d5741dde463a43d6a776ab92dcfb04bf P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5
I20251029 02:52:06.283442 1903 raft_consensus.cc:2804] T d5741dde463a43d6a776ab92dcfb04bf P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:06.283521 1903 raft_consensus.cc:493] T d5741dde463a43d6a776ab92dcfb04bf P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:06.283600 1903 raft_consensus.cc:3060] T d5741dde463a43d6a776ab92dcfb04bf P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:06.284655 1973 raft_consensus.cc:493] T 6ca9bd37de9e4be388501b15af4bda31 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:06.284715 749 tablet_service.cc:1552] Processing DeleteTablet for tablet dfa8b1b9a80e4e8689e0acb44e16346a with delete_type TABLET_DATA_DELETED (replaced by tablet 97b2ecae572a4b7ea1b9595074556708) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:06.284808 1973 raft_consensus.cc:515] T 6ca9bd37de9e4be388501b15af4bda31 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.285369 1973 leader_election.cc:290] T 6ca9bd37de9e4be388501b15af4bda31 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
W20251029 02:52:06.286051 989 leader_election.cc:343] T 6ca9bd37de9e4be388501b15af4bda31 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 6ca9bd37de9e4be388501b15af4bda31
I20251029 02:52:06.286448 1903 raft_consensus.cc:515] T d5741dde463a43d6a776ab92dcfb04bf P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.287593 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d5741dde463a43d6a776ab92dcfb04bf" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:52:06.287797 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d5741dde463a43d6a776ab92dcfb04bf" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:06.287969 768 raft_consensus.cc:3060] T d5741dde463a43d6a776ab92dcfb04bf P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:06.288030 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ca9bd37de9e4be388501b15af4bda31" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:06.288090 849 leader_election.cc:343] T d5741dde463a43d6a776ab92dcfb04bf P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:06.288211 916 raft_consensus.cc:2468] T 6ca9bd37de9e4be388501b15af4bda31 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 0.
I20251029 02:52:06.288218 1903 leader_election.cc:290] T d5741dde463a43d6a776ab92dcfb04bf P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:06.288720 990 leader_election.cc:304] T 6ca9bd37de9e4be388501b15af4bda31 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters: b495a49349324f7d81908f2da700f0d9
I20251029 02:52:06.289203 1973 raft_consensus.cc:2804] T 6ca9bd37de9e4be388501b15af4bda31 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:06.289283 1973 raft_consensus.cc:493] T 6ca9bd37de9e4be388501b15af4bda31 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:06.289347 1973 raft_consensus.cc:3060] T 6ca9bd37de9e4be388501b15af4bda31 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:06.290318 1172 tablet_bootstrap.cc:492] T f529e17e97074dca91ec2f48954cfd32 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:06.290488 1172 ts_tablet_manager.cc:1403] T f529e17e97074dca91ec2f48954cfd32 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.025s user 0.005s sys 0.002s
I20251029 02:52:06.291260 1172 raft_consensus.cc:359] T f529e17e97074dca91ec2f48954cfd32 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.291450 1172 raft_consensus.cc:385] T f529e17e97074dca91ec2f48954cfd32 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:06.291519 1172 raft_consensus.cc:740] T f529e17e97074dca91ec2f48954cfd32 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:06.291815 1172 consensus_queue.cc:260] T f529e17e97074dca91ec2f48954cfd32 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.291906 1973 raft_consensus.cc:515] T 6ca9bd37de9e4be388501b15af4bda31 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.291906 768 raft_consensus.cc:2468] T d5741dde463a43d6a776ab92dcfb04bf P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:06.292320 847 leader_election.cc:304] T d5741dde463a43d6a776ab92dcfb04bf P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5
I20251029 02:52:06.292497 1172 ts_tablet_manager.cc:1434] T f529e17e97074dca91ec2f48954cfd32 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 02:52:06.292713 1903 raft_consensus.cc:2804] T d5741dde463a43d6a776ab92dcfb04bf P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:06.292801 1172 tablet_bootstrap.cc:492] T 777992acb1bd4765886b802940db74bb P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:06.292825 1903 raft_consensus.cc:697] T d5741dde463a43d6a776ab92dcfb04bf P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:52:06.293004 2036 raft_consensus.cc:493] T 0cc1957a845f4e9cb703a553b27d1347 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:06.293061 1903 consensus_queue.cc:237] T d5741dde463a43d6a776ab92dcfb04bf P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.293107 2036 raft_consensus.cc:515] T 0cc1957a845f4e9cb703a553b27d1347 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.293702 2036 leader_election.cc:290] T 0cc1957a845f4e9cb703a553b27d1347 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:06.294421 1172 tablet_bootstrap.cc:654] T 777992acb1bd4765886b802940db74bb P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:06.295516 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0cc1957a845f4e9cb703a553b27d1347" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:06.296020 990 leader_election.cc:343] T 0cc1957a845f4e9cb703a553b27d1347 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
W20251029 02:52:06.296496 989 leader_election.cc:343] T 0cc1957a845f4e9cb703a553b27d1347 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 0cc1957a845f4e9cb703a553b27d1347
I20251029 02:52:06.296617 989 leader_election.cc:304] T 0cc1957a845f4e9cb703a553b27d1347 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5; no voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:06.296949 2036 raft_consensus.cc:2749] T 0cc1957a845f4e9cb703a553b27d1347 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:06.298122 1973 leader_election.cc:290] T 6ca9bd37de9e4be388501b15af4bda31 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Requested vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:06.298691 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6ca9bd37de9e4be388501b15af4bda31" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
I20251029 02:52:06.298610 517 catalog_manager.cc:5649] T c9e923724ca9402682f13b252dbc37aa P 0cfd4bee16f84b7c8b0ff3ce14db20d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } health_report { overall_health: HEALTHY } } }
I20251029 02:52:06.298903 916 raft_consensus.cc:3060] T 6ca9bd37de9e4be388501b15af4bda31 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:06.299042 517 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet c9e923724ca9402682f13b252dbc37aa (table default.Table_443f33587b8f428484b99e07355d69ca [id=03a25420ce6a46fc93a9fb753246bc7b]). Expected version 3 got 0
W20251029 02:52:06.300006 989 leader_election.cc:343] T 6ca9bd37de9e4be388501b15af4bda31 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 6ca9bd37de9e4be388501b15af4bda31
I20251029 02:52:06.301733 890 tablet_service.cc:1505] Processing CreateTablet for tablet d22b01dc872b4ebb8f65b586b29eea00 (DEFAULT_TABLE table=default.Table_ced93e731339479fa9ca51ef65dea8a6 [id=32a37b70496a42e8a5d670cc428070e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:06.302093 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d22b01dc872b4ebb8f65b586b29eea00. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:06.302294 1167 tablet_bootstrap.cc:492] T 4aa78f4ac38e4919864c924e50f1aaf4 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:06.302260 916 raft_consensus.cc:2468] T 6ca9bd37de9e4be388501b15af4bda31 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 1.
I20251029 02:52:06.302404 1167 ts_tablet_manager.cc:1403] T 4aa78f4ac38e4919864c924e50f1aaf4 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.043s user 0.002s sys 0.001s
I20251029 02:52:06.303162 1167 raft_consensus.cc:359] T 4aa78f4ac38e4919864c924e50f1aaf4 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.303310 1167 raft_consensus.cc:385] T 4aa78f4ac38e4919864c924e50f1aaf4 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:06.303355 1167 raft_consensus.cc:740] T 4aa78f4ac38e4919864c924e50f1aaf4 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:06.303553 1167 consensus_queue.cc:260] T 4aa78f4ac38e4919864c924e50f1aaf4 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.303978 1167 ts_tablet_manager.cc:1434] T 4aa78f4ac38e4919864c924e50f1aaf4 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 02:52:06.305219 1167 tablet_bootstrap.cc:492] T f187dbd8a55b45b292ea61db43caf47d P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:06.306591 1167 tablet_bootstrap.cc:654] T f187dbd8a55b45b292ea61db43caf47d P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:06.307081 990 leader_election.cc:304] T 6ca9bd37de9e4be388501b15af4bda31 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters: b495a49349324f7d81908f2da700f0d9
I20251029 02:52:06.307363 1973 raft_consensus.cc:2804] T 6ca9bd37de9e4be388501b15af4bda31 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:06.308245 1973 raft_consensus.cc:697] T 6ca9bd37de9e4be388501b15af4bda31 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Becoming Leader. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Running, Role: LEADER
I20251029 02:52:06.308490 1973 consensus_queue.cc:237] T 6ca9bd37de9e4be388501b15af4bda31 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.313109 1172 tablet_bootstrap.cc:492] T 777992acb1bd4765886b802940db74bb P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:06.313308 1172 ts_tablet_manager.cc:1403] T 777992acb1bd4765886b802940db74bb P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.000s
I20251029 02:52:06.314929 1174 ts_tablet_manager.cc:1929] T 24cb2a71ba3841d4a5e3e5c6fe99ae2f P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:06.315078 1174 log.cc:1199] T 24cb2a71ba3841d4a5e3e5c6fe99ae2f P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/24cb2a71ba3841d4a5e3e5c6fe99ae2f
I20251029 02:52:06.315383 1174 ts_tablet_manager.cc:1950] T 24cb2a71ba3841d4a5e3e5c6fe99ae2f P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:06.316072 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 705) took 1636 ms. Trace:
I20251029 02:52:06.316172 1174 rpcz_store.cc:276] 1029 02:52:04.679395 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:04.869886 (+190491us) service_pool.cc:225] Handling call
1029 02:52:06.268533 (+1398647us) ts_tablet_manager.cc:1151] Deleting tablet 24cb2a71ba3841d4a5e3e5c6fe99ae2f
1029 02:52:06.268541 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:06.296001 (+ 27460us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:06.314768 (+ 18767us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:06.316052 (+ 1284us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1398370}
I20251029 02:52:06.316834 1172 raft_consensus.cc:359] T 777992acb1bd4765886b802940db74bb P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.317070 1172 raft_consensus.cc:385] T 777992acb1bd4765886b802940db74bb P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:06.317159 1172 raft_consensus.cc:740] T 777992acb1bd4765886b802940db74bb P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:06.317479 1172 consensus_queue.cc:260] T 777992acb1bd4765886b802940db74bb P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.318295 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 24cb2a71ba3841d4a5e3e5c6fe99ae2f (table default.Table_4350e92f14de4ba8a04afbe12ea7422a [id=226a2d412a7b40e59b6f5340b5a9b739]) successfully deleted
W20251029 02:52:06.319402 510 consensus_peers.cc:597] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0 -> Peer e9672114a6a742e2864c16ff60ba246e (127.0.104.62:42343): Couldn't send request to peer e9672114a6a742e2864c16ff60ba246e. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.62:42343: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20251029 02:52:06.320470 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 706) took 1500 ms. Trace:
I20251029 02:52:06.320561 1174 rpcz_store.cc:276] 1029 02:52:04.819649 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:04.873723 (+ 54074us) service_pool.cc:225] Handling call
1029 02:52:06.320339 (+1446616us) ts_tablet_manager.cc:1151] Deleting tablet ba699862e6f646c6beaa39f080d0faba
1029 02:52:06.320350 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:06.320452 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1446316}
W20251029 02:52:06.321923 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet ba699862e6f646c6beaa39f080d0faba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ba699862e6f646c6beaa39f080d0faba already in progress: creating tablet
I20251029 02:52:06.322420 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 709) took 1208 ms. Trace:
I20251029 02:52:06.322808 1174 rpcz_store.cc:276] 1029 02:52:05.114352 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:05.116816 (+ 2464us) service_pool.cc:225] Handling call
1029 02:52:06.322328 (+1205512us) ts_tablet_manager.cc:1151] Deleting tablet 6feec2c8e030447fae9c292a626dcd1d
1029 02:52:06.322340 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:06.322404 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1205208}
W20251029 02:52:06.323501 512 catalog_manager.cc:4952] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 6feec2c8e030447fae9c292a626dcd1d because the tablet was not found. No further retry: Not found: Tablet not found: 6feec2c8e030447fae9c292a626dcd1d
I20251029 02:52:06.323894 1174 tablet_replica.cc:333] T 576c4e6e70e24b0399ac9597a30f0569 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:06.324311 1174 raft_consensus.cc:2243] T 576c4e6e70e24b0399ac9597a30f0569 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:06.324751 1174 raft_consensus.cc:2272] T 576c4e6e70e24b0399ac9597a30f0569 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:06.325495 1174 ts_tablet_manager.cc:1916] T 576c4e6e70e24b0399ac9597a30f0569 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:06.326831 1172 ts_tablet_manager.cc:1434] T 777992acb1bd4765886b802940db74bb P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.013s user 0.003s sys 0.000s
I20251029 02:52:06.327554 526 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 00f2e98739fa41eea23927494fe222f3
I20251029 02:52:06.327566 1039 tablet_service.cc:1505] Processing CreateTablet for tablet d22b01dc872b4ebb8f65b586b29eea00 (DEFAULT_TABLE table=default.Table_ced93e731339479fa9ca51ef65dea8a6 [id=32a37b70496a42e8a5d670cc428070e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:06.327608 749 tablet_service.cc:1505] Processing CreateTablet for tablet d22b01dc872b4ebb8f65b586b29eea00 (DEFAULT_TABLE table=default.Table_ced93e731339479fa9ca51ef65dea8a6 [id=32a37b70496a42e8a5d670cc428070e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:06.327993 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d22b01dc872b4ebb8f65b586b29eea00. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:06.328202 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d22b01dc872b4ebb8f65b586b29eea00. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:06.329272 748 tablet_service.cc:1552] Processing DeleteTablet for tablet 00f2e98739fa41eea23927494fe222f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:06 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:06.331730 1172 tablet_bootstrap.cc:492] T 1c53fc1fff1a4bb988beb46ff91fd3be P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:06.333451 1031 tablet_service.cc:1552] Processing DeleteTablet for tablet 00f2e98739fa41eea23927494fe222f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:06 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:06.333552 1172 tablet_bootstrap.cc:654] T 1c53fc1fff1a4bb988beb46ff91fd3be P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:06.336642 890 tablet_service.cc:1552] Processing DeleteTablet for tablet 00f2e98739fa41eea23927494fe222f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:06 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:06.338294 1167 tablet_bootstrap.cc:492] T f187dbd8a55b45b292ea61db43caf47d P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:06.338477 1167 ts_tablet_manager.cc:1403] T f187dbd8a55b45b292ea61db43caf47d P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.033s user 0.005s sys 0.001s
I20251029 02:52:06.339202 1167 raft_consensus.cc:359] T f187dbd8a55b45b292ea61db43caf47d P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.339425 1167 raft_consensus.cc:385] T f187dbd8a55b45b292ea61db43caf47d P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:06.339534 1167 raft_consensus.cc:740] T f187dbd8a55b45b292ea61db43caf47d P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:06.339781 1167 consensus_queue.cc:260] T f187dbd8a55b45b292ea61db43caf47d P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.340564 1167 ts_tablet_manager.cc:1434] T f187dbd8a55b45b292ea61db43caf47d P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251029 02:52:06.341161 1167 tablet_bootstrap.cc:492] T 0cc1957a845f4e9cb703a553b27d1347 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:06.343127 1167 tablet_bootstrap.cc:654] T 0cc1957a845f4e9cb703a553b27d1347 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:06.346611 1973 raft_consensus.cc:493] T e1fa4980f2624321b617b84c68b90113 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:06.346768 1973 raft_consensus.cc:515] T e1fa4980f2624321b617b84c68b90113 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.347426 1973 leader_election.cc:290] T e1fa4980f2624321b617b84c68b90113 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:52:06.348099 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1fa4980f2624321b617b84c68b90113" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:06.348305 989 leader_election.cc:343] T e1fa4980f2624321b617b84c68b90113 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: e1fa4980f2624321b617b84c68b90113
W20251029 02:52:06.348719 990 leader_election.cc:343] T e1fa4980f2624321b617b84c68b90113 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:06.348824 990 leader_election.cc:304] T e1fa4980f2624321b617b84c68b90113 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5; no voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:06.349190 1174 ts_tablet_manager.cc:1929] T 576c4e6e70e24b0399ac9597a30f0569 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:06.349295 1174 log.cc:1199] T 576c4e6e70e24b0399ac9597a30f0569 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/576c4e6e70e24b0399ac9597a30f0569
I20251029 02:52:06.349295 1973 raft_consensus.cc:2749] T e1fa4980f2624321b617b84c68b90113 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:06.349359 1167 tablet_bootstrap.cc:492] T 0cc1957a845f4e9cb703a553b27d1347 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:06.349479 1167 ts_tablet_manager.cc:1403] T 0cc1957a845f4e9cb703a553b27d1347 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20251029 02:52:06.349583 1174 ts_tablet_manager.cc:1950] T 576c4e6e70e24b0399ac9597a30f0569 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:06.350365 1167 raft_consensus.cc:359] T 0cc1957a845f4e9cb703a553b27d1347 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.350581 1167 raft_consensus.cc:385] T 0cc1957a845f4e9cb703a553b27d1347 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:06.350641 1167 raft_consensus.cc:740] T 0cc1957a845f4e9cb703a553b27d1347 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:06.350904 1167 consensus_queue.cc:260] T 0cc1957a845f4e9cb703a553b27d1347 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.351490 1167 ts_tablet_manager.cc:1434] T 0cc1957a845f4e9cb703a553b27d1347 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 02:52:06.352543 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 717) took 1104 ms. Trace:
I20251029 02:52:06.352632 1174 rpcz_store.cc:276] 1029 02:52:05.248134 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:05.248553 (+ 419us) service_pool.cc:225] Handling call
1029 02:52:06.323832 (+1075279us) ts_tablet_manager.cc:1151] Deleting tablet 576c4e6e70e24b0399ac9597a30f0569
1029 02:52:06.323842 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:06.338754 (+ 14912us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:06.349073 (+ 10319us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:06.352525 (+ 3452us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1075039}
I20251029 02:52:06.353089 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 576c4e6e70e24b0399ac9597a30f0569 (table default.Table_e8685346a99d41078ecd38cd4ce4fec5 [id=7774a12db22f414f9266236d92b3b511]) successfully deleted
I20251029 02:52:06.356976 1175 ts_tablet_manager.cc:1929] T 71068739b4384badbf305313d902f0f5 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:06.357144 1175 log.cc:1199] T 71068739b4384badbf305313d902f0f5 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/71068739b4384badbf305313d902f0f5
I20251029 02:52:06.357446 1175 ts_tablet_manager.cc:1950] T 71068739b4384badbf305313d902f0f5 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:06.358234 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 673) took 2352 ms. Trace:
I20251029 02:52:06.358920 1167 tablet_bootstrap.cc:492] T cc74e83833754f0ca19046031a24ebb9 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:06.359413 1172 tablet_bootstrap.cc:492] T 1c53fc1fff1a4bb988beb46ff91fd3be P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:06.359576 1172 ts_tablet_manager.cc:1403] T 1c53fc1fff1a4bb988beb46ff91fd3be P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.001s
I20251029 02:52:06.360376 1172 raft_consensus.cc:359] T 1c53fc1fff1a4bb988beb46ff91fd3be P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.360585 1172 raft_consensus.cc:385] T 1c53fc1fff1a4bb988beb46ff91fd3be P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:06.360628 1167 tablet_bootstrap.cc:654] T cc74e83833754f0ca19046031a24ebb9 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:06.360658 1172 raft_consensus.cc:740] T 1c53fc1fff1a4bb988beb46ff91fd3be P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:06.360962 1172 consensus_queue.cc:260] T 1c53fc1fff1a4bb988beb46ff91fd3be P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.362406 1172 ts_tablet_manager.cc:1434] T 1c53fc1fff1a4bb988beb46ff91fd3be P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251029 02:52:06.364063 1176 ts_tablet_manager.cc:1929] T 71068739b4384badbf305313d902f0f5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:06.364169 1176 log.cc:1199] T 71068739b4384badbf305313d902f0f5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/71068739b4384badbf305313d902f0f5
I20251029 02:52:06.364351 1172 tablet_bootstrap.cc:492] T 7a1866afc7564217ad777e7ce53786c5 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:06.364514 1176 ts_tablet_manager.cc:1950] T 71068739b4384badbf305313d902f0f5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:06.365443 1176 tablet_replica.cc:333] T 74873a05ef634867a30aa63b31fcfcc7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:06.365612 1176 raft_consensus.cc:2243] T 74873a05ef634867a30aa63b31fcfcc7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:06.365696 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 71068739b4384badbf305313d902f0f5 (table default.Table_4e899dcc04fe4cb1b96ced1fa1b03585 [id=992716bbc1694bd6b85e4c8a3ab51e40]) successfully deleted
I20251029 02:52:06.365751 1176 raft_consensus.cc:2272] T 74873a05ef634867a30aa63b31fcfcc7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:06.366080 1172 tablet_bootstrap.cc:654] T 7a1866afc7564217ad777e7ce53786c5 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:06.366225 1174 tablet_replica.cc:333] T 7e6c4f30be624769961c9ec17c5d5dcb P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:06.366240 1176 ts_tablet_manager.cc:1916] T 74873a05ef634867a30aa63b31fcfcc7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:06.366595 1174 raft_consensus.cc:2243] T 7e6c4f30be624769961c9ec17c5d5dcb P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:06.366930 1174 pending_rounds.cc:70] T 7e6c4f30be624769961c9ec17c5d5dcb P c756528642a543eeacd93cef3e335a7d: Trying to abort 1 pending ops.
I20251029 02:52:06.367023 1174 pending_rounds.cc:77] T 7e6c4f30be624769961c9ec17c5d5dcb P c756528642a543eeacd93cef3e335a7d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215949108640370688 op_type: NO_OP noop_request { }
I20251029 02:52:06.367163 1174 raft_consensus.cc:2889] T 7e6c4f30be624769961c9ec17c5d5dcb P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 02:52:06.367290 1174 raft_consensus.cc:2272] T 7e6c4f30be624769961c9ec17c5d5dcb P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:06.368132 1174 ts_tablet_manager.cc:1916] T 7e6c4f30be624769961c9ec17c5d5dcb P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:06.371896 2018 raft_consensus.cc:493] T dfa8b1b9a80e4e8689e0acb44e16346a P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:06.372118 2018 raft_consensus.cc:515] T dfa8b1b9a80e4e8689e0acb44e16346a P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.372767 2018 leader_election.cc:290] T dfa8b1b9a80e4e8689e0acb44e16346a P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:06.373522 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dfa8b1b9a80e4e8689e0acb44e16346a" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:06.373950 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dfa8b1b9a80e4e8689e0acb44e16346a" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:06.374645 712 leader_election.cc:343] T dfa8b1b9a80e4e8689e0acb44e16346a P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
W20251029 02:52:06.375015 711 leader_election.cc:343] T dfa8b1b9a80e4e8689e0acb44e16346a P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:06.375154 711 leader_election.cc:304] T dfa8b1b9a80e4e8689e0acb44e16346a P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:06.375617 2018 raft_consensus.cc:2749] T dfa8b1b9a80e4e8689e0acb44e16346a P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:06.380326 1176 ts_tablet_manager.cc:1929] T 74873a05ef634867a30aa63b31fcfcc7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:06.381007 749 tablet_service.cc:1505] Processing CreateTablet for tablet 874b1728fd174305977b6852921928e7 (DEFAULT_TABLE table=default.Table_3266aac3f5744a23954b5465d9af1b31 [id=6ed604ab09a04c71adcf97bcdc9edd20]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:06.381579 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 874b1728fd174305977b6852921928e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:06.380266 1039 tablet_service.cc:1505] Processing CreateTablet for tablet 874b1728fd174305977b6852921928e7 (DEFAULT_TABLE table=default.Table_3266aac3f5744a23954b5465d9af1b31 [id=6ed604ab09a04c71adcf97bcdc9edd20]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:06.383481 1176 log.cc:1199] T 74873a05ef634867a30aa63b31fcfcc7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/74873a05ef634867a30aa63b31fcfcc7
I20251029 02:52:06.383488 1174 ts_tablet_manager.cc:1929] T 7e6c4f30be624769961c9ec17c5d5dcb P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:06.383611 1174 log.cc:1199] T 7e6c4f30be624769961c9ec17c5d5dcb P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/7e6c4f30be624769961c9ec17c5d5dcb
I20251029 02:52:06.383790 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 874b1728fd174305977b6852921928e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:06.383970 1174 ts_tablet_manager.cc:1950] T 7e6c4f30be624769961c9ec17c5d5dcb P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:06.384313 1176 ts_tablet_manager.cc:1950] T 74873a05ef634867a30aa63b31fcfcc7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:06.384773 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 716) took 1136 ms. Trace:
I20251029 02:52:06.384874 1174 rpcz_store.cc:276] 1029 02:52:05.248028 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:05.252303 (+ 4275us) service_pool.cc:225] Handling call
1029 02:52:06.355139 (+1102836us) ts_tablet_manager.cc:1151] Deleting tablet 7e6c4f30be624769961c9ec17c5d5dcb
1029 02:52:06.366119 (+ 10980us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:06.371301 (+ 5182us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:06.383376 (+ 12075us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:06.384751 (+ 1375us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1102674}
I20251029 02:52:06.385933 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 74873a05ef634867a30aa63b31fcfcc7 (table default.Table_4e97f5e298434c16913bf7ca93ca30f5 [id=48dfcbe8b3654d72b51b77473dd918f6]) successfully deleted
W20251029 02:52:06.387080 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 72f9d4408f384e7093316d85f092e1b2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 72f9d4408f384e7093316d85f092e1b2 already in progress: creating tablet
W20251029 02:52:06.387979 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 8f100a12d78145c2b8586a0713fc70c4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8f100a12d78145c2b8586a0713fc70c4 already in progress: creating tablet
W20251029 02:52:06.388763 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 143f4a42001b43e683bbc6226c0550a9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 143f4a42001b43e683bbc6226c0550a9 already in progress: creating tablet
W20251029 02:52:06.389708 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet ad214c870ee649279ed7321c5b69cbee because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ad214c870ee649279ed7321c5b69cbee already in progress: creating tablet
I20251029 02:52:06.390200 1176 tablet_replica.cc:333] T 59af906061d049c99ae2153945597007 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:06.391446 1167 tablet_bootstrap.cc:492] T cc74e83833754f0ca19046031a24ebb9 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:06.391700 1167 ts_tablet_manager.cc:1403] T cc74e83833754f0ca19046031a24ebb9 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.033s user 0.008s sys 0.008s
I20251029 02:52:06.392701 1167 raft_consensus.cc:359] T cc74e83833754f0ca19046031a24ebb9 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.393025 1167 raft_consensus.cc:385] T cc74e83833754f0ca19046031a24ebb9 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:06.393146 1167 raft_consensus.cc:740] T cc74e83833754f0ca19046031a24ebb9 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:06.393503 1167 consensus_queue.cc:260] T cc74e83833754f0ca19046031a24ebb9 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.394316 1167 ts_tablet_manager.cc:1434] T cc74e83833754f0ca19046031a24ebb9 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 02:52:06.391259 1166 tablet_bootstrap.cc:492] T 4aa78f4ac38e4919864c924e50f1aaf4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:06.394593 1166 ts_tablet_manager.cc:1403] T 4aa78f4ac38e4919864c924e50f1aaf4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.158s user 0.008s sys 0.023s
I20251029 02:52:06.395504 1166 raft_consensus.cc:359] T 4aa78f4ac38e4919864c924e50f1aaf4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.395589 1039 tablet_service.cc:1505] Processing CreateTablet for tablet e0238e9cae1b40708ab326f0de7ecada (DEFAULT_TABLE table=default.Table_111cfdff9be446e09a352fd6f3265ec5 [id=9066f36249d04b01b672e86ab43b8730]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:06.395655 1166 raft_consensus.cc:385] T 4aa78f4ac38e4919864c924e50f1aaf4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:06.395714 1166 raft_consensus.cc:740] T 4aa78f4ac38e4919864c924e50f1aaf4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:06.395983 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0238e9cae1b40708ab326f0de7ecada. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:06.395897 1166 consensus_queue.cc:260] T 4aa78f4ac38e4919864c924e50f1aaf4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.396365 1166 ts_tablet_manager.cc:1434] T 4aa78f4ac38e4919864c924e50f1aaf4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 02:52:06.396538 748 tablet_service.cc:1505] Processing CreateTablet for tablet e0238e9cae1b40708ab326f0de7ecada (DEFAULT_TABLE table=default.Table_111cfdff9be446e09a352fd6f3265ec5 [id=9066f36249d04b01b672e86ab43b8730]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:06.396644 1166 tablet_bootstrap.cc:492] T cc74e83833754f0ca19046031a24ebb9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:06.396975 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0238e9cae1b40708ab326f0de7ecada. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:06.397396 1031 tablet_service.cc:1505] Processing CreateTablet for tablet f68e9c8d268e424eac1fa14b9b711738 (DEFAULT_TABLE table=default.Table_65e6cdfafc9f46e8ac0c16c6babfc7d8 [id=dbfc4c7696244c04b7f312da49fc4dae]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:06.398402 1029 tablet_service.cc:1505] Processing CreateTablet for tablet bfb9115f324f44c98f876a2d56bb23a1 (DEFAULT_TABLE table=default.Table_ddd7ee9b37df4bb4be7957aef7e82cb8 [id=91a2a75283864b60a04fa1183a6fd911]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:06.398725 1973 raft_consensus.cc:493] T 56d05e4a500043c995bb51a204cb295d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:06.398836 1973 raft_consensus.cc:515] T 56d05e4a500043c995bb51a204cb295d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.399199 747 tablet_service.cc:1505] Processing CreateTablet for tablet f68e9c8d268e424eac1fa14b9b711738 (DEFAULT_TABLE table=default.Table_65e6cdfafc9f46e8ac0c16c6babfc7d8 [id=dbfc4c7696244c04b7f312da49fc4dae]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:06.399395 1973 leader_election.cc:290] T 56d05e4a500043c995bb51a204cb295d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:06.399619 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f68e9c8d268e424eac1fa14b9b711738. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:06.400018 1166 tablet_bootstrap.cc:654] T cc74e83833754f0ca19046031a24ebb9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:06.400303 1041 tablet_service.cc:1505] Processing CreateTablet for tablet 00ebd0c68c4441f9b63c86cdab62497d (DEFAULT_TABLE table=default.Table_ba4b224d24404916b18cb61a05d08b75 [id=4e01b5a6ecd545bface69b56b8cf07d0]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 02:52:06.402042 989 leader_election.cc:343] T 56d05e4a500043c995bb51a204cb295d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 56d05e4a500043c995bb51a204cb295d
I20251029 02:52:06.402532 1167 tablet_bootstrap.cc:492] T daa05a34c0684c07a35966290fd7d44c P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:06.399534 756 tablet_service.cc:1505] Processing CreateTablet for tablet bfb9115f324f44c98f876a2d56bb23a1 (DEFAULT_TABLE table=default.Table_ddd7ee9b37df4bb4be7957aef7e82cb8 [id=91a2a75283864b60a04fa1183a6fd911]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:06.403342 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bfb9115f324f44c98f876a2d56bb23a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:06.403282 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "56d05e4a500043c995bb51a204cb295d" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:06.404218 1167 tablet_bootstrap.cc:654] T daa05a34c0684c07a35966290fd7d44c P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:06.395051 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 715) took 1147 ms. Trace:
I20251029 02:52:06.404431 1174 rpcz_store.cc:276] 1029 02:52:05.247944 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:05.252254 (+ 4310us) service_pool.cc:225] Handling call
1029 02:52:06.389914 (+1137660us) ts_tablet_manager.cc:1151] Deleting tablet c65469173762446e9725262ae998dbf3
1029 02:52:06.394946 (+ 5032us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:06.395031 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1133388}
I20251029 02:52:06.405175 1037 tablet_service.cc:1505] Processing CreateTablet for tablet 595b4073cc0446cea041838097af7471 (DEFAULT_TABLE table=default.Table_47a306977f08434993da629ff2ed490a [id=fee2c31d1a3f4da4a2b24477aefa48e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:06.405458 1174 tablet_replica.cc:333] T f81aecb20d6e498bb9b498698886d65e P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:06.405525 1176 raft_consensus.cc:2243] T 59af906061d049c99ae2153945597007 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:06.405579 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f68e9c8d268e424eac1fa14b9b711738. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:06.405632 1174 raft_consensus.cc:2243] T f81aecb20d6e498bb9b498698886d65e P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:06.405773 1174 raft_consensus.cc:2272] T f81aecb20d6e498bb9b498698886d65e P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:06.405848 1176 raft_consensus.cc:2272] T 59af906061d049c99ae2153945597007 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:06.406275 1174 ts_tablet_manager.cc:1916] T f81aecb20d6e498bb9b498698886d65e P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:06.406347 1176 ts_tablet_manager.cc:1916] T 59af906061d049c99ae2153945597007 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:06.406774 742 tablet_service.cc:1505] Processing CreateTablet for tablet 00ebd0c68c4441f9b63c86cdab62497d (DEFAULT_TABLE table=default.Table_ba4b224d24404916b18cb61a05d08b75 [id=4e01b5a6ecd545bface69b56b8cf07d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:06.407157 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00ebd0c68c4441f9b63c86cdab62497d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:06.412585 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bfb9115f324f44c98f876a2d56bb23a1. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:06.413882 990 leader_election.cc:343] T 56d05e4a500043c995bb51a204cb295d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:06.414038 990 leader_election.cc:304] T 56d05e4a500043c995bb51a204cb295d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5; no voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:06.415338 1973 raft_consensus.cc:2749] T 56d05e4a500043c995bb51a204cb295d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:06.416245 1973 raft_consensus.cc:493] T 4aa78f4ac38e4919864c924e50f1aaf4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:06.416354 1973 raft_consensus.cc:515] T 4aa78f4ac38e4919864c924e50f1aaf4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.416901 1973 leader_election.cc:290] T 4aa78f4ac38e4919864c924e50f1aaf4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:06.417165 2018 raft_consensus.cc:493] T ccdca31db2834de4b6ed0f2d67a10535 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:06.417344 2018 raft_consensus.cc:515] T ccdca31db2834de4b6ed0f2d67a10535 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.417943 2018 leader_election.cc:290] T ccdca31db2834de4b6ed0f2d67a10535 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:06.418020 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4aa78f4ac38e4919864c924e50f1aaf4" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:06.418201 916 raft_consensus.cc:2468] T 4aa78f4ac38e4919864c924e50f1aaf4 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 0.
I20251029 02:52:06.418439 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ccdca31db2834de4b6ed0f2d67a10535" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:06.418613 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00ebd0c68c4441f9b63c86cdab62497d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:06.418836 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ccdca31db2834de4b6ed0f2d67a10535" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:52:06.418985 711 leader_election.cc:343] T ccdca31db2834de4b6ed0f2d67a10535 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:06.419399 1166 tablet_bootstrap.cc:492] T cc74e83833754f0ca19046031a24ebb9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:06.419525 1166 ts_tablet_manager.cc:1403] T cc74e83833754f0ca19046031a24ebb9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.023s user 0.005s sys 0.003s
I20251029 02:52:06.419857 990 leader_election.cc:304] T 4aa78f4ac38e4919864c924e50f1aaf4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:06.420256 1166 raft_consensus.cc:359] T cc74e83833754f0ca19046031a24ebb9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.420394 1166 raft_consensus.cc:385] T cc74e83833754f0ca19046031a24ebb9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:06.420441 1166 raft_consensus.cc:740] T cc74e83833754f0ca19046031a24ebb9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:06.420614 1166 consensus_queue.cc:260] T cc74e83833754f0ca19046031a24ebb9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.421206 1166 ts_tablet_manager.cc:1434] T cc74e83833754f0ca19046031a24ebb9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 02:52:06.421779 1166 tablet_bootstrap.cc:492] T f60e7c24982d40319801869ad060864d P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:06.422569 1973 raft_consensus.cc:2804] T 4aa78f4ac38e4919864c924e50f1aaf4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:06.422647 1973 raft_consensus.cc:493] T 4aa78f4ac38e4919864c924e50f1aaf4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:06.422720 1973 raft_consensus.cc:3060] T 4aa78f4ac38e4919864c924e50f1aaf4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:06.423367 1166 tablet_bootstrap.cc:654] T f60e7c24982d40319801869ad060864d P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:06.423945 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 595b4073cc0446cea041838097af7471. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:06.425331 1973 raft_consensus.cc:515] T 4aa78f4ac38e4919864c924e50f1aaf4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.425876 1973 leader_election.cc:290] T 4aa78f4ac38e4919864c924e50f1aaf4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Requested vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:06.426026 1176 ts_tablet_manager.cc:1929] T 59af906061d049c99ae2153945597007 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251029 02:52:06.426113 1176 log.cc:1199] T 59af906061d049c99ae2153945597007 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/59af906061d049c99ae2153945597007
I20251029 02:52:06.426416 1176 ts_tablet_manager.cc:1950] T 59af906061d049c99ae2153945597007 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:06.426853 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "4aa78f4ac38e4919864c924e50f1aaf4" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
I20251029 02:52:06.427093 920 raft_consensus.cc:3060] T 4aa78f4ac38e4919864c924e50f1aaf4 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
W20251029 02:52:06.427613 989 leader_election.cc:343] T 4aa78f4ac38e4919864c924e50f1aaf4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 4aa78f4ac38e4919864c924e50f1aaf4
W20251029 02:52:06.428159 847 consensus_peers.cc:597] T b2b2d64dcf964da8be4b544dbbe31977 P c756528642a543eeacd93cef3e335a7d -> Peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Couldn't send request to peer b495a49349324f7d81908f2da700f0d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b2b2d64dcf964da8be4b544dbbe31977. This is attempt 1: this message will repeat every 5th retry.
W20251029 02:52:06.428130 712 leader_election.cc:343] T ccdca31db2834de4b6ed0f2d67a10535 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:06.428239 712 leader_election.cc:304] T ccdca31db2834de4b6ed0f2d67a10535 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:06.428480 2018 raft_consensus.cc:2749] T ccdca31db2834de4b6ed0f2d67a10535 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 02:52:06.428927 989 leader_election.cc:343] T 4aa78f4ac38e4919864c924e50f1aaf4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 4aa78f4ac38e4919864c924e50f1aaf4
I20251029 02:52:06.429414 1903 raft_consensus.cc:493] T 6bd3fac2502e479aa9f4ceaabbb39504 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:06.429536 1903 raft_consensus.cc:515] T 6bd3fac2502e479aa9f4ceaabbb39504 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.430078 1903 leader_election.cc:290] T 6bd3fac2502e479aa9f4ceaabbb39504 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:06.430320 920 raft_consensus.cc:2468] T 4aa78f4ac38e4919864c924e50f1aaf4 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 1.
W20251029 02:52:06.430812 849 leader_election.cc:343] T 6bd3fac2502e479aa9f4ceaabbb39504 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Not found: Tablet not found: 6bd3fac2502e479aa9f4ceaabbb39504
I20251029 02:52:06.430874 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 59af906061d049c99ae2153945597007 (table default.Table_6da557988d044647b5e68f09b3f00407 [id=25f16d95843d41c981d6a07ec0d4b48a]) successfully deleted
W20251029 02:52:06.431049 847 leader_election.cc:343] T 6bd3fac2502e479aa9f4ceaabbb39504 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 6bd3fac2502e479aa9f4ceaabbb39504
I20251029 02:52:06.431144 847 leader_election.cc:304] T 6bd3fac2502e479aa9f4ceaabbb39504 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9
I20251029 02:52:06.431545 1903 raft_consensus.cc:2749] T 6bd3fac2502e479aa9f4ceaabbb39504 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:06.431944 990 leader_election.cc:304] T 4aa78f4ac38e4919864c924e50f1aaf4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters: b495a49349324f7d81908f2da700f0d9
I20251029 02:52:06.432822 1973 raft_consensus.cc:2804] T 4aa78f4ac38e4919864c924e50f1aaf4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election won for term 1
W20251029 02:52:06.432983 513 catalog_manager.cc:4952] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 00f2e98739fa41eea23927494fe222f3 because the tablet was not found. No further retry: Not found: Tablet not found: 00f2e98739fa41eea23927494fe222f3
I20251029 02:52:06.433115 1973 raft_consensus.cc:697] T 4aa78f4ac38e4919864c924e50f1aaf4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Becoming Leader. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Running, Role: LEADER
I20251029 02:52:06.433354 1973 consensus_queue.cc:237] T 4aa78f4ac38e4919864c924e50f1aaf4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.435914 2036 raft_consensus.cc:493] T f246f6c202cf44d48c9f5e4f335202c5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:06.436038 2036 raft_consensus.cc:515] T f246f6c202cf44d48c9f5e4f335202c5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.436631 2036 leader_election.cc:290] T f246f6c202cf44d48c9f5e4f335202c5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
W20251029 02:52:06.438019 989 leader_election.cc:343] T f246f6c202cf44d48c9f5e4f335202c5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: f246f6c202cf44d48c9f5e4f335202c5
I20251029 02:52:06.438687 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f246f6c202cf44d48c9f5e4f335202c5" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:06.439272 990 leader_election.cc:343] T f246f6c202cf44d48c9f5e4f335202c5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:06.439393 990 leader_election.cc:304] T f246f6c202cf44d48c9f5e4f335202c5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5; no voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:06.439795 2036 raft_consensus.cc:2749] T f246f6c202cf44d48c9f5e4f335202c5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:06.441579 1174 ts_tablet_manager.cc:1929] T f81aecb20d6e498bb9b498698886d65e P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:06.445602 1174 log.cc:1199] T f81aecb20d6e498bb9b498698886d65e P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/f81aecb20d6e498bb9b498698886d65e
I20251029 02:52:06.446146 1174 ts_tablet_manager.cc:1950] T f81aecb20d6e498bb9b498698886d65e P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:06.448761 1174 tablet_replica.cc:333] T dd90fb9eb167478cbc3ec9363af86295 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:06.449239 1174 raft_consensus.cc:2243] T dd90fb9eb167478cbc3ec9363af86295 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:06.449702 1174 raft_consensus.cc:2272] T dd90fb9eb167478cbc3ec9363af86295 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:06.450486 1174 ts_tablet_manager.cc:1916] T dd90fb9eb167478cbc3ec9363af86295 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 02:52:06.450765 512 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.60:41233, user_credentials={real_user=slave}} blocked reactor thread for 84684.4us
I20251029 02:52:06.451676 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 7e6c4f30be624769961c9ec17c5d5dcb (table default.Table_dea0766540d24b5e9bd6cc8b6338b47c [id=1dfd5b42973c4d5d91e0d9014e34269f]) successfully deleted
W20251029 02:52:06.452005 512 catalog_manager.cc:4952] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet c65469173762446e9725262ae998dbf3 because the tablet was not found. No further retry: Not found: Tablet not found: c65469173762446e9725262ae998dbf3
W20251029 02:52:06.452183 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 4a55f22b0d3b4ebe98ec2e054d05b095 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4a55f22b0d3b4ebe98ec2e054d05b095 already in progress: creating tablet
W20251029 02:52:06.452374 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 6ecc552c1ef643f8bb84622d9d440ba2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6ecc552c1ef643f8bb84622d9d440ba2 already in progress: creating tablet
W20251029 02:52:06.452543 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 81f936792aae4b699b3122457d7b631d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 81f936792aae4b699b3122457d7b631d already in progress: creating tablet
W20251029 02:52:06.452730 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet ea7b977224f24bf49dadfb83ffa28310 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ea7b977224f24bf49dadfb83ffa28310 already in progress: creating tablet
W20251029 02:52:06.452911 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 46cd75ff18b14212b556b9bcb3012dd9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 46cd75ff18b14212b556b9bcb3012dd9 already in progress: creating tablet
I20251029 02:52:06.453076 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet f81aecb20d6e498bb9b498698886d65e (table default.Table_98034c2933664cefab94c26cb8349dc4 [id=efb8f381fea74cf48842af6b1a4d5a81]) successfully deleted
I20251029 02:52:06.455255 898 tablet_service.cc:1505] Processing CreateTablet for tablet 595b4073cc0446cea041838097af7471 (DEFAULT_TABLE table=default.Table_47a306977f08434993da629ff2ed490a [id=fee2c31d1a3f4da4a2b24477aefa48e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:06.455657 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 595b4073cc0446cea041838097af7471. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:06.456018 522 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 1c24459f58a6414f94c62fbd48927c11
I20251029 02:52:06.457049 900 tablet_service.cc:1505] Processing CreateTablet for tablet e0238e9cae1b40708ab326f0de7ecada (DEFAULT_TABLE table=default.Table_111cfdff9be446e09a352fd6f3265ec5 [id=9066f36249d04b01b672e86ab43b8730]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:06.457357 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e0238e9cae1b40708ab326f0de7ecada. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:06.457876 1167 tablet_bootstrap.cc:492] T daa05a34c0684c07a35966290fd7d44c P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:06.458020 1167 ts_tablet_manager.cc:1403] T daa05a34c0684c07a35966290fd7d44c P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.056s user 0.007s sys 0.011s
I20251029 02:52:06.458706 1167 raft_consensus.cc:359] T daa05a34c0684c07a35966290fd7d44c P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.458935 1167 raft_consensus.cc:385] T daa05a34c0684c07a35966290fd7d44c P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:06.459019 1167 raft_consensus.cc:740] T daa05a34c0684c07a35966290fd7d44c P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:06.459187 890 tablet_service.cc:1505] Processing CreateTablet for tablet 874b1728fd174305977b6852921928e7 (DEFAULT_TABLE table=default.Table_3266aac3f5744a23954b5465d9af1b31 [id=6ed604ab09a04c71adcf97bcdc9edd20]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:06.459508 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 874b1728fd174305977b6852921928e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:06.459405 1167 consensus_queue.cc:260] T daa05a34c0684c07a35966290fd7d44c P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.460064 1167 ts_tablet_manager.cc:1434] T daa05a34c0684c07a35966290fd7d44c P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 02:52:06.460670 892 tablet_service.cc:1505] Processing CreateTablet for tablet f68e9c8d268e424eac1fa14b9b711738 (DEFAULT_TABLE table=default.Table_65e6cdfafc9f46e8ac0c16c6babfc7d8 [id=dbfc4c7696244c04b7f312da49fc4dae]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:06.460990 1167 tablet_bootstrap.cc:492] T e1fa4980f2624321b617b84c68b90113 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:06.461225 885 tablet_service.cc:1505] Processing CreateTablet for tablet bfb9115f324f44c98f876a2d56bb23a1 (DEFAULT_TABLE table=default.Table_ddd7ee9b37df4bb4be7957aef7e82cb8 [id=91a2a75283864b60a04fa1183a6fd911]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:06.461720 895 tablet_service.cc:1505] Processing CreateTablet for tablet 00ebd0c68c4441f9b63c86cdab62497d (DEFAULT_TABLE table=default.Table_ba4b224d24404916b18cb61a05d08b75 [id=4e01b5a6ecd545bface69b56b8cf07d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:06.463181 756 tablet_service.cc:1552] Processing DeleteTablet for tablet 1c24459f58a6414f94c62fbd48927c11 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:06 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:06.460999 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f68e9c8d268e424eac1fa14b9b711738. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:06.468315 899 tablet_service.cc:1552] Processing DeleteTablet for tablet 1c24459f58a6414f94c62fbd48927c11 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:06 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:06.469115 1167 tablet_bootstrap.cc:654] T e1fa4980f2624321b617b84c68b90113 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:06.470810 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00ebd0c68c4441f9b63c86cdab62497d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:06.474826 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bfb9115f324f44c98f876a2d56bb23a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:06.479230 2036 raft_consensus.cc:493] T bfec87f690a640c39c58c64e16cc67e7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:06.479310 1039 tablet_service.cc:1552] Processing DeleteTablet for tablet 1c24459f58a6414f94c62fbd48927c11 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:06 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:06.479370 2036 raft_consensus.cc:515] T bfec87f690a640c39c58c64e16cc67e7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.480062 2036 leader_election.cc:290] T bfec87f690a640c39c58c64e16cc67e7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
W20251029 02:52:06.481417 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 1c24459f58a6414f94c62fbd48927c11 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1c24459f58a6414f94c62fbd48927c11 already in progress: creating tablet
W20251029 02:52:06.481761 989 leader_election.cc:343] T bfec87f690a640c39c58c64e16cc67e7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: bfec87f690a640c39c58c64e16cc67e7
I20251029 02:52:06.482153 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bfec87f690a640c39c58c64e16cc67e7" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:06.482908 990 leader_election.cc:343] T bfec87f690a640c39c58c64e16cc67e7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:06.483073 990 leader_election.cc:304] T bfec87f690a640c39c58c64e16cc67e7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5; no voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:06.483373 1167 tablet_bootstrap.cc:492] T e1fa4980f2624321b617b84c68b90113 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:06.483541 1167 ts_tablet_manager.cc:1403] T e1fa4980f2624321b617b84c68b90113 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.023s user 0.007s sys 0.002s
I20251029 02:52:06.483625 2036 raft_consensus.cc:2749] T bfec87f690a640c39c58c64e16cc67e7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:06.484463 1167 raft_consensus.cc:359] T e1fa4980f2624321b617b84c68b90113 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.484709 1167 raft_consensus.cc:385] T e1fa4980f2624321b617b84c68b90113 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:06.484817 1167 raft_consensus.cc:740] T e1fa4980f2624321b617b84c68b90113 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:06.485090 1167 consensus_queue.cc:260] T e1fa4980f2624321b617b84c68b90113 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.486553 1167 ts_tablet_manager.cc:1434] T e1fa4980f2624321b617b84c68b90113 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251029 02:52:06.488008 1167 tablet_bootstrap.cc:492] T da2a21c3236a4bd99872b89b12a36508 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:06.488124 520 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 4d51120775424146b6ab3e6d2e4710bf
I20251029 02:52:06.489117 518 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 8048506ced4540a5aee9f79bfc4599bf
I20251029 02:52:06.490048 524 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet b0fec06ad37d4c12a7045d424b88cc5e
I20251029 02:52:06.490324 1167 tablet_bootstrap.cc:654] T da2a21c3236a4bd99872b89b12a36508 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:06.490438 524 catalog_manager.cc:6719] ReplaceTablet: tablet b0fec06ad37d4c12a7045d424b88cc5e deleted and replaced by tablet 4eec207d19f540518bb567d7ab21d048
I20251029 02:52:06.490765 524 master_service.cc:954] ReplaceTablet: received request to replace tablet 506f8c112a3b43e1a3d6bbafb8cb81e8 from {username='slave'} at 127.0.0.1:35314
I20251029 02:52:06.491843 756 tablet_service.cc:1552] Processing DeleteTablet for tablet 4d51120775424146b6ab3e6d2e4710bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:05 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:06.492154 742 tablet_service.cc:1552] Processing DeleteTablet for tablet 8048506ced4540a5aee9f79bfc4599bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:06 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:06.492480 748 tablet_service.cc:1552] Processing DeleteTablet for tablet b0fec06ad37d4c12a7045d424b88cc5e with delete_type TABLET_DATA_DELETED (replaced by tablet 4eec207d19f540518bb567d7ab21d048) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:06.492960 1039 tablet_service.cc:1552] Processing DeleteTablet for tablet 4d51120775424146b6ab3e6d2e4710bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:05 UTC) from {username='slave'} at 127.0.0.1:37626
W20251029 02:52:06.493685 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 4d51120775424146b6ab3e6d2e4710bf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4d51120775424146b6ab3e6d2e4710bf already in progress: creating tablet
I20251029 02:52:06.494124 1041 tablet_service.cc:1552] Processing DeleteTablet for tablet b0fec06ad37d4c12a7045d424b88cc5e with delete_type TABLET_DATA_DELETED (replaced by tablet 4eec207d19f540518bb567d7ab21d048) from {username='slave'} at 127.0.0.1:37626
W20251029 02:52:06.495013 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet b0fec06ad37d4c12a7045d424b88cc5e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b0fec06ad37d4c12a7045d424b88cc5e already in progress: creating tablet
I20251029 02:52:06.495674 1037 tablet_service.cc:1552] Processing DeleteTablet for tablet 8048506ced4540a5aee9f79bfc4599bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:06 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:06.496428 900 tablet_service.cc:1552] Processing DeleteTablet for tablet 8048506ced4540a5aee9f79bfc4599bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:06 UTC) from {username='slave'} at 127.0.0.1:35702
W20251029 02:52:06.496671 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 8048506ced4540a5aee9f79bfc4599bf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8048506ced4540a5aee9f79bfc4599bf already in progress: creating tablet
I20251029 02:52:06.496722 885 tablet_service.cc:1552] Processing DeleteTablet for tablet 4d51120775424146b6ab3e6d2e4710bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:05 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:06.496515 1167 tablet_bootstrap.cc:492] T da2a21c3236a4bd99872b89b12a36508 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:06.497020 890 tablet_service.cc:1552] Processing DeleteTablet for tablet b0fec06ad37d4c12a7045d424b88cc5e with delete_type TABLET_DATA_DELETED (replaced by tablet 4eec207d19f540518bb567d7ab21d048) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:06.497346 1167 ts_tablet_manager.cc:1403] T da2a21c3236a4bd99872b89b12a36508 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.002s
I20251029 02:52:06.498337 1167 raft_consensus.cc:359] T da2a21c3236a4bd99872b89b12a36508 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.499768 1167 raft_consensus.cc:385] T da2a21c3236a4bd99872b89b12a36508 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:06.500125 1167 raft_consensus.cc:740] T da2a21c3236a4bd99872b89b12a36508 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:06.500838 1167 consensus_queue.cc:260] T da2a21c3236a4bd99872b89b12a36508 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.502916 1167 ts_tablet_manager.cc:1434] T da2a21c3236a4bd99872b89b12a36508 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251029 02:52:06.358341 1175 rpcz_store.cc:276] 1029 02:52:04.005848 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:04.005915 (+ 67us) service_pool.cc:225] Handling call
1029 02:52:06.213813 (+2207898us) ts_tablet_manager.cc:1151] Deleting tablet 71068739b4384badbf305313d902f0f5
1029 02:52:06.213823 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:06.237956 (+ 24133us) spinlock_profiling.cc:243] Waited 22.9 ms on lock 0x6080008bd650. stack: 00007f1174e9f9f6 00007f1174e9f7b0 00007f1173e7b3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f1184b0a6b8 00007f1184b101f4 00007f1184b0fecd 0000000000a8867c 00007f1174dbf43e 00007f1181f89033 00007f1182080774 000000000097cc36 000000000097cb11 0000000000b10c11
1029 02:52:06.241540 (+ 3584us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:06.356843 (+115303us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:06.358173 (+ 1330us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":52575104,"tablet-delete.queue_time_us":2207733}
I20251029 02:52:06.506198 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 674) took 2434 ms. Trace:
I20251029 02:52:06.506309 1175 rpcz_store.cc:276] 1029 02:52:04.071355 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:04.071406 (+ 51us) service_pool.cc:225] Handling call
1029 02:52:06.506043 (+2434637us) ts_tablet_manager.cc:1151] Deleting tablet 1e28a07b1a6242cd919b6ab2cb9b8a92
1029 02:52:06.506054 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:06.506173 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2434466}
I20251029 02:52:06.506336 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 71068739b4384badbf305313d902f0f5 (table default.Table_4e899dcc04fe4cb1b96ced1fa1b03585 [id=992716bbc1694bd6b85e4c8a3ab51e40]) successfully deleted
I20251029 02:52:06.506649 1175 tablet_replica.cc:333] T f60e7c24982d40319801869ad060864d P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:06.506837 1175 raft_consensus.cc:2243] T f60e7c24982d40319801869ad060864d P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:06.506982 1175 raft_consensus.cc:2272] T f60e7c24982d40319801869ad060864d P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251029 02:52:06.507000 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 1e28a07b1a6242cd919b6ab2cb9b8a92 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1e28a07b1a6242cd919b6ab2cb9b8a92 already in progress: creating tablet
I20251029 02:52:06.507470 1175 ts_tablet_manager.cc:1916] T f60e7c24982d40319801869ad060864d P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:06.508361 1172 tablet_bootstrap.cc:492] T 7a1866afc7564217ad777e7ce53786c5 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:06.508633 1172 ts_tablet_manager.cc:1403] T 7a1866afc7564217ad777e7ce53786c5 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.144s user 0.018s sys 0.010s
I20251029 02:52:06.509769 1172 raft_consensus.cc:359] T 7a1866afc7564217ad777e7ce53786c5 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.510087 1172 raft_consensus.cc:385] T 7a1866afc7564217ad777e7ce53786c5 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:06.510172 1172 raft_consensus.cc:740] T 7a1866afc7564217ad777e7ce53786c5 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:06.510561 1172 consensus_queue.cc:260] T 7a1866afc7564217ad777e7ce53786c5 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.511657 1174 ts_tablet_manager.cc:1929] T dd90fb9eb167478cbc3ec9363af86295 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20251029 02:52:06.511781 1174 log.cc:1199] T dd90fb9eb167478cbc3ec9363af86295 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/dd90fb9eb167478cbc3ec9363af86295
I20251029 02:52:06.516458 1167 tablet_bootstrap.cc:492] T 71068739b4384badbf305313d902f0f5 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:06.517850 1172 ts_tablet_manager.cc:1434] T 7a1866afc7564217ad777e7ce53786c5 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20251029 02:52:06.519549 1167 tablet_bootstrap.cc:654] T 71068739b4384badbf305313d902f0f5 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:06.520525 1174 ts_tablet_manager.cc:1950] T dd90fb9eb167478cbc3ec9363af86295 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:06.521679 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet dd90fb9eb167478cbc3ec9363af86295 (table default.Table_8b6217e43e8d467693c104ef84e78911 [id=052abf3dcc904bdcb869d5b885723dab]) successfully deleted
I20251029 02:52:06.523849 1172 tablet_bootstrap.cc:492] T 59d03a190b784ab18a3e63a572675891 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:06.525506 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_eb24c80a26b04dab8cc909a42d7184f3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:06.525883 1172 tablet_bootstrap.cc:654] T 59d03a190b784ab18a3e63a572675891 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
W20251029 02:52:06.526218 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eb24c80a26b04dab8cc909a42d7184f3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:06.528654 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_17afc4bb94e7410d8a45c9430e402e8c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:06.529193 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_17afc4bb94e7410d8a45c9430e402e8c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:06.530449 522 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_224e79c0036b470c863299755de8e9e4" } new_table_name: "default.Table_5681078bd9b54f948bd2d98511003cba" modify_external_catalogs: true
I20251029 02:52:06.530879 1174 tablet_replica.cc:333] T b2b2d64dcf964da8be4b544dbbe31977 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:06.531104 1174 raft_consensus.cc:2243] T b2b2d64dcf964da8be4b544dbbe31977 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:06.531415 1174 pending_rounds.cc:70] T b2b2d64dcf964da8be4b544dbbe31977 P c756528642a543eeacd93cef3e335a7d: Trying to abort 1 pending ops.
I20251029 02:52:06.531490 1174 pending_rounds.cc:77] T b2b2d64dcf964da8be4b544dbbe31977 P c756528642a543eeacd93cef3e335a7d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215949109853847552 op_type: NO_OP noop_request { }
I20251029 02:52:06.531613 1174 raft_consensus.cc:2889] T b2b2d64dcf964da8be4b544dbbe31977 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 02:52:06.531729 1174 raft_consensus.cc:2272] T b2b2d64dcf964da8be4b544dbbe31977 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:06.533147 1172 tablet_bootstrap.cc:492] T 59d03a190b784ab18a3e63a572675891 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:06.533411 1172 ts_tablet_manager.cc:1403] T 59d03a190b784ab18a3e63a572675891 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20251029 02:52:06.534474 1172 raft_consensus.cc:359] T 59d03a190b784ab18a3e63a572675891 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.534859 1172 raft_consensus.cc:385] T 59d03a190b784ab18a3e63a572675891 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:06.534996 1172 raft_consensus.cc:740] T 59d03a190b784ab18a3e63a572675891 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:06.535351 1172 consensus_queue.cc:260] T 59d03a190b784ab18a3e63a572675891 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.539326 525 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet f529e17e97074dca91ec2f48954cfd32
I20251029 02:52:06.540153 747 tablet_service.cc:1505] Processing CreateTablet for tablet 595b4073cc0446cea041838097af7471 (DEFAULT_TABLE table=default.Table_47a306977f08434993da629ff2ed490a [id=fee2c31d1a3f4da4a2b24477aefa48e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:06.540621 1037 tablet_service.cc:1552] Processing DeleteTablet for tablet f529e17e97074dca91ec2f48954cfd32 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:06 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:06.540714 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 595b4073cc0446cea041838097af7471. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:06.541283 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet f529e17e97074dca91ec2f48954cfd32 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f529e17e97074dca91ec2f48954cfd32 already in progress: creating tablet
I20251029 02:52:06.542637 890 tablet_service.cc:1552] Processing DeleteTablet for tablet f529e17e97074dca91ec2f48954cfd32 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:06 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:06.543736 526 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet 95f342d1cfb84c4eb31bd3d15ea3bdc2 (table default.Table_96ea715e50e74cc48759dda025a46318 [id=c63d52811db749d6badbed72fc133943]). Expected version 1 got 0
I20251029 02:52:06.548666 525 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_bafe26b1fca14cc9b908984b7d8a2442" } new_table_name: "default.Table_b708124baf534c96b3f4b8a7a4bb11cc" modify_external_catalogs: true
I20251029 02:52:06.539083 1172 ts_tablet_manager.cc:1434] T 59d03a190b784ab18a3e63a572675891 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20251029 02:52:06.551082 1172 tablet_bootstrap.cc:492] T 44b518c3a0964a6fa863b2784d80c8a4 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:06.551085 1167 tablet_bootstrap.cc:492] T 71068739b4384badbf305313d902f0f5 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:06.551240 1167 ts_tablet_manager.cc:1403] T 71068739b4384badbf305313d902f0f5 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.035s user 0.001s sys 0.008s
I20251029 02:52:06.551977 1167 raft_consensus.cc:359] T 71068739b4384badbf305313d902f0f5 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.552160 1167 raft_consensus.cc:385] T 71068739b4384badbf305313d902f0f5 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:06.552224 1167 raft_consensus.cc:740] T 71068739b4384badbf305313d902f0f5 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:06.552420 749 tablet_service.cc:1552] Processing DeleteTablet for tablet f529e17e97074dca91ec2f48954cfd32 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:06 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:06.552434 1167 consensus_queue.cc:260] T 71068739b4384badbf305313d902f0f5 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.552649 1172 tablet_bootstrap.cc:654] T 44b518c3a0964a6fa863b2784d80c8a4 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:06.552994 1167 ts_tablet_manager.cc:1434] T 71068739b4384badbf305313d902f0f5 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251029 02:52:06.553303 1167 tablet_bootstrap.cc:492] T 4add552cf4d940d2a0a36d6b58379a34 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:06.553711 1174 ts_tablet_manager.cc:1916] T b2b2d64dcf964da8be4b544dbbe31977 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:06.556980 1167 tablet_bootstrap.cc:654] T 4add552cf4d940d2a0a36d6b58379a34 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:06.559528 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_f3da247a6675435a9c881f23dd8ff376"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:06.560200 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f3da247a6675435a9c881f23dd8ff376 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:52:06.562623 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.3:45003, user_credentials={real_user=slave}} blocked reactor thread for 121401us
I20251029 02:52:06.567818 1172 tablet_bootstrap.cc:492] T 44b518c3a0964a6fa863b2784d80c8a4 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:06.568343 1172 ts_tablet_manager.cc:1403] T 44b518c3a0964a6fa863b2784d80c8a4 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.002s
I20251029 02:52:06.568835 890 tablet_service.cc:1505] Processing CreateTablet for tablet 72c5b53c2b3144f4a488411bc012a8c9 (DEFAULT_TABLE table=default.Table_46188a55c5034e5d992c14c95303ab26 [id=87b6c52d359849809ad3040401bf674b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:06.569250 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72c5b53c2b3144f4a488411bc012a8c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:06.569303 1172 raft_consensus.cc:359] T 44b518c3a0964a6fa863b2784d80c8a4 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.569450 1172 raft_consensus.cc:385] T 44b518c3a0964a6fa863b2784d80c8a4 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:06.569496 1172 raft_consensus.cc:740] T 44b518c3a0964a6fa863b2784d80c8a4 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:06.569468 1037 tablet_service.cc:1505] Processing CreateTablet for tablet 72c5b53c2b3144f4a488411bc012a8c9 (DEFAULT_TABLE table=default.Table_46188a55c5034e5d992c14c95303ab26 [id=87b6c52d359849809ad3040401bf674b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:06.569831 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72c5b53c2b3144f4a488411bc012a8c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:06.569731 1172 consensus_queue.cc:260] T 44b518c3a0964a6fa863b2784d80c8a4 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.572537 1172 ts_tablet_manager.cc:1434] T 44b518c3a0964a6fa863b2784d80c8a4 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251029 02:52:06.572804 1172 tablet_bootstrap.cc:492] T 0d84dc8dc465496f9adec798b2e03e13 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:06.574298 1172 tablet_bootstrap.cc:654] T 0d84dc8dc465496f9adec798b2e03e13 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:06.578691 1166 tablet_bootstrap.cc:492] T f60e7c24982d40319801869ad060864d P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:06.579409 1166 ts_tablet_manager.cc:1403] T f60e7c24982d40319801869ad060864d P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.158s user 0.025s sys 0.045s
I20251029 02:52:06.579811 1174 ts_tablet_manager.cc:1929] T b2b2d64dcf964da8be4b544dbbe31977 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:06.579973 1174 log.cc:1199] T b2b2d64dcf964da8be4b544dbbe31977 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/b2b2d64dcf964da8be4b544dbbe31977
I20251029 02:52:06.580387 1174 ts_tablet_manager.cc:1950] T b2b2d64dcf964da8be4b544dbbe31977 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:06.581060 1166 raft_consensus.cc:359] T f60e7c24982d40319801869ad060864d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.581559 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet b2b2d64dcf964da8be4b544dbbe31977 (table default.Table_e0e6e47b18d749e8ae80f822af1e23f0 [id=125ae5c7b5074378bcecb546e70f0c13]) successfully deleted
I20251029 02:52:06.581887 1166 raft_consensus.cc:385] T f60e7c24982d40319801869ad060864d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:06.582073 1166 raft_consensus.cc:740] T f60e7c24982d40319801869ad060864d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:06.582405 1166 consensus_queue.cc:260] T f60e7c24982d40319801869ad060864d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.583082 1166 ts_tablet_manager.cc:1434] T f60e7c24982d40319801869ad060864d P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251029 02:52:06.584215 1167 tablet_bootstrap.cc:492] T 4add552cf4d940d2a0a36d6b58379a34 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:06.584461 1167 ts_tablet_manager.cc:1403] T 4add552cf4d940d2a0a36d6b58379a34 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.031s user 0.010s sys 0.000s
I20251029 02:52:06.585418 1167 raft_consensus.cc:359] T 4add552cf4d940d2a0a36d6b58379a34 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.585716 1167 raft_consensus.cc:385] T 4add552cf4d940d2a0a36d6b58379a34 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:06.585839 1167 raft_consensus.cc:740] T 4add552cf4d940d2a0a36d6b58379a34 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:06.584215 1174 tablet_replica.cc:333] T 8df97119d0654d7d92bb5bd17c5d8562 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:06.586769 1174 raft_consensus.cc:2243] T 8df97119d0654d7d92bb5bd17c5d8562 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:06.586941 1174 raft_consensus.cc:2272] T 8df97119d0654d7d92bb5bd17c5d8562 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:06.587456 1174 ts_tablet_manager.cc:1916] T 8df97119d0654d7d92bb5bd17c5d8562 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:06.591614 747 tablet_service.cc:1505] Processing CreateTablet for tablet 72c5b53c2b3144f4a488411bc012a8c9 (DEFAULT_TABLE table=default.Table_46188a55c5034e5d992c14c95303ab26 [id=87b6c52d359849809ad3040401bf674b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:06.592049 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72c5b53c2b3144f4a488411bc012a8c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:06.592069 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_771ea2fd4396483ca70f00c7b9fb4c5f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:06.592660 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_771ea2fd4396483ca70f00c7b9fb4c5f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:06.594041 890 tablet_service.cc:1505] Processing CreateTablet for tablet 509288c033d54e4b9da3f26a30dc46a0 (DEFAULT_TABLE table=default.Table_6081d34b2c5742caa7681b275bb8e49f [id=7e07d8fb2dbc4043b5ea4a5225511254]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:06.594727 1167 consensus_queue.cc:260] T 4add552cf4d940d2a0a36d6b58379a34 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.595556 749 tablet_service.cc:1505] Processing CreateTablet for tablet 509288c033d54e4b9da3f26a30dc46a0 (DEFAULT_TABLE table=default.Table_6081d34b2c5742caa7681b275bb8e49f [id=7e07d8fb2dbc4043b5ea4a5225511254]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:06.596143 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 509288c033d54e4b9da3f26a30dc46a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:06.596261 1041 tablet_service.cc:1505] Processing CreateTablet for tablet 509288c033d54e4b9da3f26a30dc46a0 (DEFAULT_TABLE table=default.Table_6081d34b2c5742caa7681b275bb8e49f [id=7e07d8fb2dbc4043b5ea4a5225511254]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:06.598601 1167 ts_tablet_manager.cc:1434] T 4add552cf4d940d2a0a36d6b58379a34 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.014s user 0.001s sys 0.001s
I20251029 02:52:06.599689 1172 tablet_bootstrap.cc:492] T 0d84dc8dc465496f9adec798b2e03e13 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:06.599778 1167 tablet_bootstrap.cc:492] T b3de6670357c4ae59eb77469b2fb434c P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:06.599879 1172 ts_tablet_manager.cc:1403] T 0d84dc8dc465496f9adec798b2e03e13 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.002s
I20251029 02:52:06.600644 1172 raft_consensus.cc:359] T 0d84dc8dc465496f9adec798b2e03e13 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.600898 1172 raft_consensus.cc:385] T 0d84dc8dc465496f9adec798b2e03e13 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:06.601009 1172 raft_consensus.cc:740] T 0d84dc8dc465496f9adec798b2e03e13 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:06.601284 1172 consensus_queue.cc:260] T 0d84dc8dc465496f9adec798b2e03e13 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.601536 1167 tablet_bootstrap.cc:654] T b3de6670357c4ae59eb77469b2fb434c P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:06.604089 1166 tablet_bootstrap.cc:492] T c764062d3ba04d4ba8b2e367ad8ed433 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:06.604437 1172 ts_tablet_manager.cc:1434] T 0d84dc8dc465496f9adec798b2e03e13 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251029 02:52:06.604910 1172 tablet_bootstrap.cc:492] T dabb8d760aae4aa09eb83cdb586c9788 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:06.606642 1172 tablet_bootstrap.cc:654] T dabb8d760aae4aa09eb83cdb586c9788 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:06.606642 1166 tablet_bootstrap.cc:654] T c764062d3ba04d4ba8b2e367ad8ed433 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:06.610777 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 509288c033d54e4b9da3f26a30dc46a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:06.611639 2018 raft_consensus.cc:493] T 49d9ce32e4f04f11be2ecefc5d523a8b P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:06.611786 2018 raft_consensus.cc:515] T 49d9ce32e4f04f11be2ecefc5d523a8b P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.612418 2018 leader_election.cc:290] T 49d9ce32e4f04f11be2ecefc5d523a8b P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:06.612743 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 509288c033d54e4b9da3f26a30dc46a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:06.616143 885 tablet_service.cc:1505] Processing CreateTablet for tablet bf2c37f5eff740a6ae594ffc6c0e5101 (DEFAULT_TABLE table=default.Table_90cad35ff5a14d0b87b2785627781b23 [id=3fcb2116712643af892b4e153742323e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:06.616554 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf2c37f5eff740a6ae594ffc6c0e5101. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:06.617282 1037 tablet_service.cc:1505] Processing CreateTablet for tablet bf2c37f5eff740a6ae594ffc6c0e5101 (DEFAULT_TABLE table=default.Table_90cad35ff5a14d0b87b2785627781b23 [id=3fcb2116712643af892b4e153742323e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:06.617619 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf2c37f5eff740a6ae594ffc6c0e5101. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:06.619669 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "49d9ce32e4f04f11be2ecefc5d523a8b" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:52:06.620265 712 leader_election.cc:343] T 49d9ce32e4f04f11be2ecefc5d523a8b P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:06.620883 1174 ts_tablet_manager.cc:1929] T 8df97119d0654d7d92bb5bd17c5d8562 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:06.621199 1174 log.cc:1199] T 8df97119d0654d7d92bb5bd17c5d8562 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/8df97119d0654d7d92bb5bd17c5d8562
I20251029 02:52:06.621369 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "49d9ce32e4f04f11be2ecefc5d523a8b" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:06.621591 1174 ts_tablet_manager.cc:1950] T 8df97119d0654d7d92bb5bd17c5d8562 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
W20251029 02:52:06.623098 711 leader_election.cc:343] T 49d9ce32e4f04f11be2ecefc5d523a8b P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:06.623226 711 leader_election.cc:304] T 49d9ce32e4f04f11be2ecefc5d523a8b P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:06.623814 2018 raft_consensus.cc:2749] T 49d9ce32e4f04f11be2ecefc5d523a8b P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:06.624357 890 tablet_service.cc:1505] Processing CreateTablet for tablet b81f26fb2739487dbc9af7b6e9848a4a (DEFAULT_TABLE table=default.Table_ca4bf5512ff048ddb5feb708b19e3f05 [id=fea94e2e69df402c96348f2a4af6bd84]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:06.624503 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 8df97119d0654d7d92bb5bd17c5d8562 (table default.Table_99592a0735024ba7a0d8d6145b61f2e4 [id=ced78d857c314dc7802545a7909bdf46]) successfully deleted
I20251029 02:52:06.624835 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b81f26fb2739487dbc9af7b6e9848a4a. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:06.626179 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 8f100a12d78145c2b8586a0713fc70c4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8f100a12d78145c2b8586a0713fc70c4 already in progress: creating tablet
I20251029 02:52:06.626895 1166 tablet_bootstrap.cc:492] T c764062d3ba04d4ba8b2e367ad8ed433 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:06.627326 1166 ts_tablet_manager.cc:1403] T c764062d3ba04d4ba8b2e367ad8ed433 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.003s
W20251029 02:52:06.628062 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 72f9d4408f384e7093316d85f092e1b2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 72f9d4408f384e7093316d85f092e1b2 already in progress: creating tablet
I20251029 02:52:06.628159 1166 raft_consensus.cc:359] T c764062d3ba04d4ba8b2e367ad8ed433 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.628340 1166 raft_consensus.cc:385] T c764062d3ba04d4ba8b2e367ad8ed433 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:06.628399 1166 raft_consensus.cc:740] T c764062d3ba04d4ba8b2e367ad8ed433 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:06.628681 1166 consensus_queue.cc:260] T c764062d3ba04d4ba8b2e367ad8ed433 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.629175 1166 ts_tablet_manager.cc:1434] T c764062d3ba04d4ba8b2e367ad8ed433 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
W20251029 02:52:06.629736 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 143f4a42001b43e683bbc6226c0550a9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 143f4a42001b43e683bbc6226c0550a9 already in progress: creating tablet
W20251029 02:52:06.630722 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet ad214c870ee649279ed7321c5b69cbee because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ad214c870ee649279ed7321c5b69cbee already in progress: creating tablet
I20251029 02:52:06.631314 1167 tablet_bootstrap.cc:492] T b3de6670357c4ae59eb77469b2fb434c P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:06.631521 1167 ts_tablet_manager.cc:1403] T b3de6670357c4ae59eb77469b2fb434c P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.032s user 0.005s sys 0.002s
I20251029 02:52:06.632361 1167 raft_consensus.cc:359] T b3de6670357c4ae59eb77469b2fb434c P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.632612 1167 raft_consensus.cc:385] T b3de6670357c4ae59eb77469b2fb434c P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:06.632755 1167 raft_consensus.cc:740] T b3de6670357c4ae59eb77469b2fb434c P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:06.633033 1167 consensus_queue.cc:260] T b3de6670357c4ae59eb77469b2fb434c P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.633158 524 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 506f8c112a3b43e1a3d6bbafb8cb81e8
I20251029 02:52:06.633598 524 catalog_manager.cc:6719] ReplaceTablet: tablet 506f8c112a3b43e1a3d6bbafb8cb81e8 deleted and replaced by tablet ef2e9dd8f90a47389d9062d1f99a1da1
I20251029 02:52:06.633929 885 tablet_service.cc:1552] Processing DeleteTablet for tablet 506f8c112a3b43e1a3d6bbafb8cb81e8 with delete_type TABLET_DATA_DELETED (replaced by tablet ef2e9dd8f90a47389d9062d1f99a1da1) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:06.635357 1167 ts_tablet_manager.cc:1434] T b3de6670357c4ae59eb77469b2fb434c P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251029 02:52:06.635812 1039 tablet_service.cc:1552] Processing DeleteTablet for tablet 506f8c112a3b43e1a3d6bbafb8cb81e8 with delete_type TABLET_DATA_DELETED (replaced by tablet ef2e9dd8f90a47389d9062d1f99a1da1) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:06.635506 1174 tablet_replica.cc:333] T 59af906061d049c99ae2153945597007 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:06.636760 1174 raft_consensus.cc:2243] T 59af906061d049c99ae2153945597007 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:06.636988 1174 raft_consensus.cc:2272] T 59af906061d049c99ae2153945597007 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:06.638911 920 raft_consensus.cc:3060] T da2a21c3236a4bd99872b89b12a36508 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:06.640172 1172 tablet_bootstrap.cc:492] T dabb8d760aae4aa09eb83cdb586c9788 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:06.640302 1172 ts_tablet_manager.cc:1403] T dabb8d760aae4aa09eb83cdb586c9788 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.035s user 0.001s sys 0.005s
I20251029 02:52:06.641013 1172 raft_consensus.cc:359] T dabb8d760aae4aa09eb83cdb586c9788 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.641175 1172 raft_consensus.cc:385] T dabb8d760aae4aa09eb83cdb586c9788 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:06.641227 1172 raft_consensus.cc:740] T dabb8d760aae4aa09eb83cdb586c9788 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:06.641426 1172 consensus_queue.cc:260] T dabb8d760aae4aa09eb83cdb586c9788 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.641889 1172 ts_tablet_manager.cc:1434] T dabb8d760aae4aa09eb83cdb586c9788 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251029 02:52:06.642810 2036 consensus_queue.cc:1048] T da2a21c3236a4bd99872b89b12a36508 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:06.643503 1172 tablet_bootstrap.cc:492] T d61beda662d844cdaf67fd9a2e382ea2 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
W20251029 02:52:06.644392 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 506f8c112a3b43e1a3d6bbafb8cb81e8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 506f8c112a3b43e1a3d6bbafb8cb81e8 already in progress: creating tablet
I20251029 02:52:06.645087 1172 tablet_bootstrap.cc:654] T d61beda662d844cdaf67fd9a2e382ea2 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:06.645371 1039 tablet_service.cc:1505] Processing CreateTablet for tablet b81f26fb2739487dbc9af7b6e9848a4a (DEFAULT_TABLE table=default.Table_ca4bf5512ff048ddb5feb708b19e3f05 [id=fea94e2e69df402c96348f2a4af6bd84]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:06.645780 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b81f26fb2739487dbc9af7b6e9848a4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:06.646284 2036 consensus_queue.cc:1048] T da2a21c3236a4bd99872b89b12a36508 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:06.646916 1167 tablet_bootstrap.cc:492] T 86fc22bb84794149b598b97df41eb881 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:06.647683 747 tablet_service.cc:1552] Processing DeleteTablet for tablet 506f8c112a3b43e1a3d6bbafb8cb81e8 with delete_type TABLET_DATA_DELETED (replaced by tablet ef2e9dd8f90a47389d9062d1f99a1da1) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:06.648450 1167 tablet_bootstrap.cc:654] T 86fc22bb84794149b598b97df41eb881 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:06.649277 1166 tablet_bootstrap.cc:492] T 1bc45078a6a942c38ff3350c15885eff P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:06.648721 1903 raft_consensus.cc:493] T 74873a05ef634867a30aa63b31fcfcc7 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:06.650197 1903 raft_consensus.cc:515] T 74873a05ef634867a30aa63b31fcfcc7 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.650918 1166 tablet_bootstrap.cc:654] T 1bc45078a6a942c38ff3350c15885eff P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:06.650914 1903 leader_election.cc:290] T 74873a05ef634867a30aa63b31fcfcc7 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:06.649149 1174 ts_tablet_manager.cc:1916] T 59af906061d049c99ae2153945597007 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 02:52:06.651679 847 leader_election.cc:343] T 74873a05ef634867a30aa63b31fcfcc7 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 74873a05ef634867a30aa63b31fcfcc7
W20251029 02:52:06.652539 849 leader_election.cc:343] T 74873a05ef634867a30aa63b31fcfcc7 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Not found: Tablet not found: 74873a05ef634867a30aa63b31fcfcc7
I20251029 02:52:06.652719 849 leader_election.cc:304] T 74873a05ef634867a30aa63b31fcfcc7 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9
I20251029 02:52:06.653192 1903 raft_consensus.cc:2749] T 74873a05ef634867a30aa63b31fcfcc7 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:06.660656 1175 ts_tablet_manager.cc:1929] T f60e7c24982d40319801869ad060864d P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:06.662310 885 tablet_service.cc:1505] Processing CreateTablet for tablet b6436a020e3542fbb25ac2846ed42bef (DEFAULT_TABLE table=default.Table_b717a166304f42e4886196617960e361 [id=05e1cdb4be4e43ddb9bb2a468239f20c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:06.662741 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6436a020e3542fbb25ac2846ed42bef. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:06.667692 1973 consensus_queue.cc:1048] T cedfb53078d94c81b5e294ce7438325e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:06.669567 1973 raft_consensus.cc:493] T cc74e83833754f0ca19046031a24ebb9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:06.669711 1973 raft_consensus.cc:515] T cc74e83833754f0ca19046031a24ebb9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.669879 1903 raft_consensus.cc:493] T e1fa4980f2624321b617b84c68b90113 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:06.670022 1903 raft_consensus.cc:515] T e1fa4980f2624321b617b84c68b90113 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.670306 1973 leader_election.cc:290] T cc74e83833754f0ca19046031a24ebb9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:06.670377 1174 ts_tablet_manager.cc:1929] T 59af906061d049c99ae2153945597007 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251029 02:52:06.670583 1903 leader_election.cc:290] T e1fa4980f2624321b617b84c68b90113 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
W20251029 02:52:06.671494 847 leader_election.cc:343] T e1fa4980f2624321b617b84c68b90113 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: e1fa4980f2624321b617b84c68b90113
I20251029 02:52:06.672174 914 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cc74e83833754f0ca19046031a24ebb9" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:06.672216 1058 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1fa4980f2624321b617b84c68b90113" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:06.672360 1058 raft_consensus.cc:2468] T e1fa4980f2624321b617b84c68b90113 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:06.672374 914 raft_consensus.cc:2468] T cc74e83833754f0ca19046031a24ebb9 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 0.
I20251029 02:52:06.672763 849 leader_election.cc:304] T e1fa4980f2624321b617b84c68b90113 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters: b495a49349324f7d81908f2da700f0d9
I20251029 02:52:06.673197 1903 raft_consensus.cc:2804] T e1fa4980f2624321b617b84c68b90113 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:06.673167 990 leader_election.cc:304] T cc74e83833754f0ca19046031a24ebb9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:06.673280 1903 raft_consensus.cc:493] T e1fa4980f2624321b617b84c68b90113 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:06.673355 1903 raft_consensus.cc:3060] T e1fa4980f2624321b617b84c68b90113 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:06.673681 1973 raft_consensus.cc:2804] T cc74e83833754f0ca19046031a24ebb9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:06.673743 1973 raft_consensus.cc:493] T cc74e83833754f0ca19046031a24ebb9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:06.673795 1973 raft_consensus.cc:3060] T cc74e83833754f0ca19046031a24ebb9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:06.676699 1903 raft_consensus.cc:515] T e1fa4980f2624321b617b84c68b90113 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
W20251029 02:52:06.680584 847 leader_election.cc:343] T e1fa4980f2624321b617b84c68b90113 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: e1fa4980f2624321b617b84c68b90113
I20251029 02:52:06.681497 1058 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e1fa4980f2624321b617b84c68b90113" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:52:06.681710 1058 raft_consensus.cc:3060] T e1fa4980f2624321b617b84c68b90113 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:06.681736 1973 raft_consensus.cc:515] T cc74e83833754f0ca19046031a24ebb9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.682593 914 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cc74e83833754f0ca19046031a24ebb9" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
I20251029 02:52:06.682765 914 raft_consensus.cc:3060] T cc74e83833754f0ca19046031a24ebb9 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:06.684391 1058 raft_consensus.cc:2468] T e1fa4980f2624321b617b84c68b90113 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:06.684940 849 leader_election.cc:304] T e1fa4980f2624321b617b84c68b90113 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters: b495a49349324f7d81908f2da700f0d9
I20251029 02:52:06.685106 1174 log.cc:1199] T 59af906061d049c99ae2153945597007 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/59af906061d049c99ae2153945597007
I20251029 02:52:06.685283 1904 raft_consensus.cc:2804] T e1fa4980f2624321b617b84c68b90113 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:06.685464 914 raft_consensus.cc:2468] T cc74e83833754f0ca19046031a24ebb9 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 1.
I20251029 02:52:06.685662 1174 ts_tablet_manager.cc:1950] T 59af906061d049c99ae2153945597007 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:06.686067 990 leader_election.cc:304] T cc74e83833754f0ca19046031a24ebb9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:06.686453 2036 raft_consensus.cc:2804] T cc74e83833754f0ca19046031a24ebb9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:06.686604 2036 raft_consensus.cc:697] T cc74e83833754f0ca19046031a24ebb9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Becoming Leader. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Running, Role: LEADER
I20251029 02:52:06.686825 2036 consensus_queue.cc:237] T cc74e83833754f0ca19046031a24ebb9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.687110 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 59af906061d049c99ae2153945597007 (table default.Table_6da557988d044647b5e68f09b3f00407 [id=25f16d95843d41c981d6a07ec0d4b48a]) successfully deleted
W20251029 02:52:06.692003 512 catalog_manager.cc:4952] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 00f2e98739fa41eea23927494fe222f3 because the tablet was not found. No further retry: Not found: Tablet not found: 00f2e98739fa41eea23927494fe222f3
W20251029 02:52:06.699702 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 1c24459f58a6414f94c62fbd48927c11 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1c24459f58a6414f94c62fbd48927c11 already in progress: creating tablet
W20251029 02:52:06.701500 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 8048506ced4540a5aee9f79bfc4599bf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8048506ced4540a5aee9f79bfc4599bf already in progress: creating tablet
W20251029 02:52:06.703289 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 4d51120775424146b6ab3e6d2e4710bf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4d51120775424146b6ab3e6d2e4710bf already in progress: creating tablet
W20251029 02:52:06.704546 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet b0fec06ad37d4c12a7045d424b88cc5e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b0fec06ad37d4c12a7045d424b88cc5e already in progress: creating tablet
I20251029 02:52:06.705737 1903 leader_election.cc:290] T e1fa4980f2624321b617b84c68b90113 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
W20251029 02:52:06.705821 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet f529e17e97074dca91ec2f48954cfd32 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f529e17e97074dca91ec2f48954cfd32 already in progress: creating tablet
W20251029 02:52:06.707176 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 506f8c112a3b43e1a3d6bbafb8cb81e8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 506f8c112a3b43e1a3d6bbafb8cb81e8 already in progress: creating tablet
I20251029 02:52:06.715011 520 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_6081606370774be1a2233b9304083139" } new_table_name: "default.Table_7125184a813c44d997ab9d948a1156bd" modify_external_catalogs: true
I20251029 02:52:06.715036 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_11c3e33dc6ae4f9e8852d7a31b078288"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:06.715788 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_11c3e33dc6ae4f9e8852d7a31b078288 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:06.717197 1973 leader_election.cc:290] T cc74e83833754f0ca19046031a24ebb9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Requested vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:06.728048 1904 raft_consensus.cc:697] T e1fa4980f2624321b617b84c68b90113 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:52:06.728454 1904 consensus_queue.cc:237] T e1fa4980f2624321b617b84c68b90113 P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.735236 1973 raft_consensus.cc:493] T daa05a34c0684c07a35966290fd7d44c P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:06.735462 1973 raft_consensus.cc:515] T daa05a34c0684c07a35966290fd7d44c P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.735702 1041 tablet_service.cc:1505] Processing CreateTablet for tablet b6436a020e3542fbb25ac2846ed42bef (DEFAULT_TABLE table=default.Table_b717a166304f42e4886196617960e361 [id=05e1cdb4be4e43ddb9bb2a468239f20c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:06.736110 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6436a020e3542fbb25ac2846ed42bef. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:06.736617 1973 leader_election.cc:290] T daa05a34c0684c07a35966290fd7d44c P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:06.742937 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "daa05a34c0684c07a35966290fd7d44c" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:06.743817 920 raft_consensus.cc:2468] T daa05a34c0684c07a35966290fd7d44c P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 0.
I20251029 02:52:06.744627 990 leader_election.cc:304] T daa05a34c0684c07a35966290fd7d44c P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:06.754386 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_2c913689f8134363ae4a20a05f717d3e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:06.755080 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2c913689f8134363ae4a20a05f717d3e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:52:06.759393 522 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 050186302c5d4c7da393f28f4b9e4262 consensus state has no leader
I20251029 02:52:06.759826 522 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_19ec7789551742df938b5c458a04326d" } new_table_name: "default.Table_87da878e5839481a97fa117e34501d10" modify_external_catalogs: true
I20251029 02:52:06.761082 517 catalog_manager.cc:6194] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: default.Table_ad2ac3ba98784d68a94eb4eb03e530a4 [id=5e0c72957b9a491d960b33d3fffd8f70] alter complete (version 2)
I20251029 02:52:06.762487 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_a4f394d1b04146848a9a3dafc0919f0e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:06.763077 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a4f394d1b04146848a9a3dafc0919f0e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:06.763757 525 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_5c2d435abf6a4195bf1009f4e1aa725d" } new_table_name: "default.Table_f435bdcca42e4087ac8794859d9108ee" modify_external_catalogs: true
I20251029 02:52:06.765388 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_c6def3b3356a4994a768adbbdf45fb9b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:06.765691 900 tablet_service.cc:1552] Processing DeleteTablet for tablet 74873a05ef634867a30aa63b31fcfcc7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:54 UTC) from {username='slave'} at 127.0.0.1:35702
W20251029 02:52:06.765883 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c6def3b3356a4994a768adbbdf45fb9b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:06.766034 1174 tablet_replica.cc:333] T 74873a05ef634867a30aa63b31fcfcc7 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:06.766327 1174 raft_consensus.cc:2243] T 74873a05ef634867a30aa63b31fcfcc7 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:06.766561 1174 raft_consensus.cc:2272] T 74873a05ef634867a30aa63b31fcfcc7 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:06.767302 1174 ts_tablet_manager.cc:1916] T 74873a05ef634867a30aa63b31fcfcc7 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:06.768157 526 master_service.cc:954] ReplaceTablet: received request to replace tablet 31a157697e2c45ac9ba92c31c6936fa0 from {username='slave'} at 127.0.0.1:35314
I20251029 02:52:06.774303 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet f246f6c202cf44d48c9f5e4f335202c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:55 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:06.774401 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet 6ca9bd37de9e4be388501b15af4bda31 with delete_type TABLET_DATA_DELETED (replaced by tablet 20a762344c5f46f88481c4ac7ad6e43b) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:06.774571 1176 tablet_replica.cc:333] T f246f6c202cf44d48c9f5e4f335202c5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:06.774583 1031 tablet_service.cc:1552] Processing DeleteTablet for tablet e1fa4980f2624321b617b84c68b90113 with delete_type TABLET_DATA_DELETED (replaced by tablet e6e3e6a82b014d2b8de4e109b379461e) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:06.774427 1025 tablet_service.cc:1552] Processing DeleteTablet for tablet 0cc1957a845f4e9cb703a553b27d1347 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:53 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:06.774766 1176 raft_consensus.cc:2243] T f246f6c202cf44d48c9f5e4f335202c5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:06.774938 1176 raft_consensus.cc:2272] T f246f6c202cf44d48c9f5e4f335202c5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:06.775470 1176 ts_tablet_manager.cc:1916] T f246f6c202cf44d48c9f5e4f335202c5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:06.775524 1039 tablet_service.cc:1552] Processing DeleteTablet for tablet 71068739b4384badbf305313d902f0f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:56 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:06.777669 1174 ts_tablet_manager.cc:1929] T 74873a05ef634867a30aa63b31fcfcc7 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:06.777819 1174 log.cc:1199] T 74873a05ef634867a30aa63b31fcfcc7 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/74873a05ef634867a30aa63b31fcfcc7
W20251029 02:52:06.780781 989 leader_election.cc:343] T daa05a34c0684c07a35966290fd7d44c P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: daa05a34c0684c07a35966290fd7d44c
I20251029 02:52:06.781392 1174 ts_tablet_manager.cc:1950] T 74873a05ef634867a30aa63b31fcfcc7 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:06.782819 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 74873a05ef634867a30aa63b31fcfcc7 (table default.Table_4e97f5e298434c16913bf7ca93ca30f5 [id=48dfcbe8b3654d72b51b77473dd918f6]) successfully deleted
I20251029 02:52:06.783500 1176 ts_tablet_manager.cc:1929] T f246f6c202cf44d48c9f5e4f335202c5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:06.783663 1176 log.cc:1199] T f246f6c202cf44d48c9f5e4f335202c5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/f246f6c202cf44d48c9f5e4f335202c5
I20251029 02:52:06.784004 1176 ts_tablet_manager.cc:1950] T f246f6c202cf44d48c9f5e4f335202c5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:06.784584 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet f246f6c202cf44d48c9f5e4f335202c5 (table default.Table_e4de02c7c1f04704bc39e1f0efd2c9d9 [id=40738627473f4a55adc5ed250af29d6c]) successfully deleted
I20251029 02:52:06.787859 885 tablet_service.cc:1505] Processing CreateTablet for tablet c91ecaea96224e33be4d728678729886 (DEFAULT_TABLE table=default.Table_1455335d0ba84ae08eff9eca1366a8f0 [id=46e325b6a4c346d4b077030727bb6a7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:06.788430 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c91ecaea96224e33be4d728678729886. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:06.794771 1176 tablet_replica.cc:333] T 6ca9bd37de9e4be388501b15af4bda31 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:06.795095 1176 raft_consensus.cc:2243] T 6ca9bd37de9e4be388501b15af4bda31 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:06.795413 1176 pending_rounds.cc:70] T 6ca9bd37de9e4be388501b15af4bda31 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Trying to abort 1 pending ops.
I20251029 02:52:06.795512 1176 pending_rounds.cc:77] T 6ca9bd37de9e4be388501b15af4bda31 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215949112561790976 op_type: NO_OP noop_request { }
I20251029 02:52:06.795699 1176 raft_consensus.cc:2889] T 6ca9bd37de9e4be388501b15af4bda31 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 02:52:06.795861 1176 raft_consensus.cc:2272] T 6ca9bd37de9e4be388501b15af4bda31 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:06.797031 1176 ts_tablet_manager.cc:1916] T 6ca9bd37de9e4be388501b15af4bda31 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:06.662145 747 tablet_service.cc:1505] Processing CreateTablet for tablet b81f26fb2739487dbc9af7b6e9848a4a (DEFAULT_TABLE table=default.Table_ca4bf5512ff048ddb5feb708b19e3f05 [id=fea94e2e69df402c96348f2a4af6bd84]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:06.806934 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b81f26fb2739487dbc9af7b6e9848a4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:06.671186 770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cc74e83833754f0ca19046031a24ebb9" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:06.810684 770 raft_consensus.cc:2468] T cc74e83833754f0ca19046031a24ebb9 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 0.
I20251029 02:52:06.675264 2018 raft_consensus.cc:493] T f529e17e97074dca91ec2f48954cfd32 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:06.811039 2018 raft_consensus.cc:515] T f529e17e97074dca91ec2f48954cfd32 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.811766 2018 leader_election.cc:290] T f529e17e97074dca91ec2f48954cfd32 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:06.812851 1176 ts_tablet_manager.cc:1929] T 6ca9bd37de9e4be388501b15af4bda31 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:06.812647 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f529e17e97074dca91ec2f48954cfd32" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:06.813542 711 leader_election.cc:343] T f529e17e97074dca91ec2f48954cfd32 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:06.725948 2085 raft_consensus.cc:493] T 59d03a190b784ab18a3e63a572675891 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:06.771045 742 tablet_service.cc:1552] Processing DeleteTablet for tablet 49d9ce32e4f04f11be2ecefc5d523a8b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:57 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:06.771204 751 tablet_service.cc:1552] Processing DeleteTablet for tablet ccdca31db2834de4b6ed0f2d67a10535 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:54 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:06.771227 756 tablet_service.cc:1552] Processing DeleteTablet for tablet 5db519edc75d4bd0985b32322825ffae with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:55 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:06.771279 748 tablet_service.cc:1552] Processing DeleteTablet for tablet 0f80b840ed954230bfbfe7e40127cd2f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:56 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:06.771365 750 tablet_service.cc:1552] Processing DeleteTablet for tablet 785733da86ff425a90ccfab2933fc89e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:54 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:06.780061 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cc74e83833754f0ca19046031a24ebb9" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:06.784543 1172 tablet_bootstrap.cc:492] T d61beda662d844cdaf67fd9a2e382ea2 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:06.815934 768 raft_consensus.cc:3060] T cc74e83833754f0ca19046031a24ebb9 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:06.816049 1172 ts_tablet_manager.cc:1403] T d61beda662d844cdaf67fd9a2e382ea2 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.173s user 0.015s sys 0.020s
I20251029 02:52:06.817000 1175 log.cc:1199] T f60e7c24982d40319801869ad060864d P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/f60e7c24982d40319801869ad060864d
I20251029 02:52:06.817310 1175 ts_tablet_manager.cc:1950] T f60e7c24982d40319801869ad060864d P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:06.817167 1172 raft_consensus.cc:359] T d61beda662d844cdaf67fd9a2e382ea2 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.817374 1172 raft_consensus.cc:385] T d61beda662d844cdaf67fd9a2e382ea2 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:06.817437 1172 raft_consensus.cc:740] T d61beda662d844cdaf67fd9a2e382ea2 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:06.817668 1172 consensus_queue.cc:260] T d61beda662d844cdaf67fd9a2e382ea2 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.818120 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 675) took 2717 ms. Trace:
I20251029 02:52:06.818195 1175 rpcz_store.cc:276] 1029 02:52:04.100498 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:04.100552 (+ 54us) service_pool.cc:225] Handling call
1029 02:52:06.506603 (+2406051us) ts_tablet_manager.cc:1151] Deleting tablet f60e7c24982d40319801869ad060864d
1029 02:52:06.506613 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:06.586090 (+ 79477us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:06.660525 (+ 74435us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:06.818102 (+157577us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2405854}
I20251029 02:52:06.818303 1172 ts_tablet_manager.cc:1434] T d61beda662d844cdaf67fd9a2e382ea2 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 02:52:06.787401 755 tablet_service.cc:1505] Processing CreateTablet for tablet b6436a020e3542fbb25ac2846ed42bef (DEFAULT_TABLE table=default.Table_b717a166304f42e4886196617960e361 [id=05e1cdb4be4e43ddb9bb2a468239f20c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:06.818558 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 676) took 2712 ms. Trace:
I20251029 02:52:06.818641 1172 tablet_bootstrap.cc:492] T 9bb2166a590d4210b02dd779ef6654ad P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:06.818814 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6436a020e3542fbb25ac2846ed42bef. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:06.819307 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet f60e7c24982d40319801869ad060864d (table default.Table_4350e92f14de4ba8a04afbe12ea7422a [id=226a2d412a7b40e59b6f5340b5a9b739]) successfully deleted
I20251029 02:52:06.821185 1172 tablet_bootstrap.cc:654] T 9bb2166a590d4210b02dd779ef6654ad P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:06.823150 750 tablet_service.cc:1505] Processing CreateTablet for tablet c91ecaea96224e33be4d728678729886 (DEFAULT_TABLE table=default.Table_1455335d0ba84ae08eff9eca1366a8f0 [id=46e325b6a4c346d4b077030727bb6a7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:06.823458 756 tablet_service.cc:1505] Processing CreateTablet for tablet 5c40f2f9cf2b41a3b129e7ef1bf41eb3 (DEFAULT_TABLE table=default.Table_b99166d4afe04f44a80ecd035fa90d72 [id=cb1954744198426c8a8f242f6f0679c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:06.823534 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c91ecaea96224e33be4d728678729886. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:06.823829 748 tablet_service.cc:1505] Processing CreateTablet for tablet 49aaa2d8986a44df979edc1c42612189 (DEFAULT_TABLE table=default.Table_d844249d525b48428ebf40e1d6cb6c51 [id=2a13a4467a48459fb80a513cca4c4d96]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:06.824132 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49aaa2d8986a44df979edc1c42612189. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:06.824505 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c40f2f9cf2b41a3b129e7ef1bf41eb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:06.816790 1058 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f529e17e97074dca91ec2f48954cfd32" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:06.826529 1176 log.cc:1199] T 6ca9bd37de9e4be388501b15af4bda31 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/6ca9bd37de9e4be388501b15af4bda31
I20251029 02:52:06.826992 1176 ts_tablet_manager.cc:1950] T 6ca9bd37de9e4be388501b15af4bda31 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:06.827539 1176 tablet_replica.cc:333] T e1fa4980f2624321b617b84c68b90113 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:06.827970 768 raft_consensus.cc:2468] T cc74e83833754f0ca19046031a24ebb9 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 1.
I20251029 02:52:06.829314 900 tablet_service.cc:1505] Processing CreateTablet for tablet 5c40f2f9cf2b41a3b129e7ef1bf41eb3 (DEFAULT_TABLE table=default.Table_b99166d4afe04f44a80ecd035fa90d72 [id=cb1954744198426c8a8f242f6f0679c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:06.829725 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c40f2f9cf2b41a3b129e7ef1bf41eb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:06.831120 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 6ca9bd37de9e4be388501b15af4bda31 (table default.Table_3ca4a6512b174cb897b142dfe1259549 [id=a862192912a04942a8f682fc9825b014]) successfully deleted
W20251029 02:52:06.833251 712 leader_election.cc:343] T f529e17e97074dca91ec2f48954cfd32 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:06.833357 712 leader_election.cc:304] T f529e17e97074dca91ec2f48954cfd32 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:06.833585 2018 raft_consensus.cc:2749] T f529e17e97074dca91ec2f48954cfd32 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:06.834177 890 tablet_service.cc:1505] Processing CreateTablet for tablet 49aaa2d8986a44df979edc1c42612189 (DEFAULT_TABLE table=default.Table_d844249d525b48428ebf40e1d6cb6c51 [id=2a13a4467a48459fb80a513cca4c4d96]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:06.835232 1167 tablet_bootstrap.cc:492] T 86fc22bb84794149b598b97df41eb881 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:06.835397 1167 ts_tablet_manager.cc:1403] T 86fc22bb84794149b598b97df41eb881 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.199s user 0.011s sys 0.032s
I20251029 02:52:06.830559 1025 tablet_service.cc:1505] Processing CreateTablet for tablet 49aaa2d8986a44df979edc1c42612189 (DEFAULT_TABLE table=default.Table_d844249d525b48428ebf40e1d6cb6c51 [id=2a13a4467a48459fb80a513cca4c4d96]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:06.835862 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49aaa2d8986a44df979edc1c42612189. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:06.836198 1167 raft_consensus.cc:359] T 86fc22bb84794149b598b97df41eb881 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.836365 1167 raft_consensus.cc:385] T 86fc22bb84794149b598b97df41eb881 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:06.836424 1167 raft_consensus.cc:740] T 86fc22bb84794149b598b97df41eb881 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:06.836740 1167 consensus_queue.cc:260] T 86fc22bb84794149b598b97df41eb881 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.838066 2086 raft_consensus.cc:493] T f187dbd8a55b45b292ea61db43caf47d P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:06.838193 2086 raft_consensus.cc:515] T f187dbd8a55b45b292ea61db43caf47d P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.830879 1031 tablet_service.cc:1505] Processing CreateTablet for tablet 5c40f2f9cf2b41a3b129e7ef1bf41eb3 (DEFAULT_TABLE table=default.Table_b99166d4afe04f44a80ecd035fa90d72 [id=cb1954744198426c8a8f242f6f0679c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:06.838956 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5c40f2f9cf2b41a3b129e7ef1bf41eb3. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:06.839491 847 leader_election.cc:343] T f187dbd8a55b45b292ea61db43caf47d P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: f187dbd8a55b45b292ea61db43caf47d
I20251029 02:52:06.840368 1058 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f187dbd8a55b45b292ea61db43caf47d" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:52:06.840914 849 leader_election.cc:343] T f187dbd8a55b45b292ea61db43caf47d P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:06.841038 849 leader_election.cc:304] T f187dbd8a55b45b292ea61db43caf47d P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9
I20251029 02:52:06.841259 2086 leader_election.cc:290] T f187dbd8a55b45b292ea61db43caf47d P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:06.841485 2086 raft_consensus.cc:2749] T f187dbd8a55b45b292ea61db43caf47d P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:06.843369 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 49aaa2d8986a44df979edc1c42612189. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:06.818619 1175 rpcz_store.cc:276] 1029 02:52:04.106342 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:04.106392 (+ 50us) service_pool.cc:225] Handling call
1029 02:52:06.818416 (+2712024us) ts_tablet_manager.cc:1151] Deleting tablet 4929513bed104050bd6065f025303e1a
1029 02:52:06.818427 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:06.818542 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2711845}
I20251029 02:52:06.832566 1039 tablet_service.cc:1505] Processing CreateTablet for tablet c91ecaea96224e33be4d728678729886 (DEFAULT_TABLE table=default.Table_1455335d0ba84ae08eff9eca1366a8f0 [id=46e325b6a4c346d4b077030727bb6a7e]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 02:52:06.846846 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 4929513bed104050bd6065f025303e1a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4929513bed104050bd6065f025303e1a already in progress: creating tablet
I20251029 02:52:06.847764 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 677) took 2721 ms. Trace:
I20251029 02:52:06.847851 1175 rpcz_store.cc:276] 1029 02:52:04.126325 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:04.126375 (+ 50us) service_pool.cc:225] Handling call
1029 02:52:06.847454 (+2721079us) ts_tablet_manager.cc:1151] Deleting tablet 4e82c574dfa14ffbabf162a17355f841
1029 02:52:06.847462 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:06.847745 (+ 283us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2720915}
W20251029 02:52:06.848218 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 4e82c574dfa14ffbabf162a17355f841 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e82c574dfa14ffbabf162a17355f841 already in progress: creating tablet
I20251029 02:52:06.848515 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 678) took 2709 ms. Trace:
I20251029 02:52:06.848577 1175 rpcz_store.cc:276] 1029 02:52:04.138937 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:04.138990 (+ 53us) service_pool.cc:225] Handling call
1029 02:52:06.848426 (+2709436us) ts_tablet_manager.cc:1151] Deleting tablet 4929513bed104050bd6065f025303e1a
1029 02:52:06.848434 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:06.848501 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2709278}
W20251029 02:52:06.849051 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 4929513bed104050bd6065f025303e1a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4929513bed104050bd6065f025303e1a already in progress: creating tablet
I20251029 02:52:06.849354 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 682) took 2686 ms. Trace:
I20251029 02:52:06.849419 1175 rpcz_store.cc:276] 1029 02:52:04.163343 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:04.163391 (+ 48us) service_pool.cc:225] Handling call
1029 02:52:06.849264 (+2685873us) ts_tablet_manager.cc:1151] Deleting tablet 67651306f3b44e7eb14dcee9a2007a81
1029 02:52:06.849273 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:06.849340 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2685710}
W20251029 02:52:06.849774 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 67651306f3b44e7eb14dcee9a2007a81 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 67651306f3b44e7eb14dcee9a2007a81 already in progress: creating tablet
I20251029 02:52:06.850072 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 684) took 2679 ms. Trace:
I20251029 02:52:06.850147 1175 rpcz_store.cc:276] 1029 02:52:04.170683 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:04.170739 (+ 56us) service_pool.cc:225] Handling call
1029 02:52:06.849974 (+2679235us) ts_tablet_manager.cc:1151] Deleting tablet 7fb5b908db77453da5cfb32e2c955f41
1029 02:52:06.849985 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:06.850057 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2679066}
W20251029 02:52:06.850651 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 7fb5b908db77453da5cfb32e2c955f41 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7fb5b908db77453da5cfb32e2c955f41 already in progress: creating tablet
I20251029 02:52:06.815945 2085 raft_consensus.cc:515] T 59d03a190b784ab18a3e63a572675891 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.851789 2085 leader_election.cc:290] T 59d03a190b784ab18a3e63a572675891 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:06.852370 1058 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59d03a190b784ab18a3e63a572675891" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:52:06.852864 712 leader_election.cc:343] T 59d03a190b784ab18a3e63a572675891 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:06.854586 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59d03a190b784ab18a3e63a572675891" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:06.855072 711 leader_election.cc:343] T 59d03a190b784ab18a3e63a572675891 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:06.855177 711 leader_election.cc:304] T 59d03a190b784ab18a3e63a572675891 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:06.855147 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c91ecaea96224e33be4d728678729886. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:06.857758 2085 raft_consensus.cc:2749] T 59d03a190b784ab18a3e63a572675891 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:06.866613 519 catalog_manager.cc:5649] T cc74e83833754f0ca19046031a24ebb9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } health_report { overall_health: HEALTHY } } }
I20251029 02:52:06.867190 519 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet cc74e83833754f0ca19046031a24ebb9 (table default.Table_755825b17834457cb94b05163ae3e26d [id=57471769276f44299eae50b9d854b50c]). Expected version 1 got 0
I20251029 02:52:06.874920 1176 raft_consensus.cc:2243] T e1fa4980f2624321b617b84c68b90113 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:06.875530 1176 raft_consensus.cc:2272] T e1fa4980f2624321b617b84c68b90113 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:06.876382 1176 ts_tablet_manager.cc:1916] T e1fa4980f2624321b617b84c68b90113 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:06.883672 1167 ts_tablet_manager.cc:1434] T 86fc22bb84794149b598b97df41eb881 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.016s user 0.001s sys 0.001s
I20251029 02:52:06.884462 1167 tablet_bootstrap.cc:492] T 56d05e4a500043c995bb51a204cb295d P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:06.888128 1039 tablet_service.cc:1505] Processing CreateTablet for tablet c108828dc9ca4c329dcb8a0240bf675f (DEFAULT_TABLE table=default.Table_3ccbb335620141bca4af176e8a8b8c8f [id=d76bd4acdbc54d5b82f2257cf9fa9861]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:06.888770 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c108828dc9ca4c329dcb8a0240bf675f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:06.890743 750 tablet_service.cc:1505] Processing CreateTablet for tablet a5ea72303de5434fb15fc7f9009d604b (DEFAULT_TABLE table=default.Table_dbd031b128dd49d4acf13158a6352320 [id=2123035cb1284b23bd306046903d3d3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:06.891209 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5ea72303de5434fb15fc7f9009d604b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:06.894562 756 tablet_service.cc:1505] Processing CreateTablet for tablet c108828dc9ca4c329dcb8a0240bf675f (DEFAULT_TABLE table=default.Table_3ccbb335620141bca4af176e8a8b8c8f [id=d76bd4acdbc54d5b82f2257cf9fa9861]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:06.894985 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c108828dc9ca4c329dcb8a0240bf675f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:06.900378 1175 tablet_replica.cc:333] T caef1c93e33a4390bcada59a41f43465 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:06.900619 1175 raft_consensus.cc:2243] T caef1c93e33a4390bcada59a41f43465 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:06.900830 1175 raft_consensus.cc:2272] T caef1c93e33a4390bcada59a41f43465 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:06.901430 1175 ts_tablet_manager.cc:1916] T caef1c93e33a4390bcada59a41f43465 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:06.910230 1031 tablet_service.cc:1505] Processing CreateTablet for tablet a5ea72303de5434fb15fc7f9009d604b (DEFAULT_TABLE table=default.Table_dbd031b128dd49d4acf13158a6352320 [id=2123035cb1284b23bd306046903d3d3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:06.910784 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5ea72303de5434fb15fc7f9009d604b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:06.924248 2036 raft_consensus.cc:2804] T daa05a34c0684c07a35966290fd7d44c P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:06.924372 2036 raft_consensus.cc:493] T daa05a34c0684c07a35966290fd7d44c P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:06.924433 2036 raft_consensus.cc:3060] T daa05a34c0684c07a35966290fd7d44c P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
W20251029 02:52:06.927143 520 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c1222e30bc4141b3ae686d80053c4ef5 consensus state has no leader
I20251029 02:52:06.929315 749 tablet_service.cc:1505] Processing CreateTablet for tablet bf2c37f5eff740a6ae594ffc6c0e5101 (DEFAULT_TABLE table=default.Table_90cad35ff5a14d0b87b2785627781b23 [id=3fcb2116712643af892b4e153742323e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:06.929706 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bf2c37f5eff740a6ae594ffc6c0e5101. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:06.945362 520 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet 1c53fc1fff1a4bb988beb46ff91fd3be (table default.Table_dd4934aabcd442039db111954d819f6d [id=9abdb27ccc0a48e991b7a4619808fdf8]). Expected version 1 got 0
I20251029 02:52:06.945771 520 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet da2a21c3236a4bd99872b89b12a36508 (table default.Table_2a63af6894694f16b45f4bf48f73a68f [id=fb42af08ec1f48498ee5e91cce62333e]). Expected version 5 got 0
I20251029 02:52:06.946117 520 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet 44b518c3a0964a6fa863b2784d80c8a4 (table default.Table_9120016827e946cc8a145252d2569d2b [id=e0c0cf3d456444c191a32bb180e83798]). Expected version 1 got 0
I20251029 02:52:06.946388 520 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet dabb8d760aae4aa09eb83cdb586c9788 (table default.Table_d897a86961894f1aaad67274b687080e [id=b3e544a043444b77a597c6c65c15b795]). Expected version 1 got 0
I20251029 02:52:06.949079 520 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_2662f20eb8d843b8bf6c841e49863c85" } modify_external_catalogs: true
I20251029 02:52:06.950107 743 tablet_service.cc:1552] Processing DeleteTablet for tablet 0d84dc8dc465496f9adec798b2e03e13 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:56 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:06.950084 751 tablet_service.cc:1505] Processing CreateTablet for tablet 44c2ef5426704e03bd50775adcdc571b (DEFAULT_TABLE table=default.Table_ae1c37b89ad243f0a389a54fa71b3ae0 [id=947d1437043d4ac08cf809711bca8b33]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:06.950351 742 tablet_service.cc:1552] Processing DeleteTablet for tablet 777992acb1bd4765886b802940db74bb with delete_type TABLET_DATA_DELETED (replaced by tablet e65eaaae43ac435ebd665546de9d68dd) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:06.950619 742 tablet_service.cc:1552] Processing DeleteTablet for tablet 59d03a190b784ab18a3e63a572675891 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:57 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:06.950788 755 tablet_service.cc:1505] Processing CreateTablet for tablet 4372eacfc07246f3be8f7c146674a736 (DEFAULT_TABLE table=default.Table_71446f2a5631421ab14116c669e1d5ba [id=90eb310388f54ddfb92fe83066df48f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:06.951031 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44c2ef5426704e03bd50775adcdc571b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:06.951138 748 tablet_service.cc:1505] Processing CreateTablet for tablet 13af21b2952941548f1bdb2741d9e6d2 (DEFAULT_TABLE table=default.Table_11ffae1fbbae415189fd5c7bdb829875 [id=645fc5b88e2549e7b22c636f1f248f78]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:06.951537 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4372eacfc07246f3be8f7c146674a736. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:06.954739 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13af21b2952941548f1bdb2741d9e6d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:06.954958 747 tablet_service.cc:1552] Processing DeleteTablet for tablet 7a1866afc7564217ad777e7ce53786c5 with delete_type TABLET_DATA_DELETED (replaced by tablet dc33fd3404ec41c3935b02c84a12971b) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:06.961689 520 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_2662f20eb8d843b8bf6c841e49863c85" } modify_external_catalogs: true
I20251029 02:52:06.965655 2108 tablet.cc:1722] T 4bf06476c87a4549a73f5994314bb75f P b495a49349324f7d81908f2da700f0d9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 02:52:06.965991 2085 raft_consensus.cc:493] T 785733da86ff425a90ccfab2933fc89e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:06.966230 2085 raft_consensus.cc:515] T 785733da86ff425a90ccfab2933fc89e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.966979 2085 leader_election.cc:290] T 785733da86ff425a90ccfab2933fc89e P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:06.967608 1176 ts_tablet_manager.cc:1929] T e1fa4980f2624321b617b84c68b90113 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:06.967758 1176 log.cc:1199] T e1fa4980f2624321b617b84c68b90113 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/e1fa4980f2624321b617b84c68b90113
I20251029 02:52:06.968147 1176 ts_tablet_manager.cc:1950] T e1fa4980f2624321b617b84c68b90113 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:06.968554 900 tablet_service.cc:1505] Processing CreateTablet for tablet c108828dc9ca4c329dcb8a0240bf675f (DEFAULT_TABLE table=default.Table_3ccbb335620141bca4af176e8a8b8c8f [id=d76bd4acdbc54d5b82f2257cf9fa9861]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:06.968772 1176 tablet_replica.cc:333] T 0cc1957a845f4e9cb703a553b27d1347 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:06.969022 1176 raft_consensus.cc:2243] T 0cc1957a845f4e9cb703a553b27d1347 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:06.969029 2086 raft_consensus.cc:493] T 0cc1957a845f4e9cb703a553b27d1347 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:06.970285 1172 tablet_bootstrap.cc:492] T 9bb2166a590d4210b02dd779ef6654ad P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:06.970400 1172 ts_tablet_manager.cc:1403] T 9bb2166a590d4210b02dd779ef6654ad P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.152s user 0.019s sys 0.024s
I20251029 02:52:06.971102 1172 raft_consensus.cc:359] T 9bb2166a590d4210b02dd779ef6654ad P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.971254 1172 raft_consensus.cc:385] T 9bb2166a590d4210b02dd779ef6654ad P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:06.971294 1172 raft_consensus.cc:740] T 9bb2166a590d4210b02dd779ef6654ad P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:06.971473 1172 consensus_queue.cc:260] T 9bb2166a590d4210b02dd779ef6654ad P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.972079 1172 ts_tablet_manager.cc:1434] T 9bb2166a590d4210b02dd779ef6654ad P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251029 02:52:06.972375 1172 tablet_bootstrap.cc:492] T ba699862e6f646c6beaa39f080d0faba P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:06.978813 2036 raft_consensus.cc:515] T daa05a34c0684c07a35966290fd7d44c P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.981678 1172 tablet_bootstrap.cc:654] T ba699862e6f646c6beaa39f080d0faba P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:06.983518 756 tablet_service.cc:1505] Processing CreateTablet for tablet faac614af1814a2f91d66c9cf8ddc919 (DEFAULT_TABLE table=default.Table_2156469f772942ffa8a3bdfc9614409a [id=b396a7cd4a89449fb283b389f3e76655]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:06.983956 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet faac614af1814a2f91d66c9cf8ddc919. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:06.985642 892 tablet_service.cc:1505] Processing CreateTablet for tablet 4372eacfc07246f3be8f7c146674a736 (DEFAULT_TABLE table=default.Table_71446f2a5631421ab14116c669e1d5ba [id=90eb310388f54ddfb92fe83066df48f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:06.986019 895 tablet_service.cc:1505] Processing CreateTablet for tablet a5ea72303de5434fb15fc7f9009d604b (DEFAULT_TABLE table=default.Table_dbd031b128dd49d4acf13158a6352320 [id=2123035cb1284b23bd306046903d3d3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:06.987375 1172 tablet_bootstrap.cc:492] T ba699862e6f646c6beaa39f080d0faba P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:06.987504 1172 ts_tablet_manager.cc:1403] T ba699862e6f646c6beaa39f080d0faba P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.001s
I20251029 02:52:06.988292 1172 raft_consensus.cc:359] T ba699862e6f646c6beaa39f080d0faba P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.988453 1172 raft_consensus.cc:385] T ba699862e6f646c6beaa39f080d0faba P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:06.988498 1172 raft_consensus.cc:740] T ba699862e6f646c6beaa39f080d0faba P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:06.989588 1172 consensus_queue.cc:260] T ba699862e6f646c6beaa39f080d0faba P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:06.991206 1172 ts_tablet_manager.cc:1434] T ba699862e6f646c6beaa39f080d0faba P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251029 02:52:06.998481 1175 ts_tablet_manager.cc:1929] T caef1c93e33a4390bcada59a41f43465 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:06.998811 1175 log.cc:1199] T caef1c93e33a4390bcada59a41f43465 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/caef1c93e33a4390bcada59a41f43465
I20251029 02:52:06.999526 1175 ts_tablet_manager.cc:1950] T caef1c93e33a4390bcada59a41f43465 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:06.999723 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "785733da86ff425a90ccfab2933fc89e" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:07.000001 898 tablet_service.cc:1505] Processing CreateTablet for tablet 13af21b2952941548f1bdb2741d9e6d2 (DEFAULT_TABLE table=default.Table_11ffae1fbbae415189fd5c7bdb829875 [id=645fc5b88e2549e7b22c636f1f248f78]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:07.001550 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 685) took 2830 ms. Trace:
I20251029 02:52:07.001770 1175 rpcz_store.cc:276] 1029 02:52:04.171426 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:04.171474 (+ 48us) service_pool.cc:225] Handling call
1029 02:52:06.900331 (+2728857us) ts_tablet_manager.cc:1151] Deleting tablet caef1c93e33a4390bcada59a41f43465
1029 02:52:06.900341 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:06.904498 (+ 4157us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:06.998241 (+ 93743us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:07.001382 (+ 3141us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2678694}
I20251029 02:52:06.996533 1904 raft_consensus.cc:493] T 71068739b4384badbf305313d902f0f5 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:07.002722 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c108828dc9ca4c329dcb8a0240bf675f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:07.003072 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5ea72303de5434fb15fc7f9009d604b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:07.003281 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4372eacfc07246f3be8f7c146674a736. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:07.005928 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13af21b2952941548f1bdb2741d9e6d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:07.008306 2036 leader_election.cc:290] T daa05a34c0684c07a35966290fd7d44c P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Requested vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
W20251029 02:52:06.980209 989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:38481, user_credentials={real_user=slave}} blocked reactor thread for 70422.9us
I20251029 02:52:07.010505 1031 tablet_service.cc:1505] Processing CreateTablet for tablet faac614af1814a2f91d66c9cf8ddc919 (DEFAULT_TABLE table=default.Table_2156469f772942ffa8a3bdfc9614409a [id=b396a7cd4a89449fb283b389f3e76655]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:07.010967 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet faac614af1814a2f91d66c9cf8ddc919. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:07.011356 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "785733da86ff425a90ccfab2933fc89e" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:07.011854 1039 tablet_service.cc:1505] Processing CreateTablet for tablet 4372eacfc07246f3be8f7c146674a736 (DEFAULT_TABLE table=default.Table_71446f2a5631421ab14116c669e1d5ba [id=90eb310388f54ddfb92fe83066df48f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:07.012142 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4372eacfc07246f3be8f7c146674a736. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:07.014236 1904 raft_consensus.cc:515] T 71068739b4384badbf305313d902f0f5 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:07.014801 1025 tablet_service.cc:1505] Processing CreateTablet for tablet 13af21b2952941548f1bdb2741d9e6d2 (DEFAULT_TABLE table=default.Table_11ffae1fbbae415189fd5c7bdb829875 [id=645fc5b88e2549e7b22c636f1f248f78]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:07.015110 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13af21b2952941548f1bdb2741d9e6d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:07.003084 2086 raft_consensus.cc:515] T 0cc1957a845f4e9cb703a553b27d1347 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:07.016481 885 tablet_service.cc:1505] Processing CreateTablet for tablet faac614af1814a2f91d66c9cf8ddc919 (DEFAULT_TABLE table=default.Table_2156469f772942ffa8a3bdfc9614409a [id=b396a7cd4a89449fb283b389f3e76655]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 02:52:07.019933 989 leader_election.cc:343] T daa05a34c0684c07a35966290fd7d44c P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: daa05a34c0684c07a35966290fd7d44c
W20251029 02:52:07.020445 712 leader_election.cc:343] T 785733da86ff425a90ccfab2933fc89e P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:07.028096 2103 consensus_queue.cc:1048] T cedfb53078d94c81b5e294ce7438325e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:06.982069 1176 raft_consensus.cc:2272] T 0cc1957a845f4e9cb703a553b27d1347 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:07.029237 1176 ts_tablet_manager.cc:1916] T 0cc1957a845f4e9cb703a553b27d1347 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:07.033077 2113 raft_consensus.cc:493] T 0f80b840ed954230bfbfe7e40127cd2f P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:07.033205 2113 raft_consensus.cc:515] T 0f80b840ed954230bfbfe7e40127cd2f P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:07.034212 2113 leader_election.cc:290] T 0f80b840ed954230bfbfe7e40127cd2f P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:06.988272 1166 tablet_bootstrap.cc:492] T 1bc45078a6a942c38ff3350c15885eff P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:07.034796 1166 ts_tablet_manager.cc:1403] T 1bc45078a6a942c38ff3350c15885eff P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.405s user 0.100s sys 0.218s
I20251029 02:52:07.035454 1166 raft_consensus.cc:359] T 1bc45078a6a942c38ff3350c15885eff P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:07.035593 1166 raft_consensus.cc:385] T 1bc45078a6a942c38ff3350c15885eff P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:07.035660 1166 raft_consensus.cc:740] T 1bc45078a6a942c38ff3350c15885eff P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:07.035816 1166 consensus_queue.cc:260] T 1bc45078a6a942c38ff3350c15885eff P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:07.036260 1166 ts_tablet_manager.cc:1434] T 1bc45078a6a942c38ff3350c15885eff P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20251029 02:52:07.036509 1166 tablet_bootstrap.cc:492] T 519754f5d8f7484480f973c53708c5f3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:07.038056 1166 tablet_bootstrap.cc:654] T 519754f5d8f7484480f973c53708c5f3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
W20251029 02:52:06.990744 990 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:41701, user_credentials={real_user=slave}} blocked reactor thread for 246123us
I20251029 02:52:07.039526 1973 raft_consensus.cc:493] T c764062d3ba04d4ba8b2e367ad8ed433 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:07.039642 1973 raft_consensus.cc:515] T c764062d3ba04d4ba8b2e367ad8ed433 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:07.039994 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f80b840ed954230bfbfe7e40127cd2f" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:07.040180 1973 leader_election.cc:290] T c764062d3ba04d4ba8b2e367ad8ed433 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
W20251029 02:52:07.041347 712 leader_election.cc:343] T 0f80b840ed954230bfbfe7e40127cd2f P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
W20251029 02:52:07.043287 989 leader_election.cc:343] T c764062d3ba04d4ba8b2e367ad8ed433 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: c764062d3ba04d4ba8b2e367ad8ed433
W20251029 02:52:07.049990 512 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 050186302c5d4c7da393f28f4b9e4262 (table default.Table_5681078bd9b54f948bd2d98511003cba [id=8c3b50bd1e434dfebe679141ceb18be8], current schema version=4) failed: Not found: failed to reset TS proxy: No leader found: Tablet 050186302c5d4c7da393f28f4b9e4262 consensus state has no leader
I20251029 02:52:07.064095 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet caef1c93e33a4390bcada59a41f43465 (table default.Table_b9f8911906c3473da4de53896081cb2c [id=8cccaacc256b40deabe999de1d75032e]) successfully deleted
I20251029 02:52:07.064651 2118 raft_consensus.cc:493] T 95f342d1cfb84c4eb31bd3d15ea3bdc2 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:07.064880 2118 raft_consensus.cc:515] T 95f342d1cfb84c4eb31bd3d15ea3bdc2 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:07.066177 2118 leader_election.cc:290] T 95f342d1cfb84c4eb31bd3d15ea3bdc2 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:07.066228 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet e1fa4980f2624321b617b84c68b90113 (table default.Table_8a5ae35851664e0f85324ec1ca672031 [id=a0e161a5d9d94a3b936669e6dfe6fb4f]) successfully deleted
I20251029 02:52:07.066246 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "95f342d1cfb84c4eb31bd3d15ea3bdc2" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:07.066383 2118 raft_consensus.cc:493] T 1c53fc1fff1a4bb988beb46ff91fd3be P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251029 02:52:07.066769 712 leader_election.cc:343] T 95f342d1cfb84c4eb31bd3d15ea3bdc2 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:07.067610 2118 raft_consensus.cc:515] T 1c53fc1fff1a4bb988beb46ff91fd3be P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:07.068462 2118 leader_election.cc:290] T 1c53fc1fff1a4bb988beb46ff91fd3be P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:07.068807 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c53fc1fff1a4bb988beb46ff91fd3be" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:52:07.069655 712 leader_election.cc:343] T 1c53fc1fff1a4bb988beb46ff91fd3be P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:07.075181 2118 raft_consensus.cc:493] T 0d84dc8dc465496f9adec798b2e03e13 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:07.075350 2118 raft_consensus.cc:515] T 0d84dc8dc465496f9adec798b2e03e13 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:07.076270 2118 leader_election.cc:290] T 0d84dc8dc465496f9adec798b2e03e13 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:07.076339 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d84dc8dc465496f9adec798b2e03e13" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:52:07.076841 712 leader_election.cc:343] T 0d84dc8dc465496f9adec798b2e03e13 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:07.077529 518 master_service.cc:954] ReplaceTablet: received request to replace tablet e7f846b294cc4eb687a26c8b5931ac19 from {username='slave'} at 127.0.0.1:35314
W20251029 02:52:07.077943 522 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6789bc16664e4b37b86702449ab27855 consensus state has no leader
I20251029 02:52:07.078743 1176 ts_tablet_manager.cc:1929] T 0cc1957a845f4e9cb703a553b27d1347 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:07.078908 1176 log.cc:1199] T 0cc1957a845f4e9cb703a553b27d1347 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/0cc1957a845f4e9cb703a553b27d1347
I20251029 02:52:07.079257 1176 ts_tablet_manager.cc:1950] T 0cc1957a845f4e9cb703a553b27d1347 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:07.080260 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 0cc1957a845f4e9cb703a553b27d1347 (table default.Table_262db98ea44648baab95fc2b3081dc4e [id=843f681b71c64427b49becbabcad9451]) successfully deleted
W20251029 02:52:07.080998 513 catalog_manager.cc:4952] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 71068739b4384badbf305313d902f0f5 because the tablet was not found. No further retry: Not found: Tablet not found: 71068739b4384badbf305313d902f0f5
I20251029 02:52:07.003139 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 683) took 2832 ms. Trace:
I20251029 02:52:07.090801 1175 rpcz_store.cc:276] 1029 02:52:04.170532 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:04.172071 (+ 1539us) service_pool.cc:225] Handling call
1029 02:52:07.002711 (+2830640us) ts_tablet_manager.cc:1151] Deleting tablet 72e91c64a9b54cbda52ca9dfcee2ce22
1029 02:52:07.002724 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:07.003096 (+ 372us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2830422}
I20251029 02:52:07.091420 1175 tablet_replica.cc:333] T 86fc22bb84794149b598b97df41eb881 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:07.091681 1175 raft_consensus.cc:2243] T 86fc22bb84794149b598b97df41eb881 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:07.091941 1175 raft_consensus.cc:2272] T 86fc22bb84794149b598b97df41eb881 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:07.092691 1175 ts_tablet_manager.cc:1916] T 86fc22bb84794149b598b97df41eb881 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 02:52:07.093659 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 72e91c64a9b54cbda52ca9dfcee2ce22 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 72e91c64a9b54cbda52ca9dfcee2ce22 already in progress: creating tablet
I20251029 02:52:07.094357 1166 tablet_bootstrap.cc:492] T 519754f5d8f7484480f973c53708c5f3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:07.094486 1166 ts_tablet_manager.cc:1403] T 519754f5d8f7484480f973c53708c5f3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.058s user 0.002s sys 0.002s
I20251029 02:52:07.095263 1166 raft_consensus.cc:359] T 519754f5d8f7484480f973c53708c5f3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:07.095434 1166 raft_consensus.cc:385] T 519754f5d8f7484480f973c53708c5f3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:07.095490 1166 raft_consensus.cc:740] T 519754f5d8f7484480f973c53708c5f3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:07.095688 1166 consensus_queue.cc:260] T 519754f5d8f7484480f973c53708c5f3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:07.096127 1166 ts_tablet_manager.cc:1434] T 519754f5d8f7484480f973c53708c5f3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251029 02:52:07.096398 1166 tablet_bootstrap.cc:492] T c9bd3f2deb5d4af09b7918269a98509f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:07.103497 1166 tablet_bootstrap.cc:654] T c9bd3f2deb5d4af09b7918269a98509f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:07.104277 1039 tablet_service.cc:1505] Processing CreateTablet for tablet 44c2ef5426704e03bd50775adcdc571b (DEFAULT_TABLE table=default.Table_ae1c37b89ad243f0a389a54fa71b3ae0 [id=947d1437043d4ac08cf809711bca8b33]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:07.104643 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44c2ef5426704e03bd50775adcdc571b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:07.105705 1904 leader_election.cc:290] T 71068739b4384badbf305313d902f0f5 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:07.109876 2120 raft_consensus.cc:493] T 5db519edc75d4bd0985b32322825ffae P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:07.110037 2120 raft_consensus.cc:515] T 5db519edc75d4bd0985b32322825ffae P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:07.110775 2120 leader_election.cc:290] T 5db519edc75d4bd0985b32322825ffae P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:07.111383 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5db519edc75d4bd0985b32322825ffae" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:52:07.111927 712 leader_election.cc:343] T 5db519edc75d4bd0985b32322825ffae P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:07.114581 1166 tablet_bootstrap.cc:492] T c9bd3f2deb5d4af09b7918269a98509f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:07.114765 1166 ts_tablet_manager.cc:1403] T c9bd3f2deb5d4af09b7918269a98509f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.001s
I20251029 02:52:07.115793 1166 raft_consensus.cc:359] T c9bd3f2deb5d4af09b7918269a98509f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:07.116034 1166 raft_consensus.cc:385] T c9bd3f2deb5d4af09b7918269a98509f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:07.116109 1166 raft_consensus.cc:740] T c9bd3f2deb5d4af09b7918269a98509f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:07.116348 1166 consensus_queue.cc:260] T c9bd3f2deb5d4af09b7918269a98509f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:07.116990 1166 ts_tablet_manager.cc:1434] T c9bd3f2deb5d4af09b7918269a98509f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251029 02:52:07.118688 1166 tablet_bootstrap.cc:492] T 4add552cf4d940d2a0a36d6b58379a34 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:07.119062 1025 tablet_service.cc:1505] Processing CreateTablet for tablet a35569c6688647008e49fccb350c134d (DEFAULT_TABLE table=default.Table_2743cefd9b784474b3bad6e9d5f71f55 [id=02073387c4f64040a521112a0629ed0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:07.119400 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a35569c6688647008e49fccb350c134d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:07.120990 1166 tablet_bootstrap.cc:654] T 4add552cf4d940d2a0a36d6b58379a34 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:07.121250 2120 raft_consensus.cc:493] T 777992acb1bd4765886b802940db74bb P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:07.121424 2120 raft_consensus.cc:515] T 777992acb1bd4765886b802940db74bb P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:07.122014 2120 leader_election.cc:290] T 777992acb1bd4765886b802940db74bb P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:07.122606 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "777992acb1bd4765886b802940db74bb" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:52:07.123062 712 leader_election.cc:343] T 777992acb1bd4765886b802940db74bb P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:07.126456 768 raft_consensus.cc:1275] T c9e923724ca9402682f13b252dbc37aa P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Refusing update from remote peer 0cfd4bee16f84b7c8b0ff3ce14db20d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251029 02:52:07.127096 2102 consensus_queue.cc:1048] T c9e923724ca9402682f13b252dbc37aa P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:07.130448 1166 tablet_bootstrap.cc:492] T 4add552cf4d940d2a0a36d6b58379a34 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:07.130606 1166 ts_tablet_manager.cc:1403] T 4add552cf4d940d2a0a36d6b58379a34 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.001s
I20251029 02:52:07.131561 1166 raft_consensus.cc:359] T 4add552cf4d940d2a0a36d6b58379a34 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:07.131793 1166 raft_consensus.cc:385] T 4add552cf4d940d2a0a36d6b58379a34 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:07.131868 1166 raft_consensus.cc:740] T 4add552cf4d940d2a0a36d6b58379a34 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:07.132130 1166 consensus_queue.cc:260] T 4add552cf4d940d2a0a36d6b58379a34 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:07.132833 1166 ts_tablet_manager.cc:1434] T 4add552cf4d940d2a0a36d6b58379a34 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 02:52:07.133993 1166 tablet_bootstrap.cc:492] T d5741dde463a43d6a776ab92dcfb04bf P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:07.135339 756 tablet_service.cc:1505] Processing CreateTablet for tablet a35569c6688647008e49fccb350c134d (DEFAULT_TABLE table=default.Table_2743cefd9b784474b3bad6e9d5f71f55 [id=02073387c4f64040a521112a0629ed0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:07.135744 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a35569c6688647008e49fccb350c134d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:07.135830 1166 tablet_bootstrap.cc:654] T d5741dde463a43d6a776ab92dcfb04bf P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:07.136408 2112 raft_consensus.cc:493] T 519754f5d8f7484480f973c53708c5f3 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:07.136576 2112 raft_consensus.cc:515] T 519754f5d8f7484480f973c53708c5f3 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:07.137253 2112 leader_election.cc:290] T 519754f5d8f7484480f973c53708c5f3 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
W20251029 02:52:07.137670 849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:45003, user_credentials={real_user=slave}} blocked reactor thread for 112509us
I20251029 02:52:07.139194 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "519754f5d8f7484480f973c53708c5f3" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:07.139359 1060 raft_consensus.cc:2468] T 519754f5d8f7484480f973c53708c5f3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
W20251029 02:52:07.139959 849 leader_election.cc:343] T 71068739b4384badbf305313d902f0f5 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Not found: Tablet not found: 71068739b4384badbf305313d902f0f5
W20251029 02:52:07.140174 849 leader_election.cc:343] T 0cc1957a845f4e9cb703a553b27d1347 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Not found: Tablet not found: 0cc1957a845f4e9cb703a553b27d1347
I20251029 02:52:07.140372 849 leader_election.cc:304] T 519754f5d8f7484480f973c53708c5f3 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:07.144274 1031 tablet_service.cc:1505] Processing CreateTablet for tablet 3e45bf8fb2f2426f9341d61dd8f647b4 (DEFAULT_TABLE table=default.Table_ff037b99b812492d9f064ca5485e9162 [id=faa3b16f911a481c99953b39304a78b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:07.144711 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e45bf8fb2f2426f9341d61dd8f647b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:07.144945 756 tablet_service.cc:1505] Processing CreateTablet for tablet 3e45bf8fb2f2426f9341d61dd8f647b4 (DEFAULT_TABLE table=default.Table_ff037b99b812492d9f064ca5485e9162 [id=faa3b16f911a481c99953b39304a78b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:07.145269 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e45bf8fb2f2426f9341d61dd8f647b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:07.146090 1041 tablet_service.cc:1505] Processing CreateTablet for tablet 12432c84ca104b0e99926a247318906a (DEFAULT_TABLE table=default.Table_73edd4fe45f94ebab19f2a834a265b63 [id=131979e336d34f07a30a4b877d1b5794]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:07.146420 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12432c84ca104b0e99926a247318906a. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:07.148957 525 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d3faa2d4df584c6e8da426a3830a6454 has no consensus state
I20251029 02:52:07.149360 525 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_aa561368dec842f2a73780eef725fd8d" } modify_external_catalogs: true
I20251029 02:52:07.149565 525 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_aa561368dec842f2a73780eef725fd8d" } modify_external_catalogs: true
I20251029 02:52:07.151888 521 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_f26d0b6d64304372a0bc5120b663991e" } modify_external_catalogs: true
I20251029 02:52:07.152097 521 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_f26d0b6d64304372a0bc5120b663991e" } modify_external_catalogs: true
I20251029 02:52:07.152257 1166 tablet_bootstrap.cc:492] T d5741dde463a43d6a776ab92dcfb04bf P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:07.152354 1166 ts_tablet_manager.cc:1403] T d5741dde463a43d6a776ab92dcfb04bf P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.000s
I20251029 02:52:07.152886 1166 raft_consensus.cc:359] T d5741dde463a43d6a776ab92dcfb04bf P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:07.152993 1166 raft_consensus.cc:385] T d5741dde463a43d6a776ab92dcfb04bf P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:07.153025 1166 raft_consensus.cc:740] T d5741dde463a43d6a776ab92dcfb04bf P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:07.153183 1166 consensus_queue.cc:260] T d5741dde463a43d6a776ab92dcfb04bf P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:07.153529 1166 ts_tablet_manager.cc:1434] T d5741dde463a43d6a776ab92dcfb04bf P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20251029 02:52:07.154096 1166 tablet_bootstrap.cc:492] T f187dbd8a55b45b292ea61db43caf47d P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:07.155915 1166 tablet_bootstrap.cc:654] T f187dbd8a55b45b292ea61db43caf47d P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:07.157442 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_31f9807cb2934a47a4fb2f67f9507053"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:07.158107 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_31f9807cb2934a47a4fb2f67f9507053 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:07.158843 517 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_ee9b7cb33d8b4032b185338d44561a69" } modify_external_catalogs: true
I20251029 02:52:07.159039 517 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_ee9b7cb33d8b4032b185338d44561a69" } modify_external_catalogs: true
I20251029 02:52:07.159817 522 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_2156469f772942ffa8a3bdfc9614409a" } modify_external_catalogs: true
I20251029 02:52:07.159955 522 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_2156469f772942ffa8a3bdfc9614409a" } modify_external_catalogs: true
I20251029 02:52:07.161662 756 tablet_service.cc:1552] Processing DeleteTablet for tablet d61beda662d844cdaf67fd9a2e382ea2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:55 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:07.161916 2103 consensus_queue.cc:1048] T c9e923724ca9402682f13b252dbc37aa P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:07.163269 1166 tablet_bootstrap.cc:492] T f187dbd8a55b45b292ea61db43caf47d P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:07.163429 1166 ts_tablet_manager.cc:1403] T f187dbd8a55b45b292ea61db43caf47d P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20251029 02:52:07.164140 1166 raft_consensus.cc:359] T f187dbd8a55b45b292ea61db43caf47d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:07.164338 1166 raft_consensus.cc:385] T f187dbd8a55b45b292ea61db43caf47d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:07.164425 1166 raft_consensus.cc:740] T f187dbd8a55b45b292ea61db43caf47d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:07.164661 1166 consensus_queue.cc:260] T f187dbd8a55b45b292ea61db43caf47d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:07.165149 1166 ts_tablet_manager.cc:1434] T f187dbd8a55b45b292ea61db43caf47d P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251029 02:52:07.166249 749 tablet_service.cc:1505] Processing CreateTablet for tablet 658f7269e01444b0bb6ec8c3f662ef45 (DEFAULT_TABLE table=default.Table_22f2a73359784f37b6869125fe10909e [id=1850ecd938d8406287fddaee7e033655]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:07.166656 756 tablet_service.cc:1505] Processing CreateTablet for tablet 12432c84ca104b0e99926a247318906a (DEFAULT_TABLE table=default.Table_73edd4fe45f94ebab19f2a834a265b63 [id=131979e336d34f07a30a4b877d1b5794]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:07.166901 750 tablet_service.cc:1505] Processing CreateTablet for tablet 26e982a886024afca62f29c12f6ab5c0 (DEFAULT_TABLE table=default.Table_abf02c5d46974d879e8c163933cb78ab [id=92d7b64f46944d508ef48a7677c2d5c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:07.167069 1166 tablet_bootstrap.cc:492] T b3de6670357c4ae59eb77469b2fb434c P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:07.166668 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 658f7269e01444b0bb6ec8c3f662ef45. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:07.167573 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12432c84ca104b0e99926a247318906a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:07.167817 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26e982a886024afca62f29c12f6ab5c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:07.170380 526 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 31a157697e2c45ac9ba92c31c6936fa0
I20251029 02:52:07.170855 526 catalog_manager.cc:6719] ReplaceTablet: tablet 31a157697e2c45ac9ba92c31c6936fa0 deleted and replaced by tablet fe5aff0f61a44b91bfb37b42457a91f0
I20251029 02:52:07.171192 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_9e7a20ed2df8447d92cc51920fd8177e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:07.171648 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9e7a20ed2df8447d92cc51920fd8177e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:07.174965 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_b72f4af954b8442d9a8a477f69bf3b62"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:07.175469 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b72f4af954b8442d9a8a477f69bf3b62 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:07.175597 1166 tablet_bootstrap.cc:654] T b3de6670357c4ae59eb77469b2fb434c P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:07.178067 523 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet cc74e83833754f0ca19046031a24ebb9 (table default.Table_755825b17834457cb94b05163ae3e26d [id=57471769276f44299eae50b9d854b50c]). Expected version 1 got 0
I20251029 02:52:07.178298 523 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet da2a21c3236a4bd99872b89b12a36508 (table default.Table_2a63af6894694f16b45f4bf48f73a68f [id=fb42af08ec1f48498ee5e91cce62333e]). Expected version 5 got 0
I20251029 02:52:07.179710 1031 tablet_service.cc:1552] Processing DeleteTablet for tablet 31a157697e2c45ac9ba92c31c6936fa0 with delete_type TABLET_DATA_DELETED (replaced by tablet fe5aff0f61a44b91bfb37b42457a91f0) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:07.180011 1031 tablet_service.cc:1552] Processing DeleteTablet for tablet 4aa78f4ac38e4919864c924e50f1aaf4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:57 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:07.180018 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet dd6ad641fbda472fab07aa35c5e4523a with delete_type TABLET_DATA_DELETED (replaced by tablet 4f24ae08e2c34375abccb6fd187adbd4) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:07.180426 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet bfec87f690a640c39c58c64e16cc67e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:57 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:07.180483 1037 tablet_service.cc:1552] Processing DeleteTablet for tablet 6ca9bd37de9e4be388501b15af4bda31 with delete_type TABLET_DATA_DELETED (replaced by tablet 20a762344c5f46f88481c4ac7ad6e43b) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:07.180675 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet c764062d3ba04d4ba8b2e367ad8ed433 with delete_type TABLET_DATA_DELETED (replaced by tablet 372d93914abc41b38a702742b7db0bc2) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:07.180768 1031 tablet_service.cc:1552] Processing DeleteTablet for tablet 56d05e4a500043c995bb51a204cb295d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:53 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:07.180830 1032 tablet_service.cc:1552] Processing DeleteTablet for tablet f60e7c24982d40319801869ad060864d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:53 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:07.180994 1042 tablet_service.cc:1552] Processing DeleteTablet for tablet daa05a34c0684c07a35966290fd7d44c with delete_type TABLET_DATA_DELETED (replaced by tablet fcb3db8399aa4cf188b07dda220f7f5b) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:07.181377 1176 tablet_replica.cc:333] T 4aa78f4ac38e4919864c924e50f1aaf4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:07.181697 1176 raft_consensus.cc:2243] T 4aa78f4ac38e4919864c924e50f1aaf4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:07.178494 523 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet cedfb53078d94c81b5e294ce7438325e (table default.Table_e75db5cf8f5f49ea837cb6d63f4b7ebf [id=86195563fb4a4d7abd746f4ac7cd241c]). Expected version 1 got 0
I20251029 02:52:07.182128 1176 pending_rounds.cc:70] T 4aa78f4ac38e4919864c924e50f1aaf4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Trying to abort 1 pending ops.
I20251029 02:52:07.182235 1176 pending_rounds.cc:77] T 4aa78f4ac38e4919864c924e50f1aaf4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215949113073664000 op_type: NO_OP noop_request { }
I20251029 02:52:07.182122 523 catalog_manager.cc:5649] T d5741dde463a43d6a776ab92dcfb04bf P c756528642a543eeacd93cef3e335a7d reported cstate change: term changed from 0 to 1, leader changed from <none> to c756528642a543eeacd93cef3e335a7d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "c756528642a543eeacd93cef3e335a7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } health_report { overall_health: UNKNOWN } } }
I20251029 02:52:07.182494 1176 raft_consensus.cc:2889] T 4aa78f4ac38e4919864c924e50f1aaf4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 02:52:07.182570 523 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet d5741dde463a43d6a776ab92dcfb04bf (table default.Table_966779a0d37a4dc4ab3f89d2bf7c6b06 [id=08e3b2ebcf37415d9a2886ad5bbda068]). Expected version 1 got 0
I20251029 02:52:07.182734 1176 raft_consensus.cc:2272] T 4aa78f4ac38e4919864c924e50f1aaf4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251029 02:52:07.181622 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 31a157697e2c45ac9ba92c31c6936fa0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 31a157697e2c45ac9ba92c31c6936fa0 already in progress: creating tablet
I20251029 02:52:07.183905 750 tablet_service.cc:1552] Processing DeleteTablet for tablet 31a157697e2c45ac9ba92c31c6936fa0 with delete_type TABLET_DATA_DELETED (replaced by tablet fe5aff0f61a44b91bfb37b42457a91f0) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:07.184149 1176 ts_tablet_manager.cc:1916] T 4aa78f4ac38e4919864c924e50f1aaf4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:07.185497 1025 tablet_service.cc:1505] Processing CreateTablet for tablet 26e982a886024afca62f29c12f6ab5c0 (DEFAULT_TABLE table=default.Table_abf02c5d46974d879e8c163933cb78ab [id=92d7b64f46944d508ef48a7677c2d5c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:07.185866 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26e982a886024afca62f29c12f6ab5c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:07.189426 520 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 91e31f0b06924740bdf7525cae79d8ef
I20251029 02:52:07.190636 1039 tablet_service.cc:1505] Processing CreateTablet for tablet 658f7269e01444b0bb6ec8c3f662ef45 (DEFAULT_TABLE table=default.Table_22f2a73359784f37b6869125fe10909e [id=1850ecd938d8406287fddaee7e033655]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:07.190997 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 658f7269e01444b0bb6ec8c3f662ef45. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:07.192011 750 tablet_service.cc:1552] Processing DeleteTablet for tablet 91e31f0b06924740bdf7525cae79d8ef with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:06 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:07.192409 1042 tablet_service.cc:1552] Processing DeleteTablet for tablet 91e31f0b06924740bdf7525cae79d8ef with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:06 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:07.204465 525 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet e4cad991d48341cbafb5f1d9b50f4e5b
I20251029 02:52:07.206492 750 tablet_service.cc:1552] Processing DeleteTablet for tablet e4cad991d48341cbafb5f1d9b50f4e5b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:07 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:07.215888 2134 raft_consensus.cc:493] T dabb8d760aae4aa09eb83cdb586c9788 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:07.216111 2134 raft_consensus.cc:515] T dabb8d760aae4aa09eb83cdb586c9788 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:07.217592 2134 leader_election.cc:290] T dabb8d760aae4aa09eb83cdb586c9788 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:07.217696 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dabb8d760aae4aa09eb83cdb586c9788" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:07.217760 2134 raft_consensus.cc:493] T 7a1866afc7564217ad777e7ce53786c5 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251029 02:52:07.218283 712 leader_election.cc:343] T dabb8d760aae4aa09eb83cdb586c9788 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:07.220095 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet faac614af1814a2f91d66c9cf8ddc919. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:07.220275 2086 leader_election.cc:290] T 0cc1957a845f4e9cb703a553b27d1347 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:07.217905 2134 raft_consensus.cc:515] T 7a1866afc7564217ad777e7ce53786c5 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:07.223663 881 tablet_service.cc:1505] Processing CreateTablet for tablet 8af82b9846464e739d0c052b9a31ff7b (DEFAULT_TABLE table=default.Table_dc31c9a0d89947c1aeeedd96b7678134 [id=975f15222c504d1d967a3e75f2ee80b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:07.224218 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a1866afc7564217ad777e7ce53786c5" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:07.224335 1041 tablet_service.cc:1552] Processing DeleteTablet for tablet e4cad991d48341cbafb5f1d9b50f4e5b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:07 UTC) from {username='slave'} at 127.0.0.1:37626
W20251029 02:52:07.224767 712 leader_election.cc:343] T 7a1866afc7564217ad777e7ce53786c5 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:07.224385 2134 leader_election.cc:290] T 7a1866afc7564217ad777e7ce53786c5 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:07.225116 884 tablet_service.cc:1552] Processing DeleteTablet for tablet 31a157697e2c45ac9ba92c31c6936fa0 with delete_type TABLET_DATA_DELETED (replaced by tablet fe5aff0f61a44b91bfb37b42457a91f0) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:07.226470 879 tablet_service.cc:1552] Processing DeleteTablet for tablet e4cad991d48341cbafb5f1d9b50f4e5b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:07 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:07.227098 518 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet e7f846b294cc4eb687a26c8b5931ac19
I20251029 02:52:07.227653 518 catalog_manager.cc:6719] ReplaceTablet: tablet e7f846b294cc4eb687a26c8b5931ac19 deleted and replaced by tablet cdc5fb2b13e042d680fa3f2eea2d62fa
W20251029 02:52:07.227907 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 31a157697e2c45ac9ba92c31c6936fa0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 31a157697e2c45ac9ba92c31c6936fa0 already in progress: creating tablet
I20251029 02:52:07.227999 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_0c7029666c80452b8fae6945da658ec7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:07.228287 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet e4cad991d48341cbafb5f1d9b50f4e5b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e4cad991d48341cbafb5f1d9b50f4e5b already in progress: creating tablet
W20251029 02:52:07.228559 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0c7029666c80452b8fae6945da658ec7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:07.230170 521 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet e5c88e03d9324a918020688c884eb4f6
I20251029 02:52:07.230932 521 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_6d69db05345d4ecdb52c0f4b4ac5be13" } new_table_name: "default.Table_913e2dbb7d5d4692886f576858021b92" modify_external_catalogs: true
I20251029 02:52:07.232751 750 tablet_service.cc:1552] Processing DeleteTablet for tablet e7f846b294cc4eb687a26c8b5931ac19 with delete_type TABLET_DATA_DELETED (replaced by tablet cdc5fb2b13e042d680fa3f2eea2d62fa) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:07.233253 1042 tablet_service.cc:1552] Processing DeleteTablet for tablet e5c88e03d9324a918020688c884eb4f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:07 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:07.232856 749 tablet_service.cc:1552] Processing DeleteTablet for tablet e5c88e03d9324a918020688c884eb4f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:07 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:07.234953 1176 ts_tablet_manager.cc:1929] T 4aa78f4ac38e4919864c924e50f1aaf4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:07.235111 1176 log.cc:1199] T 4aa78f4ac38e4919864c924e50f1aaf4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/4aa78f4ac38e4919864c924e50f1aaf4
W20251029 02:52:07.235272 847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:38481, user_credentials={real_user=slave}} blocked reactor thread for 184181us
I20251029 02:52:07.235435 1041 tablet_service.cc:1552] Processing DeleteTablet for tablet e7f846b294cc4eb687a26c8b5931ac19 with delete_type TABLET_DATA_DELETED (replaced by tablet cdc5fb2b13e042d680fa3f2eea2d62fa) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:07.235553 1176 ts_tablet_manager.cc:1950] T 4aa78f4ac38e4919864c924e50f1aaf4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:07.235842 919 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "daa05a34c0684c07a35966290fd7d44c" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
I20251029 02:52:07.236039 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c764062d3ba04d4ba8b2e367ad8ed433" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:07.236533 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 4aa78f4ac38e4919864c924e50f1aaf4 (table default.Table_e168a30d3a1e48ed8053847b304036e9 [id=a93c34a9cd344503954bfae8752a69a6]) successfully deleted
I20251029 02:52:07.236842 1176 tablet_replica.cc:333] T dd6ad641fbda472fab07aa35c5e4523a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:07.237146 1176 raft_consensus.cc:2243] T dd6ad641fbda472fab07aa35c5e4523a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:07.237334 1176 raft_consensus.cc:2272] T dd6ad641fbda472fab07aa35c5e4523a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251029 02:52:07.237685 990 leader_election.cc:343] T c764062d3ba04d4ba8b2e367ad8ed433 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:07.237809 990 leader_election.cc:304] T c764062d3ba04d4ba8b2e367ad8ed433 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5; no voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:07.238021 1973 raft_consensus.cc:493] T 1bc45078a6a942c38ff3350c15885eff P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:07.238137 1973 raft_consensus.cc:515] T 1bc45078a6a942c38ff3350c15885eff P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:07.238660 1973 leader_election.cc:290] T 1bc45078a6a942c38ff3350c15885eff P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:07.238770 1973 raft_consensus.cc:2749] T c764062d3ba04d4ba8b2e367ad8ed433 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:07.239387 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1bc45078a6a942c38ff3350c15885eff" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:07.239555 773 raft_consensus.cc:2468] T 1bc45078a6a942c38ff3350c15885eff P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 0.
I20251029 02:52:07.239980 989 leader_election.cc:304] T 1bc45078a6a942c38ff3350c15885eff P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters:
I20251029 02:52:07.240432 1973 raft_consensus.cc:2804] T 1bc45078a6a942c38ff3350c15885eff P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:07.240500 1973 raft_consensus.cc:493] T 1bc45078a6a942c38ff3350c15885eff P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:07.240551 1973 raft_consensus.cc:3060] T 1bc45078a6a942c38ff3350c15885eff P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:07.242326 2086 raft_consensus.cc:493] T f60e7c24982d40319801869ad060864d P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:07.242467 895 tablet_service.cc:1505] Processing CreateTablet for tablet 44c2ef5426704e03bd50775adcdc571b (DEFAULT_TABLE table=default.Table_ae1c37b89ad243f0a389a54fa71b3ae0 [id=947d1437043d4ac08cf809711bca8b33]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:07.243306 890 tablet_service.cc:1505] Processing CreateTablet for tablet a35569c6688647008e49fccb350c134d (DEFAULT_TABLE table=default.Table_2743cefd9b784474b3bad6e9d5f71f55 [id=02073387c4f64040a521112a0629ed0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:07.243727 1973 raft_consensus.cc:515] T 1bc45078a6a942c38ff3350c15885eff P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:07.243930 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8af82b9846464e739d0c052b9a31ff7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:07.243955 900 tablet_service.cc:1505] Processing CreateTablet for tablet 658f7269e01444b0bb6ec8c3f662ef45 (DEFAULT_TABLE table=default.Table_22f2a73359784f37b6869125fe10909e [id=1850ecd938d8406287fddaee7e033655]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:07.244359 898 tablet_service.cc:1505] Processing CreateTablet for tablet 3e45bf8fb2f2426f9341d61dd8f647b4 (DEFAULT_TABLE table=default.Table_ff037b99b812492d9f064ca5485e9162 [id=faa3b16f911a481c99953b39304a78b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:07.245038 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1bc45078a6a942c38ff3350c15885eff" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:07.245198 773 raft_consensus.cc:3060] T 1bc45078a6a942c38ff3350c15885eff P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:07.245297 517 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet bec4214e14fe4a86a58c139d620d4da7
I20251029 02:52:07.245564 1904 raft_consensus.cc:2804] T 519754f5d8f7484480f973c53708c5f3 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:07.245715 517 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_9bed90ff07fb4ea28bc7a94b6ebfc125" } new_table_name: "default.Table_6b0b554613694e198639046c12218185" modify_external_catalogs: true
I20251029 02:52:07.245755 892 tablet_service.cc:1505] Processing CreateTablet for tablet 12432c84ca104b0e99926a247318906a (DEFAULT_TABLE table=default.Table_73edd4fe45f94ebab19f2a834a265b63 [id=131979e336d34f07a30a4b877d1b5794]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:07.247999 773 raft_consensus.cc:2468] T 1bc45078a6a942c38ff3350c15885eff P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 1.
I20251029 02:52:07.248523 989 leader_election.cc:304] T 1bc45078a6a942c38ff3350c15885eff P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters:
I20251029 02:52:07.248936 1172 tablet_bootstrap.cc:492] T 1275e192871440408a8345fef9b56260 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:07.249707 2108 tablet.cc:1722] T c9e923724ca9402682f13b252dbc37aa P b495a49349324f7d81908f2da700f0d9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251029 02:52:07.250166 1973 leader_election.cc:290] T 1bc45078a6a942c38ff3350c15885eff P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Requested vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:07.250377 1973 raft_consensus.cc:2804] T 1bc45078a6a942c38ff3350c15885eff P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:07.250702 1172 tablet_bootstrap.cc:654] T 1275e192871440408a8345fef9b56260 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:07.255501 749 tablet_service.cc:1505] Processing CreateTablet for tablet 8af82b9846464e739d0c052b9a31ff7b (DEFAULT_TABLE table=default.Table_dc31c9a0d89947c1aeeedd96b7678134 [id=975f15222c504d1d967a3e75f2ee80b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:07.255918 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8af82b9846464e739d0c052b9a31ff7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:07.257943 750 tablet_service.cc:1505] Processing CreateTablet for tablet 3b5a12416a2f49f190e008ef31519ec9 (DEFAULT_TABLE table=default.Table_bdf9abc4a0cd42f0ba2c936612afde6b [id=c6bdebdca8a84d0c81d94d59a2613fdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:07.258713 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b5a12416a2f49f190e008ef31519ec9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:07.258888 1041 tablet_service.cc:1505] Processing CreateTablet for tablet 8af82b9846464e739d0c052b9a31ff7b (DEFAULT_TABLE table=default.Table_dc31c9a0d89947c1aeeedd96b7678134 [id=975f15222c504d1d967a3e75f2ee80b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:07.258976 1042 tablet_service.cc:1505] Processing CreateTablet for tablet 3b5a12416a2f49f190e008ef31519ec9 (DEFAULT_TABLE table=default.Table_bdf9abc4a0cd42f0ba2c936612afde6b [id=c6bdebdca8a84d0c81d94d59a2613fdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:07.259229 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8af82b9846464e739d0c052b9a31ff7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:07.259441 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b5a12416a2f49f190e008ef31519ec9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:07.267408 1973 raft_consensus.cc:697] T 1bc45078a6a942c38ff3350c15885eff P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Becoming Leader. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Running, Role: LEADER
I20251029 02:52:07.268854 1973 consensus_queue.cc:237] T 1bc45078a6a942c38ff3350c15885eff P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:07.271773 919 raft_consensus.cc:3060] T daa05a34c0684c07a35966290fd7d44c P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
W20251029 02:52:07.272305 847 leader_election.cc:343] T 0cc1957a845f4e9cb703a553b27d1347 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 0cc1957a845f4e9cb703a553b27d1347
I20251029 02:52:07.275097 1172 tablet_bootstrap.cc:492] T 1275e192871440408a8345fef9b56260 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:07.275230 1172 ts_tablet_manager.cc:1403] T 1275e192871440408a8345fef9b56260 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.004s
I20251029 02:52:07.276144 1172 raft_consensus.cc:359] T 1275e192871440408a8345fef9b56260 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:07.276350 1172 raft_consensus.cc:385] T 1275e192871440408a8345fef9b56260 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:07.276401 1172 raft_consensus.cc:740] T 1275e192871440408a8345fef9b56260 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:07.276621 1172 consensus_queue.cc:260] T 1275e192871440408a8345fef9b56260 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:07.278081 1172 ts_tablet_manager.cc:1434] T 1275e192871440408a8345fef9b56260 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251029 02:52:07.279101 1172 tablet_bootstrap.cc:492] T 13ee62fc2339410b9db70dfd3d3c6fa6 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:07.279851 884 tablet_service.cc:1552] Processing DeleteTablet for tablet 91e31f0b06924740bdf7525cae79d8ef with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:06 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:07.280167 879 tablet_service.cc:1552] Processing DeleteTablet for tablet e7f846b294cc4eb687a26c8b5931ac19 with delete_type TABLET_DATA_DELETED (replaced by tablet cdc5fb2b13e042d680fa3f2eea2d62fa) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:07.280215 891 tablet_service.cc:1552] Processing DeleteTablet for tablet e5c88e03d9324a918020688c884eb4f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:07 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:07.280426 2086 raft_consensus.cc:515] T f60e7c24982d40319801869ad060864d P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:07.280900 1172 tablet_bootstrap.cc:654] T 13ee62fc2339410b9db70dfd3d3c6fa6 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:07.281030 2086 leader_election.cc:290] T f60e7c24982d40319801869ad060864d P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:07.281256 2086 raft_consensus.cc:493] T b3de6670357c4ae59eb77469b2fb434c P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:07.281378 2086 raft_consensus.cc:515] T b3de6670357c4ae59eb77469b2fb434c P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:07.281885 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1bc45078a6a942c38ff3350c15885eff" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:07.284063 2131 tablet.cc:1722] T c9e923724ca9402682f13b252dbc37aa P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251029 02:52:07.285473 1175 ts_tablet_manager.cc:1929] T 86fc22bb84794149b598b97df41eb881 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:07.285575 1175 log.cc:1199] T 86fc22bb84794149b598b97df41eb881 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/86fc22bb84794149b598b97df41eb881
I20251029 02:52:07.285905 1175 ts_tablet_manager.cc:1950] T 86fc22bb84794149b598b97df41eb881 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:07.286936 1904 raft_consensus.cc:493] T 519754f5d8f7484480f973c53708c5f3 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:07.287133 899 tablet_service.cc:1505] Processing CreateTablet for tablet 26e982a886024afca62f29c12f6ab5c0 (DEFAULT_TABLE table=default.Table_abf02c5d46974d879e8c163933cb78ab [id=92d7b64f46944d508ef48a7677c2d5c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:07.289085 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 658f7269e01444b0bb6ec8c3f662ef45. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:07.292712 1176 ts_tablet_manager.cc:1916] T dd6ad641fbda472fab07aa35c5e4523a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:07.295635 2132 tablet.cc:1722] T da2a21c3236a4bd99872b89b12a36508 P b495a49349324f7d81908f2da700f0d9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20251029 02:52:07.296344 847 leader_election.cc:304] T 0cc1957a845f4e9cb703a553b27d1347 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9
W20251029 02:52:07.296705 847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:38481, user_credentials={real_user=slave}} blocked reactor thread for 59356.5us
W20251029 02:52:07.296953 847 leader_election.cc:343] T 519754f5d8f7484480f973c53708c5f3 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 519754f5d8f7484480f973c53708c5f3
W20251029 02:52:07.297259 847 leader_election.cc:343] T 71068739b4384badbf305313d902f0f5 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 71068739b4384badbf305313d902f0f5
I20251029 02:52:07.297380 847 leader_election.cc:304] T 71068739b4384badbf305313d902f0f5 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9
I20251029 02:52:07.298358 2114 raft_consensus.cc:2749] T 0cc1957a845f4e9cb703a553b27d1347 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:07.298550 2114 raft_consensus.cc:2749] T 71068739b4384badbf305313d902f0f5 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:07.298715 1904 raft_consensus.cc:3060] T 519754f5d8f7484480f973c53708c5f3 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:07.299432 2102 raft_consensus.cc:493] T f187dbd8a55b45b292ea61db43caf47d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:07.299562 2102 raft_consensus.cc:515] T f187dbd8a55b45b292ea61db43caf47d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:07.300733 2102 leader_election.cc:290] T f187dbd8a55b45b292ea61db43caf47d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
W20251029 02:52:07.301195 989 leader_election.cc:343] T f187dbd8a55b45b292ea61db43caf47d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: f187dbd8a55b45b292ea61db43caf47d
I20251029 02:52:07.300890 2136 tablet.cc:1722] T da2a21c3236a4bd99872b89b12a36508 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20251029 02:52:07.301446 1904 raft_consensus.cc:515] T 519754f5d8f7484480f973c53708c5f3 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:07.302201 1904 leader_election.cc:290] T 519754f5d8f7484480f973c53708c5f3 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:07.302996 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e45bf8fb2f2426f9341d61dd8f647b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:07.303606 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12432c84ca104b0e99926a247318906a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:07.306748 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 688) took 3073 ms. Trace:
I20251029 02:52:07.306841 1175 rpcz_store.cc:276] 1029 02:52:04.233282 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:04.233356 (+ 74us) service_pool.cc:225] Handling call
1029 02:52:07.091346 (+2857990us) ts_tablet_manager.cc:1151] Deleting tablet 86fc22bb84794149b598b97df41eb881
1029 02:52:07.091365 (+ 19us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:07.260346 (+168981us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:07.285375 (+ 25029us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:07.306728 (+ 21353us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2857795}
W20251029 02:52:07.307083 847 leader_election.cc:343] T 519754f5d8f7484480f973c53708c5f3 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 519754f5d8f7484480f973c53708c5f3
I20251029 02:52:07.307304 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 86fc22bb84794149b598b97df41eb881 (table default.Table_77a5f27bd0ae4eb68a30c883268d2f9c [id=d62f017ad2514f29867c9cde35996faf]) successfully deleted
I20251029 02:52:07.307734 1176 ts_tablet_manager.cc:1929] T dd6ad641fbda472fab07aa35c5e4523a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:07.308023 1176 log.cc:1199] T dd6ad641fbda472fab07aa35c5e4523a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/dd6ad641fbda472fab07aa35c5e4523a
I20251029 02:52:07.308449 1176 ts_tablet_manager.cc:1950] T dd6ad641fbda472fab07aa35c5e4523a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:07.309473 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet dd6ad641fbda472fab07aa35c5e4523a (table default.Table_29766c136dbc44e486d4256e7dbba851 [id=1d39d6caa9e24fe38b5cf9a20a6c66b9]) successfully deleted
I20251029 02:52:07.309852 1176 tablet_replica.cc:333] T bfec87f690a640c39c58c64e16cc67e7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:07.310086 1176 raft_consensus.cc:2243] T bfec87f690a640c39c58c64e16cc67e7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:07.310266 1176 raft_consensus.cc:2272] T bfec87f690a640c39c58c64e16cc67e7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:07.310550 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44c2ef5426704e03bd50775adcdc571b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:07.310755 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 689) took 3050 ms. Trace:
I20251029 02:52:07.310920 1176 ts_tablet_manager.cc:1916] T bfec87f690a640c39c58c64e16cc67e7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:07.310843 1175 rpcz_store.cc:276] 1029 02:52:04.260708 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:04.260782 (+ 74us) service_pool.cc:225] Handling call
1029 02:52:07.310624 (+3049842us) ts_tablet_manager.cc:1151] Deleting tablet 78662eeae70b4148a4ad0d47be5c566d
1029 02:52:07.310635 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:07.310735 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3049632}
I20251029 02:52:07.311488 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 691) took 2985 ms. Trace:
I20251029 02:52:07.311570 1175 rpcz_store.cc:276] 1029 02:52:04.325635 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:04.325709 (+ 74us) service_pool.cc:225] Handling call
1029 02:52:07.311384 (+2985675us) ts_tablet_manager.cc:1151] Deleting tablet de7fac8751a04c19b68a52db7f167214
1029 02:52:07.311394 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:07.311472 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2985281}
W20251029 02:52:07.311966 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet de7fac8751a04c19b68a52db7f167214 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet de7fac8751a04c19b68a52db7f167214 already in progress: creating tablet
W20251029 02:52:07.314863 990 leader_election.cc:343] T 1bc45078a6a942c38ff3350c15885eff P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:07.314988 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 693) took 2969 ms. Trace:
I20251029 02:52:07.315057 1175 rpcz_store.cc:276] 1029 02:52:04.345339 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:04.345405 (+ 66us) service_pool.cc:225] Handling call
1029 02:52:07.314865 (+2969460us) ts_tablet_manager.cc:1151] Deleting tablet 1e17142dc6634db0a5288add9c199ffc
1029 02:52:07.314886 (+ 21us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:07.314972 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2969289}
W20251029 02:52:07.315451 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 1e17142dc6634db0a5288add9c199ffc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1e17142dc6634db0a5288add9c199ffc already in progress: creating tablet
W20251029 02:52:07.316231 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 91e31f0b06924740bdf7525cae79d8ef because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 91e31f0b06924740bdf7525cae79d8ef already in progress: creating tablet
W20251029 02:52:07.316501 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet e7f846b294cc4eb687a26c8b5931ac19 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e7f846b294cc4eb687a26c8b5931ac19 already in progress: creating tablet
W20251029 02:52:07.316756 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet e5c88e03d9324a918020688c884eb4f6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e5c88e03d9324a918020688c884eb4f6 already in progress: creating tablet
I20251029 02:52:07.317718 1167 tablet_bootstrap.cc:654] T 56d05e4a500043c995bb51a204cb295d P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:07.319446 1176 ts_tablet_manager.cc:1929] T bfec87f690a640c39c58c64e16cc67e7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:07.319491 2086 leader_election.cc:290] T b3de6670357c4ae59eb77469b2fb434c P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:07.319682 1176 log.cc:1199] T bfec87f690a640c39c58c64e16cc67e7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/bfec87f690a640c39c58c64e16cc67e7
I20251029 02:52:07.320010 1176 ts_tablet_manager.cc:1950] T bfec87f690a640c39c58c64e16cc67e7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:07.320195 891 tablet_service.cc:1505] Processing CreateTablet for tablet 3b5a12416a2f49f190e008ef31519ec9 (DEFAULT_TABLE table=default.Table_bdf9abc4a0cd42f0ba2c936612afde6b [id=c6bdebdca8a84d0c81d94d59a2613fdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:07.324708 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet bfec87f690a640c39c58c64e16cc67e7 (table default.Table_11de4e6789214efca8b43cbcdae81a93 [id=6506de17cec5443b8a4628bd46a39b42]) successfully deleted
W20251029 02:52:07.325541 513 catalog_manager.cc:4952] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 6ca9bd37de9e4be388501b15af4bda31 because the tablet was not found. No further retry: Not found: Tablet not found: 6ca9bd37de9e4be388501b15af4bda31
I20251029 02:52:07.326030 1176 tablet_replica.cc:333] T c764062d3ba04d4ba8b2e367ad8ed433 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:07.326262 1176 raft_consensus.cc:2243] T c764062d3ba04d4ba8b2e367ad8ed433 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:07.326488 1176 raft_consensus.cc:2272] T c764062d3ba04d4ba8b2e367ad8ed433 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:07.327072 1176 ts_tablet_manager.cc:1916] T c764062d3ba04d4ba8b2e367ad8ed433 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:07.327328 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f60e7c24982d40319801869ad060864d" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:07.327517 1060 raft_consensus.cc:2468] T f60e7c24982d40319801869ad060864d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:07.328935 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3de6670357c4ae59eb77469b2fb434c" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:07.330713 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3de6670357c4ae59eb77469b2fb434c" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:07.333523 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 694) took 2979 ms. Trace:
W20251029 02:52:07.334133 847 leader_election.cc:343] T f60e7c24982d40319801869ad060864d P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: f60e7c24982d40319801869ad060864d
I20251029 02:52:07.334184 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1bc45078a6a942c38ff3350c15885eff" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
W20251029 02:52:07.335932 990 leader_election.cc:343] T 1bc45078a6a942c38ff3350c15885eff P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:07.334688 1175 rpcz_store.cc:276] 1029 02:52:04.353768 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:04.358545 (+ 4777us) service_pool.cc:225] Handling call
1029 02:52:07.332490 (+2973945us) ts_tablet_manager.cc:1151] Deleting tablet c4b1ef53dbaa47c39361cc4ef3be71d8
1029 02:52:07.332602 (+ 112us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:07.333282 (+ 680us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2973642}
W20251029 02:52:07.336970 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet c4b1ef53dbaa47c39361cc4ef3be71d8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c4b1ef53dbaa47c39361cc4ef3be71d8 already in progress: creating tablet
I20251029 02:52:07.337739 1176 ts_tablet_manager.cc:1929] T c764062d3ba04d4ba8b2e367ad8ed433 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:07.337903 1176 log.cc:1199] T c764062d3ba04d4ba8b2e367ad8ed433 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/c764062d3ba04d4ba8b2e367ad8ed433
I20251029 02:52:07.338205 1175 tablet_replica.cc:333] T c9bd3f2deb5d4af09b7918269a98509f P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:07.338491 1176 ts_tablet_manager.cc:1950] T c764062d3ba04d4ba8b2e367ad8ed433 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:07.339762 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet c764062d3ba04d4ba8b2e367ad8ed433 (table default.Table_4e9d94609b4a4d1f94615dcc9830d717 [id=761ebdd0095c414488aa221cbc99d3bc]) successfully deleted
I20251029 02:52:07.340190 1176 tablet_replica.cc:333] T 56d05e4a500043c995bb51a204cb295d P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:07.340404 1176 raft_consensus.cc:2243] T 56d05e4a500043c995bb51a204cb295d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:07.340123 1175 raft_consensus.cc:2243] T c9bd3f2deb5d4af09b7918269a98509f P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:07.340636 1176 raft_consensus.cc:2272] T 56d05e4a500043c995bb51a204cb295d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:07.341140 1176 ts_tablet_manager.cc:1916] T 56d05e4a500043c995bb51a204cb295d P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:07.341457 1175 raft_consensus.cc:2272] T c9bd3f2deb5d4af09b7918269a98509f P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:07.346148 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f187dbd8a55b45b292ea61db43caf47d" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:07.347033 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26e982a886024afca62f29c12f6ab5c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:07.347399 1175 ts_tablet_manager.cc:1916] T c9bd3f2deb5d4af09b7918269a98509f P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:07.349531 1176 ts_tablet_manager.cc:1929] T 56d05e4a500043c995bb51a204cb295d P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:07.349684 1176 log.cc:1199] T 56d05e4a500043c995bb51a204cb295d P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/56d05e4a500043c995bb51a204cb295d
I20251029 02:52:07.350355 1176 ts_tablet_manager.cc:1950] T 56d05e4a500043c995bb51a204cb295d P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:07.351528 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 56d05e4a500043c995bb51a204cb295d (table default.Table_0c1681c9cd0347f2ba2eab73b8cf11a9 [id=fdb365438df14f12a382aa046aef41a4]) successfully deleted
I20251029 02:52:07.352182 1176 tablet_replica.cc:333] T f60e7c24982d40319801869ad060864d P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:07.352456 1176 raft_consensus.cc:2243] T f60e7c24982d40319801869ad060864d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:07.352674 1176 raft_consensus.cc:2272] T f60e7c24982d40319801869ad060864d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:07.353255 1176 ts_tablet_manager.cc:1916] T f60e7c24982d40319801869ad060864d P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:07.354576 920 raft_consensus.cc:2468] T f187dbd8a55b45b292ea61db43caf47d P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 0.
I20251029 02:52:07.355080 990 leader_election.cc:304] T f187dbd8a55b45b292ea61db43caf47d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters: b495a49349324f7d81908f2da700f0d9
I20251029 02:52:07.355475 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a35569c6688647008e49fccb350c134d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:07.356256 2103 raft_consensus.cc:2804] T f187dbd8a55b45b292ea61db43caf47d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:07.356341 2103 raft_consensus.cc:493] T f187dbd8a55b45b292ea61db43caf47d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:07.356412 2103 raft_consensus.cc:3060] T f187dbd8a55b45b292ea61db43caf47d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:07.358603 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b5a12416a2f49f190e008ef31519ec9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:07.359315 2103 raft_consensus.cc:515] T f187dbd8a55b45b292ea61db43caf47d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:07.360549 2103 leader_election.cc:290] T f187dbd8a55b45b292ea61db43caf47d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Requested vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
W20251029 02:52:07.361174 989 leader_election.cc:343] T f187dbd8a55b45b292ea61db43caf47d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: f187dbd8a55b45b292ea61db43caf47d
I20251029 02:52:07.364183 1176 ts_tablet_manager.cc:1929] T f60e7c24982d40319801869ad060864d P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:07.367556 919 raft_consensus.cc:2468] T daa05a34c0684c07a35966290fd7d44c P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 1.
I20251029 02:52:07.367522 2107 tablet.cc:1722] T 4bf06476c87a4549a73f5994314bb75f P c756528642a543eeacd93cef3e335a7d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 02:52:07.368137 990 leader_election.cc:304] T daa05a34c0684c07a35966290fd7d44c P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters: b495a49349324f7d81908f2da700f0d9
I20251029 02:52:07.374243 2103 raft_consensus.cc:2804] T daa05a34c0684c07a35966290fd7d44c P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:07.374362 2103 raft_consensus.cc:697] T daa05a34c0684c07a35966290fd7d44c P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Becoming Leader. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Running, Role: LEADER
I20251029 02:52:07.374681 1176 log.cc:1199] T f60e7c24982d40319801869ad060864d P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/f60e7c24982d40319801869ad060864d
I20251029 02:52:07.375146 1176 ts_tablet_manager.cc:1950] T f60e7c24982d40319801869ad060864d P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:07.376030 1176 tablet_replica.cc:333] T daa05a34c0684c07a35966290fd7d44c P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:07.376235 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet f60e7c24982d40319801869ad060864d (table default.Table_4350e92f14de4ba8a04afbe12ea7422a [id=226a2d412a7b40e59b6f5340b5a9b739]) successfully deleted
I20251029 02:52:07.374619 2103 consensus_queue.cc:237] T daa05a34c0684c07a35966290fd7d44c P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
W20251029 02:52:07.381153 711 leader_election.cc:343] T 785733da86ff425a90ccfab2933fc89e P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:07.381286 711 leader_election.cc:304] T 785733da86ff425a90ccfab2933fc89e P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:07.381726 2118 raft_consensus.cc:2749] T 785733da86ff425a90ccfab2933fc89e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:07.383095 2136 tablet.cc:1722] T 4bf06476c87a4549a73f5994314bb75f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 02:52:07.383592 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "519754f5d8f7484480f973c53708c5f3" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:52:07.383767 1060 raft_consensus.cc:3060] T 519754f5d8f7484480f973c53708c5f3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:07.377020 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f187dbd8a55b45b292ea61db43caf47d" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
I20251029 02:52:07.386379 1060 raft_consensus.cc:2468] T 519754f5d8f7484480f973c53708c5f3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:07.390386 2107 tablet.cc:1722] T da2a21c3236a4bd99872b89b12a36508 P c756528642a543eeacd93cef3e335a7d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20251029 02:52:07.396108 1176 raft_consensus.cc:2243] T daa05a34c0684c07a35966290fd7d44c P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:07.396488 1176 pending_rounds.cc:70] T daa05a34c0684c07a35966290fd7d44c P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Trying to abort 1 pending ops.
I20251029 02:52:07.396531 1176 pending_rounds.cc:77] T daa05a34c0684c07a35966290fd7d44c P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215949116935708672 op_type: NO_OP noop_request { }
I20251029 02:52:07.396611 1176 raft_consensus.cc:2889] T daa05a34c0684c07a35966290fd7d44c P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 02:52:07.396677 1176 raft_consensus.cc:2272] T daa05a34c0684c07a35966290fd7d44c P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251029 02:52:07.397878 2142 consensus_queue.cc:1175] T daa05a34c0684c07a35966290fd7d44c P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" status { last_received { term: 1 index: 1 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 1 } }
I20251029 02:52:07.399830 1176 ts_tablet_manager.cc:1916] T daa05a34c0684c07a35966290fd7d44c P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:07.330924 768 raft_consensus.cc:2468] T b3de6670357c4ae59eb77469b2fb434c P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:07.406090 1176 ts_tablet_manager.cc:1929] T daa05a34c0684c07a35966290fd7d44c P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:07.406184 1176 log.cc:1199] T daa05a34c0684c07a35966290fd7d44c P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/daa05a34c0684c07a35966290fd7d44c
I20251029 02:52:07.406178 1175 ts_tablet_manager.cc:1929] T c9bd3f2deb5d4af09b7918269a98509f P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:07.406275 1175 log.cc:1199] T c9bd3f2deb5d4af09b7918269a98509f P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/c9bd3f2deb5d4af09b7918269a98509f
I20251029 02:52:07.406472 1176 ts_tablet_manager.cc:1950] T daa05a34c0684c07a35966290fd7d44c P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:07.407756 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet daa05a34c0684c07a35966290fd7d44c (table default.Table_33eacbf00740465185d70b46a3782893 [id=22e43c4483ae488b81fb94a7d221bdce]) successfully deleted
W20251029 02:52:07.408077 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 91e31f0b06924740bdf7525cae79d8ef because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 91e31f0b06924740bdf7525cae79d8ef already in progress: creating tablet
W20251029 02:52:07.408282 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet e4cad991d48341cbafb5f1d9b50f4e5b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e4cad991d48341cbafb5f1d9b50f4e5b already in progress: creating tablet
W20251029 02:52:07.408474 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet e5c88e03d9324a918020688c884eb4f6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e5c88e03d9324a918020688c884eb4f6 already in progress: creating tablet
W20251029 02:52:07.408720 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet e7f846b294cc4eb687a26c8b5931ac19 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e7f846b294cc4eb687a26c8b5931ac19 already in progress: creating tablet
I20251029 02:52:07.410816 916 raft_consensus.cc:3060] T f187dbd8a55b45b292ea61db43caf47d P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:07.414997 849 leader_election.cc:304] T f60e7c24982d40319801869ad060864d P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters: b495a49349324f7d81908f2da700f0d9
I20251029 02:52:07.420677 749 tablet_service.cc:1505] Processing CreateTablet for tablet 00f2e98739fa41eea23927494fe222f3 (DEFAULT_TABLE table=default.Table_ff3d06085614492a97e31873f16f4fdc [id=e49dd0af2f6e4c91a2aa0ec24881ae48]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:07.420768 847 leader_election.cc:304] T b3de6670357c4ae59eb77469b2fb434c P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:07.421080 2109 raft_consensus.cc:2804] T b3de6670357c4ae59eb77469b2fb434c P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:07.421164 2109 raft_consensus.cc:493] T b3de6670357c4ae59eb77469b2fb434c P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:07.421175 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00f2e98739fa41eea23927494fe222f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:07.421224 2109 raft_consensus.cc:3060] T b3de6670357c4ae59eb77469b2fb434c P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:07.422699 2111 raft_consensus.cc:2804] T f60e7c24982d40319801869ad060864d P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:07.422778 2111 raft_consensus.cc:493] T f60e7c24982d40319801869ad060864d P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:07.422837 2111 raft_consensus.cc:3060] T f60e7c24982d40319801869ad060864d P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
W20251029 02:52:07.422894 849 leader_election.cc:343] T b3de6670357c4ae59eb77469b2fb434c P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:07.423342 849 leader_election.cc:304] T 519754f5d8f7484480f973c53708c5f3 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters: b495a49349324f7d81908f2da700f0d9
I20251029 02:52:07.449020 1025 tablet_service.cc:1505] Processing CreateTablet for tablet 00f2e98739fa41eea23927494fe222f3 (DEFAULT_TABLE table=default.Table_ff3d06085614492a97e31873f16f4fdc [id=e49dd0af2f6e4c91a2aa0ec24881ae48]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:07.449479 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00f2e98739fa41eea23927494fe222f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:07.456929 899 tablet_service.cc:1505] Processing CreateTablet for tablet 00f2e98739fa41eea23927494fe222f3 (DEFAULT_TABLE table=default.Table_ff3d06085614492a97e31873f16f4fdc [id=e49dd0af2f6e4c91a2aa0ec24881ae48]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:07.457394 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00f2e98739fa41eea23927494fe222f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:07.462348 2109 raft_consensus.cc:515] T b3de6670357c4ae59eb77469b2fb434c P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:07.463301 2109 leader_election.cc:290] T b3de6670357c4ae59eb77469b2fb434c P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:07.463649 2112 raft_consensus.cc:2804] T 519754f5d8f7484480f973c53708c5f3 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:07.463784 2112 raft_consensus.cc:697] T 519754f5d8f7484480f973c53708c5f3 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:52:07.464172 916 raft_consensus.cc:2468] T f187dbd8a55b45b292ea61db43caf47d P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 1.
I20251029 02:52:07.469383 520 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet da2a21c3236a4bd99872b89b12a36508 (table default.Table_2a63af6894694f16b45f4bf48f73a68f [id=fb42af08ec1f48498ee5e91cce62333e]). Expected version 5 got 0
I20251029 02:52:07.469602 520 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet d5741dde463a43d6a776ab92dcfb04bf (table default.Table_966779a0d37a4dc4ab3f89d2bf7c6b06 [id=08e3b2ebcf37415d9a2886ad5bbda068]). Expected version 1 got 0
I20251029 02:52:07.469874 520 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet cedfb53078d94c81b5e294ce7438325e (table default.Table_e75db5cf8f5f49ea837cb6d63f4b7ebf [id=86195563fb4a4d7abd746f4ac7cd241c]). Expected version 1 got 0
I20251029 02:52:07.470960 525 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet da2a21c3236a4bd99872b89b12a36508 (table default.Table_2a63af6894694f16b45f4bf48f73a68f [id=fb42af08ec1f48498ee5e91cce62333e]). Expected version 5 got 0
I20251029 02:52:07.471112 525 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet c9e923724ca9402682f13b252dbc37aa (table default.Table_443f33587b8f428484b99e07355d69ca [id=03a25420ce6a46fc93a9fb753246bc7b]). Expected version 3 got 0
I20251029 02:52:07.471289 525 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet cedfb53078d94c81b5e294ce7438325e (table default.Table_e75db5cf8f5f49ea837cb6d63f4b7ebf [id=86195563fb4a4d7abd746f4ac7cd241c]). Expected version 1 got 0
I20251029 02:52:07.473583 990 leader_election.cc:304] T f187dbd8a55b45b292ea61db43caf47d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters: b495a49349324f7d81908f2da700f0d9
I20251029 02:52:07.473915 1973 raft_consensus.cc:2804] T f187dbd8a55b45b292ea61db43caf47d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:07.474064 1973 raft_consensus.cc:697] T f187dbd8a55b45b292ea61db43caf47d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Becoming Leader. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Running, Role: LEADER
I20251029 02:52:07.474316 1973 consensus_queue.cc:237] T f187dbd8a55b45b292ea61db43caf47d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:07.475421 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3de6670357c4ae59eb77469b2fb434c" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:07.475594 768 raft_consensus.cc:3060] T b3de6670357c4ae59eb77469b2fb434c P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:07.477097 1042 tablet_service.cc:1552] Processing DeleteTablet for tablet c9bd3f2deb5d4af09b7918269a98509f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:58 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:07.477125 1041 tablet_service.cc:1552] Processing DeleteTablet for tablet 4add552cf4d940d2a0a36d6b58379a34 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:53 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:07.477272 1032 tablet_service.cc:1552] Processing DeleteTablet for tablet 519754f5d8f7484480f973c53708c5f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:53 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:07.477324 1031 tablet_service.cc:1552] Processing DeleteTablet for tablet 1bc45078a6a942c38ff3350c15885eff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:56 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:07.477377 2112 consensus_queue.cc:237] T 519754f5d8f7484480f973c53708c5f3 P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:07.478861 1176 tablet_replica.cc:333] T c9bd3f2deb5d4af09b7918269a98509f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:07.479063 1176 raft_consensus.cc:2243] T c9bd3f2deb5d4af09b7918269a98509f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:07.479235 1176 raft_consensus.cc:2272] T c9bd3f2deb5d4af09b7918269a98509f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:07.481057 1176 ts_tablet_manager.cc:1916] T c9bd3f2deb5d4af09b7918269a98509f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:07.482066 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_52cba56ce9fb44bd84c35ef2f9219e56"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:07.483769 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_52cba56ce9fb44bd84c35ef2f9219e56 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:07.485569 1039 tablet_service.cc:1552] Processing DeleteTablet for tablet f187dbd8a55b45b292ea61db43caf47d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:56 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:07.485563 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_d2ac974941f9428eaa9605606209ed8d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:07.478497 768 raft_consensus.cc:2468] T b3de6670357c4ae59eb77469b2fb434c P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:07.486786 847 leader_election.cc:304] T b3de6670357c4ae59eb77469b2fb434c P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters:
W20251029 02:52:07.488098 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d2ac974941f9428eaa9605606209ed8d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:07.492676 1176 ts_tablet_manager.cc:1929] T c9bd3f2deb5d4af09b7918269a98509f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:07.493343 1176 log.cc:1199] T c9bd3f2deb5d4af09b7918269a98509f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/c9bd3f2deb5d4af09b7918269a98509f
I20251029 02:52:07.493845 1176 ts_tablet_manager.cc:1950] T c9bd3f2deb5d4af09b7918269a98509f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:07.495280 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet c9bd3f2deb5d4af09b7918269a98509f (table default.Table_47a1a59db0444fa49f7274509385ff0c [id=283c09ba60114ba9b165e2ff144a6c35]) successfully deleted
I20251029 02:52:07.496418 1176 tablet_replica.cc:333] T 4add552cf4d940d2a0a36d6b58379a34 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:07.497028 1176 raft_consensus.cc:2243] T 4add552cf4d940d2a0a36d6b58379a34 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:07.497725 1176 raft_consensus.cc:2272] T 4add552cf4d940d2a0a36d6b58379a34 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:07.498270 881 tablet_service.cc:1552] Processing DeleteTablet for tablet e1fa4980f2624321b617b84c68b90113 with delete_type TABLET_DATA_DELETED (replaced by tablet e6e3e6a82b014d2b8de4e109b379461e) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:07.498577 881 tablet_service.cc:1552] Processing DeleteTablet for tablet 4add552cf4d940d2a0a36d6b58379a34 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:53 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:07.498605 1174 tablet_replica.cc:333] T e1fa4980f2624321b617b84c68b90113 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:07.498852 1174 raft_consensus.cc:2243] T e1fa4980f2624321b617b84c68b90113 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:07.498912 881 tablet_service.cc:1552] Processing DeleteTablet for tablet daa05a34c0684c07a35966290fd7d44c with delete_type TABLET_DATA_DELETED (replaced by tablet fcb3db8399aa4cf188b07dda220f7f5b) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:07.499123 1174 pending_rounds.cc:70] T e1fa4980f2624321b617b84c68b90113 P c756528642a543eeacd93cef3e335a7d: Trying to abort 1 pending ops.
I20251029 02:52:07.499167 1174 pending_rounds.cc:77] T e1fa4980f2624321b617b84c68b90113 P c756528642a543eeacd93cef3e335a7d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215949114283253760 op_type: NO_OP noop_request { }
I20251029 02:52:07.499257 1174 raft_consensus.cc:2889] T e1fa4980f2624321b617b84c68b90113 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 02:52:07.499356 1174 raft_consensus.cc:2272] T e1fa4980f2624321b617b84c68b90113 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:07.499305 881 tablet_service.cc:1552] Processing DeleteTablet for tablet f60e7c24982d40319801869ad060864d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:53 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:07.499646 899 tablet_service.cc:1552] Processing DeleteTablet for tablet 0cc1957a845f4e9cb703a553b27d1347 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:53 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:07.499645 879 tablet_service.cc:1552] Processing DeleteTablet for tablet 6ca9bd37de9e4be388501b15af4bda31 with delete_type TABLET_DATA_DELETED (replaced by tablet 20a762344c5f46f88481c4ac7ad6e43b) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:07.500142 895 tablet_service.cc:1552] Processing DeleteTablet for tablet 4aa78f4ac38e4919864c924e50f1aaf4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:57 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:07.500626 1174 ts_tablet_manager.cc:1916] T e1fa4980f2624321b617b84c68b90113 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:07.500788 1176 ts_tablet_manager.cc:1916] T 4add552cf4d940d2a0a36d6b58379a34 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:07.501102 900 tablet_service.cc:1552] Processing DeleteTablet for tablet 519754f5d8f7484480f973c53708c5f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:53 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:07.499305 2086 raft_consensus.cc:2804] T b3de6670357c4ae59eb77469b2fb434c P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:07.502069 2086 raft_consensus.cc:697] T b3de6670357c4ae59eb77469b2fb434c P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:52:07.502310 2086 consensus_queue.cc:237] T b3de6670357c4ae59eb77469b2fb434c P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:07.508864 898 tablet_service.cc:1552] Processing DeleteTablet for tablet b3de6670357c4ae59eb77469b2fb434c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:01 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:07.510776 892 tablet_service.cc:1552] Processing DeleteTablet for tablet 6bd3fac2502e479aa9f4ceaabbb39504 with delete_type TABLET_DATA_DELETED (replaced by tablet b125cc9aaf4248b8bfa48fb152b832a6) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:07.512364 1174 ts_tablet_manager.cc:1929] T e1fa4980f2624321b617b84c68b90113 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:07.512737 1174 log.cc:1199] T e1fa4980f2624321b617b84c68b90113 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/e1fa4980f2624321b617b84c68b90113
I20251029 02:52:07.513002 891 tablet_service.cc:1552] Processing DeleteTablet for tablet f187dbd8a55b45b292ea61db43caf47d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:56 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:07.513881 1174 ts_tablet_manager.cc:1950] T e1fa4980f2624321b617b84c68b90113 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:07.514119 1176 ts_tablet_manager.cc:1929] T 4add552cf4d940d2a0a36d6b58379a34 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:07.514308 1176 log.cc:1199] T 4add552cf4d940d2a0a36d6b58379a34 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/4add552cf4d940d2a0a36d6b58379a34
I20251029 02:52:07.514673 1176 ts_tablet_manager.cc:1950] T 4add552cf4d940d2a0a36d6b58379a34 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:07.515839 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 4add552cf4d940d2a0a36d6b58379a34 (table default.Table_c73a6bc56eb44e08aa7d2ec65ad88949 [id=903fed784c0243bbb60a39066f2dbc71]) successfully deleted
I20251029 02:52:07.516058 885 tablet_service.cc:1552] Processing DeleteTablet for tablet 71068739b4384badbf305313d902f0f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:56 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:07.516335 1176 tablet_replica.cc:333] T 519754f5d8f7484480f973c53708c5f3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:07.516561 1176 raft_consensus.cc:2243] T 519754f5d8f7484480f973c53708c5f3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:07.516884 1176 raft_consensus.cc:2272] T 519754f5d8f7484480f973c53708c5f3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:07.517704 1176 ts_tablet_manager.cc:1916] T 519754f5d8f7484480f973c53708c5f3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:07.522728 1172 tablet_bootstrap.cc:492] T 13ee62fc2339410b9db70dfd3d3c6fa6 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:07.523001 1172 ts_tablet_manager.cc:1403] T 13ee62fc2339410b9db70dfd3d3c6fa6 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.245s user 0.019s sys 0.039s
I20251029 02:52:07.524468 1176 ts_tablet_manager.cc:1929] T 519754f5d8f7484480f973c53708c5f3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:07.524583 1176 log.cc:1199] T 519754f5d8f7484480f973c53708c5f3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/519754f5d8f7484480f973c53708c5f3
I20251029 02:52:07.524610 1172 raft_consensus.cc:359] T 13ee62fc2339410b9db70dfd3d3c6fa6 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:07.524813 1172 raft_consensus.cc:385] T 13ee62fc2339410b9db70dfd3d3c6fa6 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:07.524860 1176 ts_tablet_manager.cc:1950] T 519754f5d8f7484480f973c53708c5f3 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:07.524860 1172 raft_consensus.cc:740] T 13ee62fc2339410b9db70dfd3d3c6fa6 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:07.525876 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 519754f5d8f7484480f973c53708c5f3 (table default.Table_96a1c9ac99714785a6bb853741027180 [id=d16cec879a3a45189e45088c4c0913ed]) successfully deleted
I20251029 02:52:07.526376 1176 tablet_replica.cc:333] T 1bc45078a6a942c38ff3350c15885eff P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:07.525830 1172 consensus_queue.cc:260] T 13ee62fc2339410b9db70dfd3d3c6fa6 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:07.526890 1176 raft_consensus.cc:2243] T 1bc45078a6a942c38ff3350c15885eff P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:07.527171 1176 pending_rounds.cc:70] T 1bc45078a6a942c38ff3350c15885eff P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Trying to abort 1 pending ops.
I20251029 02:52:07.527269 1176 pending_rounds.cc:77] T 1bc45078a6a942c38ff3350c15885eff P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215949116503175168 op_type: NO_OP noop_request { }
I20251029 02:52:07.527408 1176 raft_consensus.cc:2889] T 1bc45078a6a942c38ff3350c15885eff P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 02:52:07.527573 1176 raft_consensus.cc:2272] T 1bc45078a6a942c38ff3350c15885eff P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:07.528384 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet e1fa4980f2624321b617b84c68b90113 (table default.Table_8a5ae35851664e0f85324ec1ca672031 [id=a0e161a5d9d94a3b936669e6dfe6fb4f]) successfully deleted
I20251029 02:52:07.528776 1172 ts_tablet_manager.cc:1434] T 13ee62fc2339410b9db70dfd3d3c6fa6 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20251029 02:52:07.529158 1174 tablet_replica.cc:333] T 4add552cf4d940d2a0a36d6b58379a34 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:07.529367 1174 raft_consensus.cc:2243] T 4add552cf4d940d2a0a36d6b58379a34 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:07.529510 1174 raft_consensus.cc:2272] T 4add552cf4d940d2a0a36d6b58379a34 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:07.530941 1174 ts_tablet_manager.cc:1916] T 4add552cf4d940d2a0a36d6b58379a34 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:07.531322 1166 tablet_bootstrap.cc:492] T b3de6670357c4ae59eb77469b2fb434c P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:07.531442 1172 tablet_bootstrap.cc:492] T b6d4f7bf702f4834bc5f4ef2b50b92a7 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:07.531688 1166 ts_tablet_manager.cc:1403] T b3de6670357c4ae59eb77469b2fb434c P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.366s user 0.023s sys 0.032s
I20251029 02:52:07.533565 1166 raft_consensus.cc:359] T b3de6670357c4ae59eb77469b2fb434c P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:07.534068 1166 raft_consensus.cc:385] T b3de6670357c4ae59eb77469b2fb434c P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:07.534281 1166 raft_consensus.cc:740] T b3de6670357c4ae59eb77469b2fb434c P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:07.537213 1172 tablet_bootstrap.cc:654] T b6d4f7bf702f4834bc5f4ef2b50b92a7 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:07.539719 1167 tablet_bootstrap.cc:492] T 56d05e4a500043c995bb51a204cb295d P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:07.540055 1167 ts_tablet_manager.cc:1403] T 56d05e4a500043c995bb51a204cb295d P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.656s user 0.007s sys 0.003s
I20251029 02:52:07.541421 2107 tablet.cc:1722] T c9e923724ca9402682f13b252dbc37aa P c756528642a543eeacd93cef3e335a7d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251029 02:52:07.542562 2111 raft_consensus.cc:515] T f60e7c24982d40319801869ad060864d P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:07.544595 1174 ts_tablet_manager.cc:1929] T 4add552cf4d940d2a0a36d6b58379a34 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:07.544709 1174 log.cc:1199] T 4add552cf4d940d2a0a36d6b58379a34 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/4add552cf4d940d2a0a36d6b58379a34
W20251029 02:52:07.544837 847 leader_election.cc:343] T f60e7c24982d40319801869ad060864d P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: f60e7c24982d40319801869ad060864d
I20251029 02:52:07.545037 1174 ts_tablet_manager.cc:1950] T 4add552cf4d940d2a0a36d6b58379a34 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:07.545572 2111 leader_election.cc:290] T f60e7c24982d40319801869ad060864d P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:07.546862 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 4add552cf4d940d2a0a36d6b58379a34 (table default.Table_c73a6bc56eb44e08aa7d2ec65ad88949 [id=903fed784c0243bbb60a39066f2dbc71]) successfully deleted
I20251029 02:52:07.546928 2112 raft_consensus.cc:493] T 86fc22bb84794149b598b97df41eb881 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:07.547045 2112 raft_consensus.cc:515] T 86fc22bb84794149b598b97df41eb881 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:07.547226 2118 raft_consensus.cc:493] T 44b518c3a0964a6fa863b2784d80c8a4 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:07.547348 2118 raft_consensus.cc:515] T 44b518c3a0964a6fa863b2784d80c8a4 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:07.547937 2118 leader_election.cc:290] T 44b518c3a0964a6fa863b2784d80c8a4 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:07.548354 2086 consensus_queue.cc:1048] T d5741dde463a43d6a776ab92dcfb04bf P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:07.550802 1174 tablet_replica.cc:333] T daa05a34c0684c07a35966290fd7d44c P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:07.550400 1167 raft_consensus.cc:359] T 56d05e4a500043c995bb51a204cb295d P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:07.550954 1167 raft_consensus.cc:385] T 56d05e4a500043c995bb51a204cb295d P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:07.551198 1167 raft_consensus.cc:740] T 56d05e4a500043c995bb51a204cb295d P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:07.551733 1167 consensus_queue.cc:260] T 56d05e4a500043c995bb51a204cb295d P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:07.551064 1174 raft_consensus.cc:2243] T daa05a34c0684c07a35966290fd7d44c P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:07.554791 1174 raft_consensus.cc:2272] T daa05a34c0684c07a35966290fd7d44c P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:07.556035 1175 ts_tablet_manager.cc:1950] T c9bd3f2deb5d4af09b7918269a98509f P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:07.546285 1176 ts_tablet_manager.cc:1916] T 1bc45078a6a942c38ff3350c15885eff P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:07.548202 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b3de6670357c4ae59eb77469b2fb434c" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:52:07.548583 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "44b518c3a0964a6fa863b2784d80c8a4" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:52:07.561870 712 leader_election.cc:343] T 44b518c3a0964a6fa863b2784d80c8a4 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:07.534861 1166 consensus_queue.cc:260] T b3de6670357c4ae59eb77469b2fb434c P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:07.563460 1060 raft_consensus.cc:3060] T b3de6670357c4ae59eb77469b2fb434c P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
W20251029 02:52:07.561456 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.3:45003, user_credentials={real_user=slave}} blocked reactor thread for 123135us
I20251029 02:52:07.567036 1060 raft_consensus.cc:2468] T b3de6670357c4ae59eb77469b2fb434c P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:07.572108 1167 ts_tablet_manager.cc:1434] T 56d05e4a500043c995bb51a204cb295d P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.032s user 0.000s sys 0.003s
I20251029 02:52:07.572780 1167 tablet_bootstrap.cc:492] T dd6ad641fbda472fab07aa35c5e4523a P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:07.574203 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 695) took 3212 ms. Trace:
I20251029 02:52:07.576584 1167 tablet_bootstrap.cc:654] T dd6ad641fbda472fab07aa35c5e4523a P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:07.576566 1166 ts_tablet_manager.cc:1434] T b3de6670357c4ae59eb77469b2fb434c P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.044s user 0.004s sys 0.000s
I20251029 02:52:07.577818 2112 leader_election.cc:290] T 86fc22bb84794149b598b97df41eb881 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:07.575670 1175 rpcz_store.cc:276] 1029 02:52:04.361864 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:04.367556 (+ 5692us) service_pool.cc:225] Handling call
1029 02:52:07.337674 (+2970118us) ts_tablet_manager.cc:1151] Deleting tablet c9bd3f2deb5d4af09b7918269a98509f
1029 02:52:07.337799 (+ 125us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:07.403110 (+ 65311us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:07.406095 (+ 2985us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:07.573842 (+167747us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2969856}
I20251029 02:52:07.580777 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet c9bd3f2deb5d4af09b7918269a98509f (table default.Table_47a1a59db0444fa49f7274509385ff0c [id=283c09ba60114ba9b165e2ff144a6c35]) successfully deleted
W20251029 02:52:07.582494 847 leader_election.cc:343] T 86fc22bb84794149b598b97df41eb881 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 86fc22bb84794149b598b97df41eb881
I20251029 02:52:07.583880 1174 ts_tablet_manager.cc:1916] T daa05a34c0684c07a35966290fd7d44c P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:07.582568 1166 tablet_bootstrap.cc:492] T 9e6eb099648549dbbf9969e59865f51f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:07.584501 1175 tablet_replica.cc:333] T ba699862e6f646c6beaa39f080d0faba P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:07.587821 914 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f80b840ed954230bfbfe7e40127cd2f" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:07.586987 1175 raft_consensus.cc:2243] T ba699862e6f646c6beaa39f080d0faba P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:07.589113 1175 raft_consensus.cc:2272] T ba699862e6f646c6beaa39f080d0faba P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:07.596936 914 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "95f342d1cfb84c4eb31bd3d15ea3bdc2" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:07.596356 1175 ts_tablet_manager.cc:1916] T ba699862e6f646c6beaa39f080d0faba P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:07.597347 1060 raft_consensus.cc:3060] T d5741dde463a43d6a776ab92dcfb04bf P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:07.598145 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "86fc22bb84794149b598b97df41eb881" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:07.600598 914 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c53fc1fff1a4bb988beb46ff91fd3be" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:07.602960 914 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d84dc8dc465496f9adec798b2e03e13" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:07.602653 849 leader_election.cc:343] T f60e7c24982d40319801869ad060864d P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Not found: Tablet not found: f60e7c24982d40319801869ad060864d
I20251029 02:52:07.605068 914 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5db519edc75d4bd0985b32322825ffae" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:07.604300 849 leader_election.cc:304] T f60e7c24982d40319801869ad060864d P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9
I20251029 02:52:07.606890 914 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "777992acb1bd4765886b802940db74bb" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:07.607422 2112 raft_consensus.cc:2749] T f60e7c24982d40319801869ad060864d P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251029 02:52:07.609258 522 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet faac614af1814a2f91d66c9cf8ddc919
W20251029 02:52:07.609356 517 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4c17d21145e54cc78134cab37059ce1e consensus state has no leader
I20251029 02:52:07.610368 885 tablet_service.cc:1552] Processing DeleteTablet for tablet faac614af1814a2f91d66c9cf8ddc919 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:07 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:07.610839 518 master_service.cc:954] ReplaceTablet: received request to replace tablet 4bf06476c87a4549a73f5994314bb75f from {username='slave'} at 127.0.0.1:35314
W20251029 02:52:07.611749 521 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 74eafdebb4eb4ba7a0c05d99e2e93fcb consensus state has no leader
I20251029 02:52:07.612128 521 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_fdcb71c680504418a7f89d81feb372b0" } new_table_name: "default.Table_6f38863737b34dfa90488cad665a8628" modify_external_catalogs: true
I20251029 02:52:07.613231 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_8c898cd0e2f343a6ae5debb290c20b07"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:07.614444 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_aec07800b35948bba7db9168286fcf14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:07.614995 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aec07800b35948bba7db9168286fcf14 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:07.615208 914 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dabb8d760aae4aa09eb83cdb586c9788" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:07.615615 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8c898cd0e2f343a6ae5debb290c20b07 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:07.618408 1176 ts_tablet_manager.cc:1929] T 1bc45078a6a942c38ff3350c15885eff P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:07.619378 519 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_4916f7d510ec49be9480cc88f4c7d3af" } new_table_name: "default.Table_b6c35405446549b4b242600d294c4b4f" modify_external_catalogs: true
I20251029 02:52:07.622056 750 tablet_service.cc:1552] Processing DeleteTablet for tablet faac614af1814a2f91d66c9cf8ddc919 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:07 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:07.628125 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_2577a8048e3345ce82eafa3657784d8b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:07.629185 523 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_11ffae1fbbae415189fd5c7bdb829875" } new_table_name: "default.Table_f57e6863a262448b8c3d70405377e7a5" modify_external_catalogs: true
W20251029 02:52:07.629612 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2577a8048e3345ce82eafa3657784d8b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:07.628330 526 master_service.cc:954] ReplaceTablet: received request to replace tablet 050186302c5d4c7da393f28f4b9e4262 from {username='slave'} at 127.0.0.1:35314
I20251029 02:52:07.630641 518 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 4bf06476c87a4549a73f5994314bb75f
I20251029 02:52:07.631562 518 catalog_manager.cc:6719] ReplaceTablet: tablet 4bf06476c87a4549a73f5994314bb75f deleted and replaced by tablet 08ca0c889f2949408657fa60ce087608
I20251029 02:52:07.631606 1176 log.cc:1199] T 1bc45078a6a942c38ff3350c15885eff P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/1bc45078a6a942c38ff3350c15885eff
I20251029 02:52:07.631805 1031 tablet_service.cc:1552] Processing DeleteTablet for tablet 4bf06476c87a4549a73f5994314bb75f with delete_type TABLET_DATA_DELETED (replaced by tablet 08ca0c889f2949408657fa60ce087608) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:07.618935 1166 tablet_bootstrap.cc:654] T 9e6eb099648549dbbf9969e59865f51f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:07.632200 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_d61d4be2068f4a1985e207d3413141f9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:07.632645 885 tablet_service.cc:1552] Processing DeleteTablet for tablet 4bf06476c87a4549a73f5994314bb75f with delete_type TABLET_DATA_DELETED (replaced by tablet 08ca0c889f2949408657fa60ce087608) from {username='slave'} at 127.0.0.1:35702
W20251029 02:52:07.632699 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d61d4be2068f4a1985e207d3413141f9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:07.634580 517 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet cedfb53078d94c81b5e294ce7438325e (table default.Table_e75db5cf8f5f49ea837cb6d63f4b7ebf [id=86195563fb4a4d7abd746f4ac7cd241c]). Expected version 1 got 0
I20251029 02:52:07.635290 517 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet c9e923724ca9402682f13b252dbc37aa (table default.Table_443f33587b8f428484b99e07355d69ca [id=03a25420ce6a46fc93a9fb753246bc7b]). Expected version 3 got 0
I20251029 02:52:07.621850 1039 tablet_service.cc:1552] Processing DeleteTablet for tablet faac614af1814a2f91d66c9cf8ddc919 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:07 UTC) from {username='slave'} at 127.0.0.1:37626
W20251029 02:52:07.636893 521 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 15137fc1c70347b381ffdfe5653a5f22 consensus state has no leader
I20251029 02:52:07.636896 1175 ts_tablet_manager.cc:1929] T ba699862e6f646c6beaa39f080d0faba P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:07.637002 1175 log.cc:1199] T ba699862e6f646c6beaa39f080d0faba P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/ba699862e6f646c6beaa39f080d0faba
I20251029 02:52:07.637265 1175 ts_tablet_manager.cc:1950] T ba699862e6f646c6beaa39f080d0faba P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:07.637734 525 catalog_manager.cc:6194] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: default.Table_b708124baf534c96b3f4b8a7a4bb11cc [id=046010a03d774e91b0ad10df97848c56] alter complete (version 2)
I20251029 02:52:07.637856 1176 ts_tablet_manager.cc:1950] T 1bc45078a6a942c38ff3350c15885eff P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:07.637778 521 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_cb4697efc8e049be929f4e7a004879d1" } modify_external_catalogs: true
I20251029 02:52:07.638134 521 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_cb4697efc8e049be929f4e7a004879d1" } modify_external_catalogs: true
I20251029 02:52:07.638451 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 1bc45078a6a942c38ff3350c15885eff (table default.Table_3126aa57075d44dc8b73f08fe21aabdf [id=e8fa170a8a044f2fa1236949a91454de]) successfully deleted
I20251029 02:52:07.639183 750 tablet_service.cc:1552] Processing DeleteTablet for tablet 4bf06476c87a4549a73f5994314bb75f with delete_type TABLET_DATA_DELETED (replaced by tablet 08ca0c889f2949408657fa60ce087608) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:07.639587 1176 tablet_replica.cc:333] T f187dbd8a55b45b292ea61db43caf47d P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:07.639823 1176 raft_consensus.cc:2243] T f187dbd8a55b45b292ea61db43caf47d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:07.639999 914 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a1866afc7564217ad777e7ce53786c5" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:07.640102 1176 pending_rounds.cc:70] T f187dbd8a55b45b292ea61db43caf47d P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Trying to abort 1 pending ops.
I20251029 02:52:07.640157 1176 pending_rounds.cc:77] T f187dbd8a55b45b292ea61db43caf47d P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215949117337059328 op_type: NO_OP noop_request { }
I20251029 02:52:07.640249 1176 raft_consensus.cc:2889] T f187dbd8a55b45b292ea61db43caf47d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 02:52:07.640329 1176 raft_consensus.cc:2272] T f187dbd8a55b45b292ea61db43caf47d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:07.641574 1176 ts_tablet_manager.cc:1916] T f187dbd8a55b45b292ea61db43caf47d P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 02:52:07.642406 711 leader_election.cc:343] T 0f80b840ed954230bfbfe7e40127cd2f P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:07.642553 711 leader_election.cc:304] T 0f80b840ed954230bfbfe7e40127cd2f P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:07.642172 914 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "44b518c3a0964a6fa863b2784d80c8a4" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:07.643008 711 leader_election.cc:343] T 95f342d1cfb84c4eb31bd3d15ea3bdc2 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:07.643092 711 leader_election.cc:304] T 95f342d1cfb84c4eb31bd3d15ea3bdc2 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
W20251029 02:52:07.641312 849 leader_election.cc:343] T 86fc22bb84794149b598b97df41eb881 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:07.643213 849 leader_election.cc:304] T 86fc22bb84794149b598b97df41eb881 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9
W20251029 02:52:07.643399 711 leader_election.cc:343] T 1c53fc1fff1a4bb988beb46ff91fd3be P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:07.643473 711 leader_election.cc:304] T 1c53fc1fff1a4bb988beb46ff91fd3be P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
W20251029 02:52:07.643836 711 leader_election.cc:343] T 0d84dc8dc465496f9adec798b2e03e13 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:07.643923 711 leader_election.cc:304] T 0d84dc8dc465496f9adec798b2e03e13 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
W20251029 02:52:07.644268 711 leader_election.cc:343] T 5db519edc75d4bd0985b32322825ffae P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:07.644380 711 leader_election.cc:304] T 5db519edc75d4bd0985b32322825ffae P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:07.644906 2018 raft_consensus.cc:2749] T 0f80b840ed954230bfbfe7e40127cd2f P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:07.645074 2018 raft_consensus.cc:2749] T 95f342d1cfb84c4eb31bd3d15ea3bdc2 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:07.645205 2018 raft_consensus.cc:2749] T 1c53fc1fff1a4bb988beb46ff91fd3be P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:07.645320 2018 raft_consensus.cc:2749] T 0d84dc8dc465496f9adec798b2e03e13 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:07.645431 2018 raft_consensus.cc:2749] T 5db519edc75d4bd0985b32322825ffae P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 02:52:07.645900 711 leader_election.cc:343] T 777992acb1bd4765886b802940db74bb P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:07.646001 711 leader_election.cc:304] T 777992acb1bd4765886b802940db74bb P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:07.646209 2113 raft_consensus.cc:2749] T 777992acb1bd4765886b802940db74bb P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:07.644644 1903 raft_consensus.cc:2749] T 86fc22bb84794149b598b97df41eb881 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 02:52:07.646556 711 leader_election.cc:343] T dabb8d760aae4aa09eb83cdb586c9788 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:07.646477 1903 consensus_queue.cc:1048] T d5741dde463a43d6a776ab92dcfb04bf P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:07.646632 711 leader_election.cc:304] T dabb8d760aae4aa09eb83cdb586c9788 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:07.646811 2113 raft_consensus.cc:2749] T dabb8d760aae4aa09eb83cdb586c9788 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 02:52:07.647125 711 leader_election.cc:343] T 7a1866afc7564217ad777e7ce53786c5 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:07.647207 711 leader_election.cc:304] T 7a1866afc7564217ad777e7ce53786c5 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:07.647744 2113 raft_consensus.cc:2749] T 7a1866afc7564217ad777e7ce53786c5 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 02:52:07.648056 711 leader_election.cc:343] T 44b518c3a0964a6fa863b2784d80c8a4 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:07.648144 711 leader_election.cc:304] T 44b518c3a0964a6fa863b2784d80c8a4 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:07.648492 2113 raft_consensus.cc:2749] T 44b518c3a0964a6fa863b2784d80c8a4 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:07.665202 1174 ts_tablet_manager.cc:1929] T daa05a34c0684c07a35966290fd7d44c P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:07.669473 1174 log.cc:1199] T daa05a34c0684c07a35966290fd7d44c P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/daa05a34c0684c07a35966290fd7d44c
I20251029 02:52:07.671208 1166 tablet_bootstrap.cc:492] T 9e6eb099648549dbbf9969e59865f51f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:07.671367 1166 ts_tablet_manager.cc:1403] T 9e6eb099648549dbbf9969e59865f51f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.090s user 0.003s sys 0.001s
I20251029 02:52:07.672255 1166 raft_consensus.cc:359] T 9e6eb099648549dbbf9969e59865f51f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:07.672459 1166 raft_consensus.cc:385] T 9e6eb099648549dbbf9969e59865f51f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:07.672529 1166 raft_consensus.cc:740] T 9e6eb099648549dbbf9969e59865f51f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:07.672765 1166 consensus_queue.cc:260] T 9e6eb099648549dbbf9969e59865f51f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:07.673221 1174 ts_tablet_manager.cc:1950] T daa05a34c0684c07a35966290fd7d44c P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:07.674369 1166 ts_tablet_manager.cc:1434] T 9e6eb099648549dbbf9969e59865f51f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251029 02:52:07.674433 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet daa05a34c0684c07a35966290fd7d44c (table default.Table_33eacbf00740465185d70b46a3782893 [id=22e43c4483ae488b81fb94a7d221bdce]) successfully deleted
I20251029 02:52:07.675469 1166 tablet_bootstrap.cc:492] T 02e4588b03144ea2850e13a1f8f29f06 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:07.675725 1174 tablet_replica.cc:333] T f60e7c24982d40319801869ad060864d P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:07.677100 1166 tablet_bootstrap.cc:654] T 02e4588b03144ea2850e13a1f8f29f06 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:07.677357 1174 raft_consensus.cc:2243] T f60e7c24982d40319801869ad060864d P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:07.678771 1174 raft_consensus.cc:2272] T f60e7c24982d40319801869ad060864d P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:07.679390 1174 ts_tablet_manager.cc:1916] T f60e7c24982d40319801869ad060864d P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:07.682302 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 702) took 2898 ms. Trace:
I20251029 02:52:07.682407 1175 rpcz_store.cc:276] 1029 02:52:04.783482 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:04.783555 (+ 73us) service_pool.cc:225] Handling call
1029 02:52:07.582239 (+2798684us) ts_tablet_manager.cc:1151] Deleting tablet ba699862e6f646c6beaa39f080d0faba
1029 02:52:07.582253 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:07.633746 (+ 51493us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:07.636807 (+ 3061us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:07.682273 (+ 45466us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2798169}
I20251029 02:52:07.682996 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 703) took 2844 ms. Trace:
I20251029 02:52:07.683084 1175 rpcz_store.cc:276] 1029 02:52:04.838634 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:04.838714 (+ 80us) service_pool.cc:225] Handling call
1029 02:52:07.682860 (+2844146us) ts_tablet_manager.cc:1151] Deleting tablet 738046ab57334f2dab9f4aab979c5b42
1029 02:52:07.682872 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:07.682980 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2843873}
I20251029 02:52:07.683699 520 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_683d67689da74097b69f31b3d9461fe2" } modify_external_catalogs: true
I20251029 02:52:07.683907 520 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_683d67689da74097b69f31b3d9461fe2" } modify_external_catalogs: true
I20251029 02:52:07.685478 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet ba699862e6f646c6beaa39f080d0faba (table default.Table_84e9fe6549cc42ed94c0f4c0b489f726 [id=feb1f56ae44249c6adce65637017bb43]) successfully deleted
W20251029 02:52:07.685832 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 738046ab57334f2dab9f4aab979c5b42 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 738046ab57334f2dab9f4aab979c5b42 already in progress: creating tablet
I20251029 02:52:07.686719 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 704) took 2821 ms. Trace:
I20251029 02:52:07.686797 1175 rpcz_store.cc:276] 1029 02:52:04.865706 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:04.865763 (+ 57us) service_pool.cc:225] Handling call
1029 02:52:07.686607 (+2820844us) ts_tablet_manager.cc:1151] Deleting tablet c081d2eacfac4ca8a0048e2a1c4f8f83
1029 02:52:07.686618 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:07.686702 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2820598}
W20251029 02:52:07.687297 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet c081d2eacfac4ca8a0048e2a1c4f8f83 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c081d2eacfac4ca8a0048e2a1c4f8f83 already in progress: creating tablet
W20251029 02:52:07.687489 519 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 415ff17c323e4ed88da125d823ad4af1 consensus state has no leader
I20251029 02:52:07.687819 519 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_049332132e3142ca86d49635656135f5" } modify_external_catalogs: true
I20251029 02:52:07.687973 519 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_049332132e3142ca86d49635656135f5" } modify_external_catalogs: true
I20251029 02:52:07.689102 1176 ts_tablet_manager.cc:1929] T f187dbd8a55b45b292ea61db43caf47d P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:07.689232 1176 log.cc:1199] T f187dbd8a55b45b292ea61db43caf47d P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/f187dbd8a55b45b292ea61db43caf47d
I20251029 02:52:07.689553 1176 ts_tablet_manager.cc:1950] T f187dbd8a55b45b292ea61db43caf47d P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:07.690069 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet f187dbd8a55b45b292ea61db43caf47d (table default.Table_d6a9ff6b4cea442eb2394340301eac96 [id=3f2d90da0f88434592a21c0687450ded]) successfully deleted
I20251029 02:52:07.690135 1175 tablet_replica.cc:333] T 4add552cf4d940d2a0a36d6b58379a34 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:07.690300 1175 raft_consensus.cc:2243] T 4add552cf4d940d2a0a36d6b58379a34 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:07.690443 1175 raft_consensus.cc:2272] T 4add552cf4d940d2a0a36d6b58379a34 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:07.691004 1175 ts_tablet_manager.cc:1916] T 4add552cf4d940d2a0a36d6b58379a34 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:07.695303 1166 tablet_bootstrap.cc:492] T 02e4588b03144ea2850e13a1f8f29f06 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:07.695454 1166 ts_tablet_manager.cc:1403] T 02e4588b03144ea2850e13a1f8f29f06 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.001s
I20251029 02:52:07.696290 1166 raft_consensus.cc:359] T 02e4588b03144ea2850e13a1f8f29f06 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:07.696535 1166 raft_consensus.cc:385] T 02e4588b03144ea2850e13a1f8f29f06 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:07.696635 1166 raft_consensus.cc:740] T 02e4588b03144ea2850e13a1f8f29f06 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:07.696996 1176 tablet_replica.cc:333] T 4bf06476c87a4549a73f5994314bb75f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:07.696902 1166 consensus_queue.cc:260] T 02e4588b03144ea2850e13a1f8f29f06 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:07.698586 1176 raft_consensus.cc:2243] T 4bf06476c87a4549a73f5994314bb75f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:07.698685 1166 ts_tablet_manager.cc:1434] T 02e4588b03144ea2850e13a1f8f29f06 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251029 02:52:07.698966 1166 tablet_bootstrap.cc:492] T 1c53fc1fff1a4bb988beb46ff91fd3be P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
W20251029 02:52:07.699067 523 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 13af21b2952941548f1bdb2741d9e6d2 consensus state has no leader
I20251029 02:52:07.699393 523 master_service.cc:954] ReplaceTablet: received request to replace tablet 9778f924d11a48b194ccf26124bd5509 from {username='slave'} at 127.0.0.1:35314
I20251029 02:52:07.699640 1176 raft_consensus.cc:2272] T 4bf06476c87a4549a73f5994314bb75f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:07.700605 1166 tablet_bootstrap.cc:654] T 1c53fc1fff1a4bb988beb46ff91fd3be P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:07.700932 524 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_25a8f4870ca94816934d0214177fc278" } modify_external_catalogs: true
I20251029 02:52:07.701095 524 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_25a8f4870ca94816934d0214177fc278" } modify_external_catalogs: true
I20251029 02:52:07.702234 526 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 050186302c5d4c7da393f28f4b9e4262
I20251029 02:52:07.702833 526 catalog_manager.cc:6719] ReplaceTablet: tablet 050186302c5d4c7da393f28f4b9e4262 deleted and replaced by tablet aacc72b167954d4fbb14ee2bfa293e6f
I20251029 02:52:07.703063 526 master_service.cc:954] ReplaceTablet: received request to replace tablet ff3e9227b9674d31b874823cf4a62d9b from {username='slave'} at 127.0.0.1:35314
I20251029 02:52:07.703495 1039 tablet_service.cc:1552] Processing DeleteTablet for tablet 050186302c5d4c7da393f28f4b9e4262 with delete_type TABLET_DATA_DELETED (replaced by tablet aacc72b167954d4fbb14ee2bfa293e6f) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:07.704895 521 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 35d9b8dfd61e4d8db441427a49fb9dca
I20251029 02:52:07.704543 517 catalog_manager.cc:6194] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: default.Table_2a63af6894694f16b45f4bf48f73a68f [id=fb42af08ec1f48498ee5e91cce62333e] alter complete (version 5)
I20251029 02:52:07.706363 749 tablet_service.cc:1552] Processing DeleteTablet for tablet 35d9b8dfd61e4d8db441427a49fb9dca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:07 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:07.706420 756 tablet_service.cc:1552] Processing DeleteTablet for tablet ba699862e6f646c6beaa39f080d0faba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:04 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:07.706707 756 tablet_service.cc:1552] Processing DeleteTablet for tablet 9bb2166a590d4210b02dd779ef6654ad with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:55 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:07.707904 1176 ts_tablet_manager.cc:1916] T 4bf06476c87a4549a73f5994314bb75f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:07.708328 885 tablet_service.cc:1552] Processing DeleteTablet for tablet 050186302c5d4c7da393f28f4b9e4262 with delete_type TABLET_DATA_DELETED (replaced by tablet aacc72b167954d4fbb14ee2bfa293e6f) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:07.708532 891 tablet_service.cc:1552] Processing DeleteTablet for tablet 35d9b8dfd61e4d8db441427a49fb9dca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:07 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:07.708709 892 tablet_service.cc:1552] Processing DeleteTablet for tablet 86fc22bb84794149b598b97df41eb881 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:58 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:07.713547 517 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet d5741dde463a43d6a776ab92dcfb04bf (table default.Table_966779a0d37a4dc4ab3f89d2bf7c6b06 [id=08e3b2ebcf37415d9a2886ad5bbda068]). Expected version 1 got 0
I20251029 02:52:07.717828 1039 tablet_service.cc:1552] Processing DeleteTablet for tablet 35d9b8dfd61e4d8db441427a49fb9dca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:07 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:07.718365 1175 ts_tablet_manager.cc:1929] T 4add552cf4d940d2a0a36d6b58379a34 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:07.718473 1175 log.cc:1199] T 4add552cf4d940d2a0a36d6b58379a34 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/4add552cf4d940d2a0a36d6b58379a34
I20251029 02:52:07.718488 750 tablet_service.cc:1552] Processing DeleteTablet for tablet 050186302c5d4c7da393f28f4b9e4262 with delete_type TABLET_DATA_DELETED (replaced by tablet aacc72b167954d4fbb14ee2bfa293e6f) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:07.718819 1175 ts_tablet_manager.cc:1950] T 4add552cf4d940d2a0a36d6b58379a34 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:07.719558 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 705) took 2615 ms. Trace:
I20251029 02:52:07.719643 1175 rpcz_store.cc:276] 1029 02:52:05.104635 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:05.104720 (+ 85us) service_pool.cc:225] Handling call
1029 02:52:07.690087 (+2585367us) ts_tablet_manager.cc:1151] Deleting tablet 4add552cf4d940d2a0a36d6b58379a34
1029 02:52:07.690097 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:07.694198 (+ 4101us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:07.718263 (+ 24065us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:07.719537 (+ 1274us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2585162}
I20251029 02:52:07.720114 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 4add552cf4d940d2a0a36d6b58379a34 (table default.Table_c73a6bc56eb44e08aa7d2ec65ad88949 [id=903fed784c0243bbb60a39066f2dbc71]) successfully deleted
I20251029 02:52:07.723287 1166 tablet_bootstrap.cc:492] T 1c53fc1fff1a4bb988beb46ff91fd3be P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:07.723807 2113 raft_consensus.cc:493] T 9bb2166a590d4210b02dd779ef6654ad P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:07.723941 2113 raft_consensus.cc:515] T 9bb2166a590d4210b02dd779ef6654ad P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:07.724499 2113 leader_election.cc:290] T 9bb2166a590d4210b02dd779ef6654ad P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:07.724908 914 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9bb2166a590d4210b02dd779ef6654ad" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:07.724987 2036 consensus_queue.cc:1048] T cc74e83833754f0ca19046031a24ebb9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:07.725314 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9bb2166a590d4210b02dd779ef6654ad" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:52:07.725869 711 leader_election.cc:343] T 9bb2166a590d4210b02dd779ef6654ad P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
W20251029 02:52:07.726246 712 leader_election.cc:343] T 9bb2166a590d4210b02dd779ef6654ad P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:07.726348 712 leader_election.cc:304] T 9bb2166a590d4210b02dd779ef6654ad P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:07.726727 2117 raft_consensus.cc:493] T d61beda662d844cdaf67fd9a2e382ea2 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:07.726876 2117 raft_consensus.cc:515] T d61beda662d844cdaf67fd9a2e382ea2 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:07.727437 2117 leader_election.cc:290] T d61beda662d844cdaf67fd9a2e382ea2 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:07.727576 2117 raft_consensus.cc:2749] T 9bb2166a590d4210b02dd779ef6654ad P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:07.728068 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d61beda662d844cdaf67fd9a2e382ea2" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:07.728327 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d61beda662d844cdaf67fd9a2e382ea2" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:52:07.729166 712 leader_election.cc:343] T d61beda662d844cdaf67fd9a2e382ea2 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:07.729383 1166 ts_tablet_manager.cc:1403] T 1c53fc1fff1a4bb988beb46ff91fd3be P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.030s user 0.000s sys 0.003s
W20251029 02:52:07.729596 711 leader_election.cc:343] T d61beda662d844cdaf67fd9a2e382ea2 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:07.729698 711 leader_election.cc:304] T d61beda662d844cdaf67fd9a2e382ea2 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:07.729987 525 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet d5741dde463a43d6a776ab92dcfb04bf (table default.Table_966779a0d37a4dc4ab3f89d2bf7c6b06 [id=08e3b2ebcf37415d9a2886ad5bbda068]). Expected version 1 got 0
I20251029 02:52:07.730087 2113 raft_consensus.cc:2749] T d61beda662d844cdaf67fd9a2e382ea2 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:07.730197 1166 raft_consensus.cc:359] T 1c53fc1fff1a4bb988beb46ff91fd3be P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:07.730368 1166 raft_consensus.cc:385] T 1c53fc1fff1a4bb988beb46ff91fd3be P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:07.730448 1166 raft_consensus.cc:740] T 1c53fc1fff1a4bb988beb46ff91fd3be P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:07.730576 1175 tablet_replica.cc:333] T b3de6670357c4ae59eb77469b2fb434c P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:07.730865 1175 raft_consensus.cc:2243] T b3de6670357c4ae59eb77469b2fb434c P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:07.731913 1176 ts_tablet_manager.cc:1929] T 4bf06476c87a4549a73f5994314bb75f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20251029 02:52:07.732277 1167 tablet_bootstrap.cc:492] T dd6ad641fbda472fab07aa35c5e4523a P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:07.732414 1167 ts_tablet_manager.cc:1403] T dd6ad641fbda472fab07aa35c5e4523a P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.160s user 0.001s sys 0.006s
I20251029 02:52:07.733093 1167 raft_consensus.cc:359] T dd6ad641fbda472fab07aa35c5e4523a P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:07.733248 1167 raft_consensus.cc:385] T dd6ad641fbda472fab07aa35c5e4523a P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:07.733311 1167 raft_consensus.cc:740] T dd6ad641fbda472fab07aa35c5e4523a P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:07.733526 1175 raft_consensus.cc:2272] T b3de6670357c4ae59eb77469b2fb434c P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:07.733492 1167 consensus_queue.cc:260] T dd6ad641fbda472fab07aa35c5e4523a P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:07.733947 1167 ts_tablet_manager.cc:1434] T dd6ad641fbda472fab07aa35c5e4523a P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20251029 02:52:07.734215 1167 tablet_bootstrap.cc:492] T bfec87f690a640c39c58c64e16cc67e7 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:07.735764 1167 tablet_bootstrap.cc:654] T bfec87f690a640c39c58c64e16cc67e7 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:07.736238 1175 ts_tablet_manager.cc:1916] T b3de6670357c4ae59eb77469b2fb434c P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:07.731997 1166 consensus_queue.cc:260] T 1c53fc1fff1a4bb988beb46ff91fd3be P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:07.738133 1176 log.cc:1199] T 4bf06476c87a4549a73f5994314bb75f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/4bf06476c87a4549a73f5994314bb75f
I20251029 02:52:07.741132 1174 ts_tablet_manager.cc:1929] T f60e7c24982d40319801869ad060864d P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:07.743556 1176 ts_tablet_manager.cc:1950] T 4bf06476c87a4549a73f5994314bb75f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:07.744508 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 4bf06476c87a4549a73f5994314bb75f (table default.Table_b708124baf534c96b3f4b8a7a4bb11cc [id=046010a03d774e91b0ad10df97848c56]) successfully deleted
I20251029 02:52:07.738560 1166 ts_tablet_manager.cc:1434] T 1c53fc1fff1a4bb988beb46ff91fd3be P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20251029 02:52:07.750720 520 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 9e6eb099648549dbbf9969e59865f51f
I20251029 02:52:07.750787 2103 consensus_queue.cc:1048] T cc74e83833754f0ca19046031a24ebb9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:07.755685 1039 tablet_service.cc:1552] Processing DeleteTablet for tablet 9e6eb099648549dbbf9969e59865f51f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:07 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:07.756629 519 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet cdc5fb2b13e042d680fa3f2eea2d62fa
I20251029 02:52:07.757342 523 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 9778f924d11a48b194ccf26124bd5509
I20251029 02:52:07.757844 523 catalog_manager.cc:6719] ReplaceTablet: tablet 9778f924d11a48b194ccf26124bd5509 deleted and replaced by tablet 971b95ab982141e184a314e7eb5e4a6a
I20251029 02:52:07.758234 524 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 80f54d8a486a4196aca3c28c84c0872f
I20251029 02:52:07.758988 526 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet ff3e9227b9674d31b874823cf4a62d9b
I20251029 02:52:07.759382 526 catalog_manager.cc:6719] ReplaceTablet: tablet ff3e9227b9674d31b874823cf4a62d9b deleted and replaced by tablet 70850519135644d1ac3d596ecf67221f
I20251029 02:52:07.761114 1031 tablet_service.cc:1552] Processing DeleteTablet for tablet 80f54d8a486a4196aca3c28c84c0872f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:07 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:07.761543 1031 tablet_service.cc:1552] Processing DeleteTablet for tablet ff3e9227b9674d31b874823cf4a62d9b with delete_type TABLET_DATA_DELETED (replaced by tablet 70850519135644d1ac3d596ecf67221f) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:07.761763 1039 tablet_service.cc:1552] Processing DeleteTablet for tablet 9778f924d11a48b194ccf26124bd5509 with delete_type TABLET_DATA_DELETED (replaced by tablet 971b95ab982141e184a314e7eb5e4a6a) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:07.762491 1166 tablet_bootstrap.cc:492] T 7a1866afc7564217ad777e7ce53786c5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:07.754989 750 tablet_service.cc:1552] Processing DeleteTablet for tablet 9e6eb099648549dbbf9969e59865f51f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:07 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:07.758302 756 tablet_service.cc:1552] Processing DeleteTablet for tablet 9778f924d11a48b194ccf26124bd5509 with delete_type TABLET_DATA_DELETED (replaced by tablet 971b95ab982141e184a314e7eb5e4a6a) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:07.760324 749 tablet_service.cc:1552] Processing DeleteTablet for tablet 80f54d8a486a4196aca3c28c84c0872f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:07 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:07.761547 755 tablet_service.cc:1552] Processing DeleteTablet for tablet ff3e9227b9674d31b874823cf4a62d9b with delete_type TABLET_DATA_DELETED (replaced by tablet 70850519135644d1ac3d596ecf67221f) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:07.764890 517 catalog_manager.cc:6194] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: default.Table_443f33587b8f428484b99e07355d69ca [id=03a25420ce6a46fc93a9fb753246bc7b] alter complete (version 3)
I20251029 02:52:07.766799 1039 tablet_service.cc:1552] Processing DeleteTablet for tablet f187dbd8a55b45b292ea61db43caf47d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:56 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:07.767448 1031 tablet_service.cc:1552] Processing DeleteTablet for tablet 4bf06476c87a4549a73f5994314bb75f with delete_type TABLET_DATA_DELETED (replaced by tablet 08ca0c889f2949408657fa60ce087608) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:07.767689 1039 tablet_service.cc:1552] Processing DeleteTablet for tablet 1bc45078a6a942c38ff3350c15885eff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:56 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:07.769876 755 tablet_service.cc:1552] Processing DeleteTablet for tablet 1275e192871440408a8345fef9b56260 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:56 UTC) from {username='slave'} at 127.0.0.1:44580
W20251029 02:52:07.769977 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet faac614af1814a2f91d66c9cf8ddc919 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet faac614af1814a2f91d66c9cf8ddc919 already in progress: creating tablet
I20251029 02:52:07.770712 755 tablet_service.cc:1552] Processing DeleteTablet for tablet 13ee62fc2339410b9db70dfd3d3c6fa6 with delete_type TABLET_DATA_DELETED (replaced by tablet 35d9b8dfd61e4d8db441427a49fb9dca) from {username='slave'} at 127.0.0.1:44580
W20251029 02:52:07.771019 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 050186302c5d4c7da393f28f4b9e4262 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 050186302c5d4c7da393f28f4b9e4262 already in progress: creating tablet
I20251029 02:52:07.772557 522 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet cc74e83833754f0ca19046031a24ebb9 (table default.Table_755825b17834457cb94b05163ae3e26d [id=57471769276f44299eae50b9d854b50c]). Expected version 1 got 0
W20251029 02:52:07.773403 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 35d9b8dfd61e4d8db441427a49fb9dca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 35d9b8dfd61e4d8db441427a49fb9dca already in progress: creating tablet
I20251029 02:52:07.774215 1176 tablet_replica.cc:333] T 9e6eb099648549dbbf9969e59865f51f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:07.777065 1176 raft_consensus.cc:2243] T 9e6eb099648549dbbf9969e59865f51f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:07.778187 1176 raft_consensus.cc:2272] T 9e6eb099648549dbbf9969e59865f51f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:07.779325 1176 ts_tablet_manager.cc:1916] T 9e6eb099648549dbbf9969e59865f51f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:07.779773 1172 tablet_bootstrap.cc:492] T b6d4f7bf702f4834bc5f4ef2b50b92a7 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:07.779965 1172 ts_tablet_manager.cc:1403] T b6d4f7bf702f4834bc5f4ef2b50b92a7 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.249s user 0.002s sys 0.005s
I20251029 02:52:07.782871 1172 raft_consensus.cc:359] T b6d4f7bf702f4834bc5f4ef2b50b92a7 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:07.783093 1172 raft_consensus.cc:385] T b6d4f7bf702f4834bc5f4ef2b50b92a7 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:07.783200 1172 raft_consensus.cc:740] T b6d4f7bf702f4834bc5f4ef2b50b92a7 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:07.783429 1172 consensus_queue.cc:260] T b6d4f7bf702f4834bc5f4ef2b50b92a7 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:07.783978 1172 ts_tablet_manager.cc:1434] T b6d4f7bf702f4834bc5f4ef2b50b92a7 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251029 02:52:07.784288 1172 tablet_bootstrap.cc:492] T 2e97d1bc06a24e8d8ba11acb6fd280d8 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:07.784643 522 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet cc74e83833754f0ca19046031a24ebb9 (table default.Table_755825b17834457cb94b05163ae3e26d [id=57471769276f44299eae50b9d854b50c]). Expected version 1 got 0
I20251029 02:52:07.788827 1172 tablet_bootstrap.cc:654] T 2e97d1bc06a24e8d8ba11acb6fd280d8 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:07.790832 522 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet cc74e83833754f0ca19046031a24ebb9 (table default.Table_755825b17834457cb94b05163ae3e26d [id=57471769276f44299eae50b9d854b50c]). Expected version 1 got 0
I20251029 02:52:07.792356 755 tablet_service.cc:1552] Processing DeleteTablet for tablet b6d4f7bf702f4834bc5f4ef2b50b92a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:55 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:07.797681 1175 ts_tablet_manager.cc:1929] T b3de6670357c4ae59eb77469b2fb434c P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:07.752467 890 tablet_service.cc:1552] Processing DeleteTablet for tablet 9e6eb099648549dbbf9969e59865f51f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:07 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:07.798620 1174 log.cc:1199] T f60e7c24982d40319801869ad060864d P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/f60e7c24982d40319801869ad060864d
I20251029 02:52:07.759347 891 tablet_service.cc:1552] Processing DeleteTablet for tablet 80f54d8a486a4196aca3c28c84c0872f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:07 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:07.759502 892 tablet_service.cc:1552] Processing DeleteTablet for tablet 9778f924d11a48b194ccf26124bd5509 with delete_type TABLET_DATA_DELETED (replaced by tablet 971b95ab982141e184a314e7eb5e4a6a) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:07.762004 1167 tablet_bootstrap.cc:492] T bfec87f690a640c39c58c64e16cc67e7 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:07.762410 885 tablet_service.cc:1552] Processing DeleteTablet for tablet ff3e9227b9674d31b874823cf4a62d9b with delete_type TABLET_DATA_DELETED (replaced by tablet 70850519135644d1ac3d596ecf67221f) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:07.765893 898 tablet_service.cc:1552] Processing DeleteTablet for tablet 56d05e4a500043c995bb51a204cb295d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:53 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:07.773591 900 tablet_service.cc:1552] Processing DeleteTablet for tablet dd6ad641fbda472fab07aa35c5e4523a with delete_type TABLET_DATA_DELETED (replaced by tablet 4f24ae08e2c34375abccb6fd187adbd4) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:07.799103 1167 ts_tablet_manager.cc:1403] T bfec87f690a640c39c58c64e16cc67e7 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.065s user 0.004s sys 0.004s
I20251029 02:52:07.799165 1174 ts_tablet_manager.cc:1950] T f60e7c24982d40319801869ad060864d P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:07.800033 1167 raft_consensus.cc:359] T bfec87f690a640c39c58c64e16cc67e7 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:07.800205 1167 raft_consensus.cc:385] T bfec87f690a640c39c58c64e16cc67e7 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:07.800263 1167 raft_consensus.cc:740] T bfec87f690a640c39c58c64e16cc67e7 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:07.800467 1167 consensus_queue.cc:260] T bfec87f690a640c39c58c64e16cc67e7 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:07.801070 1167 ts_tablet_manager.cc:1434] T bfec87f690a640c39c58c64e16cc67e7 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 02:52:07.801409 1167 tablet_bootstrap.cc:492] T f246f6c202cf44d48c9f5e4f335202c5 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:07.777947 1166 tablet_bootstrap.cc:654] T 7a1866afc7564217ad777e7ce53786c5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:07.803732 1167 tablet_bootstrap.cc:654] T f246f6c202cf44d48c9f5e4f335202c5 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:07.805681 1174 tablet_replica.cc:333] T 0cc1957a845f4e9cb703a553b27d1347 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:07.805871 1174 raft_consensus.cc:2243] T 0cc1957a845f4e9cb703a553b27d1347 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:07.806031 1174 raft_consensus.cc:2272] T 0cc1957a845f4e9cb703a553b27d1347 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:07.806558 1174 ts_tablet_manager.cc:1916] T 0cc1957a845f4e9cb703a553b27d1347 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:07.808705 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet f60e7c24982d40319801869ad060864d (table default.Table_4350e92f14de4ba8a04afbe12ea7422a [id=226a2d412a7b40e59b6f5340b5a9b739]) successfully deleted
I20251029 02:52:07.811277 525 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_464b6de22be542b89e56489d27d2d521" } new_table_name: "default.Table_c21a6bf1d5bb40acbded1097c15ed57a" modify_external_catalogs: true
I20251029 02:52:07.813839 1175 log.cc:1199] T b3de6670357c4ae59eb77469b2fb434c P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/b3de6670357c4ae59eb77469b2fb434c
I20251029 02:52:07.813920 526 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_71446f2a5631421ab14116c669e1d5ba" } modify_external_catalogs: true
I20251029 02:52:07.814090 526 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_71446f2a5631421ab14116c669e1d5ba" } modify_external_catalogs: true
I20251029 02:52:07.814201 1175 ts_tablet_manager.cc:1950] T b3de6670357c4ae59eb77469b2fb434c P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:07.814553 1166 tablet_bootstrap.cc:492] T 7a1866afc7564217ad777e7ce53786c5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:07.814703 1166 ts_tablet_manager.cc:1403] T 7a1866afc7564217ad777e7ce53786c5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.052s user 0.000s sys 0.004s
I20251029 02:52:07.814724 523 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_6b0b554613694e198639046c12218185" } new_table_name: "default.Table_80b931e916ee46cf8704adef42106b95" modify_external_catalogs: true
I20251029 02:52:07.815045 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 720) took 2433 ms. Trace:
I20251029 02:52:07.815124 1175 rpcz_store.cc:276] 1029 02:52:05.381436 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:05.381809 (+ 373us) service_pool.cc:225] Handling call
1029 02:52:07.730380 (+2348571us) ts_tablet_manager.cc:1151] Deleting tablet b3de6670357c4ae59eb77469b2fb434c
1029 02:52:07.730396 (+ 16us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:07.793909 (+ 63513us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:07.797524 (+ 3615us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:07.815022 (+ 17498us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2342015}
I20251029 02:52:07.815536 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet b3de6670357c4ae59eb77469b2fb434c (table default.Table_8dacd5c558a04e47b424533b985aee9e [id=f9c7128476404646a9194e31f49c94e8]) successfully deleted
I20251029 02:52:07.815904 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_6a3ec6a2716e4a0994a4d721f96b488e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:07.815975 1166 raft_consensus.cc:359] T 7a1866afc7564217ad777e7ce53786c5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:07.816156 1166 raft_consensus.cc:385] T 7a1866afc7564217ad777e7ce53786c5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:07.816226 1166 raft_consensus.cc:740] T 7a1866afc7564217ad777e7ce53786c5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
W20251029 02:52:07.816390 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6a3ec6a2716e4a0994a4d721f96b488e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:07.816439 1166 consensus_queue.cc:260] T 7a1866afc7564217ad777e7ce53786c5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:07.817327 1166 ts_tablet_manager.cc:1434] T 7a1866afc7564217ad777e7ce53786c5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20251029 02:52:07.817487 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_329849bbaa3a457096d8526f70636b96"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:07.798331 1172 tablet_bootstrap.cc:492] T 2e97d1bc06a24e8d8ba11acb6fd280d8 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
W20251029 02:52:07.817937 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_329849bbaa3a457096d8526f70636b96 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:07.817981 1172 ts_tablet_manager.cc:1403] T 2e97d1bc06a24e8d8ba11acb6fd280d8 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.034s user 0.005s sys 0.005s
I20251029 02:52:07.818780 1172 raft_consensus.cc:359] T 2e97d1bc06a24e8d8ba11acb6fd280d8 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:07.818976 1172 raft_consensus.cc:385] T 2e97d1bc06a24e8d8ba11acb6fd280d8 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:07.819049 1172 raft_consensus.cc:740] T 2e97d1bc06a24e8d8ba11acb6fd280d8 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:07.819213 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_331f901095f44ebfa644aa9cf8dde711"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:07.819279 1172 consensus_queue.cc:260] T 2e97d1bc06a24e8d8ba11acb6fd280d8 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
W20251029 02:52:07.819685 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_331f901095f44ebfa644aa9cf8dde711 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:07.819756 1172 ts_tablet_manager.cc:1434] T 2e97d1bc06a24e8d8ba11acb6fd280d8 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 02:52:07.819783 1166 tablet_bootstrap.cc:492] T caef1c93e33a4390bcada59a41f43465 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
W20251029 02:52:07.821157 525 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7ff2399833e546448b85b9caeb5200fb consensus state has no leader
I20251029 02:52:07.821588 1166 tablet_bootstrap.cc:654] T caef1c93e33a4390bcada59a41f43465 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:07.825268 1172 tablet_bootstrap.cc:492] T 9269622ca92645d5aa8e73341201ccbb P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:07.826988 1172 tablet_bootstrap.cc:654] T 9269622ca92645d5aa8e73341201ccbb P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:07.830248 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_c6e9611c339e47a8b5021b69c449f141"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:07.830312 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_4d99c72b75124b20bf00ff0d75e7aada"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:07.830842 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c6e9611c339e47a8b5021b69c449f141 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:07.831007 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_2ce7a304a9184f1ebba440670ede298c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:07.831323 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4d99c72b75124b20bf00ff0d75e7aada in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:52:07.831404 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2ce7a304a9184f1ebba440670ede298c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:07.832173 755 tablet_service.cc:1552] Processing DeleteTablet for tablet 2e97d1bc06a24e8d8ba11acb6fd280d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:55 UTC) from {username='slave'} at 127.0.0.1:44580
W20251029 02:52:07.839215 523 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4c17d21145e54cc78134cab37059ce1e consensus state has no leader
I20251029 02:52:07.843113 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_f8f86e0f1d8f4ba98ba26b268e26eee2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:07.843746 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f8f86e0f1d8f4ba98ba26b268e26eee2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:07.847139 519 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_8ecb63fece7348379a7d341b1c4b5891" } new_table_name: "default.Table_5e2a6b532ed1448097f02628c1a02fc9" modify_external_catalogs: true
I20251029 02:52:07.846135 900 tablet_service.cc:1552] Processing DeleteTablet for tablet bfec87f690a640c39c58c64e16cc67e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:57 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:07.867962 1176 ts_tablet_manager.cc:1929] T 9e6eb099648549dbbf9969e59865f51f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:07.868294 1176 log.cc:1199] T 9e6eb099648549dbbf9969e59865f51f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/9e6eb099648549dbbf9969e59865f51f
I20251029 02:52:07.870021 2113 raft_consensus.cc:493] T 2e97d1bc06a24e8d8ba11acb6fd280d8 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:07.870190 2113 raft_consensus.cc:515] T 2e97d1bc06a24e8d8ba11acb6fd280d8 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:07.868745 1176 ts_tablet_manager.cc:1950] T 9e6eb099648549dbbf9969e59865f51f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:07.870939 2113 leader_election.cc:290] T 2e97d1bc06a24e8d8ba11acb6fd280d8 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:07.871865 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e97d1bc06a24e8d8ba11acb6fd280d8" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:07.872668 1166 tablet_bootstrap.cc:492] T caef1c93e33a4390bcada59a41f43465 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:07.872828 1166 ts_tablet_manager.cc:1403] T caef1c93e33a4390bcada59a41f43465 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.053s user 0.002s sys 0.003s
I20251029 02:52:07.872864 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e97d1bc06a24e8d8ba11acb6fd280d8" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:52:07.873404 711 leader_election.cc:343] T 2e97d1bc06a24e8d8ba11acb6fd280d8 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:07.873629 1166 raft_consensus.cc:359] T caef1c93e33a4390bcada59a41f43465 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:07.873817 1166 raft_consensus.cc:385] T caef1c93e33a4390bcada59a41f43465 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:07.873911 1166 raft_consensus.cc:740] T caef1c93e33a4390bcada59a41f43465 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
W20251029 02:52:07.874080 712 leader_election.cc:343] T 2e97d1bc06a24e8d8ba11acb6fd280d8 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:07.874177 712 leader_election.cc:304] T 2e97d1bc06a24e8d8ba11acb6fd280d8 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:07.874135 1166 consensus_queue.cc:260] T caef1c93e33a4390bcada59a41f43465 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:07.874662 1166 ts_tablet_manager.cc:1434] T caef1c93e33a4390bcada59a41f43465 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 02:52:07.874989 1166 tablet_bootstrap.cc:492] T 86fc22bb84794149b598b97df41eb881 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:07.876688 1166 tablet_bootstrap.cc:654] T 86fc22bb84794149b598b97df41eb881 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:07.876754 2113 raft_consensus.cc:2749] T 2e97d1bc06a24e8d8ba11acb6fd280d8 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:07.877694 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 9e6eb099648549dbbf9969e59865f51f (table default.Table_683d67689da74097b69f31b3d9461fe2 [id=0c5d3e4d34d045dc99ffc88e101cae1d]) successfully deleted
W20251029 02:52:07.877988 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 80f54d8a486a4196aca3c28c84c0872f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 80f54d8a486a4196aca3c28c84c0872f already in progress: creating tablet
W20251029 02:52:07.878213 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet ff3e9227b9674d31b874823cf4a62d9b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ff3e9227b9674d31b874823cf4a62d9b already in progress: creating tablet
W20251029 02:52:07.878396 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 9778f924d11a48b194ccf26124bd5509 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9778f924d11a48b194ccf26124bd5509 already in progress: creating tablet
W20251029 02:52:07.878857 513 catalog_manager.cc:4952] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet f187dbd8a55b45b292ea61db43caf47d because the tablet was not found. No further retry: Not found: Tablet not found: f187dbd8a55b45b292ea61db43caf47d
W20251029 02:52:07.879130 513 catalog_manager.cc:4952] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 4bf06476c87a4549a73f5994314bb75f because the tablet was not found. No further retry: Not found: Tablet not found: 4bf06476c87a4549a73f5994314bb75f
W20251029 02:52:07.879423 513 catalog_manager.cc:4952] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 1bc45078a6a942c38ff3350c15885eff because the tablet was not found. No further retry: Not found: Tablet not found: 1bc45078a6a942c38ff3350c15885eff
I20251029 02:52:07.879611 1172 tablet_bootstrap.cc:492] T 9269622ca92645d5aa8e73341201ccbb P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:07.879784 1172 ts_tablet_manager.cc:1403] T 9269622ca92645d5aa8e73341201ccbb P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.055s user 0.023s sys 0.017s
I20251029 02:52:07.880249 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 722) took 2465 ms. Trace:
I20251029 02:52:07.881016 1175 rpcz_store.cc:276] 1029 02:52:05.414336 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:05.414502 (+ 166us) service_pool.cc:225] Handling call
1029 02:52:07.880060 (+2465558us) ts_tablet_manager.cc:1151] Deleting tablet 4a55f22b0d3b4ebe98ec2e054d05b095
1029 02:52:07.880075 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:07.880219 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2465331}
W20251029 02:52:07.882014 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 4a55f22b0d3b4ebe98ec2e054d05b095 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4a55f22b0d3b4ebe98ec2e054d05b095 already in progress: creating tablet
I20251029 02:52:07.882292 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 723) took 2467 ms. Trace:
I20251029 02:52:07.882409 1175 rpcz_store.cc:276] 1029 02:52:05.414452 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:05.414826 (+ 374us) service_pool.cc:225] Handling call
1029 02:52:07.882081 (+2467255us) ts_tablet_manager.cc:1151] Deleting tablet 6ecc552c1ef643f8bb84622d9d440ba2
1029 02:52:07.882092 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:07.882273 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2467040}
W20251029 02:52:07.882891 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 6ecc552c1ef643f8bb84622d9d440ba2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6ecc552c1ef643f8bb84622d9d440ba2 already in progress: creating tablet
I20251029 02:52:07.884356 1175 tablet_replica.cc:333] T 1bc45078a6a942c38ff3350c15885eff P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
W20251029 02:52:07.886070 519 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8cae9d4ddb0744298b82f4d9cee061b3 consensus state has no leader
I20251029 02:52:07.887667 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_ffc17029e0734c73a3735b9f99cf1332"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:07.888322 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ffc17029e0734c73a3735b9f99cf1332 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:07.890399 520 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_c21a6bf1d5bb40acbded1097c15ed57a" } new_table_name: "default.Table_97a40219642245b8958e117857092c0a" modify_external_catalogs: true
I20251029 02:52:07.880591 1172 raft_consensus.cc:359] T 9269622ca92645d5aa8e73341201ccbb P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:07.891716 1175 raft_consensus.cc:2243] T 1bc45078a6a942c38ff3350c15885eff P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:07.891901 1175 raft_consensus.cc:2272] T 1bc45078a6a942c38ff3350c15885eff P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:07.892527 1175 ts_tablet_manager.cc:1916] T 1bc45078a6a942c38ff3350c15885eff P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:07.901091 519 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_287452ecd9d54f83be8e5d9186551e3c" } new_table_name: "default.Table_0427cb4a53de4e62a4b89829586a2a53" modify_external_catalogs: true
I20251029 02:52:07.902300 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_3e2da7dbc6574230bd2182d9226213fc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:07.902891 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3e2da7dbc6574230bd2182d9226213fc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:07.903149 2113 raft_consensus.cc:493] T 13ee62fc2339410b9db70dfd3d3c6fa6 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:07.903299 2113 raft_consensus.cc:515] T 13ee62fc2339410b9db70dfd3d3c6fa6 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:07.903944 2113 leader_election.cc:290] T 13ee62fc2339410b9db70dfd3d3c6fa6 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:07.904742 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "13ee62fc2339410b9db70dfd3d3c6fa6" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:07.904933 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "13ee62fc2339410b9db70dfd3d3c6fa6" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:07.905426 712 leader_election.cc:343] T 13ee62fc2339410b9db70dfd3d3c6fa6 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
W20251029 02:52:07.905594 711 leader_election.cc:343] T 13ee62fc2339410b9db70dfd3d3c6fa6 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:07.905742 711 leader_election.cc:304] T 13ee62fc2339410b9db70dfd3d3c6fa6 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:07.906273 2113 raft_consensus.cc:2749] T 13ee62fc2339410b9db70dfd3d3c6fa6 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:07.911561 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_954aab60929348dcbdbd69aa987112ac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:07.912185 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_954aab60929348dcbdbd69aa987112ac in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:07.912870 518 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_2a63af6894694f16b45f4bf48f73a68f" } modify_external_catalogs: true
I20251029 02:52:07.913137 518 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_2a63af6894694f16b45f4bf48f73a68f" } modify_external_catalogs: true
I20251029 02:52:07.913887 1174 ts_tablet_manager.cc:1929] T 0cc1957a845f4e9cb703a553b27d1347 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20251029 02:52:07.915046 520 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7ff2399833e546448b85b9caeb5200fb consensus state has no leader
W20251029 02:52:07.916139 519 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 21c73cb2f1084827b581fb09cf7239a1 has no consensus state
I20251029 02:52:07.916442 1166 tablet_bootstrap.cc:492] T 86fc22bb84794149b598b97df41eb881 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:07.916601 1166 ts_tablet_manager.cc:1403] T 86fc22bb84794149b598b97df41eb881 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.042s user 0.003s sys 0.006s
I20251029 02:52:07.917490 1166 raft_consensus.cc:359] T 86fc22bb84794149b598b97df41eb881 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:07.917690 1166 raft_consensus.cc:385] T 86fc22bb84794149b598b97df41eb881 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:07.917776 1166 raft_consensus.cc:740] T 86fc22bb84794149b598b97df41eb881 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:07.891729 1172 raft_consensus.cc:385] T 9269622ca92645d5aa8e73341201ccbb P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:07.917968 1172 raft_consensus.cc:740] T 9269622ca92645d5aa8e73341201ccbb P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:07.918022 1166 consensus_queue.cc:260] T 86fc22bb84794149b598b97df41eb881 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:07.918563 1166 ts_tablet_manager.cc:1434] T 86fc22bb84794149b598b97df41eb881 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20251029 02:52:07.918618 1172 consensus_queue.cc:260] T 9269622ca92645d5aa8e73341201ccbb P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:07.926853 1166 tablet_bootstrap.cc:492] T 0f80b840ed954230bfbfe7e40127cd2f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:07.928668 1166 tablet_bootstrap.cc:654] T 0f80b840ed954230bfbfe7e40127cd2f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:07.930088 521 master_service.cc:954] ReplaceTablet: received request to replace tablet 65e761f9f9fb490586d6415a5a8565dd from {username='slave'} at 127.0.0.1:35314
I20251029 02:52:07.914812 1167 tablet_bootstrap.cc:492] T f246f6c202cf44d48c9f5e4f335202c5 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:07.915162 2109 raft_consensus.cc:493] T bfec87f690a640c39c58c64e16cc67e7 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:07.930660 2109 raft_consensus.cc:515] T bfec87f690a640c39c58c64e16cc67e7 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:07.931346 2109 leader_election.cc:290] T bfec87f690a640c39c58c64e16cc67e7 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:07.932883 1172 ts_tablet_manager.cc:1434] T 9269622ca92645d5aa8e73341201ccbb P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.053s user 0.001s sys 0.001s
W20251029 02:52:07.932874 849 leader_election.cc:343] T bfec87f690a640c39c58c64e16cc67e7 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Not found: Tablet not found: bfec87f690a640c39c58c64e16cc67e7
I20251029 02:52:07.933269 1172 tablet_bootstrap.cc:492] T 30d03e1e42d34242bd354f9e9ec4f127 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
W20251029 02:52:07.935660 847 leader_election.cc:343] T bfec87f690a640c39c58c64e16cc67e7 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: bfec87f690a640c39c58c64e16cc67e7
I20251029 02:52:07.935796 847 leader_election.cc:304] T bfec87f690a640c39c58c64e16cc67e7 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9
I20251029 02:52:07.936215 1167 ts_tablet_manager.cc:1403] T f246f6c202cf44d48c9f5e4f335202c5 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.135s user 0.006s sys 0.005s
I20251029 02:52:07.937399 1172 tablet_bootstrap.cc:654] T 30d03e1e42d34242bd354f9e9ec4f127 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:07.937399 1167 raft_consensus.cc:359] T f246f6c202cf44d48c9f5e4f335202c5 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:07.937563 1167 raft_consensus.cc:385] T f246f6c202cf44d48c9f5e4f335202c5 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:07.937621 1167 raft_consensus.cc:740] T f246f6c202cf44d48c9f5e4f335202c5 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:07.937822 2109 raft_consensus.cc:2749] T bfec87f690a640c39c58c64e16cc67e7 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:07.938642 1167 consensus_queue.cc:260] T f246f6c202cf44d48c9f5e4f335202c5 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:07.940697 1167 ts_tablet_manager.cc:1434] T f246f6c202cf44d48c9f5e4f335202c5 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251029 02:52:07.943555 1167 tablet_bootstrap.cc:492] T 1bc45078a6a942c38ff3350c15885eff P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:07.947715 1167 tablet_bootstrap.cc:654] T 1bc45078a6a942c38ff3350c15885eff P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:07.949384 1174 log.cc:1199] T 0cc1957a845f4e9cb703a553b27d1347 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/0cc1957a845f4e9cb703a553b27d1347
I20251029 02:52:07.952898 1175 ts_tablet_manager.cc:1929] T 1bc45078a6a942c38ff3350c15885eff P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:07.953017 1175 log.cc:1199] T 1bc45078a6a942c38ff3350c15885eff P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/1bc45078a6a942c38ff3350c15885eff
I20251029 02:52:07.953281 1175 ts_tablet_manager.cc:1950] T 1bc45078a6a942c38ff3350c15885eff P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:07.953733 1174 ts_tablet_manager.cc:1950] T 0cc1957a845f4e9cb703a553b27d1347 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:07.955451 1174 tablet_replica.cc:333] T 6ca9bd37de9e4be388501b15af4bda31 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:07.955727 1174 raft_consensus.cc:2243] T 6ca9bd37de9e4be388501b15af4bda31 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:07.955976 1174 raft_consensus.cc:2272] T 6ca9bd37de9e4be388501b15af4bda31 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:07.956370 1166 tablet_bootstrap.cc:492] T 0f80b840ed954230bfbfe7e40127cd2f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:07.956532 1166 ts_tablet_manager.cc:1403] T 0f80b840ed954230bfbfe7e40127cd2f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.030s user 0.002s sys 0.001s
I20251029 02:52:07.956533 522 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_e48aadc66a0e48658d45d0812c29cbdf" } modify_external_catalogs: true
I20251029 02:52:07.956730 522 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_e48aadc66a0e48658d45d0812c29cbdf" } modify_external_catalogs: true
I20251029 02:52:07.956940 1174 ts_tablet_manager.cc:1916] T 6ca9bd37de9e4be388501b15af4bda31 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:07.957335 1166 raft_consensus.cc:359] T 0f80b840ed954230bfbfe7e40127cd2f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:07.957505 1166 raft_consensus.cc:385] T 0f80b840ed954230bfbfe7e40127cd2f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:07.957572 1166 raft_consensus.cc:740] T 0f80b840ed954230bfbfe7e40127cd2f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:07.957787 1166 consensus_queue.cc:260] T 0f80b840ed954230bfbfe7e40127cd2f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:07.958184 1166 ts_tablet_manager.cc:1434] T 0f80b840ed954230bfbfe7e40127cd2f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251029 02:52:07.958465 1166 tablet_bootstrap.cc:492] T 95f342d1cfb84c4eb31bd3d15ea3bdc2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:07.956341 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 0cc1957a845f4e9cb703a553b27d1347 (table default.Table_262db98ea44648baab95fc2b3081dc4e [id=843f681b71c64427b49becbabcad9451]) successfully deleted
I20251029 02:52:07.960116 1166 tablet_bootstrap.cc:654] T 95f342d1cfb84c4eb31bd3d15ea3bdc2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:07.961678 524 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_d897a86961894f1aaad67274b687080e" } modify_external_catalogs: true
I20251029 02:52:07.961900 524 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_d897a86961894f1aaad67274b687080e" } modify_external_catalogs: true
I20251029 02:52:07.962007 518 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet da2a21c3236a4bd99872b89b12a36508
I20251029 02:52:07.967620 1025 tablet_service.cc:1552] Processing DeleteTablet for tablet da2a21c3236a4bd99872b89b12a36508 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:07 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:07.968314 900 tablet_service.cc:1552] Processing DeleteTablet for tablet f246f6c202cf44d48c9f5e4f335202c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:55 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:07.968407 755 tablet_service.cc:1552] Processing DeleteTablet for tablet da2a21c3236a4bd99872b89b12a36508 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:07 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:07.968538 898 tablet_service.cc:1552] Processing DeleteTablet for tablet da2a21c3236a4bd99872b89b12a36508 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:07 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:07.969177 518 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_99f14277f09b47438fb0522c896dd4b4" } modify_external_catalogs: true
I20251029 02:52:07.969374 518 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_99f14277f09b47438fb0522c896dd4b4" } modify_external_catalogs: true
I20251029 02:52:07.972167 1176 tablet_replica.cc:333] T da2a21c3236a4bd99872b89b12a36508 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:07.972391 1176 raft_consensus.cc:2243] T da2a21c3236a4bd99872b89b12a36508 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:07.972726 1176 raft_consensus.cc:2272] T da2a21c3236a4bd99872b89b12a36508 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:07.973301 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 724) took 2551 ms. Trace:
I20251029 02:52:07.974444 1176 ts_tablet_manager.cc:1916] T da2a21c3236a4bd99872b89b12a36508 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:07.975351 1166 tablet_bootstrap.cc:492] T 95f342d1cfb84c4eb31bd3d15ea3bdc2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:07.975507 1166 ts_tablet_manager.cc:1403] T 95f342d1cfb84c4eb31bd3d15ea3bdc2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.000s
I20251029 02:52:07.975553 1175 rpcz_store.cc:276] 1029 02:52:05.422015 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:05.422069 (+ 54us) service_pool.cc:225] Handling call
1029 02:52:07.884308 (+2462239us) ts_tablet_manager.cc:1151] Deleting tablet 1bc45078a6a942c38ff3350c15885eff
1029 02:52:07.884317 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:07.939183 (+ 54866us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:07.952784 (+ 13601us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:07.973278 (+ 20494us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2462017}
I20251029 02:52:07.976207 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 726) took 2546 ms. Trace:
I20251029 02:52:07.976289 1166 raft_consensus.cc:359] T 95f342d1cfb84c4eb31bd3d15ea3bdc2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:07.976478 1166 raft_consensus.cc:385] T 95f342d1cfb84c4eb31bd3d15ea3bdc2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:07.976567 1166 raft_consensus.cc:740] T 95f342d1cfb84c4eb31bd3d15ea3bdc2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:07.976786 1166 consensus_queue.cc:260] T 95f342d1cfb84c4eb31bd3d15ea3bdc2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:07.976368 1175 rpcz_store.cc:276] 1029 02:52:05.429304 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:05.429483 (+ 179us) service_pool.cc:225] Handling call
1029 02:52:07.976018 (+2546535us) ts_tablet_manager.cc:1151] Deleting tablet ea7b977224f24bf49dadfb83ffa28310
1029 02:52:07.976029 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:07.976187 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2546308}
I20251029 02:52:07.977078 2103 raft_consensus.cc:493] T caef1c93e33a4390bcada59a41f43465 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:07.977296 1166 ts_tablet_manager.cc:1434] T 95f342d1cfb84c4eb31bd3d15ea3bdc2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251029 02:52:07.977280 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 727) took 2547 ms. Trace:
I20251029 02:52:07.977265 2103 raft_consensus.cc:515] T caef1c93e33a4390bcada59a41f43465 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:07.977602 1166 tablet_bootstrap.cc:492] T ccdca31db2834de4b6ed0f2d67a10535 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:07.977873 2103 leader_election.cc:290] T caef1c93e33a4390bcada59a41f43465 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:07.978718 1172 tablet_bootstrap.cc:492] T 30d03e1e42d34242bd354f9e9ec4f127 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:07.978850 1172 ts_tablet_manager.cc:1403] T 30d03e1e42d34242bd354f9e9ec4f127 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.046s user 0.003s sys 0.003s
I20251029 02:52:07.979490 1172 raft_consensus.cc:359] T 30d03e1e42d34242bd354f9e9ec4f127 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:07.979652 1172 raft_consensus.cc:385] T 30d03e1e42d34242bd354f9e9ec4f127 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:07.979709 1172 raft_consensus.cc:740] T 30d03e1e42d34242bd354f9e9ec4f127 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:07.979897 1172 consensus_queue.cc:260] T 30d03e1e42d34242bd354f9e9ec4f127 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:07.980418 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 1bc45078a6a942c38ff3350c15885eff (table default.Table_3126aa57075d44dc8b73f08fe21aabdf [id=e8fa170a8a044f2fa1236949a91454de]) successfully deleted
W20251029 02:52:07.980799 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet ea7b977224f24bf49dadfb83ffa28310 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ea7b977224f24bf49dadfb83ffa28310 already in progress: creating tablet
I20251029 02:52:07.981244 1172 ts_tablet_manager.cc:1434] T 30d03e1e42d34242bd354f9e9ec4f127 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 02:52:07.981613 1172 tablet_bootstrap.cc:492] T 738046ab57334f2dab9f4aab979c5b42 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:07.977355 1175 rpcz_store.cc:276] 1029 02:52:05.429433 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:05.429747 (+ 314us) service_pool.cc:225] Handling call
1029 02:52:07.977158 (+2547411us) ts_tablet_manager.cc:1151] Deleting tablet 81f936792aae4b699b3122457d7b631d
1029 02:52:07.977168 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:07.977261 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2547184}
W20251029 02:52:07.982473 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 81f936792aae4b699b3122457d7b631d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 81f936792aae4b699b3122457d7b631d already in progress: creating tablet
I20251029 02:52:07.982609 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "caef1c93e33a4390bcada59a41f43465" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:07.982626 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 728) took 2545 ms. Trace:
W20251029 02:52:07.983511 989 leader_election.cc:343] T caef1c93e33a4390bcada59a41f43465 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: caef1c93e33a4390bcada59a41f43465
I20251029 02:52:07.983462 1175 rpcz_store.cc:276] 1029 02:52:05.437025 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:05.437568 (+ 543us) service_pool.cc:225] Handling call
1029 02:52:07.982469 (+2544901us) ts_tablet_manager.cc:1151] Deleting tablet 46cd75ff18b14212b556b9bcb3012dd9
1029 02:52:07.982479 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:07.982608 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2544696}
W20251029 02:52:07.983853 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 46cd75ff18b14212b556b9bcb3012dd9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 46cd75ff18b14212b556b9bcb3012dd9 already in progress: creating tablet
W20251029 02:52:07.983913 990 leader_election.cc:343] T caef1c93e33a4390bcada59a41f43465 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:07.984036 990 leader_election.cc:304] T caef1c93e33a4390bcada59a41f43465 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5; no voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:07.984400 1172 tablet_bootstrap.cc:654] T 738046ab57334f2dab9f4aab979c5b42 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:07.984444 1973 raft_consensus.cc:2749] T caef1c93e33a4390bcada59a41f43465 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 02:52:07.989161 510 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 15137fc1c70347b381ffdfe5653a5f22 (table default.Table_6f38863737b34dfa90488cad665a8628 [id=4bacb12c51d34f78bf4b1aba964c38ce], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 15137fc1c70347b381ffdfe5653a5f22 consensus state has no leader
I20251029 02:52:07.992590 1172 tablet_bootstrap.cc:492] T 738046ab57334f2dab9f4aab979c5b42 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:07.992815 1172 ts_tablet_manager.cc:1403] T 738046ab57334f2dab9f4aab979c5b42 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.007s
I20251029 02:52:07.993659 520 master_service.cc:954] ReplaceTablet: received request to replace tablet 59d03a190b784ab18a3e63a572675891 from {username='slave'} at 127.0.0.1:35314
I20251029 02:52:07.993599 1172 raft_consensus.cc:359] T 738046ab57334f2dab9f4aab979c5b42 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:07.993863 1172 raft_consensus.cc:385] T 738046ab57334f2dab9f4aab979c5b42 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:07.993929 1172 raft_consensus.cc:740] T 738046ab57334f2dab9f4aab979c5b42 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:07.994194 1172 consensus_queue.cc:260] T 738046ab57334f2dab9f4aab979c5b42 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:07.994814 1172 ts_tablet_manager.cc:1434] T 738046ab57334f2dab9f4aab979c5b42 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251029 02:52:07.995229 1172 tablet_bootstrap.cc:492] T 6d830d30b26b4ba799785fec8a6f0769 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:07.995920 1175 tablet_replica.cc:333] T dd90fb9eb167478cbc3ec9363af86295 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:07.996754 1175 raft_consensus.cc:2243] T dd90fb9eb167478cbc3ec9363af86295 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:07.996985 1175 raft_consensus.cc:2272] T dd90fb9eb167478cbc3ec9363af86295 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:07.997520 1175 ts_tablet_manager.cc:1916] T dd90fb9eb167478cbc3ec9363af86295 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:08.000062 1172 tablet_bootstrap.cc:654] T 6d830d30b26b4ba799785fec8a6f0769 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:08.003924 1176 ts_tablet_manager.cc:1929] T da2a21c3236a4bd99872b89b12a36508 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251029 02:52:08.004127 1176 log.cc:1199] T da2a21c3236a4bd99872b89b12a36508 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/da2a21c3236a4bd99872b89b12a36508
I20251029 02:52:08.005549 522 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 41a6139fd9b84707a8c7eebc0381e383
I20251029 02:52:08.006474 755 tablet_service.cc:1552] Processing DeleteTablet for tablet 41a6139fd9b84707a8c7eebc0381e383 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:07 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:08.007252 898 tablet_service.cc:1552] Processing DeleteTablet for tablet 41a6139fd9b84707a8c7eebc0381e383 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:07 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:08.011406 1172 tablet_bootstrap.cc:492] T 6d830d30b26b4ba799785fec8a6f0769 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:08.011551 1172 ts_tablet_manager.cc:1403] T 6d830d30b26b4ba799785fec8a6f0769 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.005s
I20251029 02:52:08.012342 1167 tablet_bootstrap.cc:492] T 1bc45078a6a942c38ff3350c15885eff P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:08.012367 1172 raft_consensus.cc:359] T 6d830d30b26b4ba799785fec8a6f0769 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.012534 1172 raft_consensus.cc:385] T 6d830d30b26b4ba799785fec8a6f0769 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:08.012595 1172 raft_consensus.cc:740] T 6d830d30b26b4ba799785fec8a6f0769 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:08.012645 1167 ts_tablet_manager.cc:1403] T 1bc45078a6a942c38ff3350c15885eff P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.069s user 0.003s sys 0.002s
I20251029 02:52:08.012833 1172 consensus_queue.cc:260] T 6d830d30b26b4ba799785fec8a6f0769 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.013319 1172 ts_tablet_manager.cc:1434] T 6d830d30b26b4ba799785fec8a6f0769 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251029 02:52:08.013636 1167 raft_consensus.cc:359] T 1bc45078a6a942c38ff3350c15885eff P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.014016 1172 tablet_bootstrap.cc:492] T 061f3d38c4604fa4b57e74ff6f6fc7da P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:08.014043 1167 raft_consensus.cc:385] T 1bc45078a6a942c38ff3350c15885eff P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:08.014204 1167 raft_consensus.cc:740] T 1bc45078a6a942c38ff3350c15885eff P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:08.014478 1167 consensus_queue.cc:260] T 1bc45078a6a942c38ff3350c15885eff P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.015400 1167 ts_tablet_manager.cc:1434] T 1bc45078a6a942c38ff3350c15885eff P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251029 02:52:08.017153 1167 tablet_bootstrap.cc:492] T 785733da86ff425a90ccfab2933fc89e P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:08.019912 1025 tablet_service.cc:1552] Processing DeleteTablet for tablet 41a6139fd9b84707a8c7eebc0381e383 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:07 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:08.023149 898 tablet_service.cc:1552] Processing DeleteTablet for tablet 1bc45078a6a942c38ff3350c15885eff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:56 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:08.024314 1172 tablet_bootstrap.cc:654] T 061f3d38c4604fa4b57e74ff6f6fc7da P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:08.028817 522 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_be523b14f1164046a23c062a7907de02" } new_table_name: "default.Table_6e41519351e8439d868fa058f1a820c6" modify_external_catalogs: true
I20251029 02:52:08.030382 1175 ts_tablet_manager.cc:1929] T dd90fb9eb167478cbc3ec9363af86295 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20251029 02:52:08.030539 1175 log.cc:1199] T dd90fb9eb167478cbc3ec9363af86295 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/dd90fb9eb167478cbc3ec9363af86295
I20251029 02:52:08.030939 1175 ts_tablet_manager.cc:1950] T dd90fb9eb167478cbc3ec9363af86295 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:08.033110 1176 ts_tablet_manager.cc:1950] T da2a21c3236a4bd99872b89b12a36508 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:08.033926 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet da2a21c3236a4bd99872b89b12a36508 (table default.Table_2a63af6894694f16b45f4bf48f73a68f [id=fb42af08ec1f48498ee5e91cce62333e]) successfully deleted
I20251029 02:52:08.034340 1176 tablet_replica.cc:333] T 41a6139fd9b84707a8c7eebc0381e383 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:08.034335 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 735) took 2443 ms. Trace:
I20251029 02:52:08.034433 1175 rpcz_store.cc:276] 1029 02:52:05.590828 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:05.596210 (+ 5382us) service_pool.cc:225] Handling call
1029 02:52:07.995135 (+2398925us) ts_tablet_manager.cc:1151] Deleting tablet dd90fb9eb167478cbc3ec9363af86295
1029 02:52:07.995876 (+ 741us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:08.012326 (+ 16450us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:08.030267 (+ 17941us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:08.034312 (+ 4045us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2398144}
I20251029 02:52:08.034842 1176 raft_consensus.cc:2243] T 41a6139fd9b84707a8c7eebc0381e383 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:08.035216 1176 raft_consensus.cc:2272] T 41a6139fd9b84707a8c7eebc0381e383 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:08.035248 2113 raft_consensus.cc:493] T 1275e192871440408a8345fef9b56260 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:08.035382 2113 raft_consensus.cc:515] T 1275e192871440408a8345fef9b56260 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.035921 1176 ts_tablet_manager.cc:1916] T 41a6139fd9b84707a8c7eebc0381e383 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:08.036022 2113 leader_election.cc:290] T 1275e192871440408a8345fef9b56260 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:08.035063 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 738) took 2407 ms. Trace:
I20251029 02:52:08.036531 1175 rpcz_store.cc:276] 1029 02:52:05.628078 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:05.661571 (+ 33493us) service_pool.cc:225] Handling call
1029 02:52:08.034927 (+2373356us) ts_tablet_manager.cc:1151] Deleting tablet 76287edc126a403dbbec4a3b41effe84
1029 02:52:08.034937 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:08.035044 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2373079}
I20251029 02:52:08.036820 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1275e192871440408a8345fef9b56260" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:08.036995 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 742) took 2191 ms. Trace:
I20251029 02:52:08.037067 1175 rpcz_store.cc:276] 1029 02:52:05.845789 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:05.846169 (+ 380us) service_pool.cc:225] Handling call
1029 02:52:08.036886 (+2190717us) ts_tablet_manager.cc:1151] Deleting tablet 8f100a12d78145c2b8586a0713fc70c4
1029 02:52:08.036895 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:08.036981 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2190303}
I20251029 02:52:08.037357 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 743) took 2191 ms. Trace:
W20251029 02:52:08.037431 711 leader_election.cc:343] T 1275e192871440408a8345fef9b56260 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:08.037474 1175 rpcz_store.cc:276] 1029 02:52:05.845980 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:05.854038 (+ 8058us) service_pool.cc:225] Handling call
1029 02:52:08.037250 (+2183212us) ts_tablet_manager.cc:1151] Deleting tablet 72f9d4408f384e7093316d85f092e1b2
1029 02:52:08.037257 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:08.037345 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2150157}
I20251029 02:52:08.036562 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1275e192871440408a8345fef9b56260" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:52:08.038225 522 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 09e3ce70eb9a4a6c9035dabbf7434636 consensus state has no leader
I20251029 02:52:08.037863 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 744) took 2161 ms. Trace:
W20251029 02:52:08.038429 712 leader_election.cc:343] T 1275e192871440408a8345fef9b56260 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:08.039825 712 leader_election.cc:304] T 1275e192871440408a8345fef9b56260 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:08.040230 2113 raft_consensus.cc:2749] T 1275e192871440408a8345fef9b56260 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:08.039768 1175 rpcz_store.cc:276] 1029 02:52:05.876361 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:05.876437 (+ 76us) service_pool.cc:225] Handling call
1029 02:52:08.037775 (+2161338us) ts_tablet_manager.cc:1151] Deleting tablet 143f4a42001b43e683bbc6226c0550a9
1029 02:52:08.037783 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:08.037850 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2150218}
I20251029 02:52:08.043983 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 745) took 2094 ms. Trace:
I20251029 02:52:08.043953 522 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_8746e5337ced40038d965fae5bcb4acd" } new_table_name: "default.Table_ec3a665b859f46abb37e1e8675867d13" modify_external_catalogs: true
I20251029 02:52:08.044063 1175 rpcz_store.cc:276] 1029 02:52:05.949267 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:05.949334 (+ 67us) service_pool.cc:225] Handling call
1029 02:52:08.043867 (+2094533us) ts_tablet_manager.cc:1151] Deleting tablet ad214c870ee649279ed7321c5b69cbee
1029 02:52:08.043878 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:08.043967 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2094302}
I20251029 02:52:08.044415 1175 tablet_replica.cc:333] T 59af906061d049c99ae2153945597007 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:08.044580 1175 raft_consensus.cc:2243] T 59af906061d049c99ae2153945597007 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:08.044754 1175 raft_consensus.cc:2272] T 59af906061d049c99ae2153945597007 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:08.045260 1175 ts_tablet_manager.cc:1916] T 59af906061d049c99ae2153945597007 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 02:52:08.050352 522 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7bcb1cb3cb0949edbfd96cd7f6b11f1c consensus state has no leader
I20251029 02:52:08.051057 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_13f9fb5682214cc28f12dce88109d831"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:08.051472 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_5d956728da054438a1bd400dac5dac1e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:08.052165 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_dbb86af86c304f02ae3a9ece7bb4998b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:08.059432 525 master_service.cc:954] ReplaceTablet: received request to replace tablet 0ffd22d8610b4329a0a4f753d9e60149 from {username='slave'} at 127.0.0.1:35314
I20251029 02:52:08.067873 1175 ts_tablet_manager.cc:1929] T 59af906061d049c99ae2153945597007 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251029 02:52:08.068419 1175 log.cc:1199] T 59af906061d049c99ae2153945597007 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/59af906061d049c99ae2153945597007
I20251029 02:52:08.069844 1175 ts_tablet_manager.cc:1950] T 59af906061d049c99ae2153945597007 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:08.073657 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 746) took 1952 ms. Trace:
I20251029 02:52:08.074280 1167 tablet_bootstrap.cc:654] T 785733da86ff425a90ccfab2933fc89e P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:08.074054 1175 rpcz_store.cc:276] 1029 02:52:06.121502 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:06.121561 (+ 59us) service_pool.cc:225] Handling call
1029 02:52:08.044345 (+1922784us) ts_tablet_manager.cc:1151] Deleting tablet 59af906061d049c99ae2153945597007
1029 02:52:08.044381 (+ 36us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:08.055155 (+ 10774us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:08.067468 (+ 12313us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:08.073541 (+ 6073us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1922577}
I20251029 02:52:08.075466 1175 tablet_replica.cc:333] T dfa8b1b9a80e4e8689e0acb44e16346a P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:08.036681 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet dd90fb9eb167478cbc3ec9363af86295 (table default.Table_8b6217e43e8d467693c104ef84e78911 [id=052abf3dcc904bdcb869d5b885723dab]) successfully deleted
W20251029 02:52:08.076462 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dbb86af86c304f02ae3a9ece7bb4998b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:52:08.076457 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_13f9fb5682214cc28f12dce88109d831 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:52:08.077008 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 8f100a12d78145c2b8586a0713fc70c4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8f100a12d78145c2b8586a0713fc70c4 already in progress: creating tablet
W20251029 02:52:08.077260 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 72f9d4408f384e7093316d85f092e1b2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 72f9d4408f384e7093316d85f092e1b2 already in progress: creating tablet
W20251029 02:52:08.077461 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 143f4a42001b43e683bbc6226c0550a9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 143f4a42001b43e683bbc6226c0550a9 already in progress: creating tablet
W20251029 02:52:08.077690 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet ad214c870ee649279ed7321c5b69cbee because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ad214c870ee649279ed7321c5b69cbee already in progress: creating tablet
W20251029 02:52:08.077855 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5d956728da054438a1bd400dac5dac1e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:08.077880 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 59af906061d049c99ae2153945597007 (table default.Table_6da557988d044647b5e68f09b3f00407 [id=25f16d95843d41c981d6a07ec0d4b48a]) successfully deleted
I20251029 02:52:08.076299 1175 raft_consensus.cc:2243] T dfa8b1b9a80e4e8689e0acb44e16346a P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:08.091555 1175 raft_consensus.cc:2272] T dfa8b1b9a80e4e8689e0acb44e16346a P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:08.092263 1166 tablet_bootstrap.cc:654] T ccdca31db2834de4b6ed0f2d67a10535 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:08.092311 1175 ts_tablet_manager.cc:1916] T dfa8b1b9a80e4e8689e0acb44e16346a P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:08.102339 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_1c456ca0e187431baa7d2054b49ac76c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:08.103302 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_e9a962d143c94976af9285475ea40631"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:08.109526 2113 raft_consensus.cc:493] T 30d03e1e42d34242bd354f9e9ec4f127 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:08.109694 2113 raft_consensus.cc:515] T 30d03e1e42d34242bd354f9e9ec4f127 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.110296 2113 leader_election.cc:290] T 30d03e1e42d34242bd354f9e9ec4f127 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
W20251029 02:52:08.110602 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e9a962d143c94976af9285475ea40631 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:08.110661 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_3bc0c202b81d42c5a788393fe07d02a1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:08.110962 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30d03e1e42d34242bd354f9e9ec4f127" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:08.111343 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3bc0c202b81d42c5a788393fe07d02a1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:52:08.111763 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1c456ca0e187431baa7d2054b49ac76c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:08.111781 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30d03e1e42d34242bd354f9e9ec4f127" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:52:08.113068 711 leader_election.cc:343] T 30d03e1e42d34242bd354f9e9ec4f127 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
W20251029 02:52:08.113273 712 leader_election.cc:343] T 30d03e1e42d34242bd354f9e9ec4f127 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:08.113344 712 leader_election.cc:304] T 30d03e1e42d34242bd354f9e9ec4f127 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:08.113858 2113 raft_consensus.cc:2749] T 30d03e1e42d34242bd354f9e9ec4f127 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:08.114449 1176 ts_tablet_manager.cc:1929] T 41a6139fd9b84707a8c7eebc0381e383 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251029 02:52:08.114643 1176 log.cc:1199] T 41a6139fd9b84707a8c7eebc0381e383 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/41a6139fd9b84707a8c7eebc0381e383
I20251029 02:52:08.115048 1176 ts_tablet_manager.cc:1950] T 41a6139fd9b84707a8c7eebc0381e383 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:08.117893 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 41a6139fd9b84707a8c7eebc0381e383 (table default.Table_e48aadc66a0e48658d45d0812c29cbdf [id=f45315b686594867aae0114f4ce2f26c]) successfully deleted
I20251029 02:52:08.119103 1167 tablet_bootstrap.cc:492] T 785733da86ff425a90ccfab2933fc89e P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:08.119292 1167 ts_tablet_manager.cc:1403] T 785733da86ff425a90ccfab2933fc89e P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.102s user 0.005s sys 0.000s
I20251029 02:52:08.120182 1167 raft_consensus.cc:359] T 785733da86ff425a90ccfab2933fc89e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.120463 1167 raft_consensus.cc:385] T 785733da86ff425a90ccfab2933fc89e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:08.120568 1167 raft_consensus.cc:740] T 785733da86ff425a90ccfab2933fc89e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:08.120864 1167 consensus_queue.cc:260] T 785733da86ff425a90ccfab2933fc89e P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.121052 1172 tablet_bootstrap.cc:492] T 061f3d38c4604fa4b57e74ff6f6fc7da P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:08.121754 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_9af081e1d784401792ad8964bdad2e73"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:08.122342 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9af081e1d784401792ad8964bdad2e73 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:08.123236 1172 ts_tablet_manager.cc:1403] T 061f3d38c4604fa4b57e74ff6f6fc7da P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.109s user 0.023s sys 0.015s
I20251029 02:52:08.124059 1172 raft_consensus.cc:359] T 061f3d38c4604fa4b57e74ff6f6fc7da P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.124221 1172 raft_consensus.cc:385] T 061f3d38c4604fa4b57e74ff6f6fc7da P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:08.124272 1172 raft_consensus.cc:740] T 061f3d38c4604fa4b57e74ff6f6fc7da P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:08.124490 1172 consensus_queue.cc:260] T 061f3d38c4604fa4b57e74ff6f6fc7da P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.124998 1172 ts_tablet_manager.cc:1434] T 061f3d38c4604fa4b57e74ff6f6fc7da P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251029 02:52:08.125228 1172 tablet_bootstrap.cc:492] T e7f846b294cc4eb687a26c8b5931ac19 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:08.126976 1172 tablet_bootstrap.cc:654] T e7f846b294cc4eb687a26c8b5931ac19 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:08.127646 2103 raft_consensus.cc:493] T 0f80b840ed954230bfbfe7e40127cd2f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:08.127775 2103 raft_consensus.cc:515] T 0f80b840ed954230bfbfe7e40127cd2f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.128345 2103 leader_election.cc:290] T 0f80b840ed954230bfbfe7e40127cd2f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:08.129107 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f80b840ed954230bfbfe7e40127cd2f" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:08.130975 990 leader_election.cc:343] T 0f80b840ed954230bfbfe7e40127cd2f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:08.131985 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f80b840ed954230bfbfe7e40127cd2f" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:08.132231 768 raft_consensus.cc:2468] T 0f80b840ed954230bfbfe7e40127cd2f P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 0.
I20251029 02:52:08.132810 989 leader_election.cc:304] T 0f80b840ed954230bfbfe7e40127cd2f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters: c756528642a543eeacd93cef3e335a7d
I20251029 02:52:08.133239 1973 raft_consensus.cc:2804] T 0f80b840ed954230bfbfe7e40127cd2f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:08.133311 1973 raft_consensus.cc:493] T 0f80b840ed954230bfbfe7e40127cd2f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:08.133370 1973 raft_consensus.cc:3060] T 0f80b840ed954230bfbfe7e40127cd2f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:08.136289 1973 raft_consensus.cc:515] T 0f80b840ed954230bfbfe7e40127cd2f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.137975 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f80b840ed954230bfbfe7e40127cd2f" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
W20251029 02:52:08.138731 990 leader_election.cc:343] T 0f80b840ed954230bfbfe7e40127cd2f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:08.139915 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0f80b840ed954230bfbfe7e40127cd2f" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:08.140089 768 raft_consensus.cc:3060] T 0f80b840ed954230bfbfe7e40127cd2f P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:08.143227 1973 leader_election.cc:290] T 0f80b840ed954230bfbfe7e40127cd2f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Requested vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:08.156752 768 raft_consensus.cc:2468] T 0f80b840ed954230bfbfe7e40127cd2f P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 1.
I20251029 02:52:08.157303 989 leader_election.cc:304] T 0f80b840ed954230bfbfe7e40127cd2f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters: c756528642a543eeacd93cef3e335a7d
I20251029 02:52:08.158322 1973 raft_consensus.cc:2804] T 0f80b840ed954230bfbfe7e40127cd2f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:08.158344 1175 ts_tablet_manager.cc:1929] T dfa8b1b9a80e4e8689e0acb44e16346a P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:08.158780 1973 raft_consensus.cc:697] T 0f80b840ed954230bfbfe7e40127cd2f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Becoming Leader. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Running, Role: LEADER
I20251029 02:52:08.159107 1973 consensus_queue.cc:237] T 0f80b840ed954230bfbfe7e40127cd2f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.163213 1166 tablet_bootstrap.cc:492] T ccdca31db2834de4b6ed0f2d67a10535 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:08.163403 1166 ts_tablet_manager.cc:1403] T ccdca31db2834de4b6ed0f2d67a10535 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.186s user 0.021s sys 0.017s
I20251029 02:52:08.163990 520 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_8e43b2bad80b45ceb71aada04cdbc4b8" } new_table_name: "default.Table_902ea0b243c7400c86250cfb5fae5b54" modify_external_catalogs: true
I20251029 02:52:08.164100 1166 raft_consensus.cc:359] T ccdca31db2834de4b6ed0f2d67a10535 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.164252 1166 raft_consensus.cc:385] T ccdca31db2834de4b6ed0f2d67a10535 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:08.164306 1166 raft_consensus.cc:740] T ccdca31db2834de4b6ed0f2d67a10535 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:08.164490 1166 consensus_queue.cc:260] T ccdca31db2834de4b6ed0f2d67a10535 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.165199 1166 ts_tablet_manager.cc:1434] T ccdca31db2834de4b6ed0f2d67a10535 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 02:52:08.165469 1166 tablet_bootstrap.cc:492] T 5db519edc75d4bd0985b32322825ffae P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:08.166293 1167 ts_tablet_manager.cc:1434] T 785733da86ff425a90ccfab2933fc89e P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.047s user 0.002s sys 0.000s
I20251029 02:52:08.166911 1166 tablet_bootstrap.cc:654] T 5db519edc75d4bd0985b32322825ffae P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:08.170044 1167 tablet_bootstrap.cc:492] T c764062d3ba04d4ba8b2e367ad8ed433 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:08.171806 523 master_service.cc:954] ReplaceTablet: received request to replace tablet 91e31f0b06924740bdf7525cae79d8ef from {username='slave'} at 127.0.0.1:35314
I20251029 02:52:08.171850 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_9c6ff90fbbd9431c9ae4dbf96e02b307"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:08.172397 523 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_7cfcc5277f9f4b1d9bc979e2b218efaf" } new_table_name: "default.Table_81009c7f23bd483fac10943bb273ae87" modify_external_catalogs: true
W20251029 02:52:08.172533 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9c6ff90fbbd9431c9ae4dbf96e02b307 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:52:08.176411 849 consensus_peers.cc:597] T 4bf06476c87a4549a73f5994314bb75f P c756528642a543eeacd93cef3e335a7d -> Peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Couldn't send request to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4bf06476c87a4549a73f5994314bb75f. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:08.177810 898 tablet_service.cc:1552] Processing DeleteTablet for tablet 785733da86ff425a90ccfab2933fc89e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:54 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:08.178925 1174 ts_tablet_manager.cc:1929] T 6ca9bd37de9e4be388501b15af4bda31 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:08.182139 1166 tablet_bootstrap.cc:492] T 5db519edc75d4bd0985b32322825ffae P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:08.182289 1166 ts_tablet_manager.cc:1403] T 5db519edc75d4bd0985b32322825ffae P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.003s
I20251029 02:52:08.183086 1166 raft_consensus.cc:359] T 5db519edc75d4bd0985b32322825ffae P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.183250 1166 raft_consensus.cc:385] T 5db519edc75d4bd0985b32322825ffae P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:08.183295 1166 raft_consensus.cc:740] T 5db519edc75d4bd0985b32322825ffae P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:08.183591 1166 consensus_queue.cc:260] T 5db519edc75d4bd0985b32322825ffae P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.184194 1166 ts_tablet_manager.cc:1434] T 5db519edc75d4bd0985b32322825ffae P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 02:52:08.184476 1166 tablet_bootstrap.cc:492] T 777992acb1bd4765886b802940db74bb P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:08.185897 1166 tablet_bootstrap.cc:654] T 777992acb1bd4765886b802940db74bb P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:08.181185 1174 log.cc:1199] T 6ca9bd37de9e4be388501b15af4bda31 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/6ca9bd37de9e4be388501b15af4bda31
W20251029 02:52:08.188120 523 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 688af6580c4d4637a39f817b4e324618 consensus state has no leader
I20251029 02:52:08.191393 523 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_a6472497e2e940c3b6a79f15665aa636" } new_table_name: "default.Table_a9e50760813844af85a85c5f8d22f8e0" modify_external_catalogs: true
I20251029 02:52:08.195873 1166 tablet_bootstrap.cc:492] T 777992acb1bd4765886b802940db74bb P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:08.196043 1166 ts_tablet_manager.cc:1403] T 777992acb1bd4765886b802940db74bb P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.002s
I20251029 02:52:08.158497 1175 log.cc:1199] T dfa8b1b9a80e4e8689e0acb44e16346a P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/dfa8b1b9a80e4e8689e0acb44e16346a
I20251029 02:52:08.196792 1166 raft_consensus.cc:359] T 777992acb1bd4765886b802940db74bb P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.196961 1166 raft_consensus.cc:385] T 777992acb1bd4765886b802940db74bb P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:08.173648 2113 raft_consensus.cc:493] T 061f3d38c4604fa4b57e74ff6f6fc7da P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:08.197017 1166 raft_consensus.cc:740] T 777992acb1bd4765886b802940db74bb P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:08.197144 1175 ts_tablet_manager.cc:1950] T dfa8b1b9a80e4e8689e0acb44e16346a P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:08.197103 2113 raft_consensus.cc:515] T 061f3d38c4604fa4b57e74ff6f6fc7da P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.197276 1166 consensus_queue.cc:260] T 777992acb1bd4765886b802940db74bb P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.197688 2113 leader_election.cc:290] T 061f3d38c4604fa4b57e74ff6f6fc7da P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:08.197898 1166 ts_tablet_manager.cc:1434] T 777992acb1bd4765886b802940db74bb P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 02:52:08.198164 1166 tablet_bootstrap.cc:492] T 44b518c3a0964a6fa863b2784d80c8a4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:08.198494 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "061f3d38c4604fa4b57e74ff6f6fc7da" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:08.199250 711 leader_election.cc:343] T 061f3d38c4604fa4b57e74ff6f6fc7da P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:08.199853 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 747) took 1915 ms. Trace:
I20251029 02:52:08.199957 1175 rpcz_store.cc:276] 1029 02:52:06.284051 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:06.284580 (+ 529us) service_pool.cc:225] Handling call
1029 02:52:08.075312 (+1790732us) ts_tablet_manager.cc:1151] Deleting tablet dfa8b1b9a80e4e8689e0acb44e16346a
1029 02:52:08.075325 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:08.153788 (+ 78463us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:08.158208 (+ 4420us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:08.199831 (+ 41623us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1790373}
I20251029 02:52:08.200439 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet dfa8b1b9a80e4e8689e0acb44e16346a (table default.Table_3b4bdfaf4ddf43cfa2d2287594c57816 [id=222ef1b283554565b4930328c2c8975b]) successfully deleted
I20251029 02:52:08.200642 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 749) took 1871 ms. Trace:
I20251029 02:52:08.200726 1175 rpcz_store.cc:276] 1029 02:52:06.329128 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:06.329181 (+ 53us) service_pool.cc:225] Handling call
1029 02:52:08.200518 (+1871337us) ts_tablet_manager.cc:1151] Deleting tablet 00f2e98739fa41eea23927494fe222f3
1029 02:52:08.200530 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:08.200625 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1871120}
I20251029 02:52:08.201035 1174 ts_tablet_manager.cc:1950] T 6ca9bd37de9e4be388501b15af4bda31 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
W20251029 02:52:08.201125 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 00f2e98739fa41eea23927494fe222f3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 00f2e98739fa41eea23927494fe222f3 already in progress: creating tablet
I20251029 02:52:08.202454 1167 tablet_bootstrap.cc:654] T c764062d3ba04d4ba8b2e367ad8ed433 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:08.203176 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_8a944a4e396a4f7cb27c0b3c01ab3ef2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:08.203835 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8a944a4e396a4f7cb27c0b3c01ab3ef2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:08.206962 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "061f3d38c4604fa4b57e74ff6f6fc7da" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:52:08.207682 712 leader_election.cc:343] T 061f3d38c4604fa4b57e74ff6f6fc7da P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:08.207825 712 leader_election.cc:304] T 061f3d38c4604fa4b57e74ff6f6fc7da P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:08.208273 2113 raft_consensus.cc:2749] T 061f3d38c4604fa4b57e74ff6f6fc7da P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:08.213626 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 755) took 1750 ms. Trace:
I20251029 02:52:08.213755 1175 rpcz_store.cc:276] 1029 02:52:06.463040 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:06.463115 (+ 75us) service_pool.cc:225] Handling call
1029 02:52:08.213473 (+1750358us) ts_tablet_manager.cc:1151] Deleting tablet 1c24459f58a6414f94c62fbd48927c11
1029 02:52:08.213487 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:08.213602 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1750140}
I20251029 02:52:08.214210 1166 tablet_bootstrap.cc:654] T 44b518c3a0964a6fa863b2784d80c8a4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:08.214197 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 756) took 1722 ms. Trace:
W20251029 02:52:08.214272 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 1c24459f58a6414f94c62fbd48927c11 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1c24459f58a6414f94c62fbd48927c11 already in progress: creating tablet
I20251029 02:52:08.214272 1175 rpcz_store.cc:276] 1029 02:52:06.491453 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:06.491774 (+ 321us) service_pool.cc:225] Handling call
1029 02:52:08.214078 (+1722304us) ts_tablet_manager.cc:1151] Deleting tablet 4d51120775424146b6ab3e6d2e4710bf
1029 02:52:08.214090 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:08.214180 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1722042}
W20251029 02:52:08.214594 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 4d51120775424146b6ab3e6d2e4710bf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4d51120775424146b6ab3e6d2e4710bf already in progress: creating tablet
I20251029 02:52:08.214597 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 757) took 1723 ms. Trace:
I20251029 02:52:08.214673 1175 rpcz_store.cc:276] 1029 02:52:06.491606 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:06.492092 (+ 486us) service_pool.cc:225] Handling call
1029 02:52:08.214479 (+1722387us) ts_tablet_manager.cc:1151] Deleting tablet 8048506ced4540a5aee9f79bfc4599bf
1029 02:52:08.214488 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:08.214580 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1722219}
I20251029 02:52:08.214993 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 758) took 1723 ms. Trace:
W20251029 02:52:08.215065 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 8048506ced4540a5aee9f79bfc4599bf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8048506ced4540a5aee9f79bfc4599bf already in progress: creating tablet
I20251029 02:52:08.215073 1175 rpcz_store.cc:276] 1029 02:52:06.491712 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:06.492417 (+ 705us) service_pool.cc:225] Handling call
1029 02:52:08.214905 (+1722488us) ts_tablet_manager.cc:1151] Deleting tablet b0fec06ad37d4c12a7045d424b88cc5e
1029 02:52:08.214915 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:08.214981 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1722305}
W20251029 02:52:08.215381 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet b0fec06ad37d4c12a7045d424b88cc5e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b0fec06ad37d4c12a7045d424b88cc5e already in progress: creating tablet
I20251029 02:52:08.220404 1175 tablet_replica.cc:333] T f529e17e97074dca91ec2f48954cfd32 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:08.220624 1175 raft_consensus.cc:2243] T f529e17e97074dca91ec2f48954cfd32 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:08.220767 1175 raft_consensus.cc:2272] T f529e17e97074dca91ec2f48954cfd32 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:08.221256 1175 ts_tablet_manager.cc:1916] T f529e17e97074dca91ec2f48954cfd32 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:08.225291 522 master_service.cc:954] ReplaceTablet: received request to replace tablet 9bb2166a590d4210b02dd779ef6654ad from {username='slave'} at 127.0.0.1:35314
I20251029 02:52:08.225945 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_53892f35056440d89815e7f4997f08d2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:08.226557 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_53892f35056440d89815e7f4997f08d2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:52:08.226047 523 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 27b08f88b99643d28ac609e45bf8104e consensus state has no leader
I20251029 02:52:08.229722 523 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_3ca4a6512b174cb897b142dfe1259549" } modify_external_catalogs: true
I20251029 02:52:08.229921 523 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_3ca4a6512b174cb897b142dfe1259549" } modify_external_catalogs: true
I20251029 02:52:08.232314 1973 raft_consensus.cc:493] T ccdca31db2834de4b6ed0f2d67a10535 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:08.232553 1973 raft_consensus.cc:515] T ccdca31db2834de4b6ed0f2d67a10535 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.233954 1973 leader_election.cc:290] T ccdca31db2834de4b6ed0f2d67a10535 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:08.235139 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ccdca31db2834de4b6ed0f2d67a10535" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:08.235423 768 raft_consensus.cc:2468] T ccdca31db2834de4b6ed0f2d67a10535 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 0.
I20251029 02:52:08.235932 989 leader_election.cc:304] T ccdca31db2834de4b6ed0f2d67a10535 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters:
I20251029 02:52:08.236153 522 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_a37ba50ba83a43668c7fe1af7cca7f0d" } modify_external_catalogs: true
I20251029 02:52:08.236330 522 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_a37ba50ba83a43668c7fe1af7cca7f0d" } modify_external_catalogs: true
I20251029 02:52:08.239799 1174 tablet_replica.cc:333] T 4aa78f4ac38e4919864c924e50f1aaf4 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:08.241106 1973 raft_consensus.cc:2804] T ccdca31db2834de4b6ed0f2d67a10535 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:08.241254 1973 raft_consensus.cc:493] T ccdca31db2834de4b6ed0f2d67a10535 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:08.241381 1973 raft_consensus.cc:3060] T ccdca31db2834de4b6ed0f2d67a10535 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:08.241681 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 6ca9bd37de9e4be388501b15af4bda31 (table default.Table_3ca4a6512b174cb897b142dfe1259549 [id=a862192912a04942a8f682fc9825b014]) successfully deleted
I20251029 02:52:08.241298 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ccdca31db2834de4b6ed0f2d67a10535" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:08.243273 990 leader_election.cc:343] T ccdca31db2834de4b6ed0f2d67a10535 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:08.244176 1174 raft_consensus.cc:2243] T 4aa78f4ac38e4919864c924e50f1aaf4 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:08.244134 1973 raft_consensus.cc:515] T ccdca31db2834de4b6ed0f2d67a10535 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.244418 1174 raft_consensus.cc:2272] T 4aa78f4ac38e4919864c924e50f1aaf4 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:08.245111 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ccdca31db2834de4b6ed0f2d67a10535" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
I20251029 02:52:08.245575 1174 ts_tablet_manager.cc:1916] T 4aa78f4ac38e4919864c924e50f1aaf4 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 02:52:08.245708 990 leader_election.cc:343] T ccdca31db2834de4b6ed0f2d67a10535 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:08.246065 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ccdca31db2834de4b6ed0f2d67a10535" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:08.246294 768 raft_consensus.cc:3060] T ccdca31db2834de4b6ed0f2d67a10535 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:08.249084 768 raft_consensus.cc:2468] T ccdca31db2834de4b6ed0f2d67a10535 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 1.
I20251029 02:52:08.249555 989 leader_election.cc:304] T ccdca31db2834de4b6ed0f2d67a10535 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters: c756528642a543eeacd93cef3e335a7d
I20251029 02:52:08.249871 2036 raft_consensus.cc:2804] T ccdca31db2834de4b6ed0f2d67a10535 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:08.249987 2036 raft_consensus.cc:697] T ccdca31db2834de4b6ed0f2d67a10535 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Becoming Leader. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Running, Role: LEADER
I20251029 02:52:08.250234 2036 consensus_queue.cc:237] T ccdca31db2834de4b6ed0f2d67a10535 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.250984 519 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet 9269622ca92645d5aa8e73341201ccbb (table default.Table_8e43b2bad80b45ceb71aada04cdbc4b8 [id=d0ca7c93c8cf4dc19962a1957321c5b7]). Expected version 4 got 0
W20251029 02:52:08.251521 520 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9269622ca92645d5aa8e73341201ccbb consensus state has no leader
I20251029 02:52:08.251943 520 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_94ed5141b07a4cd7b94c14174c30ed99" } new_table_name: "default.Table_316affb3c558411eb1aa14af326a27c2" modify_external_catalogs: true
I20251029 02:52:08.256280 519 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_4d82d8b113274cf1841f2141772815e0" } new_table_name: "default.Table_abdff89e238347a69b825355bd66f2cd" modify_external_catalogs: true
I20251029 02:52:08.260355 1166 tablet_bootstrap.cc:492] T 44b518c3a0964a6fa863b2784d80c8a4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:08.260504 1166 ts_tablet_manager.cc:1403] T 44b518c3a0964a6fa863b2784d80c8a4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.062s user 0.003s sys 0.000s
I20251029 02:52:08.261299 1166 raft_consensus.cc:359] T 44b518c3a0964a6fa863b2784d80c8a4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.261492 1166 raft_consensus.cc:385] T 44b518c3a0964a6fa863b2784d80c8a4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:08.261551 1166 raft_consensus.cc:740] T 44b518c3a0964a6fa863b2784d80c8a4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:08.261770 1166 consensus_queue.cc:260] T 44b518c3a0964a6fa863b2784d80c8a4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.262439 1973 leader_election.cc:290] T ccdca31db2834de4b6ed0f2d67a10535 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Requested vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:08.263828 1166 ts_tablet_manager.cc:1434] T 44b518c3a0964a6fa863b2784d80c8a4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251029 02:52:08.264303 517 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet 1c53fc1fff1a4bb988beb46ff91fd3be (table default.Table_dd4934aabcd442039db111954d819f6d [id=9abdb27ccc0a48e991b7a4619808fdf8]). Expected version 1 got 0
I20251029 02:52:08.264541 517 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet 02e4588b03144ea2850e13a1f8f29f06 (table default.Table_0772f9db5d6142d988f6ffca2e96bdd2 [id=fd6d31d379394857a35422211af00642]). Expected version 3 got 0
I20251029 02:52:08.264679 1166 tablet_bootstrap.cc:492] T 785733da86ff425a90ccfab2933fc89e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:08.264780 517 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet cc74e83833754f0ca19046031a24ebb9 (table default.Table_755825b17834457cb94b05163ae3e26d [id=57471769276f44299eae50b9d854b50c]). Expected version 1 got 0
I20251029 02:52:08.265079 517 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet d5741dde463a43d6a776ab92dcfb04bf (table default.Table_966779a0d37a4dc4ab3f89d2bf7c6b06 [id=08e3b2ebcf37415d9a2886ad5bbda068]). Expected version 1 got 0
I20251029 02:52:08.270143 525 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 0ffd22d8610b4329a0a4f753d9e60149
I20251029 02:52:08.270860 525 catalog_manager.cc:6719] ReplaceTablet: tablet 0ffd22d8610b4329a0a4f753d9e60149 deleted and replaced by tablet 7fba607c566c4fa1a803cba8d86ca646
I20251029 02:52:08.270892 526 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 4372eacfc07246f3be8f7c146674a736
I20251029 02:52:08.271430 526 master_service.cc:954] ReplaceTablet: received request to replace tablet d2990e9f626c40618c118266bc595684 from {username='slave'} at 127.0.0.1:35314
I20251029 02:52:08.271744 518 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 50ab70b30aaa4f6bbae15855431159c8
I20251029 02:52:08.272249 900 tablet_service.cc:1552] Processing DeleteTablet for tablet 4372eacfc07246f3be8f7c146674a736 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:07 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:08.272322 898 tablet_service.cc:1552] Processing DeleteTablet for tablet 0ffd22d8610b4329a0a4f753d9e60149 with delete_type TABLET_DATA_DELETED (replaced by tablet 7fba607c566c4fa1a803cba8d86ca646) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:08.273469 525 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet 9269622ca92645d5aa8e73341201ccbb (table default.Table_902ea0b243c7400c86250cfb5fae5b54 [id=d0ca7c93c8cf4dc19962a1957321c5b7]). Expected version 5 got 0
I20251029 02:52:08.273646 523 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 20a762344c5f46f88481c4ac7ad6e43b
I20251029 02:52:08.275416 755 tablet_service.cc:1552] Processing DeleteTablet for tablet 0ffd22d8610b4329a0a4f753d9e60149 with delete_type TABLET_DATA_DELETED (replaced by tablet 7fba607c566c4fa1a803cba8d86ca646) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:08.275725 749 tablet_service.cc:1552] Processing DeleteTablet for tablet 4372eacfc07246f3be8f7c146674a736 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:07 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:08.275904 750 tablet_service.cc:1552] Processing DeleteTablet for tablet 20a762344c5f46f88481c4ac7ad6e43b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:08 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:08.276281 750 tablet_service.cc:1552] Processing DeleteTablet for tablet 738046ab57334f2dab9f4aab979c5b42 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:04 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:08.276427 749 tablet_service.cc:1552] Processing DeleteTablet for tablet 061f3d38c4604fa4b57e74ff6f6fc7da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:58 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:08.276468 756 tablet_service.cc:1552] Processing DeleteTablet for tablet 50ab70b30aaa4f6bbae15855431159c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:07 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:08.276700 756 tablet_service.cc:1552] Processing DeleteTablet for tablet 30d03e1e42d34242bd354f9e9ec4f127 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:58 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:08.276750 1025 tablet_service.cc:1552] Processing DeleteTablet for tablet 0ffd22d8610b4329a0a4f753d9e60149 with delete_type TABLET_DATA_DELETED (replaced by tablet 7fba607c566c4fa1a803cba8d86ca646) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:08.276984 756 tablet_service.cc:1552] Processing DeleteTablet for tablet 6d830d30b26b4ba799785fec8a6f0769 with delete_type TABLET_DATA_DELETED (replaced by tablet abf18e06df9b482a8b25c078b9495272) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:08.277000 1039 tablet_service.cc:1552] Processing DeleteTablet for tablet 4372eacfc07246f3be8f7c146674a736 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:07 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:08.277182 1031 tablet_service.cc:1552] Processing DeleteTablet for tablet 50ab70b30aaa4f6bbae15855431159c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:07 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:08.277333 1032 tablet_service.cc:1552] Processing DeleteTablet for tablet 20a762344c5f46f88481c4ac7ad6e43b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:08 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:08.279806 900 tablet_service.cc:1552] Processing DeleteTablet for tablet 20a762344c5f46f88481c4ac7ad6e43b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:08 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:08.280068 898 tablet_service.cc:1552] Processing DeleteTablet for tablet 50ab70b30aaa4f6bbae15855431159c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:07 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:08.281549 1032 tablet_service.cc:1552] Processing DeleteTablet for tablet b3de6670357c4ae59eb77469b2fb434c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:01 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:08.282248 1032 tablet_service.cc:1552] Processing DeleteTablet for tablet 9e6eb099648549dbbf9969e59865f51f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:07 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:08.283221 522 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 48aa20640dfe43288326ed6d6b9bc1cf
I20251029 02:52:08.283862 1032 tablet_service.cc:1552] Processing DeleteTablet for tablet 48aa20640dfe43288326ed6d6b9bc1cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:08 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:08.284219 756 tablet_service.cc:1552] Processing DeleteTablet for tablet 48aa20640dfe43288326ed6d6b9bc1cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:08 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:08.281602 1166 tablet_bootstrap.cc:654] T 785733da86ff425a90ccfab2933fc89e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:08.284400 898 tablet_service.cc:1552] Processing DeleteTablet for tablet 48aa20640dfe43288326ed6d6b9bc1cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:08 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:08.287400 521 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 65e761f9f9fb490586d6415a5a8565dd
W20251029 02:52:08.287659 520 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4e89d72dd274498783dc815f6f2ee031 consensus state has no leader
I20251029 02:52:08.287950 521 catalog_manager.cc:6719] ReplaceTablet: tablet 65e761f9f9fb490586d6415a5a8565dd deleted and replaced by tablet 8de7380e93dd4a9abdb84522a18abda6
I20251029 02:52:08.288422 756 tablet_service.cc:1552] Processing DeleteTablet for tablet 65e761f9f9fb490586d6415a5a8565dd with delete_type TABLET_DATA_DELETED (replaced by tablet 8de7380e93dd4a9abdb84522a18abda6) from {username='slave'} at 127.0.0.1:44580
W20251029 02:52:08.288734 519 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c7a3e6d8e3fb45898d8a5e094dbf0fd7 consensus state has no leader
I20251029 02:52:08.289172 1032 tablet_service.cc:1552] Processing DeleteTablet for tablet 65e761f9f9fb490586d6415a5a8565dd with delete_type TABLET_DATA_DELETED (replaced by tablet 8de7380e93dd4a9abdb84522a18abda6) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:08.287959 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_d76042aac8cb46c38a05fdf710a406cc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:08.289824 898 tablet_service.cc:1552] Processing DeleteTablet for tablet 65e761f9f9fb490586d6415a5a8565dd with delete_type TABLET_DATA_DELETED (replaced by tablet 8de7380e93dd4a9abdb84522a18abda6) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:08.290052 1973 raft_consensus.cc:493] T 86fc22bb84794149b598b97df41eb881 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251029 02:52:08.290117 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d76042aac8cb46c38a05fdf710a406cc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:08.290190 1973 raft_consensus.cc:515] T 86fc22bb84794149b598b97df41eb881 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.287400 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_482767bf59bf442a99a8675f678afd06"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:08.290831 1973 leader_election.cc:290] T 86fc22bb84794149b598b97df41eb881 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
W20251029 02:52:08.291213 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_482767bf59bf442a99a8675f678afd06 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:52:08.292111 989 leader_election.cc:343] T 86fc22bb84794149b598b97df41eb881 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 86fc22bb84794149b598b97df41eb881
I20251029 02:52:08.293020 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "86fc22bb84794149b598b97df41eb881" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:08.293228 916 raft_consensus.cc:2468] T 86fc22bb84794149b598b97df41eb881 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 0.
I20251029 02:52:08.293735 990 leader_election.cc:304] T 86fc22bb84794149b598b97df41eb881 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters: b495a49349324f7d81908f2da700f0d9
I20251029 02:52:08.294209 1175 ts_tablet_manager.cc:1929] T f529e17e97074dca91ec2f48954cfd32 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:08.294237 1973 raft_consensus.cc:2804] T 86fc22bb84794149b598b97df41eb881 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:08.294304 1973 raft_consensus.cc:493] T 86fc22bb84794149b598b97df41eb881 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:08.294332 1175 log.cc:1199] T f529e17e97074dca91ec2f48954cfd32 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/f529e17e97074dca91ec2f48954cfd32
I20251029 02:52:08.294371 1973 raft_consensus.cc:3060] T 86fc22bb84794149b598b97df41eb881 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:08.294664 1175 ts_tablet_manager.cc:1950] T f529e17e97074dca91ec2f48954cfd32 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:08.295476 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 760) took 1747 ms. Trace:
I20251029 02:52:08.295558 1175 rpcz_store.cc:276] 1029 02:52:06.547545 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:06.552324 (+ 4779us) service_pool.cc:225] Handling call
1029 02:52:08.220343 (+1668019us) ts_tablet_manager.cc:1151] Deleting tablet f529e17e97074dca91ec2f48954cfd32
1029 02:52:08.220350 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:08.239674 (+ 19324us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:08.294083 (+ 54409us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:08.295453 (+ 1370us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1667787}
I20251029 02:52:08.296201 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet f529e17e97074dca91ec2f48954cfd32 (table default.Table_570f159e617548ec974b41a654c1d800 [id=835aa84a7f8d426d9560b819e6864715]) successfully deleted
I20251029 02:52:08.297991 1166 tablet_bootstrap.cc:492] T 785733da86ff425a90ccfab2933fc89e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:08.298141 1166 ts_tablet_manager.cc:1403] T 785733da86ff425a90ccfab2933fc89e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.034s user 0.002s sys 0.002s
I20251029 02:52:08.298369 1973 raft_consensus.cc:515] T 86fc22bb84794149b598b97df41eb881 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.298817 1166 raft_consensus.cc:359] T 785733da86ff425a90ccfab2933fc89e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.298991 1166 raft_consensus.cc:385] T 785733da86ff425a90ccfab2933fc89e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:08.299055 1166 raft_consensus.cc:740] T 785733da86ff425a90ccfab2933fc89e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:08.299252 1166 consensus_queue.cc:260] T 785733da86ff425a90ccfab2933fc89e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.299544 914 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "86fc22bb84794149b598b97df41eb881" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
I20251029 02:52:08.299853 914 raft_consensus.cc:3060] T 86fc22bb84794149b598b97df41eb881 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:08.299934 1166 ts_tablet_manager.cc:1434] T 785733da86ff425a90ccfab2933fc89e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 02:52:08.301146 1166 tablet_bootstrap.cc:492] T d61beda662d844cdaf67fd9a2e382ea2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:08.301595 526 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet d2990e9f626c40618c118266bc595684
I20251029 02:52:08.302120 526 catalog_manager.cc:6719] ReplaceTablet: tablet d2990e9f626c40618c118266bc595684 deleted and replaced by tablet 13670ea867e349b2b5a25021469c32d3
W20251029 02:52:08.299679 849 consensus_peers.cc:597] T 519754f5d8f7484480f973c53708c5f3 P c756528642a543eeacd93cef3e335a7d -> Peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Couldn't send request to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 519754f5d8f7484480f973c53708c5f3. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:08.303032 756 tablet_service.cc:1552] Processing DeleteTablet for tablet d2990e9f626c40618c118266bc595684 with delete_type TABLET_DATA_DELETED (replaced by tablet 13670ea867e349b2b5a25021469c32d3) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:08.303524 1032 tablet_service.cc:1552] Processing DeleteTablet for tablet d2990e9f626c40618c118266bc595684 with delete_type TABLET_DATA_DELETED (replaced by tablet 13670ea867e349b2b5a25021469c32d3) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:08.303824 914 raft_consensus.cc:2468] T 86fc22bb84794149b598b97df41eb881 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 1.
I20251029 02:52:08.304267 1166 tablet_bootstrap.cc:654] T d61beda662d844cdaf67fd9a2e382ea2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:08.304347 898 tablet_service.cc:1552] Processing DeleteTablet for tablet d2990e9f626c40618c118266bc595684 with delete_type TABLET_DATA_DELETED (replaced by tablet 13670ea867e349b2b5a25021469c32d3) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:08.304591 990 leader_election.cc:304] T 86fc22bb84794149b598b97df41eb881 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:08.304944 2179 raft_consensus.cc:2804] T 86fc22bb84794149b598b97df41eb881 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election won for term 1
W20251029 02:52:08.306316 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 0ffd22d8610b4329a0a4f753d9e60149 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0ffd22d8610b4329a0a4f753d9e60149 already in progress: creating tablet
I20251029 02:52:08.306784 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 764) took 1659 ms. Trace:
I20251029 02:52:08.306907 1973 leader_election.cc:290] T 86fc22bb84794149b598b97df41eb881 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Requested vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:08.306881 1175 rpcz_store.cc:276] 1029 02:52:06.647581 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:06.647634 (+ 53us) service_pool.cc:225] Handling call
1029 02:52:08.306631 (+1658997us) ts_tablet_manager.cc:1151] Deleting tablet 506f8c112a3b43e1a3d6bbafb8cb81e8
1029 02:52:08.306646 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:08.306762 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1658804}
W20251029 02:52:08.307309 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 506f8c112a3b43e1a3d6bbafb8cb81e8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 506f8c112a3b43e1a3d6bbafb8cb81e8 already in progress: creating tablet
W20251029 02:52:08.308111 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 4372eacfc07246f3be8f7c146674a736 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4372eacfc07246f3be8f7c146674a736 already in progress: creating tablet
W20251029 02:52:08.308362 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 50ab70b30aaa4f6bbae15855431159c8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 50ab70b30aaa4f6bbae15855431159c8 already in progress: creating tablet
I20251029 02:52:08.308701 2109 raft_consensus.cc:493] T dd6ad641fbda472fab07aa35c5e4523a P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:08.308862 2109 raft_consensus.cc:515] T dd6ad641fbda472fab07aa35c5e4523a P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.309468 2109 leader_election.cc:290] T dd6ad641fbda472fab07aa35c5e4523a P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
W20251029 02:52:08.310418 847 leader_election.cc:343] T dd6ad641fbda472fab07aa35c5e4523a P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: dd6ad641fbda472fab07aa35c5e4523a
I20251029 02:52:08.310899 1175 tablet_replica.cc:333] T 49d9ce32e4f04f11be2ecefc5d523a8b P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:08.311120 1175 raft_consensus.cc:2243] T 49d9ce32e4f04f11be2ecefc5d523a8b P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:08.311285 1175 raft_consensus.cc:2272] T 49d9ce32e4f04f11be2ecefc5d523a8b P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251029 02:52:08.311690 849 leader_election.cc:343] T dd6ad641fbda472fab07aa35c5e4523a P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Not found: Tablet not found: dd6ad641fbda472fab07aa35c5e4523a
I20251029 02:52:08.311805 849 leader_election.cc:304] T dd6ad641fbda472fab07aa35c5e4523a P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9
I20251029 02:52:08.311864 1175 ts_tablet_manager.cc:1916] T 49d9ce32e4f04f11be2ecefc5d523a8b P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:08.312288 2109 raft_consensus.cc:2749] T dd6ad641fbda472fab07aa35c5e4523a P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 02:52:08.312779 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 20a762344c5f46f88481c4ac7ad6e43b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 20a762344c5f46f88481c4ac7ad6e43b already in progress: creating tablet
I20251029 02:52:08.314097 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_1f3338455b1a4eef85ecea36443bc329"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:08.314904 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1f3338455b1a4eef85ecea36443bc329 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:08.315995 1973 raft_consensus.cc:493] T 1c53fc1fff1a4bb988beb46ff91fd3be P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:08.316131 1973 raft_consensus.cc:515] T 1c53fc1fff1a4bb988beb46ff91fd3be P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.317134 914 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c53fc1fff1a4bb988beb46ff91fd3be" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:08.317718 990 leader_election.cc:343] T 1c53fc1fff1a4bb988beb46ff91fd3be P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:08.318074 1973 leader_election.cc:290] T 1c53fc1fff1a4bb988beb46ff91fd3be P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:08.307935 1176 tablet_replica.cc:333] T b3de6670357c4ae59eb77469b2fb434c P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
W20251029 02:52:08.312723 989 leader_election.cc:343] T 86fc22bb84794149b598b97df41eb881 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 86fc22bb84794149b598b97df41eb881
I20251029 02:52:08.319347 1176 raft_consensus.cc:2243] T b3de6670357c4ae59eb77469b2fb434c P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:08.319511 1176 raft_consensus.cc:2272] T b3de6670357c4ae59eb77469b2fb434c P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:08.319700 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c53fc1fff1a4bb988beb46ff91fd3be" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:08.319967 768 raft_consensus.cc:2468] T 1c53fc1fff1a4bb988beb46ff91fd3be P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 0.
I20251029 02:52:08.320042 1176 ts_tablet_manager.cc:1916] T b3de6670357c4ae59eb77469b2fb434c P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:08.320571 989 leader_election.cc:304] T 1c53fc1fff1a4bb988beb46ff91fd3be P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters: c756528642a543eeacd93cef3e335a7d
I20251029 02:52:08.321054 1973 raft_consensus.cc:2804] T 1c53fc1fff1a4bb988beb46ff91fd3be P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:08.321180 1973 raft_consensus.cc:493] T 1c53fc1fff1a4bb988beb46ff91fd3be P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:08.321305 1973 raft_consensus.cc:3060] T 1c53fc1fff1a4bb988beb46ff91fd3be P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:08.322089 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_f7fa4c862f364b6eb8510a09d693001b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:08.322733 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f7fa4c862f364b6eb8510a09d693001b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:08.324028 1973 raft_consensus.cc:515] T 1c53fc1fff1a4bb988beb46ff91fd3be P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.325868 1166 tablet_bootstrap.cc:492] T d61beda662d844cdaf67fd9a2e382ea2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:08.326026 1166 ts_tablet_manager.cc:1403] T d61beda662d844cdaf67fd9a2e382ea2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.001s
I20251029 02:52:08.326005 914 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c53fc1fff1a4bb988beb46ff91fd3be" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
W20251029 02:52:08.326653 990 leader_election.cc:343] T 1c53fc1fff1a4bb988beb46ff91fd3be P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:08.326766 1166 raft_consensus.cc:359] T d61beda662d844cdaf67fd9a2e382ea2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.327046 1166 raft_consensus.cc:385] T d61beda662d844cdaf67fd9a2e382ea2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:08.327112 1166 raft_consensus.cc:740] T d61beda662d844cdaf67fd9a2e382ea2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:08.327339 1166 consensus_queue.cc:260] T d61beda662d844cdaf67fd9a2e382ea2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.328599 1166 ts_tablet_manager.cc:1434] T d61beda662d844cdaf67fd9a2e382ea2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 02:52:08.328531 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1c53fc1fff1a4bb988beb46ff91fd3be" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:08.328770 768 raft_consensus.cc:3060] T 1c53fc1fff1a4bb988beb46ff91fd3be P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:08.328918 1166 tablet_bootstrap.cc:492] T 1bdb67125493430398ff5c80b73de2ac P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:08.330106 1167 tablet_bootstrap.cc:492] T c764062d3ba04d4ba8b2e367ad8ed433 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:08.330245 1167 ts_tablet_manager.cc:1403] T c764062d3ba04d4ba8b2e367ad8ed433 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.160s user 0.002s sys 0.002s
I20251029 02:52:08.331053 1167 raft_consensus.cc:359] T c764062d3ba04d4ba8b2e367ad8ed433 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.331228 1167 raft_consensus.cc:385] T c764062d3ba04d4ba8b2e367ad8ed433 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:08.331279 1167 raft_consensus.cc:740] T c764062d3ba04d4ba8b2e367ad8ed433 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:08.331298 1166 tablet_bootstrap.cc:654] T 1bdb67125493430398ff5c80b73de2ac P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:08.331476 1167 consensus_queue.cc:260] T c764062d3ba04d4ba8b2e367ad8ed433 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.333060 768 raft_consensus.cc:2468] T 1c53fc1fff1a4bb988beb46ff91fd3be P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 1.
I20251029 02:52:08.333631 1167 ts_tablet_manager.cc:1434] T c764062d3ba04d4ba8b2e367ad8ed433 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251029 02:52:08.334079 1167 tablet_bootstrap.cc:492] T dfa8b1b9a80e4e8689e0acb44e16346a P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:08.336525 989 leader_election.cc:304] T 1c53fc1fff1a4bb988beb46ff91fd3be P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters: c756528642a543eeacd93cef3e335a7d
I20251029 02:52:08.337013 2103 raft_consensus.cc:2804] T 1c53fc1fff1a4bb988beb46ff91fd3be P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:08.337127 2103 raft_consensus.cc:697] T 1c53fc1fff1a4bb988beb46ff91fd3be P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Becoming Leader. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Running, Role: LEADER
I20251029 02:52:08.337373 2103 consensus_queue.cc:237] T 1c53fc1fff1a4bb988beb46ff91fd3be P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.338856 1167 tablet_bootstrap.cc:654] T dfa8b1b9a80e4e8689e0acb44e16346a P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:08.344298 1175 ts_tablet_manager.cc:1929] T 49d9ce32e4f04f11be2ecefc5d523a8b P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:08.344455 1175 log.cc:1199] T 49d9ce32e4f04f11be2ecefc5d523a8b P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/49d9ce32e4f04f11be2ecefc5d523a8b
I20251029 02:52:08.344802 1175 ts_tablet_manager.cc:1950] T 49d9ce32e4f04f11be2ecefc5d523a8b P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:08.345156 898 tablet_service.cc:1552] Processing DeleteTablet for tablet c764062d3ba04d4ba8b2e367ad8ed433 with delete_type TABLET_DATA_DELETED (replaced by tablet 372d93914abc41b38a702742b7db0bc2) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:08.345816 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 767) took 1574 ms. Trace:
I20251029 02:52:08.345937 1175 rpcz_store.cc:276] 1029 02:52:06.770862 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:06.770961 (+ 99us) service_pool.cc:225] Handling call
1029 02:52:08.310837 (+1539876us) ts_tablet_manager.cc:1151] Deleting tablet 49d9ce32e4f04f11be2ecefc5d523a8b
1029 02:52:08.310848 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:08.333032 (+ 22184us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:08.344144 (+ 11112us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:08.345787 (+ 1643us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1495181}
I20251029 02:52:08.346484 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 49d9ce32e4f04f11be2ecefc5d523a8b (table default.Table_af1ab75f523d41338d4471824997f935 [id=468c76c8c99a4aa59c8b1f6c8e4bfa30]) successfully deleted
I20251029 02:52:08.346844 1167 tablet_bootstrap.cc:492] T dfa8b1b9a80e4e8689e0acb44e16346a P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:08.346983 1167 ts_tablet_manager.cc:1403] T dfa8b1b9a80e4e8689e0acb44e16346a P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20251029 02:52:08.347765 1167 raft_consensus.cc:359] T dfa8b1b9a80e4e8689e0acb44e16346a P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.347978 1167 raft_consensus.cc:385] T dfa8b1b9a80e4e8689e0acb44e16346a P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:08.348045 1167 raft_consensus.cc:740] T dfa8b1b9a80e4e8689e0acb44e16346a P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:08.348269 1167 consensus_queue.cc:260] T dfa8b1b9a80e4e8689e0acb44e16346a P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.351806 1973 leader_election.cc:290] T 1c53fc1fff1a4bb988beb46ff91fd3be P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Requested vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:08.352523 1167 ts_tablet_manager.cc:1434] T dfa8b1b9a80e4e8689e0acb44e16346a P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251029 02:52:08.352854 1167 tablet_bootstrap.cc:492] T 9e6eb099648549dbbf9969e59865f51f P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:08.357355 1167 tablet_bootstrap.cc:654] T 9e6eb099648549dbbf9969e59865f51f P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:08.358734 524 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet dabb8d760aae4aa09eb83cdb586c9788
I20251029 02:52:08.358759 1166 tablet_bootstrap.cc:492] T 1bdb67125493430398ff5c80b73de2ac P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:08.358928 1166 ts_tablet_manager.cc:1403] T 1bdb67125493430398ff5c80b73de2ac P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.001s
I20251029 02:52:08.360082 756 tablet_service.cc:1552] Processing DeleteTablet for tablet dabb8d760aae4aa09eb83cdb586c9788 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:07 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:08.360196 1032 tablet_service.cc:1552] Processing DeleteTablet for tablet dabb8d760aae4aa09eb83cdb586c9788 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:07 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:08.360574 2179 raft_consensus.cc:697] T 86fc22bb84794149b598b97df41eb881 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Becoming Leader. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Running, Role: LEADER
I20251029 02:52:08.360607 1166 raft_consensus.cc:359] T 1bdb67125493430398ff5c80b73de2ac P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.360774 1166 raft_consensus.cc:385] T 1bdb67125493430398ff5c80b73de2ac P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:08.360834 1166 raft_consensus.cc:740] T 1bdb67125493430398ff5c80b73de2ac P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:08.361045 1166 consensus_queue.cc:260] T 1bdb67125493430398ff5c80b73de2ac P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.361094 2179 consensus_queue.cc:237] T 86fc22bb84794149b598b97df41eb881 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.361564 1166 ts_tablet_manager.cc:1434] T 1bdb67125493430398ff5c80b73de2ac P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20251029 02:52:08.362044 1175 tablet_replica.cc:333] T ccdca31db2834de4b6ed0f2d67a10535 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:08.362244 1175 raft_consensus.cc:2243] T ccdca31db2834de4b6ed0f2d67a10535 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:08.362397 1175 raft_consensus.cc:2272] T ccdca31db2834de4b6ed0f2d67a10535 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:08.363021 1175 ts_tablet_manager.cc:1916] T ccdca31db2834de4b6ed0f2d67a10535 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:08.363551 898 tablet_service.cc:1552] Processing DeleteTablet for tablet dfa8b1b9a80e4e8689e0acb44e16346a with delete_type TABLET_DATA_DELETED (replaced by tablet 97b2ecae572a4b7ea1b9595074556708) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:08.364022 898 tablet_service.cc:1552] Processing DeleteTablet for tablet dabb8d760aae4aa09eb83cdb586c9788 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:07 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:08.366837 1167 tablet_bootstrap.cc:492] T 9e6eb099648549dbbf9969e59865f51f P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:08.366990 1167 ts_tablet_manager.cc:1403] T 9e6eb099648549dbbf9969e59865f51f P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.001s
I20251029 02:52:08.368533 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_cfbb812339124da1b8e6e8a95d816a5e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:08.368624 1167 raft_consensus.cc:359] T 9e6eb099648549dbbf9969e59865f51f P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.368818 1167 raft_consensus.cc:385] T 9e6eb099648549dbbf9969e59865f51f P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:08.368866 1167 raft_consensus.cc:740] T 9e6eb099648549dbbf9969e59865f51f P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:08.368861 1166 tablet_bootstrap.cc:492] T f529e17e97074dca91ec2f48954cfd32 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:08.369064 1167 consensus_queue.cc:260] T 9e6eb099648549dbbf9969e59865f51f P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
W20251029 02:52:08.369341 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cfbb812339124da1b8e6e8a95d816a5e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:08.370709 1166 tablet_bootstrap.cc:654] T f529e17e97074dca91ec2f48954cfd32 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:08.371140 1167 ts_tablet_manager.cc:1434] T 9e6eb099648549dbbf9969e59865f51f P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20251029 02:52:08.373778 1167 tablet_bootstrap.cc:492] T 02e4588b03144ea2850e13a1f8f29f06 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:08.375862 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_e64c529c8e2d48b487caaa29ed3d17d8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:08.376370 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_ff347dc490d441baa678753aa33b710b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:08.376544 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e64c529c8e2d48b487caaa29ed3d17d8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:52:08.376847 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ff347dc490d441baa678753aa33b710b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:08.380146 1167 tablet_bootstrap.cc:654] T 02e4588b03144ea2850e13a1f8f29f06 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:08.386648 1167 tablet_bootstrap.cc:492] T 02e4588b03144ea2850e13a1f8f29f06 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:08.386811 1167 ts_tablet_manager.cc:1403] T 02e4588b03144ea2850e13a1f8f29f06 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.001s
I20251029 02:52:08.388198 1167 raft_consensus.cc:359] T 02e4588b03144ea2850e13a1f8f29f06 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.388408 1167 raft_consensus.cc:385] T 02e4588b03144ea2850e13a1f8f29f06 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:08.388465 1167 raft_consensus.cc:740] T 02e4588b03144ea2850e13a1f8f29f06 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:08.388707 1167 consensus_queue.cc:260] T 02e4588b03144ea2850e13a1f8f29f06 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.391185 524 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_ae1c37b89ad243f0a389a54fa71b3ae0" } new_table_name: "default.Table_13bf62ac663d4b28b91d2942bc03d6b1" modify_external_catalogs: true
W20251029 02:52:08.394903 847 consensus_peers.cc:597] T b3de6670357c4ae59eb77469b2fb434c P c756528642a543eeacd93cef3e335a7d -> Peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Couldn't send request to peer b495a49349324f7d81908f2da700f0d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b3de6670357c4ae59eb77469b2fb434c. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:08.397526 1175 ts_tablet_manager.cc:1929] T ccdca31db2834de4b6ed0f2d67a10535 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:08.397650 1175 log.cc:1199] T ccdca31db2834de4b6ed0f2d67a10535 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/ccdca31db2834de4b6ed0f2d67a10535
I20251029 02:52:08.397949 1175 ts_tablet_manager.cc:1950] T ccdca31db2834de4b6ed0f2d67a10535 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:08.398824 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 769) took 1627 ms. Trace:
I20251029 02:52:08.398927 1175 rpcz_store.cc:276] 1029 02:52:06.771112 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:06.771158 (+ 46us) service_pool.cc:225] Handling call
1029 02:52:08.361969 (+1590811us) ts_tablet_manager.cc:1151] Deleting tablet ccdca31db2834de4b6ed0f2d67a10535
1029 02:52:08.361984 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:08.366463 (+ 4479us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:08.397407 (+ 30944us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:08.398802 (+ 1395us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1546250}
I20251029 02:52:08.399401 1175 tablet_replica.cc:333] T 5db519edc75d4bd0985b32322825ffae P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:08.399407 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet ccdca31db2834de4b6ed0f2d67a10535 (table default.Table_75341a6f40d84ad3938442923cf3131e [id=72e9fc37abc14519b6b87ebfc83c292c]) successfully deleted
I20251029 02:52:08.399576 1175 raft_consensus.cc:2243] T 5db519edc75d4bd0985b32322825ffae P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:08.399772 1175 raft_consensus.cc:2272] T 5db519edc75d4bd0985b32322825ffae P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:08.400305 1175 ts_tablet_manager.cc:1916] T 5db519edc75d4bd0985b32322825ffae P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:08.405301 1167 ts_tablet_manager.cc:1434] T 02e4588b03144ea2850e13a1f8f29f06 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.018s user 0.001s sys 0.001s
I20251029 02:52:08.407131 2113 raft_consensus.cc:493] T 9269622ca92645d5aa8e73341201ccbb P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:08.407289 2113 raft_consensus.cc:515] T 9269622ca92645d5aa8e73341201ccbb P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.407893 2113 leader_election.cc:290] T 9269622ca92645d5aa8e73341201ccbb P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:08.408612 1167 tablet_bootstrap.cc:492] T f529e17e97074dca91ec2f48954cfd32 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:08.408490 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_47178701cd044f8fa5357cd607154988"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:08.409251 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_47178701cd044f8fa5357cd607154988 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:08.411278 1032 tablet_service.cc:1505] Processing CreateTablet for tablet f17b76f5dd4c4dd78d1fd45cc490763c (DEFAULT_TABLE table=default.Table_c6def3b3356a4994a768adbbdf45fb9b [id=914eab99e023490791ee42dc8436fb03]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:08.411649 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f17b76f5dd4c4dd78d1fd45cc490763c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:08.414896 519 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet 44b518c3a0964a6fa863b2784d80c8a4 (table default.Table_9120016827e946cc8a145252d2569d2b [id=e0c0cf3d456444c191a32bb180e83798]). Expected version 1 got 0
I20251029 02:52:08.415261 519 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet cedfb53078d94c81b5e294ce7438325e (table default.Table_e75db5cf8f5f49ea837cb6d63f4b7ebf [id=86195563fb4a4d7abd746f4ac7cd241c]). Expected version 1 got 0
I20251029 02:52:08.415532 519 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet 95f342d1cfb84c4eb31bd3d15ea3bdc2 (table default.Table_96ea715e50e74cc48759dda025a46318 [id=c63d52811db749d6badbed72fc133943]). Expected version 1 got 0
I20251029 02:52:08.417860 1031 tablet_service.cc:1552] Processing DeleteTablet for tablet caef1c93e33a4390bcada59a41f43465 with delete_type TABLET_DATA_DELETED (replaced by tablet 49492d91b1e94e9ca086533dbd0440f6) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:08.418007 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9269622ca92645d5aa8e73341201ccbb" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:08.418403 914 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9269622ca92645d5aa8e73341201ccbb" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:08.418555 712 leader_election.cc:343] T 9269622ca92645d5aa8e73341201ccbb P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
W20251029 02:52:08.419173 711 leader_election.cc:343] T 9269622ca92645d5aa8e73341201ccbb P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:08.419276 711 leader_election.cc:304] T 9269622ca92645d5aa8e73341201ccbb P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:08.419872 756 tablet_service.cc:1505] Processing CreateTablet for tablet f17b76f5dd4c4dd78d1fd45cc490763c (DEFAULT_TABLE table=default.Table_c6def3b3356a4994a768adbbdf45fb9b [id=914eab99e023490791ee42dc8436fb03]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:08.420272 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f17b76f5dd4c4dd78d1fd45cc490763c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:08.421456 1167 tablet_bootstrap.cc:654] T f529e17e97074dca91ec2f48954cfd32 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:08.422818 2113 raft_consensus.cc:2749] T 9269622ca92645d5aa8e73341201ccbb P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:08.423683 749 tablet_service.cc:1505] Processing CreateTablet for tablet 34d141bee474452693dba1b9d7f2b023 (DEFAULT_TABLE table=default.Table_11c3e33dc6ae4f9e8852d7a31b078288 [id=041e1f62e8654eb19a8231123a813cdd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:08.424052 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34d141bee474452693dba1b9d7f2b023. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:08.426918 750 tablet_service.cc:1505] Processing CreateTablet for tablet 2a61593bfbfe4430ac85f8e5c65e56c3 (DEFAULT_TABLE table=default.Table_771ea2fd4396483ca70f00c7b9fb4c5f [id=593ef93f30844fa4abbfff654e3202a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:08.427332 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a61593bfbfe4430ac85f8e5c65e56c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:08.427690 1166 tablet_bootstrap.cc:492] T f529e17e97074dca91ec2f48954cfd32 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:08.427861 1166 ts_tablet_manager.cc:1403] T f529e17e97074dca91ec2f48954cfd32 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.059s user 0.005s sys 0.002s
I20251029 02:52:08.428609 1166 raft_consensus.cc:359] T f529e17e97074dca91ec2f48954cfd32 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.428809 1166 raft_consensus.cc:385] T f529e17e97074dca91ec2f48954cfd32 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:08.428856 1166 raft_consensus.cc:740] T f529e17e97074dca91ec2f48954cfd32 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:08.428936 1167 tablet_bootstrap.cc:492] T f529e17e97074dca91ec2f48954cfd32 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:08.429067 1167 ts_tablet_manager.cc:1403] T f529e17e97074dca91ec2f48954cfd32 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.002s
I20251029 02:52:08.429080 1166 consensus_queue.cc:260] T f529e17e97074dca91ec2f48954cfd32 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.429525 1166 ts_tablet_manager.cc:1434] T f529e17e97074dca91ec2f48954cfd32 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 02:52:08.429773 1167 raft_consensus.cc:359] T f529e17e97074dca91ec2f48954cfd32 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.429920 1167 raft_consensus.cc:385] T f529e17e97074dca91ec2f48954cfd32 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:08.429963 1167 raft_consensus.cc:740] T f529e17e97074dca91ec2f48954cfd32 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:08.431917 1167 consensus_queue.cc:260] T f529e17e97074dca91ec2f48954cfd32 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.432111 1172 tablet_bootstrap.cc:492] T e7f846b294cc4eb687a26c8b5931ac19 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:08.432266 1172 ts_tablet_manager.cc:1403] T e7f846b294cc4eb687a26c8b5931ac19 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.307s user 0.037s sys 0.052s
I20251029 02:52:08.433451 1172 raft_consensus.cc:359] T e7f846b294cc4eb687a26c8b5931ac19 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.433715 1172 raft_consensus.cc:385] T e7f846b294cc4eb687a26c8b5931ac19 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:08.433777 1172 raft_consensus.cc:740] T e7f846b294cc4eb687a26c8b5931ac19 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:08.433962 1172 consensus_queue.cc:260] T e7f846b294cc4eb687a26c8b5931ac19 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.434410 1172 ts_tablet_manager.cc:1434] T e7f846b294cc4eb687a26c8b5931ac19 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 02:52:08.435307 1166 tablet_bootstrap.cc:492] T 6f35412a66b74051a475dce5dafa7792 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:08.435528 1167 ts_tablet_manager.cc:1434] T f529e17e97074dca91ec2f48954cfd32 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251029 02:52:08.435797 1167 tablet_bootstrap.cc:492] T c9bd3f2deb5d4af09b7918269a98509f P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:08.436688 1172 tablet_bootstrap.cc:492] T 28f7ceb91a34450cad4b5b45f5285b41 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:08.436977 1166 tablet_bootstrap.cc:654] T 6f35412a66b74051a475dce5dafa7792 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:08.438326 1172 tablet_bootstrap.cc:654] T 28f7ceb91a34450cad4b5b45f5285b41 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:08.442334 1167 tablet_bootstrap.cc:654] T c9bd3f2deb5d4af09b7918269a98509f P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
W20251029 02:52:08.445751 524 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 44c2ef5426704e03bd50775adcdc571b consensus state has no leader
I20251029 02:52:08.450745 1167 tablet_bootstrap.cc:492] T c9bd3f2deb5d4af09b7918269a98509f P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:08.450924 1167 ts_tablet_manager.cc:1403] T c9bd3f2deb5d4af09b7918269a98509f P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.004s
I20251029 02:52:08.451223 755 tablet_service.cc:1505] Processing CreateTablet for tablet 938e7292913f456ba8588a4647b78ff9 (DEFAULT_TABLE table=default.Table_786b080d63354a4cb479483f172ba0af [id=e9a01f22f5914bd1af0898fa67bb95db]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:08.451632 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 938e7292913f456ba8588a4647b78ff9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:08.451694 1167 raft_consensus.cc:359] T c9bd3f2deb5d4af09b7918269a98509f P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.451875 1167 raft_consensus.cc:385] T c9bd3f2deb5d4af09b7918269a98509f P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:08.451922 1167 raft_consensus.cc:740] T c9bd3f2deb5d4af09b7918269a98509f P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:08.455691 1167 consensus_queue.cc:260] T c9bd3f2deb5d4af09b7918269a98509f P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.456205 526 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_9bf4e25d441f4d4dbd6651386eec512e" } new_table_name: "default.Table_8449a7a46e2a4059a6c41c42bdee0c70" modify_external_catalogs: true
I20251029 02:52:08.458664 521 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_53892f35056440d89815e7f4997f08d2" } new_table_name: "default.Table_fe6f038b462644e7964ea2e48df2a331" modify_external_catalogs: true
I20251029 02:52:08.460614 1031 tablet_service.cc:1505] Processing CreateTablet for tablet 34d141bee474452693dba1b9d7f2b023 (DEFAULT_TABLE table=default.Table_11c3e33dc6ae4f9e8852d7a31b078288 [id=041e1f62e8654eb19a8231123a813cdd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:08.461061 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34d141bee474452693dba1b9d7f2b023. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:08.462911 1032 tablet_service.cc:1505] Processing CreateTablet for tablet 2a61593bfbfe4430ac85f8e5c65e56c3 (DEFAULT_TABLE table=default.Table_771ea2fd4396483ca70f00c7b9fb4c5f [id=593ef93f30844fa4abbfff654e3202a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:08.463286 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a61593bfbfe4430ac85f8e5c65e56c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:08.465377 2179 raft_consensus.cc:493] T 02e4588b03144ea2850e13a1f8f29f06 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:08.465502 2179 raft_consensus.cc:515] T 02e4588b03144ea2850e13a1f8f29f06 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.465812 1039 tablet_service.cc:1505] Processing CreateTablet for tablet 938e7292913f456ba8588a4647b78ff9 (DEFAULT_TABLE table=default.Table_786b080d63354a4cb479483f172ba0af [id=e9a01f22f5914bd1af0898fa67bb95db]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:08.466114 2179 leader_election.cc:290] T 02e4588b03144ea2850e13a1f8f29f06 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:08.466158 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 938e7292913f456ba8588a4647b78ff9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:08.466691 2109 raft_consensus.cc:493] T 56d05e4a500043c995bb51a204cb295d P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:08.466822 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "02e4588b03144ea2850e13a1f8f29f06" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:08.466836 2109 raft_consensus.cc:515] T 56d05e4a500043c995bb51a204cb295d P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.467006 768 raft_consensus.cc:2468] T 02e4588b03144ea2850e13a1f8f29f06 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 0.
I20251029 02:52:08.467396 989 leader_election.cc:304] T 02e4588b03144ea2850e13a1f8f29f06 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters:
I20251029 02:52:08.467641 2179 raft_consensus.cc:2804] T 02e4588b03144ea2850e13a1f8f29f06 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:08.467682 2109 leader_election.cc:290] T 56d05e4a500043c995bb51a204cb295d P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:08.467702 2179 raft_consensus.cc:493] T 02e4588b03144ea2850e13a1f8f29f06 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:08.467768 2179 raft_consensus.cc:3060] T 02e4588b03144ea2850e13a1f8f29f06 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
W20251029 02:52:08.468750 849 leader_election.cc:343] T 56d05e4a500043c995bb51a204cb295d P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Not found: Tablet not found: 56d05e4a500043c995bb51a204cb295d
W20251029 02:52:08.469555 847 leader_election.cc:343] T 56d05e4a500043c995bb51a204cb295d P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 56d05e4a500043c995bb51a204cb295d
I20251029 02:52:08.469650 847 leader_election.cc:304] T 56d05e4a500043c995bb51a204cb295d P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9
I20251029 02:52:08.470026 2109 raft_consensus.cc:2749] T 56d05e4a500043c995bb51a204cb295d P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:08.470579 2179 raft_consensus.cc:515] T 02e4588b03144ea2850e13a1f8f29f06 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.470819 1025 tablet_service.cc:1505] Processing CreateTablet for tablet 8d5d02c38f2a48cc94358bcaa9fe5065 (DEFAULT_TABLE table=default.Table_e3aae8a6f0314d75b52739cd81d573d9 [id=7f20f69c0a6e4355bd2c997249a05fc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:08.471185 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d5d02c38f2a48cc94358bcaa9fe5065. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:08.471405 2179 leader_election.cc:290] T 02e4588b03144ea2850e13a1f8f29f06 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Requested vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:08.471966 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "02e4588b03144ea2850e13a1f8f29f06" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:08.472157 768 raft_consensus.cc:3060] T 02e4588b03144ea2850e13a1f8f29f06 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:08.474220 2113 raft_consensus.cc:493] T 738046ab57334f2dab9f4aab979c5b42 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:08.474352 2113 raft_consensus.cc:515] T 738046ab57334f2dab9f4aab979c5b42 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.475000 2113 leader_election.cc:290] T 738046ab57334f2dab9f4aab979c5b42 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:08.475296 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "02e4588b03144ea2850e13a1f8f29f06" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:08.475391 919 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "02e4588b03144ea2850e13a1f8f29f06" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
I20251029 02:52:08.475484 916 raft_consensus.cc:2468] T 02e4588b03144ea2850e13a1f8f29f06 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 0.
I20251029 02:52:08.475590 898 tablet_service.cc:1505] Processing CreateTablet for tablet f17b76f5dd4c4dd78d1fd45cc490763c (DEFAULT_TABLE table=default.Table_c6def3b3356a4994a768adbbdf45fb9b [id=914eab99e023490791ee42dc8436fb03]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:08.475880 1167 ts_tablet_manager.cc:1434] T c9bd3f2deb5d4af09b7918269a98509f P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.025s user 0.002s sys 0.000s
I20251029 02:52:08.478066 2036 raft_consensus.cc:493] T 7a1866afc7564217ad777e7ce53786c5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:08.478201 2036 raft_consensus.cc:515] T 7a1866afc7564217ad777e7ce53786c5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.478921 2036 leader_election.cc:290] T 7a1866afc7564217ad777e7ce53786c5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:08.480033 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "738046ab57334f2dab9f4aab979c5b42" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:08.480062 766 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a1866afc7564217ad777e7ce53786c5" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:08.480288 766 raft_consensus.cc:2468] T 7a1866afc7564217ad777e7ce53786c5 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 0.
I20251029 02:52:08.480901 989 leader_election.cc:304] T 7a1866afc7564217ad777e7ce53786c5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters:
W20251029 02:52:08.481201 712 leader_election.cc:343] T 738046ab57334f2dab9f4aab979c5b42 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:08.481391 2036 raft_consensus.cc:2804] T 7a1866afc7564217ad777e7ce53786c5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:08.481459 2036 raft_consensus.cc:493] T 7a1866afc7564217ad777e7ce53786c5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:08.481520 2036 raft_consensus.cc:3060] T 7a1866afc7564217ad777e7ce53786c5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:08.484323 2036 raft_consensus.cc:515] T 7a1866afc7564217ad777e7ce53786c5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.485128 2036 leader_election.cc:290] T 7a1866afc7564217ad777e7ce53786c5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Requested vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:08.476173 1167 tablet_bootstrap.cc:492] T 777992acb1bd4765886b802940db74bb P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:08.476511 892 tablet_service.cc:1505] Processing CreateTablet for tablet 938e7292913f456ba8588a4647b78ff9 (DEFAULT_TABLE table=default.Table_786b080d63354a4cb479483f172ba0af [id=e9a01f22f5914bd1af0898fa67bb95db]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:08.476987 891 tablet_service.cc:1505] Processing CreateTablet for tablet 8d5d02c38f2a48cc94358bcaa9fe5065 (DEFAULT_TABLE table=default.Table_e3aae8a6f0314d75b52739cd81d573d9 [id=7f20f69c0a6e4355bd2c997249a05fc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:08.477461 900 tablet_service.cc:1505] Processing CreateTablet for tablet 34d141bee474452693dba1b9d7f2b023 (DEFAULT_TABLE table=default.Table_11c3e33dc6ae4f9e8852d7a31b078288 [id=041e1f62e8654eb19a8231123a813cdd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:08.477759 885 tablet_service.cc:1505] Processing CreateTablet for tablet 2a61593bfbfe4430ac85f8e5c65e56c3 (DEFAULT_TABLE table=default.Table_771ea2fd4396483ca70f00c7b9fb4c5f [id=593ef93f30844fa4abbfff654e3202a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:08.481873 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "738046ab57334f2dab9f4aab979c5b42" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:08.486110 1175 ts_tablet_manager.cc:1929] T 5db519edc75d4bd0985b32322825ffae P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:08.482777 2109 raft_consensus.cc:493] T f529e17e97074dca91ec2f48954cfd32 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:08.486217 1175 log.cc:1199] T 5db519edc75d4bd0985b32322825ffae P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/5db519edc75d4bd0985b32322825ffae
I20251029 02:52:08.486299 2109 raft_consensus.cc:515] T f529e17e97074dca91ec2f48954cfd32 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.486534 1175 ts_tablet_manager.cc:1950] T 5db519edc75d4bd0985b32322825ffae P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:08.486915 2109 leader_election.cc:290] T f529e17e97074dca91ec2f48954cfd32 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:08.487709 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f529e17e97074dca91ec2f48954cfd32" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:08.487870 1060 raft_consensus.cc:2468] T f529e17e97074dca91ec2f48954cfd32 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:08.488235 849 leader_election.cc:304] T f529e17e97074dca91ec2f48954cfd32 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters:
W20251029 02:52:08.488938 847 leader_election.cc:343] T f529e17e97074dca91ec2f48954cfd32 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: f529e17e97074dca91ec2f48954cfd32
I20251029 02:52:08.488873 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_aaecc1c9efd04549a2484850ff8efc60"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:08.489559 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aaecc1c9efd04549a2484850ff8efc60 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:08.490883 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_126f4f49037f46628fa5e1d33b4fcf24"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:08.491492 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_126f4f49037f46628fa5e1d33b4fcf24 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:08.493064 2109 raft_consensus.cc:2804] T f529e17e97074dca91ec2f48954cfd32 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:08.493139 2109 raft_consensus.cc:493] T f529e17e97074dca91ec2f48954cfd32 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:08.493201 2109 raft_consensus.cc:3060] T f529e17e97074dca91ec2f48954cfd32 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
W20251029 02:52:08.494665 711 leader_election.cc:343] T 738046ab57334f2dab9f4aab979c5b42 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:08.494797 711 leader_election.cc:304] T 738046ab57334f2dab9f4aab979c5b42 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:08.494958 522 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet 02e4588b03144ea2850e13a1f8f29f06 (table default.Table_0772f9db5d6142d988f6ffca2e96bdd2 [id=fd6d31d379394857a35422211af00642]). Expected version 3 got 0
I20251029 02:52:08.495191 2113 raft_consensus.cc:2749] T 738046ab57334f2dab9f4aab979c5b42 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:08.496014 766 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a1866afc7564217ad777e7ce53786c5" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:08.496176 766 raft_consensus.cc:3060] T 7a1866afc7564217ad777e7ce53786c5 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:08.496908 2109 raft_consensus.cc:515] T f529e17e97074dca91ec2f48954cfd32 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.497598 768 raft_consensus.cc:2468] T 02e4588b03144ea2850e13a1f8f29f06 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 1.
W20251029 02:52:08.498363 847 leader_election.cc:343] T f529e17e97074dca91ec2f48954cfd32 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: f529e17e97074dca91ec2f48954cfd32
W20251029 02:52:08.499243 526 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a2d08c252a934cf9a45d2f60e62f76aa consensus state has no leader
I20251029 02:52:08.499171 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f529e17e97074dca91ec2f48954cfd32" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:52:08.499353 1060 raft_consensus.cc:3060] T f529e17e97074dca91ec2f48954cfd32 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:08.502187 1060 raft_consensus.cc:2468] T f529e17e97074dca91ec2f48954cfd32 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:08.502574 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 768) took 1731 ms. Trace:
I20251029 02:52:08.502710 849 leader_election.cc:304] T f529e17e97074dca91ec2f48954cfd32 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters: b495a49349324f7d81908f2da700f0d9
I20251029 02:52:08.502671 1175 rpcz_store.cc:276] 1029 02:52:06.770989 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:06.771193 (+ 204us) service_pool.cc:225] Handling call
1029 02:52:08.399352 (+1628159us) ts_tablet_manager.cc:1151] Deleting tablet 5db519edc75d4bd0985b32322825ffae
1029 02:52:08.399362 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:08.403649 (+ 4287us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:08.485991 (+ 82342us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:08.502552 (+ 16561us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1583575}
I20251029 02:52:08.502969 1175 tablet_replica.cc:333] T 0f80b840ed954230bfbfe7e40127cd2f P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:08.504557 766 raft_consensus.cc:2468] T 7a1866afc7564217ad777e7ce53786c5 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 1.
I20251029 02:52:08.504738 1175 raft_consensus.cc:2243] T 0f80b840ed954230bfbfe7e40127cd2f P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:08.504890 1175 raft_consensus.cc:2272] T 0f80b840ed954230bfbfe7e40127cd2f P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:08.505410 1175 ts_tablet_manager.cc:1916] T 0f80b840ed954230bfbfe7e40127cd2f P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 02:52:08.506033 521 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bc9fc1c90eaf4046a4022beab31f2902 has no consensus state
I20251029 02:52:08.507642 524 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_f435bdcca42e4087ac8794859d9108ee" } modify_external_catalogs: true
I20251029 02:52:08.507844 524 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_f435bdcca42e4087ac8794859d9108ee" } modify_external_catalogs: true
I20251029 02:52:08.509522 521 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_282e9d0ae9384d3284e662fa864925a9" } modify_external_catalogs: true
I20251029 02:52:08.509784 521 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_282e9d0ae9384d3284e662fa864925a9" } modify_external_catalogs: true
I20251029 02:52:08.509855 520 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_3e2da7dbc6574230bd2182d9226213fc" } modify_external_catalogs: true
I20251029 02:52:08.510003 520 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_3e2da7dbc6574230bd2182d9226213fc" } modify_external_catalogs: true
I20251029 02:52:08.511147 526 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_457a3835f7d84fd7bf722dfc1d18d301" } modify_external_catalogs: true
I20251029 02:52:08.511307 519 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_47dc7c63ef104aec894c06e91376a212" } new_table_name: "default.Table_af62893fe44b437ba6c5753df4b31da9" modify_external_catalogs: true
I20251029 02:52:08.511366 526 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_457a3835f7d84fd7bf722dfc1d18d301" } modify_external_catalogs: true
I20251029 02:52:08.511444 525 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_90cf8deabd5c47c1b162ce513d4c1fe4" } new_table_name: "default.Table_b869c013d33246d59a16bf6ce55bb96a" modify_external_catalogs: true
I20251029 02:52:08.515228 2109 leader_election.cc:290] T f529e17e97074dca91ec2f48954cfd32 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:08.515643 2109 raft_consensus.cc:2804] T f529e17e97074dca91ec2f48954cfd32 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:08.515663 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 5db519edc75d4bd0985b32322825ffae (table default.Table_b8ddec1d965444948171ef0cca0ff011 [id=60a528f028d24bbabd6f5a927fb7853b]) successfully deleted
I20251029 02:52:08.516299 2109 raft_consensus.cc:697] T f529e17e97074dca91ec2f48954cfd32 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:52:08.517203 1166 tablet_bootstrap.cc:492] T 6f35412a66b74051a475dce5dafa7792 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:08.517346 1166 ts_tablet_manager.cc:1403] T 6f35412a66b74051a475dce5dafa7792 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.082s user 0.002s sys 0.002s
I20251029 02:52:08.517333 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_b19f559f0b7048ecbbb93a9d3ceae749"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:08.517889 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b19f559f0b7048ecbbb93a9d3ceae749 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:08.518184 1166 raft_consensus.cc:359] T 6f35412a66b74051a475dce5dafa7792 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.518343 1166 raft_consensus.cc:385] T 6f35412a66b74051a475dce5dafa7792 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:08.518394 1166 raft_consensus.cc:740] T 6f35412a66b74051a475dce5dafa7792 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:08.518602 1166 consensus_queue.cc:260] T 6f35412a66b74051a475dce5dafa7792 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.519093 1166 ts_tablet_manager.cc:1434] T 6f35412a66b74051a475dce5dafa7792 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251029 02:52:08.519398 1166 tablet_bootstrap.cc:492] T dfa8b1b9a80e4e8689e0acb44e16346a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:08.521235 1166 tablet_bootstrap.cc:654] T dfa8b1b9a80e4e8689e0acb44e16346a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:08.523101 2109 consensus_queue.cc:237] T f529e17e97074dca91ec2f48954cfd32 P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.525506 1175 ts_tablet_manager.cc:1929] T 0f80b840ed954230bfbfe7e40127cd2f P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:08.525642 1175 log.cc:1199] T 0f80b840ed954230bfbfe7e40127cd2f P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/0f80b840ed954230bfbfe7e40127cd2f
I20251029 02:52:08.526093 1175 ts_tablet_manager.cc:1950] T 0f80b840ed954230bfbfe7e40127cd2f P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
W20251029 02:52:08.528159 519 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8de7380e93dd4a9abdb84522a18abda6 has no consensus state
W20251029 02:52:08.528505 525 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0d2077a74c5342eb813accde2d667017 consensus state has no leader
I20251029 02:52:08.529556 1032 tablet_service.cc:1552] Processing DeleteTablet for tablet ccdca31db2834de4b6ed0f2d67a10535 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:54 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:08.529594 1041 tablet_service.cc:1552] Processing DeleteTablet for tablet 5db519edc75d4bd0985b32322825ffae with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:55 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:08.529899 1042 tablet_service.cc:1552] Processing DeleteTablet for tablet 86fc22bb84794149b598b97df41eb881 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:58 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:08.529934 526 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 71b4b91071a54d6aa66160e7955bc915
I20251029 02:52:08.530207 989 leader_election.cc:304] T 02e4588b03144ea2850e13a1f8f29f06 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters: c756528642a543eeacd93cef3e335a7d
I20251029 02:52:08.530699 989 leader_election.cc:304] T 7a1866afc7564217ad777e7ce53786c5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters:
I20251029 02:52:08.530966 524 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet d3faa2d4df584c6e8da426a3830a6454
I20251029 02:52:08.531008 2179 raft_consensus.cc:2804] T 02e4588b03144ea2850e13a1f8f29f06 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:08.531019 755 tablet_service.cc:1552] Processing DeleteTablet for tablet 71b4b91071a54d6aa66160e7955bc915 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:08 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:08.531152 2188 raft_consensus.cc:2804] T 7a1866afc7564217ad777e7ce53786c5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:08.531229 2188 raft_consensus.cc:697] T 7a1866afc7564217ad777e7ce53786c5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Becoming Leader. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Running, Role: LEADER
I20251029 02:52:08.531453 2188 consensus_queue.cc:237] T 7a1866afc7564217ad777e7ce53786c5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.531656 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 766) took 1756 ms. Trace:
I20251029 02:52:08.531759 1175 rpcz_store.cc:276] 1029 02:52:06.770719 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:06.771252 (+ 533us) service_pool.cc:225] Handling call
1029 02:52:08.502915 (+1731663us) ts_tablet_manager.cc:1151] Deleting tablet 0f80b840ed954230bfbfe7e40127cd2f
1029 02:52:08.502923 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:08.508601 (+ 5678us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:08.525379 (+ 16778us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:08.527038 (+ 1659us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1588,"tablet-delete.queue_time_us":1687116}
I20251029 02:52:08.531914 755 tablet_service.cc:1552] Processing DeleteTablet for tablet d3faa2d4df584c6e8da426a3830a6454 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:08 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:08.532078 1175 tablet_replica.cc:333] T 785733da86ff425a90ccfab2933fc89e P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:08.532251 1175 raft_consensus.cc:2243] T 785733da86ff425a90ccfab2933fc89e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:08.532335 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 0f80b840ed954230bfbfe7e40127cd2f (table default.Table_5902a11255a049be9dd07acf8b50adda [id=5bd71ce398e84db28f54262cd3c5528b]) successfully deleted
I20251029 02:52:08.532394 1175 raft_consensus.cc:2272] T 785733da86ff425a90ccfab2933fc89e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:08.532954 1175 ts_tablet_manager.cc:1916] T 785733da86ff425a90ccfab2933fc89e P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:08.533675 521 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet c9dff4dff82f49e7ad49caad5e404f09
I20251029 02:52:08.534659 755 tablet_service.cc:1552] Processing DeleteTablet for tablet c9dff4dff82f49e7ad49caad5e404f09 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:08 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:08.534758 2179 raft_consensus.cc:697] T 02e4588b03144ea2850e13a1f8f29f06 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Becoming Leader. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Running, Role: LEADER
I20251029 02:52:08.535017 2179 consensus_queue.cc:237] T 02e4588b03144ea2850e13a1f8f29f06 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.538600 1167 tablet_bootstrap.cc:654] T 777992acb1bd4765886b802940db74bb P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:08.539570 1025 tablet_service.cc:1552] Processing DeleteTablet for tablet 7a1866afc7564217ad777e7ce53786c5 with delete_type TABLET_DATA_DELETED (replaced by tablet dc33fd3404ec41c3935b02c84a12971b) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:08.541786 1039 tablet_service.cc:1552] Processing DeleteTablet for tablet 0f80b840ed954230bfbfe7e40127cd2f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:56 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:08.546844 520 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 906f1f20b80a4596af9c9ab20dfe7927
I20251029 02:52:08.547371 522 catalog_manager.cc:5649] T 1c53fc1fff1a4bb988beb46ff91fd3be P 0cfd4bee16f84b7c8b0ff3ce14db20d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } health_report { overall_health: HEALTHY } } }
I20251029 02:52:08.547871 1031 tablet_service.cc:1552] Processing DeleteTablet for tablet 777992acb1bd4765886b802940db74bb with delete_type TABLET_DATA_DELETED (replaced by tablet e65eaaae43ac435ebd665546de9d68dd) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:08.548832 1176 ts_tablet_manager.cc:1929] T b3de6670357c4ae59eb77469b2fb434c P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:08.549032 1176 log.cc:1199] T b3de6670357c4ae59eb77469b2fb434c P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/b3de6670357c4ae59eb77469b2fb434c
I20251029 02:52:08.549314 2201 raft_consensus.cc:493] T 785733da86ff425a90ccfab2933fc89e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:08.549459 2201 raft_consensus.cc:515] T 785733da86ff425a90ccfab2933fc89e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.550496 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f17b76f5dd4c4dd78d1fd45cc490763c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:08.550824 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 34d141bee474452693dba1b9d7f2b023. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:08.551121 2201 leader_election.cc:290] T 785733da86ff425a90ccfab2933fc89e P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
W20251029 02:52:08.551627 849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:45003, user_credentials={real_user=slave}} blocked reactor thread for 48952us
I20251029 02:52:08.551725 766 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "785733da86ff425a90ccfab2933fc89e" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:08.552269 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "785733da86ff425a90ccfab2933fc89e" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:52:08.552369 847 leader_election.cc:343] T 785733da86ff425a90ccfab2933fc89e P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Illegal state: Tablet not RUNNING: STOPPED
I20251029 02:52:08.552434 1060 raft_consensus.cc:2468] T 785733da86ff425a90ccfab2933fc89e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:08.553442 1166 tablet_bootstrap.cc:492] T dfa8b1b9a80e4e8689e0acb44e16346a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:08.553557 1166 ts_tablet_manager.cc:1403] T dfa8b1b9a80e4e8689e0acb44e16346a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.034s user 0.003s sys 0.005s
I20251029 02:52:08.554657 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a61593bfbfe4430ac85f8e5c65e56c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:08.558060 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d5d02c38f2a48cc94358bcaa9fe5065. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:08.558274 1175 ts_tablet_manager.cc:1929] T 785733da86ff425a90ccfab2933fc89e P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:08.558435 1175 log.cc:1199] T 785733da86ff425a90ccfab2933fc89e P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/785733da86ff425a90ccfab2933fc89e
I20251029 02:52:08.558828 1175 ts_tablet_manager.cc:1950] T 785733da86ff425a90ccfab2933fc89e P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:08.562849 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 938e7292913f456ba8588a4647b78ff9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:08.563270 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 770) took 1792 ms. Trace:
I20251029 02:52:08.563385 1175 rpcz_store.cc:276] 1029 02:52:06.771202 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:06.771323 (+ 121us) service_pool.cc:225] Handling call
1029 02:52:08.532024 (+1760701us) ts_tablet_manager.cc:1151] Deleting tablet 785733da86ff425a90ccfab2933fc89e
1029 02:52:08.532036 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:08.539589 (+ 7553us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:08.558166 (+ 18577us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:08.563242 (+ 5076us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1716166}
I20251029 02:52:08.563885 1025 tablet_service.cc:1552] Processing DeleteTablet for tablet c9dff4dff82f49e7ad49caad5e404f09 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:08 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:08.564116 1039 tablet_service.cc:1552] Processing DeleteTablet for tablet d3faa2d4df584c6e8da426a3830a6454 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:08 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:08.565104 1175 tablet_replica.cc:333] T 0d84dc8dc465496f9adec798b2e03e13 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:08.565037 1166 raft_consensus.cc:359] T dfa8b1b9a80e4e8689e0acb44e16346a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.565207 1166 raft_consensus.cc:385] T dfa8b1b9a80e4e8689e0acb44e16346a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:08.565260 1166 raft_consensus.cc:740] T dfa8b1b9a80e4e8689e0acb44e16346a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:08.565452 1166 consensus_queue.cc:260] T dfa8b1b9a80e4e8689e0acb44e16346a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.565963 1166 ts_tablet_manager.cc:1434] T dfa8b1b9a80e4e8689e0acb44e16346a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.012s user 0.000s sys 0.000s
I20251029 02:52:08.566202 1175 raft_consensus.cc:2243] T 0d84dc8dc465496f9adec798b2e03e13 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:08.566236 1166 tablet_bootstrap.cc:492] T 59d03a190b784ab18a3e63a572675891 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:08.566666 1175 raft_consensus.cc:2272] T 0d84dc8dc465496f9adec798b2e03e13 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:08.567752 1175 ts_tablet_manager.cc:1916] T 0d84dc8dc465496f9adec798b2e03e13 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:08.570487 1166 tablet_bootstrap.cc:654] T 59d03a190b784ab18a3e63a572675891 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:08.578296 1166 tablet_bootstrap.cc:492] T 59d03a190b784ab18a3e63a572675891 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:08.578447 1166 ts_tablet_manager.cc:1403] T 59d03a190b784ab18a3e63a572675891 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20251029 02:52:08.578461 1031 tablet_service.cc:1552] Processing DeleteTablet for tablet 71b4b91071a54d6aa66160e7955bc915 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:08 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:08.578893 1174 ts_tablet_manager.cc:1929] T 4aa78f4ac38e4919864c924e50f1aaf4 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:08.579056 1174 log.cc:1199] T 4aa78f4ac38e4919864c924e50f1aaf4 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/4aa78f4ac38e4919864c924e50f1aaf4
I20251029 02:52:08.579268 1166 raft_consensus.cc:359] T 59d03a190b784ab18a3e63a572675891 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.579448 1166 raft_consensus.cc:385] T 59d03a190b784ab18a3e63a572675891 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:08.579509 1166 raft_consensus.cc:740] T 59d03a190b784ab18a3e63a572675891 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:08.579494 1174 ts_tablet_manager.cc:1950] T 4aa78f4ac38e4919864c924e50f1aaf4 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:08.579751 1166 consensus_queue.cc:260] T 59d03a190b784ab18a3e63a572675891 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.580255 1166 ts_tablet_manager.cc:1434] T 59d03a190b784ab18a3e63a572675891 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251029 02:52:08.580520 1166 tablet_bootstrap.cc:492] T 49d9ce32e4f04f11be2ecefc5d523a8b P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:08.580631 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 801) took 1082 ms. Trace:
I20251029 02:52:08.580757 1174 rpcz_store.cc:276] 1029 02:52:07.498315 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:07.500056 (+ 1741us) service_pool.cc:225] Handling call
1029 02:52:08.239722 (+739666us) ts_tablet_manager.cc:1151] Deleting tablet 4aa78f4ac38e4919864c924e50f1aaf4
1029 02:52:08.239735 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:08.427957 (+188222us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:08.578795 (+150838us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:08.580555 (+ 1760us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":739138}
I20251029 02:52:08.581197 1174 tablet_replica.cc:333] T 519754f5d8f7484480f973c53708c5f3 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:08.581434 1174 raft_consensus.cc:2243] T 519754f5d8f7484480f973c53708c5f3 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:08.582240 1174 pending_rounds.cc:70] T 519754f5d8f7484480f973c53708c5f3 P c756528642a543eeacd93cef3e335a7d: Trying to abort 1 pending ops.
I20251029 02:52:08.582264 1166 tablet_bootstrap.cc:654] T 49d9ce32e4f04f11be2ecefc5d523a8b P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:08.582369 1174 pending_rounds.cc:77] T 519754f5d8f7484480f973c53708c5f3 P c756528642a543eeacd93cef3e335a7d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215949117349736448 op_type: NO_OP noop_request { }
I20251029 02:52:08.582855 1174 raft_consensus.cc:2889] T 519754f5d8f7484480f973c53708c5f3 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 02:52:08.560089 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_957971e954a244309b62d05bb172dc9d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:08.583016 1174 raft_consensus.cc:2272] T 519754f5d8f7484480f973c53708c5f3 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
W20251029 02:52:08.583428 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_957971e954a244309b62d05bb172dc9d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:08.583683 1174 ts_tablet_manager.cc:1916] T 519754f5d8f7484480f973c53708c5f3 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:08.584204 1175 ts_tablet_manager.cc:1929] T 0d84dc8dc465496f9adec798b2e03e13 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:08.584308 1175 log.cc:1199] T 0d84dc8dc465496f9adec798b2e03e13 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/0d84dc8dc465496f9adec798b2e03e13
I20251029 02:52:08.584589 1175 ts_tablet_manager.cc:1950] T 0d84dc8dc465496f9adec798b2e03e13 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:08.585096 522 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet 1c53fc1fff1a4bb988beb46ff91fd3be (table default.Table_dd4934aabcd442039db111954d819f6d [id=9abdb27ccc0a48e991b7a4619808fdf8]). Expected version 1 got 0
I20251029 02:52:08.585326 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 783) took 1635 ms. Trace:
I20251029 02:52:08.585414 1175 rpcz_store.cc:276] 1029 02:52:06.949995 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:06.950040 (+ 45us) service_pool.cc:225] Handling call
1029 02:52:08.565036 (+1614996us) ts_tablet_manager.cc:1151] Deleting tablet 0d84dc8dc465496f9adec798b2e03e13
1029 02:52:08.565053 (+ 17us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:08.576089 (+ 11036us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:08.584104 (+ 8015us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:08.585305 (+ 1201us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1614781}
I20251029 02:52:08.562078 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_f060d401cf984a96b90cf4735c9f34d2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:08.562529 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_0a146ce6c31f4dbf994a2442f2386d93"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:08.564358 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 785733da86ff425a90ccfab2933fc89e (table default.Table_bdefc7775234406788f3f7e220769b43 [id=b81476e2d6db40ef9f97211bdb79ce21]) successfully deleted
I20251029 02:52:08.565565 523 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_b72f4af954b8442d9a8a477f69bf3b62" } modify_external_catalogs: true
I20251029 02:52:08.588120 755 tablet_service.cc:1505] Processing CreateTablet for tablet 8d5d02c38f2a48cc94358bcaa9fe5065 (DEFAULT_TABLE table=default.Table_e3aae8a6f0314d75b52739cd81d573d9 [id=7f20f69c0a6e4355bd2c997249a05fc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:08.588578 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d5d02c38f2a48cc94358bcaa9fe5065. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:08.587532 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 0d84dc8dc465496f9adec798b2e03e13 (table default.Table_0d5730d385074a70b295b233e6ecd79e [id=423993f75a9948aebe1dee01d3e12702]) successfully deleted
W20251029 02:52:08.587560 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0a146ce6c31f4dbf994a2442f2386d93 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:08.591326 750 tablet_service.cc:1505] Processing CreateTablet for tablet 753446e48bed460cb1f9e03868b0d3a4 (DEFAULT_TABLE table=default.Table_f8a743f1d5174309921ff1ff7f6b10af [id=2f787bfc9fc342ca8f70d6f82cd38c20]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:08.591677 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 753446e48bed460cb1f9e03868b0d3a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:08.585562 522 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet 1bdb67125493430398ff5c80b73de2ac (table default.Table_2846af457a4b4e27a739af50c7f8c7f7 [id=8c3d3d88395e4712af0f62c4ad14edea]). Expected version 3 got 0
I20251029 02:52:08.594810 749 tablet_service.cc:1505] Processing CreateTablet for tablet c3ce63a996da494ba0775ba533c1a007 (DEFAULT_TABLE table=default.Table_c0fb3766a9c04aa9b2e3caca2e7b9b3a [id=b0f2cffd7c83453bbc48a89508a6f479]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:08.595196 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3ce63a996da494ba0775ba533c1a007. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:08.598641 1176 ts_tablet_manager.cc:1950] T b3de6670357c4ae59eb77469b2fb434c P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:08.598855 756 tablet_service.cc:1505] Processing CreateTablet for tablet 195ddd7cae484f488d9d6277db9fdd15 (DEFAULT_TABLE table=default.Table_f6c47489fbba4d65bbab23c587dddb63 [id=01cc6886c0ef425c80b39d3d7d6dcee5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:08.599275 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 195ddd7cae484f488d9d6277db9fdd15. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:08.600293 1176 tablet_replica.cc:333] T d2990e9f626c40618c118266bc595684 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:08.600500 1176 raft_consensus.cc:2243] T d2990e9f626c40618c118266bc595684 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:08.600829 1176 raft_consensus.cc:2272] T d2990e9f626c40618c118266bc595684 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:08.601500 1176 ts_tablet_manager.cc:1916] T d2990e9f626c40618c118266bc595684 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:08.601752 891 tablet_service.cc:1552] Processing DeleteTablet for tablet f529e17e97074dca91ec2f48954cfd32 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:06 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:08.601984 898 tablet_service.cc:1552] Processing DeleteTablet for tablet c9dff4dff82f49e7ad49caad5e404f09 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:08 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:08.601979 885 tablet_service.cc:1552] Processing DeleteTablet for tablet d3faa2d4df584c6e8da426a3830a6454 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:08 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:08.602243 892 tablet_service.cc:1552] Processing DeleteTablet for tablet 71b4b91071a54d6aa66160e7955bc915 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:08 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:08.602667 891 tablet_service.cc:1505] Processing CreateTablet for tablet 753446e48bed460cb1f9e03868b0d3a4 (DEFAULT_TABLE table=default.Table_f8a743f1d5174309921ff1ff7f6b10af [id=2f787bfc9fc342ca8f70d6f82cd38c20]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:08.603122 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 753446e48bed460cb1f9e03868b0d3a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:08.603191 2113 raft_consensus.cc:493] T b6d4f7bf702f4834bc5f4ef2b50b92a7 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:08.603323 2113 raft_consensus.cc:515] T b6d4f7bf702f4834bc5f4ef2b50b92a7 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.603904 2113 leader_election.cc:290] T b6d4f7bf702f4834bc5f4ef2b50b92a7 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:08.604538 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b6d4f7bf702f4834bc5f4ef2b50b92a7" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:08.587641 523 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_b72f4af954b8442d9a8a477f69bf3b62" } modify_external_catalogs: true
I20251029 02:52:08.605659 890 tablet_service.cc:1505] Processing CreateTablet for tablet c3ce63a996da494ba0775ba533c1a007 (DEFAULT_TABLE table=default.Table_c0fb3766a9c04aa9b2e3caca2e7b9b3a [id=b0f2cffd7c83453bbc48a89508a6f479]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:08.606014 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3ce63a996da494ba0775ba533c1a007. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:08.606194 711 leader_election.cc:343] T b6d4f7bf702f4834bc5f4ef2b50b92a7 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:08.606684 1166 tablet_bootstrap.cc:492] T 49d9ce32e4f04f11be2ecefc5d523a8b P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:08.606604 900 tablet_service.cc:1552] Processing DeleteTablet for tablet c9bd3f2deb5d4af09b7918269a98509f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:58 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:08.606813 1166 ts_tablet_manager.cc:1403] T 49d9ce32e4f04f11be2ecefc5d523a8b P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.026s user 0.000s sys 0.009s
I20251029 02:52:08.607636 1166 raft_consensus.cc:359] T 49d9ce32e4f04f11be2ecefc5d523a8b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.607797 1166 raft_consensus.cc:385] T 49d9ce32e4f04f11be2ecefc5d523a8b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:08.607846 1166 raft_consensus.cc:740] T 49d9ce32e4f04f11be2ecefc5d523a8b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:08.608074 1166 consensus_queue.cc:260] T 49d9ce32e4f04f11be2ecefc5d523a8b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.608531 1166 ts_tablet_manager.cc:1434] T 49d9ce32e4f04f11be2ecefc5d523a8b P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251029 02:52:08.608850 1166 tablet_bootstrap.cc:492] T 0d84dc8dc465496f9adec798b2e03e13 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:08.609676 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet b3de6670357c4ae59eb77469b2fb434c (table default.Table_8dacd5c558a04e47b424533b985aee9e [id=f9c7128476404646a9194e31f49c94e8]) successfully deleted
W20251029 02:52:08.609941 513 catalog_manager.cc:4952] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 9e6eb099648549dbbf9969e59865f51f because the tablet was not found. No further retry: Not found: Tablet not found: 9e6eb099648549dbbf9969e59865f51f
W20251029 02:52:08.610105 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 48aa20640dfe43288326ed6d6b9bc1cf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 48aa20640dfe43288326ed6d6b9bc1cf already in progress: creating tablet
W20251029 02:52:08.610275 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 65e761f9f9fb490586d6415a5a8565dd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 65e761f9f9fb490586d6415a5a8565dd already in progress: creating tablet
I20251029 02:52:08.611583 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_95f035fda47144eb9d79be23614b41b6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:08.611661 849 leader_election.cc:304] T 785733da86ff425a90ccfab2933fc89e P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters: b495a49349324f7d81908f2da700f0d9
I20251029 02:52:08.611927 2201 raft_consensus.cc:2804] T 785733da86ff425a90ccfab2933fc89e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:08.611996 2201 raft_consensus.cc:493] T 785733da86ff425a90ccfab2933fc89e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:08.612056 2201 raft_consensus.cc:3060] T 785733da86ff425a90ccfab2933fc89e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
W20251029 02:52:08.612355 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_95f035fda47144eb9d79be23614b41b6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:08.614780 2201 raft_consensus.cc:515] T 785733da86ff425a90ccfab2933fc89e P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.615127 890 tablet_service.cc:1505] Processing CreateTablet for tablet 195ddd7cae484f488d9d6277db9fdd15 (DEFAULT_TABLE table=default.Table_f6c47489fbba4d65bbab23c587dddb63 [id=01cc6886c0ef425c80b39d3d7d6dcee5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:08.615190 900 tablet_service.cc:1505] Processing CreateTablet for tablet 57898c0257894f899613d7578088cbb3 (DEFAULT_TABLE table=default.Table_603240f584a74f1d965ee97ee1a2d2dc [id=2f93923f0a254ad1ba33ef8b44f2014b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:08.615491 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 195ddd7cae484f488d9d6277db9fdd15. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:08.615628 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a1866afc7564217ad777e7ce53786c5" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:08.615995 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57898c0257894f899613d7578088cbb3. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:08.616108 847 leader_election.cc:343] T 785733da86ff425a90ccfab2933fc89e P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 785733da86ff425a90ccfab2933fc89e
I20251029 02:52:08.616489 2201 leader_election.cc:290] T 785733da86ff425a90ccfab2933fc89e P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:08.617187 1167 tablet_bootstrap.cc:492] T 777992acb1bd4765886b802940db74bb P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:08.617357 1167 ts_tablet_manager.cc:1403] T 777992acb1bd4765886b802940db74bb P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.141s user 0.000s sys 0.003s
I20251029 02:52:08.618192 1167 raft_consensus.cc:359] T 777992acb1bd4765886b802940db74bb P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.618417 1167 raft_consensus.cc:385] T 777992acb1bd4765886b802940db74bb P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:08.618986 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7a1866afc7564217ad777e7ce53786c5" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
I20251029 02:52:08.619903 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "785733da86ff425a90ccfab2933fc89e" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:52:08.620100 1060 raft_consensus.cc:3060] T 785733da86ff425a90ccfab2933fc89e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:08.621364 1167 raft_consensus.cc:740] T 777992acb1bd4765886b802940db74bb P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:08.621598 1167 consensus_queue.cc:260] T 777992acb1bd4765886b802940db74bb P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.622088 1167 ts_tablet_manager.cc:1434] T 777992acb1bd4765886b802940db74bb P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20251029 02:52:08.622416 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 4aa78f4ac38e4919864c924e50f1aaf4 (table default.Table_e168a30d3a1e48ed8053847b304036e9 [id=a93c34a9cd344503954bfae8752a69a6]) successfully deleted
I20251029 02:52:08.622764 1167 tablet_bootstrap.cc:492] T ccdca31db2834de4b6ed0f2d67a10535 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:08.622844 1060 raft_consensus.cc:2468] T 785733da86ff425a90ccfab2933fc89e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
W20251029 02:52:08.623092 990 leader_election.cc:343] T 7a1866afc7564217ad777e7ce53786c5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:08.624186 849 leader_election.cc:304] T 785733da86ff425a90ccfab2933fc89e P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters: b495a49349324f7d81908f2da700f0d9
I20251029 02:52:08.624748 2109 raft_consensus.cc:2804] T 785733da86ff425a90ccfab2933fc89e P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:08.624869 2109 raft_consensus.cc:697] T 785733da86ff425a90ccfab2933fc89e P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:52:08.625097 2109 consensus_queue.cc:237] T 785733da86ff425a90ccfab2933fc89e P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.570107 521 master_service.cc:954] ReplaceTablet: received request to replace tablet 372d93914abc41b38a702742b7db0bc2 from {username='slave'} at 127.0.0.1:35314
I20251029 02:52:08.581930 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_fdb0a130609a45878b3a6f07cf79947c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:08.629858 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fdb0a130609a45878b3a6f07cf79947c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:08.633168 1166 tablet_bootstrap.cc:654] T 0d84dc8dc465496f9adec798b2e03e13 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:08.634951 1166 tablet_bootstrap.cc:492] T 0d84dc8dc465496f9adec798b2e03e13 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:08.635159 1166 ts_tablet_manager.cc:1403] T 0d84dc8dc465496f9adec798b2e03e13 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.002s
I20251029 02:52:08.636013 1166 raft_consensus.cc:359] T 0d84dc8dc465496f9adec798b2e03e13 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.636169 1166 raft_consensus.cc:385] T 0d84dc8dc465496f9adec798b2e03e13 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:08.636212 1166 raft_consensus.cc:740] T 0d84dc8dc465496f9adec798b2e03e13 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:08.636401 1166 consensus_queue.cc:260] T 0d84dc8dc465496f9adec798b2e03e13 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.636907 1166 ts_tablet_manager.cc:1434] T 0d84dc8dc465496f9adec798b2e03e13 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251029 02:52:08.637249 1166 tablet_bootstrap.cc:492] T 13ee62fc2339410b9db70dfd3d3c6fa6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:08.639097 1166 tablet_bootstrap.cc:654] T 13ee62fc2339410b9db70dfd3d3c6fa6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:08.640285 748 tablet_service.cc:1505] Processing CreateTablet for tablet e60b029eb3ec466cbdced20d536f4f47 (DEFAULT_TABLE table=default.Table_27e1e56b7ff0486facdfdd318dc845fc [id=0b1aa07832b94a9f990a564ff353a0f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:08.640841 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e60b029eb3ec466cbdced20d536f4f47. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:08.588044 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f060d401cf984a96b90cf4735c9f34d2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:08.642462 2113 raft_consensus.cc:493] T e7f846b294cc4eb687a26c8b5931ac19 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:08.642663 2113 raft_consensus.cc:515] T e7f846b294cc4eb687a26c8b5931ac19 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.643373 2113 leader_election.cc:290] T e7f846b294cc4eb687a26c8b5931ac19 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:08.650436 2205 raft_consensus.cc:493] T 777992acb1bd4765886b802940db74bb P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:08.650758 2205 raft_consensus.cc:515] T 777992acb1bd4765886b802940db74bb P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.651397 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e7f846b294cc4eb687a26c8b5931ac19" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:08.652019 711 leader_election.cc:343] T e7f846b294cc4eb687a26c8b5931ac19 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:08.652171 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "777992acb1bd4765886b802940db74bb" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:08.652415 768 raft_consensus.cc:2468] T 777992acb1bd4765886b802940db74bb P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:08.653069 847 leader_election.cc:304] T 777992acb1bd4765886b802940db74bb P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:08.653787 2201 raft_consensus.cc:2804] T 777992acb1bd4765886b802940db74bb P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:08.653888 2201 raft_consensus.cc:493] T 777992acb1bd4765886b802940db74bb P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:08.653966 2201 raft_consensus.cc:3060] T 777992acb1bd4765886b802940db74bb P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:08.657181 2201 raft_consensus.cc:515] T 777992acb1bd4765886b802940db74bb P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.657305 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_2aba26f0828343a6981aad6b2e3ac97f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:08.657918 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2aba26f0828343a6981aad6b2e3ac97f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:08.658385 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "777992acb1bd4765886b802940db74bb" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:08.658605 768 raft_consensus.cc:3060] T 777992acb1bd4765886b802940db74bb P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:08.659235 518 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_dd6aaf5ed87548009b3025e430874d1e" } new_table_name: "default.Table_2c15df0a33124ce69282c48d6455d1b2" modify_external_catalogs: true
I20251029 02:52:08.661310 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_8d301ab34d9e48b69964f88339f88120"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:08.662272 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8d301ab34d9e48b69964f88339f88120 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:08.662613 2188 raft_consensus.cc:493] T d61beda662d844cdaf67fd9a2e382ea2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:08.662752 2188 raft_consensus.cc:515] T d61beda662d844cdaf67fd9a2e382ea2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.664217 2188 leader_election.cc:290] T d61beda662d844cdaf67fd9a2e382ea2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:08.665203 768 raft_consensus.cc:2468] T 777992acb1bd4765886b802940db74bb P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:08.665881 847 leader_election.cc:304] T 777992acb1bd4765886b802940db74bb P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:08.668841 2201 leader_election.cc:290] T 777992acb1bd4765886b802940db74bb P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:08.669613 2201 raft_consensus.cc:2804] T 777992acb1bd4765886b802940db74bb P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:08.670028 2201 raft_consensus.cc:697] T 777992acb1bd4765886b802940db74bb P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:52:08.670580 2201 consensus_queue.cc:237] T 777992acb1bd4765886b802940db74bb P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.671142 750 tablet_service.cc:1505] Processing CreateTablet for tablet 57898c0257894f899613d7578088cbb3 (DEFAULT_TABLE table=default.Table_603240f584a74f1d965ee97ee1a2d2dc [id=2f93923f0a254ad1ba33ef8b44f2014b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:08.671555 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57898c0257894f899613d7578088cbb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:08.673403 2205 leader_election.cc:290] T 777992acb1bd4765886b802940db74bb P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:08.677121 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "777992acb1bd4765886b802940db74bb" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:52:08.677403 1060 raft_consensus.cc:3060] T 777992acb1bd4765886b802940db74bb P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:08.680485 1039 tablet_service.cc:1505] Processing CreateTablet for tablet c3ce63a996da494ba0775ba533c1a007 (DEFAULT_TABLE table=default.Table_c0fb3766a9c04aa9b2e3caca2e7b9b3a [id=b0f2cffd7c83453bbc48a89508a6f479]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:08.681186 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3ce63a996da494ba0775ba533c1a007. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:08.681229 1060 raft_consensus.cc:2468] T 777992acb1bd4765886b802940db74bb P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:08.682337 1031 tablet_service.cc:1505] Processing CreateTablet for tablet 753446e48bed460cb1f9e03868b0d3a4 (DEFAULT_TABLE table=default.Table_f8a743f1d5174309921ff1ff7f6b10af [id=2f787bfc9fc342ca8f70d6f82cd38c20]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:08.683009 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 753446e48bed460cb1f9e03868b0d3a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:08.687093 1025 tablet_service.cc:1505] Processing CreateTablet for tablet 195ddd7cae484f488d9d6277db9fdd15 (DEFAULT_TABLE table=default.Table_f6c47489fbba4d65bbab23c587dddb63 [id=01cc6886c0ef425c80b39d3d7d6dcee5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:08.687137 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d61beda662d844cdaf67fd9a2e382ea2" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:08.687346 768 raft_consensus.cc:2468] T d61beda662d844cdaf67fd9a2e382ea2 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 0.
I20251029 02:52:08.687604 1042 tablet_service.cc:1505] Processing CreateTablet for tablet 57898c0257894f899613d7578088cbb3 (DEFAULT_TABLE table=default.Table_603240f584a74f1d965ee97ee1a2d2dc [id=2f93923f0a254ad1ba33ef8b44f2014b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:08.688372 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "777992acb1bd4765886b802940db74bb" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:08.688537 1060 raft_consensus.cc:2376] T 777992acb1bd4765886b802940db74bb P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1. Re-sending same reply.
I20251029 02:52:08.689381 2201 raft_consensus.cc:493] T dfa8b1b9a80e4e8689e0acb44e16346a P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:08.689582 2201 raft_consensus.cc:515] T dfa8b1b9a80e4e8689e0acb44e16346a P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.692031 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57898c0257894f899613d7578088cbb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:08.692288 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 195ddd7cae484f488d9d6277db9fdd15. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:08.693859 523 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 1762d94243854844bc4305d5a44b5e61
I20251029 02:52:08.696938 1032 tablet_service.cc:1552] Processing DeleteTablet for tablet d61beda662d844cdaf67fd9a2e382ea2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:55 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:08.697304 1032 tablet_service.cc:1552] Processing DeleteTablet for tablet f529e17e97074dca91ec2f48954cfd32 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:06 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:08.697543 1032 tablet_service.cc:1552] Processing DeleteTablet for tablet 785733da86ff425a90ccfab2933fc89e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:54 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:08.698035 1032 tablet_service.cc:1505] Processing CreateTablet for tablet e60b029eb3ec466cbdced20d536f4f47 (DEFAULT_TABLE table=default.Table_27e1e56b7ff0486facdfdd318dc845fc [id=0b1aa07832b94a9f990a564ff353a0f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:08.698364 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e60b029eb3ec466cbdced20d536f4f47. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:08.699478 2201 leader_election.cc:290] T dfa8b1b9a80e4e8689e0acb44e16346a P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:08.703298 1167 tablet_bootstrap.cc:654] T ccdca31db2834de4b6ed0f2d67a10535 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:08.703469 1172 tablet_bootstrap.cc:492] T 28f7ceb91a34450cad4b5b45f5285b41 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:08.703629 1172 ts_tablet_manager.cc:1403] T 28f7ceb91a34450cad4b5b45f5285b41 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.267s user 0.002s sys 0.003s
I20251029 02:52:08.704411 1172 raft_consensus.cc:359] T 28f7ceb91a34450cad4b5b45f5285b41 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.704592 1172 raft_consensus.cc:385] T 28f7ceb91a34450cad4b5b45f5285b41 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:08.704655 1172 raft_consensus.cc:740] T 28f7ceb91a34450cad4b5b45f5285b41 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:08.704849 1172 consensus_queue.cc:260] T 28f7ceb91a34450cad4b5b45f5285b41 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.705356 1172 ts_tablet_manager.cc:1434] T 28f7ceb91a34450cad4b5b45f5285b41 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251029 02:52:08.704932 1032 tablet_service.cc:1505] Processing CreateTablet for tablet 349628fa90ca48ae906070deb3f9ca79 (DEFAULT_TABLE table=default.Table_b7096d022c484e079ae5c12c056da371 [id=8835dbcc777a49eba346b58bb11d6478]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:08.706477 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 349628fa90ca48ae906070deb3f9ca79. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:08.707945 523 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_7d8a27a8c05d4e9686a1fd616c5697bd" } new_table_name: "default.Table_aae2119a32f548aa829d7f408c573777" modify_external_catalogs: true
I20251029 02:52:08.711382 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dfa8b1b9a80e4e8689e0acb44e16346a" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:08.711560 1056 raft_consensus.cc:2468] T dfa8b1b9a80e4e8689e0acb44e16346a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:08.712162 525 master_service.cc:954] ReplaceTablet: received request to replace tablet fcb3db8399aa4cf188b07dda220f7f5b from {username='slave'} at 127.0.0.1:35314
I20251029 02:52:08.714654 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_7fd7b56a9c09475a80cce932d6e15cc1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:08.714852 519 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_045fe56124d445f987ef1b91282c7db5" } new_table_name: "default.Table_96e017f445bc4f1fbebddf65c722b8de" modify_external_catalogs: true
I20251029 02:52:08.715191 2214 raft_consensus.cc:493] T 9e6eb099648549dbbf9969e59865f51f P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251029 02:52:08.715235 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7fd7b56a9c09475a80cce932d6e15cc1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:08.715332 2214 raft_consensus.cc:515] T 9e6eb099648549dbbf9969e59865f51f P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.715957 2214 leader_election.cc:290] T 9e6eb099648549dbbf9969e59865f51f P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:08.717669 849 leader_election.cc:304] T dfa8b1b9a80e4e8689e0acb44e16346a P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters:
W20251029 02:52:08.718046 849 consensus_peers.cc:597] T b3de6670357c4ae59eb77469b2fb434c P c756528642a543eeacd93cef3e335a7d -> Peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Couldn't send request to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b3de6670357c4ae59eb77469b2fb434c. This is attempt 1: this message will repeat every 5th retry.
W20251029 02:52:08.718250 849 leader_election.cc:343] T 9e6eb099648549dbbf9969e59865f51f P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Not found: Tablet not found: 9e6eb099648549dbbf9969e59865f51f
I20251029 02:52:08.718461 2214 raft_consensus.cc:2804] T dfa8b1b9a80e4e8689e0acb44e16346a P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:08.718595 2214 raft_consensus.cc:493] T dfa8b1b9a80e4e8689e0acb44e16346a P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:08.718868 2214 raft_consensus.cc:3060] T dfa8b1b9a80e4e8689e0acb44e16346a P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:08.719453 748 tablet_service.cc:1505] Processing CreateTablet for tablet d14a67fa17fe481baffc9f56bfaf5d3c (DEFAULT_TABLE table=default.Table_f908888068114a9083c998cf80a8441a [id=d7f17b2899964e69b96000482b3e0571]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:08.719853 756 tablet_service.cc:1505] Processing CreateTablet for tablet c92a3c77be8943fb929d321559e3a46c (DEFAULT_TABLE table=default.Table_f3da247a6675435a9c881f23dd8ff376 [id=e6beb2f778414c4bb1987db73a86a017]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:08.720178 755 tablet_service.cc:1505] Processing CreateTablet for tablet 21c73cb2f1084827b581fb09cf7239a1 (DEFAULT_TABLE table=default.Table_0427cb4a53de4e62a4b89829586a2a53 [id=3681cfb9581f415b9fe97c2a8467cc28]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:08.720511 749 tablet_service.cc:1505] Processing CreateTablet for tablet 349628fa90ca48ae906070deb3f9ca79 (DEFAULT_TABLE table=default.Table_b7096d022c484e079ae5c12c056da371 [id=8835dbcc777a49eba346b58bb11d6478]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:08.722262 2214 raft_consensus.cc:515] T dfa8b1b9a80e4e8689e0acb44e16346a P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.723044 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b6d4f7bf702f4834bc5f4ef2b50b92a7" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:08.723594 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e7f846b294cc4eb687a26c8b5931ac19" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:08.730649 766 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e6eb099648549dbbf9969e59865f51f" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:08.730881 766 raft_consensus.cc:2468] T 9e6eb099648549dbbf9969e59865f51f P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:08.731561 1172 tablet_bootstrap.cc:492] T 8048506ced4540a5aee9f79bfc4599bf P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:08.733033 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d14a67fa17fe481baffc9f56bfaf5d3c. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:08.733808 712 leader_election.cc:343] T b6d4f7bf702f4834bc5f4ef2b50b92a7 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:08.733948 712 leader_election.cc:304] T b6d4f7bf702f4834bc5f4ef2b50b92a7 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:08.734485 2188 raft_consensus.cc:493] T 5db519edc75d4bd0985b32322825ffae P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:08.734602 2113 raft_consensus.cc:2749] T b6d4f7bf702f4834bc5f4ef2b50b92a7 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:08.734658 2188 raft_consensus.cc:515] T 5db519edc75d4bd0985b32322825ffae P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.735270 2188 leader_election.cc:290] T 5db519edc75d4bd0985b32322825ffae P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:08.735754 1175 tablet_replica.cc:333] T 777992acb1bd4765886b802940db74bb P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:08.736003 1175 raft_consensus.cc:2243] T 777992acb1bd4765886b802940db74bb P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:08.736203 1175 raft_consensus.cc:2272] T 777992acb1bd4765886b802940db74bb P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:08.736768 1175 ts_tablet_manager.cc:1916] T 777992acb1bd4765886b802940db74bb P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:08.737287 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21c73cb2f1084827b581fb09cf7239a1. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:08.737382 712 leader_election.cc:343] T e7f846b294cc4eb687a26c8b5931ac19 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:08.742704 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c92a3c77be8943fb929d321559e3a46c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:08.743371 1172 tablet_bootstrap.cc:654] T 8048506ced4540a5aee9f79bfc4599bf P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:08.746985 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 349628fa90ca48ae906070deb3f9ca79. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:08.753540 989 leader_election.cc:304] T d61beda662d844cdaf67fd9a2e382ea2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters:
W20251029 02:52:08.753968 512 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.60:41233, user_credentials={real_user=slave}} blocked reactor thread for 54584.2us
I20251029 02:52:08.756527 1175 ts_tablet_manager.cc:1929] T 777992acb1bd4765886b802940db74bb P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:08.756654 1175 log.cc:1199] T 777992acb1bd4765886b802940db74bb P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/777992acb1bd4765886b802940db74bb
I20251029 02:52:08.757011 1175 ts_tablet_manager.cc:1950] T 777992acb1bd4765886b802940db74bb P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:08.757787 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 781) took 1807 ms. Trace:
I20251029 02:52:08.757871 1175 rpcz_store.cc:276] 1029 02:52:06.949848 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:06.950318 (+ 470us) service_pool.cc:225] Handling call
1029 02:52:08.711628 (+1761310us) ts_tablet_manager.cc:1151] Deleting tablet 777992acb1bd4765886b802940db74bb
1029 02:52:08.735680 (+ 24052us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:08.741395 (+ 5715us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:08.756408 (+ 15013us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:08.757763 (+ 1355us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1761170}
I20251029 02:52:08.758204 1175 tablet_replica.cc:333] T 59d03a190b784ab18a3e63a572675891 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:08.759708 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 777992acb1bd4765886b802940db74bb (table default.Table_d9fe74a49f4843b089aae7fca9e4a826 [id=152882c742b2459ba863fd41083284c1]) successfully deleted
I20251029 02:52:08.759774 521 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 372d93914abc41b38a702742b7db0bc2
I20251029 02:52:08.760437 751 tablet_service.cc:1552] Processing DeleteTablet for tablet 372d93914abc41b38a702742b7db0bc2 with delete_type TABLET_DATA_DELETED (replaced by tablet dab6ed2266ed42508738399d388c4da3) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:08.760957 521 catalog_manager.cc:6719] ReplaceTablet: tablet 372d93914abc41b38a702742b7db0bc2 deleted and replaced by tablet dab6ed2266ed42508738399d388c4da3
I20251029 02:52:08.762080 751 tablet_service.cc:1505] Processing CreateTablet for tablet 2e37717c241c4d4a8fecc32394764b5c (DEFAULT_TABLE table=default.Table_2c913689f8134363ae4a20a05f717d3e [id=beb2ccbc9cae46bea0d1d75544a5f77a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:08.762454 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e37717c241c4d4a8fecc32394764b5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:08.766783 747 tablet_service.cc:1505] Processing CreateTablet for tablet ad58c7f066f6408490c5398abf077ee7 (DEFAULT_TABLE table=default.Table_6a444f431d374afa8f25c801ef2a92d7 [id=c2437d0dcf3d4774b60d4ff6adb638e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:08.767163 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad58c7f066f6408490c5398abf077ee7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:08.769253 1175 raft_consensus.cc:2243] T 59d03a190b784ab18a3e63a572675891 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:08.769410 1175 raft_consensus.cc:2272] T 59d03a190b784ab18a3e63a572675891 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:08.769904 1175 ts_tablet_manager.cc:1916] T 59d03a190b784ab18a3e63a572675891 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:08.770817 738 tablet_service.cc:1505] Processing CreateTablet for tablet 4eec207d19f540518bb567d7ab21d048 (DEFAULT_TABLE table=default.Table_c4333d8c61d346b1ac1a809d3ccb1915 [id=6d576d3499e649d78000c31fc5dab58c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:08.770817 742 tablet_service.cc:1505] Processing CreateTablet for tablet a8cef3074aef43b2981eaffdc5a5b5bb (DEFAULT_TABLE table=default.Table_ec7e5a0085cd43e28618d7b9d3d30c65 [id=09edb028a40a4418b5624916a9223c66]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:08.771266 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8cef3074aef43b2981eaffdc5a5b5bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:08.771338 745 tablet_service.cc:1505] Processing CreateTablet for tablet d3faa2d4df584c6e8da426a3830a6454 (DEFAULT_TABLE table=default.Table_f435bdcca42e4087ac8794859d9108ee [id=bce593f4f7934589b71f8a2d694cd9b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:08.771732 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3faa2d4df584c6e8da426a3830a6454. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:08.772038 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4eec207d19f540518bb567d7ab21d048. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:08.772327 741 tablet_service.cc:1505] Processing CreateTablet for tablet fe5aff0f61a44b91bfb37b42457a91f0 (DEFAULT_TABLE table=default.Table_df7d932b50b64537b4dbc68a846665fc [id=d7c62808bc9a4e74b85ba6a0ddb01495]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:08.772737 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe5aff0f61a44b91bfb37b42457a91f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:08.774726 743 tablet_service.cc:1505] Processing CreateTablet for tablet 26ae44429dc24164b00489dcd90b7d8a (DEFAULT_TABLE table=default.Table_e72155685e4942c5916842ac8547cf57 [id=9eea17ba5cba4d6dba8091486ce2779a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:08.774948 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26ae44429dc24164b00489dcd90b7d8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:08.783928 756 tablet_service.cc:1505] Processing CreateTablet for tablet ef2e9dd8f90a47389d9062d1f99a1da1 (DEFAULT_TABLE table=default.Table_d7cbd7b4c4bf4886a4ccf4c89da3cbce [id=9107fd53f5c94d2d80be50ef1f7c8e9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:08.784451 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef2e9dd8f90a47389d9062d1f99a1da1. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:08.791004 510 consensus_peers.cc:597] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0 -> Peer e9672114a6a742e2864c16ff60ba246e (127.0.104.62:42343): Couldn't send request to peer e9672114a6a742e2864c16ff60ba246e. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.62:42343: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20251029 02:52:08.792390 2214 leader_election.cc:290] T dfa8b1b9a80e4e8689e0acb44e16346a P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:08.793047 1174 ts_tablet_manager.cc:1929] T 519754f5d8f7484480f973c53708c5f3 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:08.793161 1174 log.cc:1199] T 519754f5d8f7484480f973c53708c5f3 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/519754f5d8f7484480f973c53708c5f3
I20251029 02:52:08.793512 1174 ts_tablet_manager.cc:1950] T 519754f5d8f7484480f973c53708c5f3 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:08.794396 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 804) took 1295 ms. Trace:
I20251029 02:52:08.794487 1174 rpcz_store.cc:276] 1029 02:52:07.498588 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:07.500801 (+ 2213us) service_pool.cc:225] Handling call
1029 02:52:08.581145 (+1080344us) ts_tablet_manager.cc:1151] Deleting tablet 519754f5d8f7484480f973c53708c5f3
1029 02:52:08.581157 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:08.723777 (+142620us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:08.792930 (+ 69153us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:08.794371 (+ 1441us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1079556}
I20251029 02:52:08.794886 1174 tablet_replica.cc:333] T b3de6670357c4ae59eb77469b2fb434c P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:08.795092 1174 raft_consensus.cc:2243] T b3de6670357c4ae59eb77469b2fb434c P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:08.795398 1174 pending_rounds.cc:70] T b3de6670357c4ae59eb77469b2fb434c P c756528642a543eeacd93cef3e335a7d: Trying to abort 1 pending ops.
I20251029 02:52:08.795446 1174 pending_rounds.cc:77] T b3de6670357c4ae59eb77469b2fb434c P c756528642a543eeacd93cef3e335a7d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215949117486166016 op_type: NO_OP noop_request { }
I20251029 02:52:08.795544 1174 raft_consensus.cc:2889] T b3de6670357c4ae59eb77469b2fb434c P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 02:52:08.795625 1174 raft_consensus.cc:2272] T b3de6670357c4ae59eb77469b2fb434c P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:08.796231 1174 ts_tablet_manager.cc:1916] T b3de6670357c4ae59eb77469b2fb434c P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:08.731576 847 leader_election.cc:304] T 9e6eb099648549dbbf9969e59865f51f P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5
I20251029 02:52:08.760756 1167 tablet_bootstrap.cc:492] T ccdca31db2834de4b6ed0f2d67a10535 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:08.797912 1167 ts_tablet_manager.cc:1403] T ccdca31db2834de4b6ed0f2d67a10535 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.175s user 0.003s sys 0.000s
W20251029 02:52:08.798009 847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:38481, user_credentials={real_user=slave}} blocked reactor thread for 66458.5us
I20251029 02:52:08.798696 1167 raft_consensus.cc:359] T ccdca31db2834de4b6ed0f2d67a10535 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.798863 1167 raft_consensus.cc:385] T ccdca31db2834de4b6ed0f2d67a10535 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:08.798913 1167 raft_consensus.cc:740] T ccdca31db2834de4b6ed0f2d67a10535 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:08.799105 1167 consensus_queue.cc:260] T ccdca31db2834de4b6ed0f2d67a10535 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.799340 2109 raft_consensus.cc:2804] T 9e6eb099648549dbbf9969e59865f51f P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:08.799404 2109 raft_consensus.cc:493] T 9e6eb099648549dbbf9969e59865f51f P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:08.799459 2109 raft_consensus.cc:3060] T 9e6eb099648549dbbf9969e59865f51f P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:08.802357 1172 tablet_bootstrap.cc:492] T 8048506ced4540a5aee9f79bfc4599bf P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:08.802306 2109 raft_consensus.cc:515] T 9e6eb099648549dbbf9969e59865f51f P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.802497 1172 ts_tablet_manager.cc:1403] T 8048506ced4540a5aee9f79bfc4599bf P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.071s user 0.002s sys 0.002s
I20251029 02:52:08.802925 2201 raft_consensus.cc:493] T f246f6c202cf44d48c9f5e4f335202c5 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:08.803035 2201 raft_consensus.cc:515] T f246f6c202cf44d48c9f5e4f335202c5 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.803170 900 tablet_service.cc:1505] Processing CreateTablet for tablet e60b029eb3ec466cbdced20d536f4f47 (DEFAULT_TABLE table=default.Table_27e1e56b7ff0486facdfdd318dc845fc [id=0b1aa07832b94a9f990a564ff353a0f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:08.803303 1172 raft_consensus.cc:359] T 8048506ced4540a5aee9f79bfc4599bf P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.803452 1172 raft_consensus.cc:385] T 8048506ced4540a5aee9f79bfc4599bf P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:08.803510 1172 raft_consensus.cc:740] T 8048506ced4540a5aee9f79bfc4599bf P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:08.803524 2201 leader_election.cc:290] T f246f6c202cf44d48c9f5e4f335202c5 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:08.803623 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e60b029eb3ec466cbdced20d536f4f47. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:08.803699 1172 consensus_queue.cc:260] T 8048506ced4540a5aee9f79bfc4599bf P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.804332 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 519754f5d8f7484480f973c53708c5f3 (table default.Table_96a1c9ac99714785a6bb853741027180 [id=d16cec879a3a45189e45088c4c0913ed]) successfully deleted
I20251029 02:52:08.804914 2109 leader_election.cc:290] T 9e6eb099648549dbbf9969e59865f51f P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
W20251029 02:52:08.806638 849 leader_election.cc:343] T f246f6c202cf44d48c9f5e4f335202c5 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Not found: Tablet not found: f246f6c202cf44d48c9f5e4f335202c5
W20251029 02:52:08.806881 849 leader_election.cc:343] T 9e6eb099648549dbbf9969e59865f51f P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Not found: Tablet not found: 9e6eb099648549dbbf9969e59865f51f
I20251029 02:52:08.807484 1167 ts_tablet_manager.cc:1434] T ccdca31db2834de4b6ed0f2d67a10535 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20251029 02:52:08.808169 1167 tablet_bootstrap.cc:492] T 9bb2166a590d4210b02dd779ef6654ad P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
W20251029 02:52:08.809616 847 leader_election.cc:343] T dfa8b1b9a80e4e8689e0acb44e16346a P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: dfa8b1b9a80e4e8689e0acb44e16346a
I20251029 02:52:08.809805 1167 tablet_bootstrap.cc:654] T 9bb2166a590d4210b02dd779ef6654ad P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
W20251029 02:52:08.810227 847 leader_election.cc:343] T dfa8b1b9a80e4e8689e0acb44e16346a P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: dfa8b1b9a80e4e8689e0acb44e16346a
I20251029 02:52:08.813071 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9e6eb099648549dbbf9969e59865f51f" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:08.813242 768 raft_consensus.cc:3060] T 9e6eb099648549dbbf9969e59865f51f P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:08.757539 1176 ts_tablet_manager.cc:1929] T d2990e9f626c40618c118266bc595684 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20251029 02:52:08.768946 990 leader_election.cc:343] T 7a1866afc7564217ad777e7ce53786c5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
W20251029 02:52:08.818787 990 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:41701, user_credentials={real_user=slave}} blocked reactor thread for 49845.3us
I20251029 02:52:08.819490 1176 log.cc:1199] T d2990e9f626c40618c118266bc595684 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/d2990e9f626c40618c118266bc595684
I20251029 02:52:08.819950 1176 ts_tablet_manager.cc:1950] T d2990e9f626c40618c118266bc595684 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:08.820533 1176 tablet_replica.cc:333] T caef1c93e33a4390bcada59a41f43465 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:08.820753 1176 raft_consensus.cc:2243] T caef1c93e33a4390bcada59a41f43465 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:08.820860 768 raft_consensus.cc:2468] T 9e6eb099648549dbbf9969e59865f51f P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:08.820930 1176 raft_consensus.cc:2272] T caef1c93e33a4390bcada59a41f43465 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:08.821044 1175 ts_tablet_manager.cc:1929] T 59d03a190b784ab18a3e63a572675891 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:08.821208 1175 log.cc:1199] T 59d03a190b784ab18a3e63a572675891 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/59d03a190b784ab18a3e63a572675891
I20251029 02:52:08.821507 1176 ts_tablet_manager.cc:1916] T caef1c93e33a4390bcada59a41f43465 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:08.793394 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dfa8b1b9a80e4e8689e0acb44e16346a" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:52:08.821568 1175 ts_tablet_manager.cc:1950] T 59d03a190b784ab18a3e63a572675891 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:08.811240 1166 tablet_bootstrap.cc:492] T 13ee62fc2339410b9db70dfd3d3c6fa6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:08.821751 1060 raft_consensus.cc:3060] T dfa8b1b9a80e4e8689e0acb44e16346a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:08.821777 1166 ts_tablet_manager.cc:1403] T 13ee62fc2339410b9db70dfd3d3c6fa6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.185s user 0.031s sys 0.046s
I20251029 02:52:08.822613 1166 raft_consensus.cc:359] T 13ee62fc2339410b9db70dfd3d3c6fa6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.822800 1166 raft_consensus.cc:385] T 13ee62fc2339410b9db70dfd3d3c6fa6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:08.822877 1166 raft_consensus.cc:740] T 13ee62fc2339410b9db70dfd3d3c6fa6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:08.823128 1166 consensus_queue.cc:260] T 13ee62fc2339410b9db70dfd3d3c6fa6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.823580 1166 ts_tablet_manager.cc:1434] T 13ee62fc2339410b9db70dfd3d3c6fa6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 02:52:08.823884 1166 tablet_bootstrap.cc:492] T 9bb2166a590d4210b02dd779ef6654ad P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:08.824024 524 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_b0b0cdffdbbc4f89a6117b857e80df26" } new_table_name: "default.Table_d7efa6c76da9426a89093166507d7bd3" modify_external_catalogs: true
I20251029 02:52:08.828775 1060 raft_consensus.cc:2468] T dfa8b1b9a80e4e8689e0acb44e16346a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:08.829334 849 leader_election.cc:304] T dfa8b1b9a80e4e8689e0acb44e16346a P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters: b495a49349324f7d81908f2da700f0d9
I20251029 02:52:08.829763 2205 raft_consensus.cc:2804] T dfa8b1b9a80e4e8689e0acb44e16346a P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:08.829879 2205 raft_consensus.cc:697] T dfa8b1b9a80e4e8689e0acb44e16346a P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:52:08.830132 2205 consensus_queue.cc:237] T dfa8b1b9a80e4e8689e0acb44e16346a P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.841935 520 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_ba4b224d24404916b18cb61a05d08b75" } modify_external_catalogs: true
I20251029 02:52:08.842212 520 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_ba4b224d24404916b18cb61a05d08b75" } modify_external_catalogs: true
W20251029 02:52:08.842803 989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:38481, user_credentials={real_user=slave}} blocked reactor thread for 89265.2us
I20251029 02:52:08.842873 2188 raft_consensus.cc:2804] T d61beda662d844cdaf67fd9a2e382ea2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:08.842993 2188 raft_consensus.cc:493] T d61beda662d844cdaf67fd9a2e382ea2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:08.843093 2188 raft_consensus.cc:3060] T d61beda662d844cdaf67fd9a2e382ea2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:08.843088 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 784) took 1892 ms. Trace:
I20251029 02:52:08.843494 1031 tablet_service.cc:1505] Processing CreateTablet for tablet 21c73cb2f1084827b581fb09cf7239a1 (DEFAULT_TABLE table=default.Table_0427cb4a53de4e62a4b89829586a2a53 [id=3681cfb9581f415b9fe97c2a8467cc28]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:08.843175 1175 rpcz_store.cc:276] 1029 02:52:06.950543 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:06.950582 (+ 39us) service_pool.cc:225] Handling call
1029 02:52:08.758154 (+1807572us) ts_tablet_manager.cc:1151] Deleting tablet 59d03a190b784ab18a3e63a572675891
1029 02:52:08.758166 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:08.781081 (+ 22915us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:08.820900 (+ 39819us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:08.843056 (+ 22156us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10839,"tablet-delete.queue_time_us":1807428}
I20251029 02:52:08.846117 900 tablet_service.cc:1552] Processing DeleteTablet for tablet 777992acb1bd4765886b802940db74bb with delete_type TABLET_DATA_DELETED (replaced by tablet e65eaaae43ac435ebd665546de9d68dd) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:08.847003 1042 tablet_service.cc:1505] Processing CreateTablet for tablet c92a3c77be8943fb929d321559e3a46c (DEFAULT_TABLE table=default.Table_f3da247a6675435a9c881f23dd8ff376 [id=e6beb2f778414c4bb1987db73a86a017]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:08.848754 522 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet cc74e83833754f0ca19046031a24ebb9 (table default.Table_755825b17834457cb94b05163ae3e26d [id=57471769276f44299eae50b9d854b50c]). Expected version 1 got 0
I20251029 02:52:08.849192 900 tablet_service.cc:1505] Processing CreateTablet for tablet 349628fa90ca48ae906070deb3f9ca79 (DEFAULT_TABLE table=default.Table_b7096d022c484e079ae5c12c056da371 [id=8835dbcc777a49eba346b58bb11d6478]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:08.849144 522 catalog_manager.cc:5649] T 02e4588b03144ea2850e13a1f8f29f06 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } health_report { overall_health: HEALTHY } } }
I20251029 02:52:08.849166 1175 tablet_replica.cc:333] T 7a1866afc7564217ad777e7ce53786c5 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:08.849553 522 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet 02e4588b03144ea2850e13a1f8f29f06 (table default.Table_0772f9db5d6142d988f6ffca2e96bdd2 [id=fd6d31d379394857a35422211af00642]). Expected version 3 got 0
I20251029 02:52:08.850165 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 349628fa90ca48ae906070deb3f9ca79. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:08.850436 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 59d03a190b784ab18a3e63a572675891 (table default.Table_2ec9c2215f7147adab6bbfa2fd06d968 [id=3ede90ae95fb46b2ab910914a49a35a5]) successfully deleted
I20251029 02:52:08.851640 517 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_796f8f19849d4b01922da96435e6833c" } modify_external_catalogs: true
I20251029 02:52:08.851892 517 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_796f8f19849d4b01922da96435e6833c" } modify_external_catalogs: true
I20251029 02:52:08.851605 1175 raft_consensus.cc:2243] T 7a1866afc7564217ad777e7ce53786c5 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:08.854413 1175 raft_consensus.cc:2272] T 7a1866afc7564217ad777e7ce53786c5 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:08.853735 743 tablet_service.cc:1505] Processing CreateTablet for tablet 5a1aa131cf4e4edca5bb2515edd7556a (DEFAULT_TABLE table=default.Table_17afc4bb94e7410d8a45c9430e402e8c [id=85553cc5b6ea4decbbc5dd52532fff7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:08.854960 1025 tablet_service.cc:1505] Processing CreateTablet for tablet d14a67fa17fe481baffc9f56bfaf5d3c (DEFAULT_TABLE table=default.Table_f908888068114a9083c998cf80a8441a [id=d7f17b2899964e69b96000482b3e0571]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:08.855340 521 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_104d5aaf444b4e5e813288013737aa15" } modify_external_catalogs: true
I20251029 02:52:08.855484 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a1aa131cf4e4edca5bb2515edd7556a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:08.855695 521 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_104d5aaf444b4e5e813288013737aa15" } modify_external_catalogs: true
I20251029 02:52:08.856145 741 tablet_service.cc:1552] Processing DeleteTablet for tablet 28f7ceb91a34450cad4b5b45f5285b41 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:59 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:08.857668 1172 ts_tablet_manager.cc:1434] T 8048506ced4540a5aee9f79bfc4599bf P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.055s user 0.000s sys 0.002s
I20251029 02:52:08.859666 891 tablet_service.cc:1505] Processing CreateTablet for tablet 21c73cb2f1084827b581fb09cf7239a1 (DEFAULT_TABLE table=default.Table_0427cb4a53de4e62a4b89829586a2a53 [id=3681cfb9581f415b9fe97c2a8467cc28]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:08.860086 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21c73cb2f1084827b581fb09cf7239a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:08.860713 1172 tablet_bootstrap.cc:492] T af4860c6b72f4d988b206b5ef2f22906 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:08.861397 756 tablet_service.cc:1505] Processing CreateTablet for tablet 51d4ee74e3e144658fa63451f144f418 (DEFAULT_TABLE table=default.Table_9037a876e81e47ec96334f1dc2022e2b [id=3bd7f691809d4786ad0c90dd5730a3e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:08.861714 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51d4ee74e3e144658fa63451f144f418. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:08.862308 847 leader_election.cc:343] T f246f6c202cf44d48c9f5e4f335202c5 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: f246f6c202cf44d48c9f5e4f335202c5
I20251029 02:52:08.862421 847 leader_election.cc:304] T f246f6c202cf44d48c9f5e4f335202c5 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9
I20251029 02:52:08.862709 2201 raft_consensus.cc:2749] T f246f6c202cf44d48c9f5e4f335202c5 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:08.862874 847 leader_election.cc:304] T 9e6eb099648549dbbf9969e59865f51f P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5
I20251029 02:52:08.863101 2201 raft_consensus.cc:2804] T 9e6eb099648549dbbf9969e59865f51f P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:08.863236 2201 raft_consensus.cc:697] T 9e6eb099648549dbbf9969e59865f51f P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:52:08.863512 2201 consensus_queue.cc:237] T 9e6eb099648549dbbf9969e59865f51f P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.863812 745 tablet_service.cc:1505] Processing CreateTablet for tablet 591a6c86fc984bac9fb9eb62e35dc6c1 (DEFAULT_TABLE table=default.Table_3bd4c012655a48aa970875bb07006ac2 [id=0c67073b8bed4af9a40a7788194d9598]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:08.864146 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 591a6c86fc984bac9fb9eb62e35dc6c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:08.872138 747 tablet_service.cc:1505] Processing CreateTablet for tablet 862d268a526545ba8f25bb91c2992321 (DEFAULT_TABLE table=default.Table_eb24c80a26b04dab8cc909a42d7184f3 [id=81467c832387409fb5195849a73d8142]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:08.872692 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 862d268a526545ba8f25bb91c2992321. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:08.876022 2188 raft_consensus.cc:515] T d61beda662d844cdaf67fd9a2e382ea2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.876718 2188 leader_election.cc:290] T d61beda662d844cdaf67fd9a2e382ea2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Requested vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:08.879657 712 leader_election.cc:304] T e7f846b294cc4eb687a26c8b5931ac19 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:08.880400 2113 raft_consensus.cc:2749] T e7f846b294cc4eb687a26c8b5931ac19 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 02:52:08.880959 712 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:45003, user_credentials={real_user=slave}} blocked reactor thread for 143575us
I20251029 02:52:08.881522 2113 raft_consensus.cc:493] T 6d830d30b26b4ba799785fec8a6f0769 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:08.881680 2113 raft_consensus.cc:515] T 6d830d30b26b4ba799785fec8a6f0769 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.882457 2113 leader_election.cc:290] T 6d830d30b26b4ba799785fec8a6f0769 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:08.883524 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d830d30b26b4ba799785fec8a6f0769" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:08.884222 711 leader_election.cc:343] T 6d830d30b26b4ba799785fec8a6f0769 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:08.885284 738 tablet_service.cc:1505] Processing CreateTablet for tablet bc3cfa2d7fe8411b954ccb3e16d643a9 (DEFAULT_TABLE table=default.Table_6da557988d044647b5e68f09b3f00407 [id=25f16d95843d41c981d6a07ec0d4b48a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:08.885740 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc3cfa2d7fe8411b954ccb3e16d643a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:08.886286 1175 ts_tablet_manager.cc:1916] T 7a1866afc7564217ad777e7ce53786c5 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:08.891882 890 tablet_service.cc:1505] Processing CreateTablet for tablet d14a67fa17fe481baffc9f56bfaf5d3c (DEFAULT_TABLE table=default.Table_f908888068114a9083c998cf80a8441a [id=d7f17b2899964e69b96000482b3e0571]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:08.875741 742 tablet_service.cc:1505] Processing CreateTablet for tablet 9099fb43dbec46cd83f2cb3f4796ecc6 (DEFAULT_TABLE table=default.Table_a4f394d1b04146848a9a3dafc0919f0e [id=ec98b9beb8f74a2db50ad6c4c37c49c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:08.893082 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9099fb43dbec46cd83f2cb3f4796ecc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:08.893123 1041 tablet_service.cc:1552] Processing DeleteTablet for tablet 372d93914abc41b38a702742b7db0bc2 with delete_type TABLET_DATA_DELETED (replaced by tablet dab6ed2266ed42508738399d388c4da3) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:08.893663 1028 tablet_service.cc:1505] Processing CreateTablet for tablet ad58c7f066f6408490c5398abf077ee7 (DEFAULT_TABLE table=default.Table_6a444f431d374afa8f25c801ef2a92d7 [id=c2437d0dcf3d4774b60d4ff6adb638e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:08.893944 1041 tablet_service.cc:1505] Processing CreateTablet for tablet a8cef3074aef43b2981eaffdc5a5b5bb (DEFAULT_TABLE table=default.Table_ec7e5a0085cd43e28618d7b9d3d30c65 [id=09edb028a40a4418b5624916a9223c66]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:08.894574 1037 tablet_service.cc:1505] Processing CreateTablet for tablet 26ae44429dc24164b00489dcd90b7d8a (DEFAULT_TABLE table=default.Table_e72155685e4942c5916842ac8547cf57 [id=9eea17ba5cba4d6dba8091486ce2779a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:08.895774 1029 tablet_service.cc:1505] Processing CreateTablet for tablet fe5aff0f61a44b91bfb37b42457a91f0 (DEFAULT_TABLE table=default.Table_df7d932b50b64537b4dbc68a846665fc [id=d7c62808bc9a4e74b85ba6a0ddb01495]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:08.896771 1036 tablet_service.cc:1505] Processing CreateTablet for tablet d3faa2d4df584c6e8da426a3830a6454 (DEFAULT_TABLE table=default.Table_f435bdcca42e4087ac8794859d9108ee [id=bce593f4f7934589b71f8a2d694cd9b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:08.898043 1026 tablet_service.cc:1505] Processing CreateTablet for tablet 4eec207d19f540518bb567d7ab21d048 (DEFAULT_TABLE table=default.Table_c4333d8c61d346b1ac1a809d3ccb1915 [id=6d576d3499e649d78000c31fc5dab58c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:08.898115 1039 tablet_service.cc:1505] Processing CreateTablet for tablet 2e37717c241c4d4a8fecc32394764b5c (DEFAULT_TABLE table=default.Table_2c913689f8134363ae4a20a05f717d3e [id=beb2ccbc9cae46bea0d1d75544a5f77a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:08.898772 751 tablet_service.cc:1505] Processing CreateTablet for tablet 4b29426988b043e29783fecc16e7f13e (DEFAULT_TABLE table=default.Table_5557b4fcc4f34c0cacd4af39e9d24e35 [id=adde6f5f632b447f9f1037e4e14ada8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:08.899199 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b29426988b043e29783fecc16e7f13e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:08.910719 891 tablet_service.cc:1505] Processing CreateTablet for tablet c92a3c77be8943fb929d321559e3a46c (DEFAULT_TABLE table=default.Table_f3da247a6675435a9c881f23dd8ff376 [id=e6beb2f778414c4bb1987db73a86a017]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:08.913115 892 tablet_service.cc:1552] Processing DeleteTablet for tablet 372d93914abc41b38a702742b7db0bc2 with delete_type TABLET_DATA_DELETED (replaced by tablet dab6ed2266ed42508738399d388c4da3) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:08.914444 892 tablet_service.cc:1505] Processing CreateTablet for tablet 2e37717c241c4d4a8fecc32394764b5c (DEFAULT_TABLE table=default.Table_2c913689f8134363ae4a20a05f717d3e [id=beb2ccbc9cae46bea0d1d75544a5f77a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:08.914690 2113 raft_consensus.cc:493] T 28f7ceb91a34450cad4b5b45f5285b41 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:08.914826 2113 raft_consensus.cc:515] T 28f7ceb91a34450cad4b5b45f5285b41 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.915531 2113 leader_election.cc:290] T 28f7ceb91a34450cad4b5b45f5285b41 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:08.916033 885 tablet_service.cc:1505] Processing CreateTablet for tablet ad58c7f066f6408490c5398abf077ee7 (DEFAULT_TABLE table=default.Table_6a444f431d374afa8f25c801ef2a92d7 [id=c2437d0dcf3d4774b60d4ff6adb638e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:08.916160 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "28f7ceb91a34450cad4b5b45f5285b41" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:08.916587 898 tablet_service.cc:1505] Processing CreateTablet for tablet a8cef3074aef43b2981eaffdc5a5b5bb (DEFAULT_TABLE table=default.Table_ec7e5a0085cd43e28618d7b9d3d30c65 [id=09edb028a40a4418b5624916a9223c66]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 02:52:08.916826 711 leader_election.cc:343] T 28f7ceb91a34450cad4b5b45f5285b41 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:08.917232 895 tablet_service.cc:1505] Processing CreateTablet for tablet 26ae44429dc24164b00489dcd90b7d8a (DEFAULT_TABLE table=default.Table_e72155685e4942c5916842ac8547cf57 [id=9eea17ba5cba4d6dba8091486ce2779a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:08.917786 879 tablet_service.cc:1505] Processing CreateTablet for tablet fe5aff0f61a44b91bfb37b42457a91f0 (DEFAULT_TABLE table=default.Table_df7d932b50b64537b4dbc68a846665fc [id=d7c62808bc9a4e74b85ba6a0ddb01495]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:08.918386 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e37717c241c4d4a8fecc32394764b5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:08.918452 1174 ts_tablet_manager.cc:1929] T b3de6670357c4ae59eb77469b2fb434c P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:08.918565 1174 log.cc:1199] T b3de6670357c4ae59eb77469b2fb434c P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/b3de6670357c4ae59eb77469b2fb434c
I20251029 02:52:08.918589 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d14a67fa17fe481baffc9f56bfaf5d3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:08.918911 1174 ts_tablet_manager.cc:1950] T b3de6670357c4ae59eb77469b2fb434c P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:08.919199 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8cef3074aef43b2981eaffdc5a5b5bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:08.919516 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe5aff0f61a44b91bfb37b42457a91f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:08.919692 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad58c7f066f6408490c5398abf077ee7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:08.920140 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26ae44429dc24164b00489dcd90b7d8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:08.924090 899 tablet_service.cc:1505] Processing CreateTablet for tablet d3faa2d4df584c6e8da426a3830a6454 (DEFAULT_TABLE table=default.Table_f435bdcca42e4087ac8794859d9108ee [id=bce593f4f7934589b71f8a2d694cd9b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:08.931937 2188 raft_consensus.cc:493] T 1bdb67125493430398ff5c80b73de2ac P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:08.931948 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c92a3c77be8943fb929d321559e3a46c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:08.932123 2188 raft_consensus.cc:515] T 1bdb67125493430398ff5c80b73de2ac P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.932776 2188 leader_election.cc:290] T 1bdb67125493430398ff5c80b73de2ac P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:08.935563 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 800) took 1437 ms. Trace:
I20251029 02:52:08.935685 1174 rpcz_store.cc:276] 1029 02:52:07.498223 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:07.498787 (+ 564us) service_pool.cc:225] Handling call
1029 02:52:08.794828 (+1296041us) ts_tablet_manager.cc:1151] Deleting tablet b3de6670357c4ae59eb77469b2fb434c
1029 02:52:08.794842 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:08.833734 (+ 38892us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:08.918326 (+ 84592us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:08.935538 (+ 17212us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1285732}
I20251029 02:52:08.936057 1174 tablet_replica.cc:333] T 6bd3fac2502e479aa9f4ceaabbb39504 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:08.936286 1174 raft_consensus.cc:2243] T 6bd3fac2502e479aa9f4ceaabbb39504 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:08.936455 1174 raft_consensus.cc:2272] T 6bd3fac2502e479aa9f4ceaabbb39504 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:08.937062 1174 ts_tablet_manager.cc:1916] T 6bd3fac2502e479aa9f4ceaabbb39504 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:08.945363 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3faa2d4df584c6e8da426a3830a6454. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:08.953087 1167 tablet_bootstrap.cc:492] T 9bb2166a590d4210b02dd779ef6654ad P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:08.953454 1167 ts_tablet_manager.cc:1403] T 9bb2166a590d4210b02dd779ef6654ad P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.145s user 0.047s sys 0.051s
I20251029 02:52:08.954357 1167 raft_consensus.cc:359] T 9bb2166a590d4210b02dd779ef6654ad P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.954547 1167 raft_consensus.cc:385] T 9bb2166a590d4210b02dd779ef6654ad P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:08.954695 1167 raft_consensus.cc:740] T 9bb2166a590d4210b02dd779ef6654ad P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:08.954927 1167 consensus_queue.cc:260] T 9bb2166a590d4210b02dd779ef6654ad P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.955430 1167 ts_tablet_manager.cc:1434] T 9bb2166a590d4210b02dd779ef6654ad P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251029 02:52:08.955685 1167 tablet_bootstrap.cc:492] T b6d4f7bf702f4834bc5f4ef2b50b92a7 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:08.956714 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d3faa2d4df584c6e8da426a3830a6454. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:08.957384 1167 tablet_bootstrap.cc:654] T b6d4f7bf702f4834bc5f4ef2b50b92a7 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:08.957247 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e37717c241c4d4a8fecc32394764b5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:08.957829 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26ae44429dc24164b00489dcd90b7d8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:08.958052 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4eec207d19f540518bb567d7ab21d048. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:08.958638 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe5aff0f61a44b91bfb37b42457a91f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:08.962714 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8cef3074aef43b2981eaffdc5a5b5bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:08.966637 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad58c7f066f6408490c5398abf077ee7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:08.967576 1167 tablet_bootstrap.cc:492] T b6d4f7bf702f4834bc5f4ef2b50b92a7 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:08.967694 1167 ts_tablet_manager.cc:1403] T b6d4f7bf702f4834bc5f4ef2b50b92a7 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.008s
I20251029 02:52:08.968585 1167 raft_consensus.cc:359] T b6d4f7bf702f4834bc5f4ef2b50b92a7 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.968775 1167 raft_consensus.cc:385] T b6d4f7bf702f4834bc5f4ef2b50b92a7 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:08.968827 1167 raft_consensus.cc:740] T b6d4f7bf702f4834bc5f4ef2b50b92a7 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:08.969007 1167 consensus_queue.cc:260] T b6d4f7bf702f4834bc5f4ef2b50b92a7 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.969439 1167 ts_tablet_manager.cc:1434] T b6d4f7bf702f4834bc5f4ef2b50b92a7 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 02:52:08.969702 1167 tablet_bootstrap.cc:492] T 1275e192871440408a8345fef9b56260 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:08.971338 1167 tablet_bootstrap.cc:654] T 1275e192871440408a8345fef9b56260 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:08.972170 1166 tablet_bootstrap.cc:654] T 9bb2166a590d4210b02dd779ef6654ad P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
W20251029 02:52:08.973248 518 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e3ac6597d52441569a7aeef051087ad2 consensus state has no leader
I20251029 02:52:08.975569 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21c73cb2f1084827b581fb09cf7239a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:08.975687 2188 raft_consensus.cc:493] T 95f342d1cfb84c4eb31bd3d15ea3bdc2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:08.975832 2188 raft_consensus.cc:515] T 95f342d1cfb84c4eb31bd3d15ea3bdc2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.976421 2188 leader_election.cc:290] T 95f342d1cfb84c4eb31bd3d15ea3bdc2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:08.980120 743 tablet_service.cc:1552] Processing DeleteTablet for tablet 8048506ced4540a5aee9f79bfc4599bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:06 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:08.981329 518 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet d5741dde463a43d6a776ab92dcfb04bf (table default.Table_966779a0d37a4dc4ab3f89d2bf7c6b06 [id=08e3b2ebcf37415d9a2886ad5bbda068]). Expected version 1 got 0
I20251029 02:52:08.985666 2188 raft_consensus.cc:493] T 13ee62fc2339410b9db70dfd3d3c6fa6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:08.985833 2188 raft_consensus.cc:515] T 13ee62fc2339410b9db70dfd3d3c6fa6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.986383 2188 leader_election.cc:290] T 13ee62fc2339410b9db70dfd3d3c6fa6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:08.988461 1167 tablet_bootstrap.cc:492] T 1275e192871440408a8345fef9b56260 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:08.988643 1167 ts_tablet_manager.cc:1403] T 1275e192871440408a8345fef9b56260 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.002s
I20251029 02:52:08.989408 1167 raft_consensus.cc:359] T 1275e192871440408a8345fef9b56260 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.989636 1167 raft_consensus.cc:385] T 1275e192871440408a8345fef9b56260 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:08.989717 1167 raft_consensus.cc:740] T 1275e192871440408a8345fef9b56260 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:08.989951 1167 consensus_queue.cc:260] T 1275e192871440408a8345fef9b56260 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.990447 1167 ts_tablet_manager.cc:1434] T 1275e192871440408a8345fef9b56260 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 02:52:08.990792 1167 tablet_bootstrap.cc:492] T 1bdb67125493430398ff5c80b73de2ac P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
W20251029 02:52:08.991472 512 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet a9e0fcc759a64679b3f59a6c964877b6 (table default.Table_9dce7f70d51d4a3c8d7f98a2435cbce3 [id=657afc87656d4250b4ca87b473fbb752], current schema version=3) failed: Not found: failed to reset TS proxy: No leader found: Tablet a9e0fcc759a64679b3f59a6c964877b6 consensus state has no leader
I20251029 02:52:08.992558 1167 tablet_bootstrap.cc:654] T 1bdb67125493430398ff5c80b73de2ac P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:08.993076 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c92a3c77be8943fb929d321559e3a46c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:08.993559 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d14a67fa17fe481baffc9f56bfaf5d3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:08.994629 1166 tablet_bootstrap.cc:492] T 9bb2166a590d4210b02dd779ef6654ad P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:08.994807 1166 ts_tablet_manager.cc:1403] T 9bb2166a590d4210b02dd779ef6654ad P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.171s user 0.002s sys 0.016s
I20251029 02:52:08.995596 1166 raft_consensus.cc:359] T 9bb2166a590d4210b02dd779ef6654ad P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.995824 1166 raft_consensus.cc:385] T 9bb2166a590d4210b02dd779ef6654ad P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:08.995930 1166 raft_consensus.cc:740] T 9bb2166a590d4210b02dd779ef6654ad P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:08.996171 1166 consensus_queue.cc:260] T 9bb2166a590d4210b02dd779ef6654ad P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:08.997273 1166 ts_tablet_manager.cc:1434] T 9bb2166a590d4210b02dd779ef6654ad P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 02:52:08.997922 1166 tablet_bootstrap.cc:492] T ba699862e6f646c6beaa39f080d0faba P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:08.998561 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_a3886404687640698bd3302e39ee6854"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:09.000169 1166 tablet_bootstrap.cc:654] T ba699862e6f646c6beaa39f080d0faba P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:09.000341 1026 tablet_service.cc:1505] Processing CreateTablet for tablet 591a6c86fc984bac9fb9eb62e35dc6c1 (DEFAULT_TABLE table=default.Table_3bd4c012655a48aa970875bb07006ac2 [id=0c67073b8bed4af9a40a7788194d9598]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:09.000844 1028 tablet_service.cc:1505] Processing CreateTablet for tablet 862d268a526545ba8f25bb91c2992321 (DEFAULT_TABLE table=default.Table_eb24c80a26b04dab8cc909a42d7184f3 [id=81467c832387409fb5195849a73d8142]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:09.001289 1041 tablet_service.cc:1505] Processing CreateTablet for tablet ef2e9dd8f90a47389d9062d1f99a1da1 (DEFAULT_TABLE table=default.Table_d7cbd7b4c4bf4886a4ccf4c89da3cbce [id=9107fd53f5c94d2d80be50ef1f7c8e9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:09.001766 1031 tablet_service.cc:1505] Processing CreateTablet for tablet bc3cfa2d7fe8411b954ccb3e16d643a9 (DEFAULT_TABLE table=default.Table_6da557988d044647b5e68f09b3f00407 [id=25f16d95843d41c981d6a07ec0d4b48a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:09.002456 1036 tablet_service.cc:1505] Processing CreateTablet for tablet 51d4ee74e3e144658fa63451f144f418 (DEFAULT_TABLE table=default.Table_9037a876e81e47ec96334f1dc2022e2b [id=3bd7f691809d4786ad0c90dd5730a3e3]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 02:52:09.003357 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a3886404687640698bd3302e39ee6854 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:52:09.012908 523 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 81ee839d811a4f46b93a57ede397bd93 consensus state has no leader
W20251029 02:52:09.013593 519 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6f8cf09f23984b879b60ab6b27c7d52e consensus state has no leader
I20251029 02:52:09.013924 895 tablet_service.cc:1505] Processing CreateTablet for tablet 4eec207d19f540518bb567d7ab21d048 (DEFAULT_TABLE table=default.Table_c4333d8c61d346b1ac1a809d3ccb1915 [id=6d576d3499e649d78000c31fc5dab58c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:09.014034 1166 tablet_bootstrap.cc:492] T ba699862e6f646c6beaa39f080d0faba P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:09.014199 1166 ts_tablet_manager.cc:1403] T ba699862e6f646c6beaa39f080d0faba P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.003s
I20251029 02:52:09.014446 898 tablet_service.cc:1505] Processing CreateTablet for tablet 862d268a526545ba8f25bb91c2992321 (DEFAULT_TABLE table=default.Table_eb24c80a26b04dab8cc909a42d7184f3 [id=81467c832387409fb5195849a73d8142]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:09.014592 884 tablet_service.cc:1505] Processing CreateTablet for tablet 4b29426988b043e29783fecc16e7f13e (DEFAULT_TABLE table=default.Table_5557b4fcc4f34c0cacd4af39e9d24e35 [id=adde6f5f632b447f9f1037e4e14ada8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:09.014902 893 tablet_service.cc:1552] Processing DeleteTablet for tablet ccdca31db2834de4b6ed0f2d67a10535 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:54 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:09.014982 1166 raft_consensus.cc:359] T ba699862e6f646c6beaa39f080d0faba P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.015096 881 tablet_service.cc:1505] Processing CreateTablet for tablet 9099fb43dbec46cd83f2cb3f4796ecc6 (DEFAULT_TABLE table=default.Table_a4f394d1b04146848a9a3dafc0919f0e [id=ec98b9beb8f74a2db50ad6c4c37c49c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:09.015195 1166 raft_consensus.cc:385] T ba699862e6f646c6beaa39f080d0faba P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:09.015144 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d61beda662d844cdaf67fd9a2e382ea2" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:09.015266 1166 raft_consensus.cc:740] T ba699862e6f646c6beaa39f080d0faba P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:09.015504 1166 consensus_queue.cc:260] T ba699862e6f646c6beaa39f080d0faba P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.016091 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5db519edc75d4bd0985b32322825ffae" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:09.017040 1175 ts_tablet_manager.cc:1929] T 7a1866afc7564217ad777e7ce53786c5 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:09.017151 1175 log.cc:1199] T 7a1866afc7564217ad777e7ce53786c5 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/7a1866afc7564217ad777e7ce53786c5
I20251029 02:52:09.017411 1175 ts_tablet_manager.cc:1950] T 7a1866afc7564217ad777e7ce53786c5 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:09.014916 885 tablet_service.cc:1505] Processing CreateTablet for tablet ef2e9dd8f90a47389d9062d1f99a1da1 (DEFAULT_TABLE table=default.Table_d7cbd7b4c4bf4886a4ccf4c89da3cbce [id=9107fd53f5c94d2d80be50ef1f7c8e9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:09.018210 1166 ts_tablet_manager.cc:1434] T ba699862e6f646c6beaa39f080d0faba P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251029 02:52:09.018288 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b29426988b043e29783fecc16e7f13e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:09.018529 1166 tablet_bootstrap.cc:492] T 8048506ced4540a5aee9f79bfc4599bf P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:09.019111 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4eec207d19f540518bb567d7ab21d048. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:09.019503 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 862d268a526545ba8f25bb91c2992321. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:09.019960 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9099fb43dbec46cd83f2cb3f4796ecc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:09.021664 1166 tablet_bootstrap.cc:654] T 8048506ced4540a5aee9f79bfc4599bf P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:09.021868 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d61beda662d844cdaf67fd9a2e382ea2" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
I20251029 02:52:09.022287 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1bdb67125493430398ff5c80b73de2ac" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:09.022727 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "95f342d1cfb84c4eb31bd3d15ea3bdc2" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:09.023166 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef2e9dd8f90a47389d9062d1f99a1da1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:09.023202 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "13ee62fc2339410b9db70dfd3d3c6fa6" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:09.024106 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet b3de6670357c4ae59eb77469b2fb434c (table default.Table_8dacd5c558a04e47b424533b985aee9e [id=f9c7128476404646a9194e31f49c94e8]) successfully deleted
I20251029 02:52:09.024895 1167 tablet_bootstrap.cc:492] T 1bdb67125493430398ff5c80b73de2ac P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:09.025039 1167 ts_tablet_manager.cc:1403] T 1bdb67125493430398ff5c80b73de2ac P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.034s user 0.006s sys 0.002s
W20251029 02:52:09.025972 990 leader_election.cc:343] T d61beda662d844cdaf67fd9a2e382ea2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
W20251029 02:52:09.026408 990 leader_election.cc:343] T 5db519edc75d4bd0985b32322825ffae P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:09.026495 1027 tablet_service.cc:1505] Processing CreateTablet for tablet 9099fb43dbec46cd83f2cb3f4796ecc6 (DEFAULT_TABLE table=default.Table_a4f394d1b04146848a9a3dafc0919f0e [id=ec98b9beb8f74a2db50ad6c4c37c49c8]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 02:52:09.026834 990 leader_election.cc:343] T d61beda662d844cdaf67fd9a2e382ea2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
W20251029 02:52:09.027035 990 leader_election.cc:343] T 1bdb67125493430398ff5c80b73de2ac P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
W20251029 02:52:09.027233 990 leader_election.cc:343] T 95f342d1cfb84c4eb31bd3d15ea3bdc2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
W20251029 02:52:09.027432 990 leader_election.cc:343] T 13ee62fc2339410b9db70dfd3d3c6fa6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:09.027681 1167 raft_consensus.cc:359] T 1bdb67125493430398ff5c80b73de2ac P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.027885 1167 raft_consensus.cc:385] T 1bdb67125493430398ff5c80b73de2ac P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:09.027973 1167 raft_consensus.cc:740] T 1bdb67125493430398ff5c80b73de2ac P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:09.028244 1167 consensus_queue.cc:260] T 1bdb67125493430398ff5c80b73de2ac P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.034812 890 tablet_service.cc:1505] Processing CreateTablet for tablet 5a1aa131cf4e4edca5bb2515edd7556a (DEFAULT_TABLE table=default.Table_17afc4bb94e7410d8a45c9430e402e8c [id=85553cc5b6ea4decbbc5dd52532fff7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:09.035425 879 tablet_service.cc:1505] Processing CreateTablet for tablet 51d4ee74e3e144658fa63451f144f418 (DEFAULT_TABLE table=default.Table_9037a876e81e47ec96334f1dc2022e2b [id=3bd7f691809d4786ad0c90dd5730a3e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:09.035940 899 tablet_service.cc:1505] Processing CreateTablet for tablet bc3cfa2d7fe8411b954ccb3e16d643a9 (DEFAULT_TABLE table=default.Table_6da557988d044647b5e68f09b3f00407 [id=25f16d95843d41c981d6a07ec0d4b48a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:09.036414 892 tablet_service.cc:1505] Processing CreateTablet for tablet 591a6c86fc984bac9fb9eb62e35dc6c1 (DEFAULT_TABLE table=default.Table_3bd4c012655a48aa970875bb07006ac2 [id=0c67073b8bed4af9a40a7788194d9598]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:09.039351 1172 tablet_bootstrap.cc:654] T af4860c6b72f4d988b206b5ef2f22906 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:09.040338 2103 raft_consensus.cc:493] T 44b518c3a0964a6fa863b2784d80c8a4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:09.040468 2103 raft_consensus.cc:515] T 44b518c3a0964a6fa863b2784d80c8a4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.040964 2103 leader_election.cc:290] T 44b518c3a0964a6fa863b2784d80c8a4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:09.041426 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 591a6c86fc984bac9fb9eb62e35dc6c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:09.041965 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9099fb43dbec46cd83f2cb3f4796ecc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:09.042001 1166 tablet_bootstrap.cc:492] T 8048506ced4540a5aee9f79bfc4599bf P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:09.042132 1166 ts_tablet_manager.cc:1403] T 8048506ced4540a5aee9f79bfc4599bf P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.024s user 0.009s sys 0.003s
I20251029 02:52:09.042718 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef2e9dd8f90a47389d9062d1f99a1da1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:09.042830 1166 raft_consensus.cc:359] T 8048506ced4540a5aee9f79bfc4599bf P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.042990 1166 raft_consensus.cc:385] T 8048506ced4540a5aee9f79bfc4599bf P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:09.043071 1166 raft_consensus.cc:740] T 8048506ced4540a5aee9f79bfc4599bf P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:09.043179 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 782) took 2093 ms. Trace:
I20251029 02:52:09.043280 1166 consensus_queue.cc:260] T 8048506ced4540a5aee9f79bfc4599bf P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.043288 1175 rpcz_store.cc:276] 1029 02:52:06.949923 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:06.954917 (+ 4994us) service_pool.cc:225] Handling call
1029 02:52:08.848379 (+1893462us) ts_tablet_manager.cc:1151] Deleting tablet 7a1866afc7564217ad777e7ce53786c5
1029 02:52:08.848629 (+ 250us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:08.922626 (+ 73997us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:09.016923 (+ 94297us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:09.043146 (+ 26223us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1893299}
I20251029 02:52:09.043380 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc3cfa2d7fe8411b954ccb3e16d643a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:09.043678 1175 tablet_replica.cc:333] T d61beda662d844cdaf67fd9a2e382ea2 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:09.043823 1166 ts_tablet_manager.cc:1434] T 8048506ced4540a5aee9f79bfc4599bf P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 02:52:09.043861 1175 raft_consensus.cc:2243] T d61beda662d844cdaf67fd9a2e382ea2 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:09.044062 1166 tablet_bootstrap.cc:492] T 28f7ceb91a34450cad4b5b45f5285b41 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:09.044021 1175 raft_consensus.cc:2272] T d61beda662d844cdaf67fd9a2e382ea2 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:09.044215 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51d4ee74e3e144658fa63451f144f418. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:09.044595 1175 ts_tablet_manager.cc:1916] T d61beda662d844cdaf67fd9a2e382ea2 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:09.045436 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 7a1866afc7564217ad777e7ce53786c5 (table default.Table_46e2f96551d3497c919e1cab3521d63b [id=09ee821707fa4082889bfccb8667b0b7]) successfully deleted
I20251029 02:52:09.047070 1166 tablet_bootstrap.cc:654] T 28f7ceb91a34450cad4b5b45f5285b41 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:09.047811 1172 tablet_bootstrap.cc:492] T af4860c6b72f4d988b206b5ef2f22906 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:09.047963 1172 ts_tablet_manager.cc:1403] T af4860c6b72f4d988b206b5ef2f22906 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.187s user 0.005s sys 0.000s
I20251029 02:52:09.048791 1172 raft_consensus.cc:359] T af4860c6b72f4d988b206b5ef2f22906 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.049021 1172 raft_consensus.cc:385] T af4860c6b72f4d988b206b5ef2f22906 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:09.049096 1172 raft_consensus.cc:740] T af4860c6b72f4d988b206b5ef2f22906 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:09.049360 1172 consensus_queue.cc:260] T af4860c6b72f4d988b206b5ef2f22906 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.050010 1172 ts_tablet_manager.cc:1434] T af4860c6b72f4d988b206b5ef2f22906 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251029 02:52:09.050355 1172 tablet_bootstrap.cc:492] T c32c28b0e74a4af3a0e69bbf24ab1a1f P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:09.052281 1172 tablet_bootstrap.cc:654] T c32c28b0e74a4af3a0e69bbf24ab1a1f P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:09.055251 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 862d268a526545ba8f25bb91c2992321. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:09.055564 1166 tablet_bootstrap.cc:492] T 28f7ceb91a34450cad4b5b45f5285b41 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:09.055691 1166 ts_tablet_manager.cc:1403] T 28f7ceb91a34450cad4b5b45f5285b41 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
I20251029 02:52:09.056607 1166 raft_consensus.cc:359] T 28f7ceb91a34450cad4b5b45f5285b41 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.056783 1166 raft_consensus.cc:385] T 28f7ceb91a34450cad4b5b45f5285b41 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:09.056833 1166 raft_consensus.cc:740] T 28f7ceb91a34450cad4b5b45f5285b41 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:09.057001 1166 consensus_queue.cc:260] T 28f7ceb91a34450cad4b5b45f5285b41 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.058037 1166 ts_tablet_manager.cc:1434] T 28f7ceb91a34450cad4b5b45f5285b41 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 02:52:09.058295 1166 tablet_bootstrap.cc:492] T 061f3d38c4604fa4b57e74ff6f6fc7da P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:09.060947 1166 tablet_bootstrap.cc:654] T 061f3d38c4604fa4b57e74ff6f6fc7da P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:09.061106 1172 tablet_bootstrap.cc:492] T c32c28b0e74a4af3a0e69bbf24ab1a1f P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:09.061265 1172 ts_tablet_manager.cc:1403] T c32c28b0e74a4af3a0e69bbf24ab1a1f P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20251029 02:52:09.062011 1172 raft_consensus.cc:359] T c32c28b0e74a4af3a0e69bbf24ab1a1f P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:09.062247 1172 raft_consensus.cc:385] T c32c28b0e74a4af3a0e69bbf24ab1a1f P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:09.062323 1172 raft_consensus.cc:740] T c32c28b0e74a4af3a0e69bbf24ab1a1f P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:09.062601 1172 consensus_queue.cc:260] T c32c28b0e74a4af3a0e69bbf24ab1a1f P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:09.063208 1172 ts_tablet_manager.cc:1434] T c32c28b0e74a4af3a0e69bbf24ab1a1f P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 02:52:09.063608 1172 tablet_bootstrap.cc:492] T 44ae2b07edca44afbe1b1fb3a5b1115d P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:09.063889 1166 tablet_bootstrap.cc:492] T 061f3d38c4604fa4b57e74ff6f6fc7da P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:09.064008 1166 ts_tablet_manager.cc:1403] T 061f3d38c4604fa4b57e74ff6f6fc7da P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20251029 02:52:09.065423 492 master_runner.cc:186] Error getting master registration for 127.0.104.62:42343: Network error: Client connection negotiation failed: client connection to 127.0.104.62:42343: connect: Connection refused (error 111),
I20251029 02:52:09.065407 1166 raft_consensus.cc:359] T 061f3d38c4604fa4b57e74ff6f6fc7da P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.065574 492 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251029 02:52:09.065586 1166 raft_consensus.cc:385] T 061f3d38c4604fa4b57e74ff6f6fc7da P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:09.065644 1166 raft_consensus.cc:740] T 061f3d38c4604fa4b57e74ff6f6fc7da P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:09.065668 1172 tablet_bootstrap.cc:654] T 44ae2b07edca44afbe1b1fb3a5b1115d P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:09.065888 1166 consensus_queue.cc:260] T 061f3d38c4604fa4b57e74ff6f6fc7da P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.067124 1166 ts_tablet_manager.cc:1434] T 061f3d38c4604fa4b57e74ff6f6fc7da P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251029 02:52:09.067790 1166 tablet_bootstrap.cc:492] T 738046ab57334f2dab9f4aab979c5b42 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:09.069751 1166 tablet_bootstrap.cc:654] T 738046ab57334f2dab9f4aab979c5b42 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:09.077764 1166 tablet_bootstrap.cc:492] T 738046ab57334f2dab9f4aab979c5b42 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:09.077908 1172 tablet_bootstrap.cc:492] T 44ae2b07edca44afbe1b1fb3a5b1115d P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:09.077920 1166 ts_tablet_manager.cc:1403] T 738046ab57334f2dab9f4aab979c5b42 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.003s
I20251029 02:52:09.078045 1172 ts_tablet_manager.cc:1403] T 44ae2b07edca44afbe1b1fb3a5b1115d P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.004s
I20251029 02:52:09.078715 1172 raft_consensus.cc:359] T 44ae2b07edca44afbe1b1fb3a5b1115d P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:09.078893 1172 raft_consensus.cc:385] T 44ae2b07edca44afbe1b1fb3a5b1115d P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:09.078960 1172 raft_consensus.cc:740] T 44ae2b07edca44afbe1b1fb3a5b1115d P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:09.079161 1172 consensus_queue.cc:260] T 44ae2b07edca44afbe1b1fb3a5b1115d P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:09.078715 1166 raft_consensus.cc:359] T 738046ab57334f2dab9f4aab979c5b42 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.079380 1166 raft_consensus.cc:385] T 738046ab57334f2dab9f4aab979c5b42 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:09.079432 1166 raft_consensus.cc:740] T 738046ab57334f2dab9f4aab979c5b42 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:09.079659 1172 ts_tablet_manager.cc:1434] T 44ae2b07edca44afbe1b1fb3a5b1115d P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251029 02:52:09.079592 1166 consensus_queue.cc:260] T 738046ab57334f2dab9f4aab979c5b42 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.079941 1172 tablet_bootstrap.cc:492] T 946f65014d97425aab3851009b742807 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:09.080613 1166 ts_tablet_manager.cc:1434] T 738046ab57334f2dab9f4aab979c5b42 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251029 02:52:09.081509 1172 tablet_bootstrap.cc:654] T 946f65014d97425aab3851009b742807 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:09.082122 1166 tablet_bootstrap.cc:492] T dabb8d760aae4aa09eb83cdb586c9788 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:09.083784 1166 tablet_bootstrap.cc:654] T dabb8d760aae4aa09eb83cdb586c9788 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
W20251029 02:52:09.090207 524 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fe9e660de57a46c7ba1d82f25e12b4b0 consensus state has no leader
I20251029 02:52:09.091615 1166 tablet_bootstrap.cc:492] T dabb8d760aae4aa09eb83cdb586c9788 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:09.091745 1166 ts_tablet_manager.cc:1403] T dabb8d760aae4aa09eb83cdb586c9788 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.005s
I20251029 02:52:09.091730 520 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 00ebd0c68c4441f9b63c86cdab62497d
I20251029 02:52:09.091967 1172 tablet_bootstrap.cc:492] T 946f65014d97425aab3851009b742807 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:09.092087 1172 ts_tablet_manager.cc:1403] T 946f65014d97425aab3851009b742807 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.001s
I20251029 02:52:09.092381 520 master_service.cc:954] ReplaceTablet: received request to replace tablet e65eaaae43ac435ebd665546de9d68dd from {username='slave'} at 127.0.0.1:35314
I20251029 02:52:09.092538 1166 raft_consensus.cc:359] T dabb8d760aae4aa09eb83cdb586c9788 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.092691 1166 raft_consensus.cc:385] T dabb8d760aae4aa09eb83cdb586c9788 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:09.092749 1166 raft_consensus.cc:740] T dabb8d760aae4aa09eb83cdb586c9788 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:09.092720 1172 raft_consensus.cc:359] T 946f65014d97425aab3851009b742807 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:09.092895 1172 raft_consensus.cc:385] T 946f65014d97425aab3851009b742807 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:09.092941 1172 raft_consensus.cc:740] T 946f65014d97425aab3851009b742807 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:09.092978 1166 consensus_queue.cc:260] T dabb8d760aae4aa09eb83cdb586c9788 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.093058 743 tablet_service.cc:1552] Processing DeleteTablet for tablet 00ebd0c68c4441f9b63c86cdab62497d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:08 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:09.093160 1172 consensus_queue.cc:260] T 946f65014d97425aab3851009b742807 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:09.094121 1172 ts_tablet_manager.cc:1434] T 946f65014d97425aab3851009b742807 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 02:52:09.094280 1166 ts_tablet_manager.cc:1434] T dabb8d760aae4aa09eb83cdb586c9788 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 02:52:09.094389 1172 tablet_bootstrap.cc:492] T 97b2ecae572a4b7ea1b9595074556708 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:09.095160 1166 tablet_bootstrap.cc:492] T 1275e192871440408a8345fef9b56260 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:09.095906 1172 tablet_bootstrap.cc:654] T 97b2ecae572a4b7ea1b9595074556708 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:09.096808 1166 tablet_bootstrap.cc:654] T 1275e192871440408a8345fef9b56260 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:09.099772 1166 tablet_bootstrap.cc:492] T 1275e192871440408a8345fef9b56260 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:09.099835 1172 tablet_bootstrap.cc:492] T 97b2ecae572a4b7ea1b9595074556708 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:09.099905 1166 ts_tablet_manager.cc:1403] T 1275e192871440408a8345fef9b56260 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20251029 02:52:09.099938 1172 ts_tablet_manager.cc:1403] T 97b2ecae572a4b7ea1b9595074556708 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20251029 02:52:09.100540 1166 raft_consensus.cc:359] T 1275e192871440408a8345fef9b56260 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.100540 1172 raft_consensus.cc:359] T 97b2ecae572a4b7ea1b9595074556708 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.100701 1166 raft_consensus.cc:385] T 1275e192871440408a8345fef9b56260 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:09.100700 1172 raft_consensus.cc:385] T 97b2ecae572a4b7ea1b9595074556708 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:09.100757 1166 raft_consensus.cc:740] T 1275e192871440408a8345fef9b56260 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:09.100764 1172 raft_consensus.cc:740] T 97b2ecae572a4b7ea1b9595074556708 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:09.100960 1172 consensus_queue.cc:260] T 97b2ecae572a4b7ea1b9595074556708 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.100970 1166 consensus_queue.cc:260] T 1275e192871440408a8345fef9b56260 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.101418 1172 ts_tablet_manager.cc:1434] T 97b2ecae572a4b7ea1b9595074556708 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251029 02:52:09.101672 1172 tablet_bootstrap.cc:492] T 71b4b91071a54d6aa66160e7955bc915 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:09.103394 1172 tablet_bootstrap.cc:654] T 71b4b91071a54d6aa66160e7955bc915 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:09.104204 1029 tablet_service.cc:1505] Processing CreateTablet for tablet 4b29426988b043e29783fecc16e7f13e (DEFAULT_TABLE table=default.Table_5557b4fcc4f34c0cacd4af39e9d24e35 [id=adde6f5f632b447f9f1037e4e14ada8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:09.104645 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b29426988b043e29783fecc16e7f13e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:09.105871 1166 ts_tablet_manager.cc:1434] T 1275e192871440408a8345fef9b56260 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251029 02:52:09.106798 2205 raft_consensus.cc:493] T 1bc45078a6a942c38ff3350c15885eff P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:09.107223 1167 ts_tablet_manager.cc:1434] T 1bdb67125493430398ff5c80b73de2ac P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.082s user 0.001s sys 0.001s
I20251029 02:52:09.107522 1167 tablet_bootstrap.cc:492] T 5db519edc75d4bd0985b32322825ffae P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:09.107082 2205 raft_consensus.cc:515] T 1bc45078a6a942c38ff3350c15885eff P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.108335 2205 leader_election.cc:290] T 1bc45078a6a942c38ff3350c15885eff P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:09.109247 1167 tablet_bootstrap.cc:654] T 5db519edc75d4bd0985b32322825ffae P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:09.109514 1174 ts_tablet_manager.cc:1929] T 6bd3fac2502e479aa9f4ceaabbb39504 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:09.109622 1174 log.cc:1199] T 6bd3fac2502e479aa9f4ceaabbb39504 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/6bd3fac2502e479aa9f4ceaabbb39504
I20251029 02:52:09.109911 1174 ts_tablet_manager.cc:1950] T 6bd3fac2502e479aa9f4ceaabbb39504 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:09.106602 1166 tablet_bootstrap.cc:492] T 6d830d30b26b4ba799785fec8a6f0769 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:09.115473 1166 tablet_bootstrap.cc:654] T 6d830d30b26b4ba799785fec8a6f0769 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
W20251029 02:52:09.116057 847 leader_election.cc:343] T 1bc45078a6a942c38ff3350c15885eff P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 1bc45078a6a942c38ff3350c15885eff
W20251029 02:52:09.117141 849 leader_election.cc:343] T 1bc45078a6a942c38ff3350c15885eff P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Not found: Tablet not found: 1bc45078a6a942c38ff3350c15885eff
I20251029 02:52:09.117246 849 leader_election.cc:304] T 1bc45078a6a942c38ff3350c15885eff P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9
I20251029 02:52:09.117710 2205 raft_consensus.cc:2749] T 1bc45078a6a942c38ff3350c15885eff P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:09.119450 884 tablet_service.cc:1552] Processing DeleteTablet for tablet 00ebd0c68c4441f9b63c86cdab62497d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:08 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:09.120628 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "44b518c3a0964a6fa863b2784d80c8a4" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:09.124046 990 leader_election.cc:343] T 44b518c3a0964a6fa863b2784d80c8a4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:09.124188 2103 raft_consensus.cc:493] T 49d9ce32e4f04f11be2ecefc5d523a8b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:09.124303 2103 raft_consensus.cc:515] T 49d9ce32e4f04f11be2ecefc5d523a8b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.124461 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a1aa131cf4e4edca5bb2515edd7556a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:09.124912 2103 leader_election.cc:290] T 49d9ce32e4f04f11be2ecefc5d523a8b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:09.127501 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "49d9ce32e4f04f11be2ecefc5d523a8b" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:09.128000 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 807) took 1629 ms. Trace:
I20251029 02:52:09.128124 1166 tablet_bootstrap.cc:492] T 6d830d30b26b4ba799785fec8a6f0769 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:09.128084 1174 rpcz_store.cc:276] 1029 02:52:07.498896 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:07.499770 (+ 874us) service_pool.cc:225] Handling call
1029 02:52:08.936000 (+1436230us) ts_tablet_manager.cc:1151] Deleting tablet 6bd3fac2502e479aa9f4ceaabbb39504
1029 02:52:08.936012 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:09.016460 (+ 80448us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:09.109404 (+ 92944us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:09.127979 (+ 18575us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1425072}
I20251029 02:52:09.128232 1166 ts_tablet_manager.cc:1403] T 6d830d30b26b4ba799785fec8a6f0769 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.002s
I20251029 02:52:09.128329 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51d4ee74e3e144658fa63451f144f418. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:09.128377 1174 tablet_replica.cc:333] T f187dbd8a55b45b292ea61db43caf47d P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:09.128559 1174 raft_consensus.cc:2243] T f187dbd8a55b45b292ea61db43caf47d P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:09.128582 523 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_6a3ec6a2716e4a0994a4d721f96b488e" } modify_external_catalogs: true
I20251029 02:52:09.128707 1174 raft_consensus.cc:2272] T f187dbd8a55b45b292ea61db43caf47d P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:09.128845 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc3cfa2d7fe8411b954ccb3e16d643a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:09.128826 523 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_6a3ec6a2716e4a0994a4d721f96b488e" } modify_external_catalogs: true
I20251029 02:52:09.129014 1166 raft_consensus.cc:359] T 6d830d30b26b4ba799785fec8a6f0769 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.129159 1166 raft_consensus.cc:385] T 6d830d30b26b4ba799785fec8a6f0769 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:09.129195 1174 ts_tablet_manager.cc:1916] T f187dbd8a55b45b292ea61db43caf47d P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:09.129273 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 591a6c86fc984bac9fb9eb62e35dc6c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:09.129251 1166 raft_consensus.cc:740] T 6d830d30b26b4ba799785fec8a6f0769 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:09.129469 1166 consensus_queue.cc:260] T 6d830d30b26b4ba799785fec8a6f0769 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.130004 1166 ts_tablet_manager.cc:1434] T 6d830d30b26b4ba799785fec8a6f0769 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251029 02:52:09.130437 524 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_b19f559f0b7048ecbbb93a9d3ceae749" } new_table_name: "default.Table_6442d40b6bad4ac9a2dbd42a8a3cb096" modify_external_catalogs: true
I20251029 02:52:09.132407 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_345b52c4609e43c78920c2f13d1e7b3f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:09.132952 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_345b52c4609e43c78920c2f13d1e7b3f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:09.133658 1166 tablet_bootstrap.cc:492] T b6d4f7bf702f4834bc5f4ef2b50b92a7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:09.134775 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_715191bdf4ad4b2fbf349552331f0f1e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:09.135428 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_715191bdf4ad4b2fbf349552331f0f1e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:09.137004 1166 tablet_bootstrap.cc:654] T b6d4f7bf702f4834bc5f4ef2b50b92a7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:09.137192 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_41dc74911c3747a7a2763f31d7ca40e1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:09.140973 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_41dc74911c3747a7a2763f31d7ca40e1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:09.142319 743 tablet_service.cc:1552] Processing DeleteTablet for tablet af4860c6b72f4d988b206b5ef2f22906 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:56 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:09.142781 1166 tablet_bootstrap.cc:492] T b6d4f7bf702f4834bc5f4ef2b50b92a7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:09.142959 1166 ts_tablet_manager.cc:1403] T b6d4f7bf702f4834bc5f4ef2b50b92a7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.002s
I20251029 02:52:09.144006 1166 raft_consensus.cc:359] T b6d4f7bf702f4834bc5f4ef2b50b92a7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.144273 1166 raft_consensus.cc:385] T b6d4f7bf702f4834bc5f4ef2b50b92a7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:09.144371 1166 raft_consensus.cc:740] T b6d4f7bf702f4834bc5f4ef2b50b92a7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:09.144657 1166 consensus_queue.cc:260] T b6d4f7bf702f4834bc5f4ef2b50b92a7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
W20251029 02:52:09.149148 990 leader_election.cc:343] T 49d9ce32e4f04f11be2ecefc5d523a8b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:09.152364 1172 tablet_bootstrap.cc:492] T 71b4b91071a54d6aa66160e7955bc915 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:09.152508 1172 ts_tablet_manager.cc:1403] T 71b4b91071a54d6aa66160e7955bc915 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.051s user 0.001s sys 0.005s
I20251029 02:52:09.152664 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 6bd3fac2502e479aa9f4ceaabbb39504 (table default.Table_110e1b64737c4040ada22360d5a373cb [id=3513c69f6b3941849ac5b8ae0a5846d3]) successfully deleted
I20251029 02:52:09.153304 1172 raft_consensus.cc:359] T 71b4b91071a54d6aa66160e7955bc915 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:09.153491 1172 raft_consensus.cc:385] T 71b4b91071a54d6aa66160e7955bc915 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:09.153541 1172 raft_consensus.cc:740] T 71b4b91071a54d6aa66160e7955bc915 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:09.153797 1172 consensus_queue.cc:260] T 71b4b91071a54d6aa66160e7955bc915 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:09.154299 1172 ts_tablet_manager.cc:1434] T 71b4b91071a54d6aa66160e7955bc915 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 02:52:09.154590 1172 tablet_bootstrap.cc:492] T 82289e0f823c4d43bfd8c4e878572b2b P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:09.156725 1172 tablet_bootstrap.cc:654] T 82289e0f823c4d43bfd8c4e878572b2b P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:09.159168 1032 tablet_service.cc:1505] Processing CreateTablet for tablet 5a1aa131cf4e4edca5bb2515edd7556a (DEFAULT_TABLE table=default.Table_17afc4bb94e7410d8a45c9430e402e8c [id=85553cc5b6ea4decbbc5dd52532fff7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:09.160859 2103 consensus_queue.cc:1048] T 86fc22bb84794149b598b97df41eb881 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:09.163441 1167 tablet_bootstrap.cc:492] T 5db519edc75d4bd0985b32322825ffae P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:09.163604 1167 ts_tablet_manager.cc:1403] T 5db519edc75d4bd0985b32322825ffae P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.056s user 0.003s sys 0.015s
I20251029 02:52:09.163944 1172 tablet_bootstrap.cc:492] T 82289e0f823c4d43bfd8c4e878572b2b P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:09.164072 1172 ts_tablet_manager.cc:1403] T 82289e0f823c4d43bfd8c4e878572b2b P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20251029 02:52:09.164397 1167 raft_consensus.cc:359] T 5db519edc75d4bd0985b32322825ffae P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.164568 1167 raft_consensus.cc:385] T 5db519edc75d4bd0985b32322825ffae P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:09.164629 1167 raft_consensus.cc:740] T 5db519edc75d4bd0985b32322825ffae P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:09.164822 1167 consensus_queue.cc:260] T 5db519edc75d4bd0985b32322825ffae P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.164769 1172 raft_consensus.cc:359] T 82289e0f823c4d43bfd8c4e878572b2b P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:09.165128 1172 raft_consensus.cc:385] T 82289e0f823c4d43bfd8c4e878572b2b P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:09.165189 1172 raft_consensus.cc:740] T 82289e0f823c4d43bfd8c4e878572b2b P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:09.165313 1167 ts_tablet_manager.cc:1434] T 5db519edc75d4bd0985b32322825ffae P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251029 02:52:09.165405 1172 consensus_queue.cc:260] T 82289e0f823c4d43bfd8c4e878572b2b P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:09.165571 1167 tablet_bootstrap.cc:492] T ba699862e6f646c6beaa39f080d0faba P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:09.166080 1172 ts_tablet_manager.cc:1434] T 82289e0f823c4d43bfd8c4e878572b2b P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20251029 02:52:09.166378 1172 tablet_bootstrap.cc:492] T 217144957930495aa3e9dfcfde268e71 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:09.167236 1167 tablet_bootstrap.cc:654] T ba699862e6f646c6beaa39f080d0faba P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:09.168119 1166 ts_tablet_manager.cc:1434] T b6d4f7bf702f4834bc5f4ef2b50b92a7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.025s user 0.003s sys 0.000s
I20251029 02:52:09.168156 1172 tablet_bootstrap.cc:654] T 217144957930495aa3e9dfcfde268e71 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:09.168656 1166 tablet_bootstrap.cc:492] T 2e97d1bc06a24e8d8ba11acb6fd280d8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:09.170625 1166 tablet_bootstrap.cc:654] T 2e97d1bc06a24e8d8ba11acb6fd280d8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:09.174878 2205 raft_consensus.cc:493] T 1bdb67125493430398ff5c80b73de2ac P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:09.175033 2205 raft_consensus.cc:515] T 1bdb67125493430398ff5c80b73de2ac P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.176012 2205 leader_election.cc:290] T 1bdb67125493430398ff5c80b73de2ac P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:09.176802 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1bdb67125493430398ff5c80b73de2ac" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:09.177000 768 raft_consensus.cc:2468] T 1bdb67125493430398ff5c80b73de2ac P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:09.177039 1166 tablet_bootstrap.cc:492] T 2e97d1bc06a24e8d8ba11acb6fd280d8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:09.177174 1166 ts_tablet_manager.cc:1403] T 2e97d1bc06a24e8d8ba11acb6fd280d8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.003s
I20251029 02:52:09.177554 847 leader_election.cc:304] T 1bdb67125493430398ff5c80b73de2ac P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:09.177877 1166 raft_consensus.cc:359] T 2e97d1bc06a24e8d8ba11acb6fd280d8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.178063 1166 raft_consensus.cc:385] T 2e97d1bc06a24e8d8ba11acb6fd280d8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:09.178151 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1bdb67125493430398ff5c80b73de2ac" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:09.178421 1060 raft_consensus.cc:2468] T 1bdb67125493430398ff5c80b73de2ac P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:09.179123 2205 raft_consensus.cc:2804] T 1bdb67125493430398ff5c80b73de2ac P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:09.179258 2205 raft_consensus.cc:493] T 1bdb67125493430398ff5c80b73de2ac P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:09.179368 2205 raft_consensus.cc:3060] T 1bdb67125493430398ff5c80b73de2ac P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:09.179788 1167 tablet_bootstrap.cc:492] T ba699862e6f646c6beaa39f080d0faba P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:09.179913 1167 ts_tablet_manager.cc:1403] T ba699862e6f646c6beaa39f080d0faba P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.002s
I20251029 02:52:09.180647 1167 raft_consensus.cc:359] T ba699862e6f646c6beaa39f080d0faba P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.180811 1167 raft_consensus.cc:385] T ba699862e6f646c6beaa39f080d0faba P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:09.180869 1167 raft_consensus.cc:740] T ba699862e6f646c6beaa39f080d0faba P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:09.181061 1167 consensus_queue.cc:260] T ba699862e6f646c6beaa39f080d0faba P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.181588 1167 ts_tablet_manager.cc:1434] T ba699862e6f646c6beaa39f080d0faba P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251029 02:52:09.182062 1167 tablet_bootstrap.cc:492] T caef1c93e33a4390bcada59a41f43465 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:09.184015 1167 tablet_bootstrap.cc:654] T caef1c93e33a4390bcada59a41f43465 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:09.184392 1172 tablet_bootstrap.cc:492] T 217144957930495aa3e9dfcfde268e71 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:09.184554 1172 ts_tablet_manager.cc:1403] T 217144957930495aa3e9dfcfde268e71 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.000s
I20251029 02:52:09.185384 1172 raft_consensus.cc:359] T 217144957930495aa3e9dfcfde268e71 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:09.185573 1172 raft_consensus.cc:385] T 217144957930495aa3e9dfcfde268e71 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:09.185623 1172 raft_consensus.cc:740] T 217144957930495aa3e9dfcfde268e71 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:09.185830 1172 consensus_queue.cc:260] T 217144957930495aa3e9dfcfde268e71 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:09.186328 1172 ts_tablet_manager.cc:1434] T 217144957930495aa3e9dfcfde268e71 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251029 02:52:09.187229 1172 tablet_bootstrap.cc:492] T e65eaaae43ac435ebd665546de9d68dd P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:09.189025 1172 tablet_bootstrap.cc:654] T e65eaaae43ac435ebd665546de9d68dd P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:09.178153 1166 raft_consensus.cc:740] T 2e97d1bc06a24e8d8ba11acb6fd280d8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:09.190328 1166 consensus_queue.cc:260] T 2e97d1bc06a24e8d8ba11acb6fd280d8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.190945 1166 ts_tablet_manager.cc:1434] T 2e97d1bc06a24e8d8ba11acb6fd280d8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20251029 02:52:09.191278 1166 tablet_bootstrap.cc:492] T 9269622ca92645d5aa8e73341201ccbb P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:09.193234 1166 tablet_bootstrap.cc:654] T 9269622ca92645d5aa8e73341201ccbb P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:09.194931 517 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 5815626b05204b578cc4a110f246bf10
I20251029 02:52:09.195375 1172 tablet_bootstrap.cc:492] T e65eaaae43ac435ebd665546de9d68dd P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:09.195508 1172 ts_tablet_manager.cc:1403] T e65eaaae43ac435ebd665546de9d68dd P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.001s
I20251029 02:52:09.196312 1172 raft_consensus.cc:359] T e65eaaae43ac435ebd665546de9d68dd P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.196507 1172 raft_consensus.cc:385] T e65eaaae43ac435ebd665546de9d68dd P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:09.196566 1172 raft_consensus.cc:740] T e65eaaae43ac435ebd665546de9d68dd P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:09.196805 1172 consensus_queue.cc:260] T e65eaaae43ac435ebd665546de9d68dd P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.197345 1172 ts_tablet_manager.cc:1434] T e65eaaae43ac435ebd665546de9d68dd P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251029 02:52:09.197829 1167 tablet_bootstrap.cc:492] T caef1c93e33a4390bcada59a41f43465 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:09.197877 1172 tablet_bootstrap.cc:492] T d7abaf84ba5d445aa17433b3be891548 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:09.197971 1167 ts_tablet_manager.cc:1403] T caef1c93e33a4390bcada59a41f43465 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.000s
I20251029 02:52:09.198879 1167 raft_consensus.cc:359] T caef1c93e33a4390bcada59a41f43465 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.199074 1167 raft_consensus.cc:385] T caef1c93e33a4390bcada59a41f43465 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:09.199126 1167 raft_consensus.cc:740] T caef1c93e33a4390bcada59a41f43465 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:09.199330 1167 consensus_queue.cc:260] T caef1c93e33a4390bcada59a41f43465 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.199656 1172 tablet_bootstrap.cc:654] T d7abaf84ba5d445aa17433b3be891548 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:09.199810 1167 ts_tablet_manager.cc:1434] T caef1c93e33a4390bcada59a41f43465 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251029 02:52:09.200136 1167 tablet_bootstrap.cc:492] T 49d9ce32e4f04f11be2ecefc5d523a8b P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:09.200356 526 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_3bd4c012655a48aa970875bb07006ac2" } modify_external_catalogs: true
I20251029 02:52:09.200567 526 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_3bd4c012655a48aa970875bb07006ac2" } modify_external_catalogs: true
I20251029 02:52:09.202332 743 tablet_service.cc:1552] Processing DeleteTablet for tablet 5815626b05204b578cc4a110f246bf10 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:08 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:09.203999 1175 ts_tablet_manager.cc:1929] T d61beda662d844cdaf67fd9a2e382ea2 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:09.204113 1175 log.cc:1199] T d61beda662d844cdaf67fd9a2e382ea2 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/d61beda662d844cdaf67fd9a2e382ea2
I20251029 02:52:09.204401 1175 ts_tablet_manager.cc:1950] T d61beda662d844cdaf67fd9a2e382ea2 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:09.204715 1166 tablet_bootstrap.cc:492] T 9269622ca92645d5aa8e73341201ccbb P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:09.204879 1166 ts_tablet_manager.cc:1403] T 9269622ca92645d5aa8e73341201ccbb P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.004s
I20251029 02:52:09.205603 1166 raft_consensus.cc:359] T 9269622ca92645d5aa8e73341201ccbb P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.205785 1166 raft_consensus.cc:385] T 9269622ca92645d5aa8e73341201ccbb P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:09.205845 1166 raft_consensus.cc:740] T 9269622ca92645d5aa8e73341201ccbb P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:09.206059 1166 consensus_queue.cc:260] T 9269622ca92645d5aa8e73341201ccbb P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.206560 1166 ts_tablet_manager.cc:1434] T 9269622ca92645d5aa8e73341201ccbb P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251029 02:52:09.206857 1166 tablet_bootstrap.cc:492] T e7f846b294cc4eb687a26c8b5931ac19 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:09.207711 1174 ts_tablet_manager.cc:1929] T f187dbd8a55b45b292ea61db43caf47d P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:09.207826 1174 log.cc:1199] T f187dbd8a55b45b292ea61db43caf47d P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/f187dbd8a55b45b292ea61db43caf47d
I20251029 02:52:09.208096 1174 ts_tablet_manager.cc:1950] T f187dbd8a55b45b292ea61db43caf47d P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:09.208529 1166 tablet_bootstrap.cc:654] T e7f846b294cc4eb687a26c8b5931ac19 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:09.209924 2214 raft_consensus.cc:493] T c9bd3f2deb5d4af09b7918269a98509f P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:09.210052 2214 raft_consensus.cc:515] T c9bd3f2deb5d4af09b7918269a98509f P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.210696 2214 leader_election.cc:290] T c9bd3f2deb5d4af09b7918269a98509f P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:09.211907 1176 ts_tablet_manager.cc:1929] T caef1c93e33a4390bcada59a41f43465 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:09.212046 1176 log.cc:1199] T caef1c93e33a4390bcada59a41f43465 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/caef1c93e33a4390bcada59a41f43465
I20251029 02:52:09.212411 1176 ts_tablet_manager.cc:1950] T caef1c93e33a4390bcada59a41f43465 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:09.212550 1167 tablet_bootstrap.cc:654] T 49d9ce32e4f04f11be2ecefc5d523a8b P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:09.213534 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a1aa131cf4e4edca5bb2515edd7556a. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:09.214239 847 leader_election.cc:343] T c9bd3f2deb5d4af09b7918269a98509f P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: c9bd3f2deb5d4af09b7918269a98509f
I20251029 02:52:09.217615 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_17c5072bc19c40b0ac7348fdf90a1e8d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:09.218245 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_17c5072bc19c40b0ac7348fdf90a1e8d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:09.219703 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 787) took 2058 ms. Trace:
I20251029 02:52:09.219801 1175 rpcz_store.cc:276] 1029 02:52:07.161565 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:07.161615 (+ 50us) service_pool.cc:225] Handling call
1029 02:52:09.043631 (+1882016us) ts_tablet_manager.cc:1151] Deleting tablet d61beda662d844cdaf67fd9a2e382ea2
1029 02:52:09.043642 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:09.115055 (+ 71413us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:09.203888 (+ 88833us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:09.219674 (+ 15786us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1881839}
I20251029 02:52:09.220216 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 791) took 2039 ms. Trace:
I20251029 02:52:09.220280 1175 rpcz_store.cc:276] 1029 02:52:07.180586 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:07.183845 (+ 3259us) service_pool.cc:225] Handling call
1029 02:52:09.220102 (+2036257us) ts_tablet_manager.cc:1151] Deleting tablet 31a157697e2c45ac9ba92c31c6936fa0
1029 02:52:09.220110 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:09.220203 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2034879}
I20251029 02:52:09.220510 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 792) took 2028 ms. Trace:
I20251029 02:52:09.220559 1175 rpcz_store.cc:276] 1029 02:52:07.191860 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:07.191920 (+ 60us) service_pool.cc:225] Handling call
1029 02:52:09.220443 (+2028523us) ts_tablet_manager.cc:1151] Deleting tablet 91e31f0b06924740bdf7525cae79d8ef
1029 02:52:09.220449 (+ 6us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:09.220500 (+ 51us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2028298}
I20251029 02:52:09.220778 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 793) took 2014 ms. Trace:
I20251029 02:52:09.220827 1175 rpcz_store.cc:276] 1029 02:52:07.206370 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:07.206431 (+ 61us) service_pool.cc:225] Handling call
1029 02:52:09.220702 (+2014271us) ts_tablet_manager.cc:1151] Deleting tablet e4cad991d48341cbafb5f1d9b50f4e5b
1029 02:52:09.220708 (+ 6us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:09.220768 (+ 60us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2013937}
I20251029 02:52:09.221019 1175 tablet_replica.cc:333] T e7f846b294cc4eb687a26c8b5931ac19 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:09.221375 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet d61beda662d844cdaf67fd9a2e382ea2 (table default.Table_ca4e3b5646244cafa5a0429d5a8f2ac6 [id=481e8e175f364d31bb7dbbaa2df2dad1]) successfully deleted
I20251029 02:52:09.221544 1175 raft_consensus.cc:2243] T e7f846b294cc4eb687a26c8b5931ac19 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:09.221688 1175 raft_consensus.cc:2272] T e7f846b294cc4eb687a26c8b5931ac19 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251029 02:52:09.221689 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 31a157697e2c45ac9ba92c31c6936fa0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 31a157697e2c45ac9ba92c31c6936fa0 already in progress: creating tablet
W20251029 02:52:09.221935 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 91e31f0b06924740bdf7525cae79d8ef because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 91e31f0b06924740bdf7525cae79d8ef already in progress: creating tablet
I20251029 02:52:09.222163 1175 ts_tablet_manager.cc:1916] T e7f846b294cc4eb687a26c8b5931ac19 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 02:52:09.222174 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet e4cad991d48341cbafb5f1d9b50f4e5b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e4cad991d48341cbafb5f1d9b50f4e5b already in progress: creating tablet
I20251029 02:52:09.223413 1166 tablet_bootstrap.cc:492] T e7f846b294cc4eb687a26c8b5931ac19 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
W20251029 02:52:09.223441 849 leader_election.cc:343] T c9bd3f2deb5d4af09b7918269a98509f P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Not found: Tablet not found: c9bd3f2deb5d4af09b7918269a98509f
I20251029 02:52:09.223559 1166 ts_tablet_manager.cc:1403] T e7f846b294cc4eb687a26c8b5931ac19 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.003s
I20251029 02:52:09.223551 849 leader_election.cc:304] T c9bd3f2deb5d4af09b7918269a98509f P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9
I20251029 02:52:09.224036 2214 raft_consensus.cc:2749] T c9bd3f2deb5d4af09b7918269a98509f P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:09.224289 1166 raft_consensus.cc:359] T e7f846b294cc4eb687a26c8b5931ac19 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.224474 1166 raft_consensus.cc:385] T e7f846b294cc4eb687a26c8b5931ac19 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:09.224568 1166 raft_consensus.cc:740] T e7f846b294cc4eb687a26c8b5931ac19 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:09.225247 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_6d9b81aa4e3a4880ac256ecf0fe9f010"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:09.219856 1172 tablet_bootstrap.cc:492] T d7abaf84ba5d445aa17433b3be891548 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:09.225745 1172 ts_tablet_manager.cc:1403] T d7abaf84ba5d445aa17433b3be891548 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.028s user 0.000s sys 0.006s
W20251029 02:52:09.225822 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6d9b81aa4e3a4880ac256ecf0fe9f010 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:09.225955 1167 tablet_bootstrap.cc:492] T 49d9ce32e4f04f11be2ecefc5d523a8b P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:09.226076 1167 ts_tablet_manager.cc:1403] T 49d9ce32e4f04f11be2ecefc5d523a8b P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.001s
I20251029 02:52:09.226501 1172 raft_consensus.cc:359] T d7abaf84ba5d445aa17433b3be891548 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:09.226671 1172 raft_consensus.cc:385] T d7abaf84ba5d445aa17433b3be891548 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:09.226719 1172 raft_consensus.cc:740] T d7abaf84ba5d445aa17433b3be891548 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:09.226838 1167 raft_consensus.cc:359] T 49d9ce32e4f04f11be2ecefc5d523a8b P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.226981 1167 raft_consensus.cc:385] T 49d9ce32e4f04f11be2ecefc5d523a8b P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:09.227041 1167 raft_consensus.cc:740] T 49d9ce32e4f04f11be2ecefc5d523a8b P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:09.227221 1167 consensus_queue.cc:260] T 49d9ce32e4f04f11be2ecefc5d523a8b P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.227705 1167 ts_tablet_manager.cc:1434] T 49d9ce32e4f04f11be2ecefc5d523a8b P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251029 02:52:09.227699 743 tablet_service.cc:1552] Processing DeleteTablet for tablet c32c28b0e74a4af3a0e69bbf24ab1a1f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:57 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:09.227896 751 tablet_service.cc:1552] Processing DeleteTablet for tablet 44ae2b07edca44afbe1b1fb3a5b1115d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:00 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:09.227977 1167 tablet_bootstrap.cc:492] T 95f342d1cfb84c4eb31bd3d15ea3bdc2 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:09.228045 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 97b2ecae572a4b7ea1b9595074556708 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:00 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:09.226969 1172 consensus_queue.cc:260] T d7abaf84ba5d445aa17433b3be891548 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:09.228497 2214 raft_consensus.cc:493] T ccdca31db2834de4b6ed0f2d67a10535 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:09.228611 2214 raft_consensus.cc:515] T ccdca31db2834de4b6ed0f2d67a10535 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.229166 2214 leader_election.cc:290] T ccdca31db2834de4b6ed0f2d67a10535 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:09.229301 1172 ts_tablet_manager.cc:1434] T d7abaf84ba5d445aa17433b3be891548 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251029 02:52:09.229584 1172 tablet_bootstrap.cc:492] T 44c0488eba0543078ff4f0b3d69366f2 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:09.229854 1058 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ccdca31db2834de4b6ed0f2d67a10535" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:09.230195 1167 tablet_bootstrap.cc:654] T 95f342d1cfb84c4eb31bd3d15ea3bdc2 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:09.230706 1166 consensus_queue.cc:260] T e7f846b294cc4eb687a26c8b5931ac19 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.231235 1166 ts_tablet_manager.cc:1434] T e7f846b294cc4eb687a26c8b5931ac19 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20251029 02:52:09.231856 1166 tablet_bootstrap.cc:492] T 30d03e1e42d34242bd354f9e9ec4f127 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:09.233450 1172 tablet_bootstrap.cc:654] T 44c0488eba0543078ff4f0b3d69366f2 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
W20251029 02:52:09.234834 847 leader_election.cc:343] T ccdca31db2834de4b6ed0f2d67a10535 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: ccdca31db2834de4b6ed0f2d67a10535
I20251029 02:52:09.234947 847 leader_election.cc:304] T ccdca31db2834de4b6ed0f2d67a10535 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9
I20251029 02:52:09.235718 2214 raft_consensus.cc:2749] T ccdca31db2834de4b6ed0f2d67a10535 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:09.236641 2214 raft_consensus.cc:493] T c764062d3ba04d4ba8b2e367ad8ed433 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:09.236759 2214 raft_consensus.cc:515] T c764062d3ba04d4ba8b2e367ad8ed433 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.237258 1166 tablet_bootstrap.cc:654] T 30d03e1e42d34242bd354f9e9ec4f127 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:09.237313 2214 leader_election.cc:290] T c764062d3ba04d4ba8b2e367ad8ed433 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
W20251029 02:52:09.238858 849 leader_election.cc:343] T c764062d3ba04d4ba8b2e367ad8ed433 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Not found: Tablet not found: c764062d3ba04d4ba8b2e367ad8ed433
W20251029 02:52:09.239106 847 leader_election.cc:343] T c764062d3ba04d4ba8b2e367ad8ed433 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: c764062d3ba04d4ba8b2e367ad8ed433
I20251029 02:52:09.239192 847 leader_election.cc:304] T c764062d3ba04d4ba8b2e367ad8ed433 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9
I20251029 02:52:09.239668 2214 raft_consensus.cc:2749] T c764062d3ba04d4ba8b2e367ad8ed433 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:09.242240 2205 raft_consensus.cc:515] T 1bdb67125493430398ff5c80b73de2ac P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.243160 2205 leader_election.cc:290] T 1bdb67125493430398ff5c80b73de2ac P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:09.243252 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1bdb67125493430398ff5c80b73de2ac" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:09.243418 768 raft_consensus.cc:3060] T 1bdb67125493430398ff5c80b73de2ac P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:09.244841 1172 tablet_bootstrap.cc:492] T 44c0488eba0543078ff4f0b3d69366f2 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:09.244989 1172 ts_tablet_manager.cc:1403] T 44c0488eba0543078ff4f0b3d69366f2 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
W20251029 02:52:09.245437 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.3:45003, user_credentials={real_user=slave}} blocked reactor thread for 67137.2us
I20251029 02:52:09.245906 1172 raft_consensus.cc:359] T 44c0488eba0543078ff4f0b3d69366f2 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:09.246073 1172 raft_consensus.cc:385] T 44c0488eba0543078ff4f0b3d69366f2 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:09.246127 1172 raft_consensus.cc:740] T 44c0488eba0543078ff4f0b3d69366f2 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:09.246337 1172 consensus_queue.cc:260] T 44c0488eba0543078ff4f0b3d69366f2 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:09.246850 1172 ts_tablet_manager.cc:1434] T 44c0488eba0543078ff4f0b3d69366f2 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 02:52:09.247103 1172 tablet_bootstrap.cc:492] T afb19ba39399410bb90b6225d7b5b5b1 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:09.247210 1166 tablet_bootstrap.cc:492] T 30d03e1e42d34242bd354f9e9ec4f127 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:09.247153 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1bdb67125493430398ff5c80b73de2ac" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:52:09.247370 1166 ts_tablet_manager.cc:1403] T 30d03e1e42d34242bd354f9e9ec4f127 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.004s
I20251029 02:52:09.247390 1060 raft_consensus.cc:3060] T 1bdb67125493430398ff5c80b73de2ac P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:09.248144 1166 raft_consensus.cc:359] T 30d03e1e42d34242bd354f9e9ec4f127 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.248330 1166 raft_consensus.cc:385] T 30d03e1e42d34242bd354f9e9ec4f127 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:09.248400 1166 raft_consensus.cc:740] T 30d03e1e42d34242bd354f9e9ec4f127 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:09.248623 1166 consensus_queue.cc:260] T 30d03e1e42d34242bd354f9e9ec4f127 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.248855 1172 tablet_bootstrap.cc:654] T afb19ba39399410bb90b6225d7b5b5b1 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:09.250389 1167 tablet_bootstrap.cc:492] T 95f342d1cfb84c4eb31bd3d15ea3bdc2 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:09.250530 1167 ts_tablet_manager.cc:1403] T 95f342d1cfb84c4eb31bd3d15ea3bdc2 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.004s
I20251029 02:52:09.251302 1167 raft_consensus.cc:359] T 95f342d1cfb84c4eb31bd3d15ea3bdc2 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.251454 1167 raft_consensus.cc:385] T 95f342d1cfb84c4eb31bd3d15ea3bdc2 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:09.251503 1167 raft_consensus.cc:740] T 95f342d1cfb84c4eb31bd3d15ea3bdc2 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:09.251677 1167 consensus_queue.cc:260] T 95f342d1cfb84c4eb31bd3d15ea3bdc2 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.252130 1167 ts_tablet_manager.cc:1434] T 95f342d1cfb84c4eb31bd3d15ea3bdc2 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251029 02:52:09.252377 1167 tablet_bootstrap.cc:492] T 0f80b840ed954230bfbfe7e40127cd2f P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:09.253906 1167 tablet_bootstrap.cc:654] T 0f80b840ed954230bfbfe7e40127cd2f P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:09.258360 1166 ts_tablet_manager.cc:1434] T 30d03e1e42d34242bd354f9e9ec4f127 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.011s user 0.000s sys 0.002s
I20251029 02:52:09.258664 879 tablet_service.cc:1552] Processing DeleteTablet for tablet 5815626b05204b578cc4a110f246bf10 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:08 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:09.258845 1166 tablet_bootstrap.cc:492] T af4860c6b72f4d988b206b5ef2f22906 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:09.259305 879 tablet_service.cc:1552] Processing DeleteTablet for tablet 9bb2166a590d4210b02dd779ef6654ad with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:55 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:09.260118 879 tablet_service.cc:1552] Processing DeleteTablet for tablet 1275e192871440408a8345fef9b56260 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:56 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:09.260579 879 tablet_service.cc:1552] Processing DeleteTablet for tablet b6d4f7bf702f4834bc5f4ef2b50b92a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:55 UTC) from {username='slave'} at 127.0.0.1:35702
W20251029 02:52:09.261885 510 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.62:42343: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20251029 02:52:09.262161 919 raft_consensus.cc:3060] T ccdca31db2834de4b6ed0f2d67a10535 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:09.262331 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 805) took 1763 ms. Trace:
I20251029 02:52:09.262418 1174 rpcz_store.cc:276] 1029 02:52:07.498717 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:07.499925 (+ 1208us) service_pool.cc:225] Handling call
1029 02:52:09.128331 (+1628406us) ts_tablet_manager.cc:1151] Deleting tablet f187dbd8a55b45b292ea61db43caf47d
1029 02:52:09.128340 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:09.202682 (+ 74342us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:09.207598 (+ 4916us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:09.262310 (+ 54712us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1615140}
I20251029 02:52:09.262737 1174 tablet_replica.cc:333] T 71068739b4384badbf305313d902f0f5 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:09.262980 1174 raft_consensus.cc:2243] T 71068739b4384badbf305313d902f0f5 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:09.263119 1174 raft_consensus.cc:2272] T 71068739b4384badbf305313d902f0f5 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:09.263628 1174 ts_tablet_manager.cc:1916] T 71068739b4384badbf305313d902f0f5 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:09.266024 1166 tablet_bootstrap.cc:654] T af4860c6b72f4d988b206b5ef2f22906 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:09.266817 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet f187dbd8a55b45b292ea61db43caf47d (table default.Table_d6a9ff6b4cea442eb2394340301eac96 [id=3f2d90da0f88434592a21c0687450ded]) successfully deleted
I20251029 02:52:09.267663 1172 tablet_bootstrap.cc:492] T afb19ba39399410bb90b6225d7b5b5b1 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:09.267812 1172 ts_tablet_manager.cc:1403] T afb19ba39399410bb90b6225d7b5b5b1 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.004s
I20251029 02:52:09.268646 1172 raft_consensus.cc:359] T afb19ba39399410bb90b6225d7b5b5b1 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:09.268867 1172 raft_consensus.cc:385] T afb19ba39399410bb90b6225d7b5b5b1 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:09.268934 1172 raft_consensus.cc:740] T afb19ba39399410bb90b6225d7b5b5b1 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:09.269169 1172 consensus_queue.cc:260] T afb19ba39399410bb90b6225d7b5b5b1 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:09.269717 1172 ts_tablet_manager.cc:1434] T afb19ba39399410bb90b6225d7b5b5b1 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 02:52:09.270027 1172 tablet_bootstrap.cc:492] T 2f198c3259864f7ab3625cb7bfae20b0 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:09.270488 1167 tablet_bootstrap.cc:492] T 0f80b840ed954230bfbfe7e40127cd2f P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:09.270650 1167 ts_tablet_manager.cc:1403] T 0f80b840ed954230bfbfe7e40127cd2f P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.001s
I20251029 02:52:09.271355 1167 raft_consensus.cc:359] T 0f80b840ed954230bfbfe7e40127cd2f P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.271497 1167 raft_consensus.cc:385] T 0f80b840ed954230bfbfe7e40127cd2f P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:09.271569 1167 raft_consensus.cc:740] T 0f80b840ed954230bfbfe7e40127cd2f P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:09.271652 1172 tablet_bootstrap.cc:654] T 2f198c3259864f7ab3625cb7bfae20b0 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:09.271780 1167 consensus_queue.cc:260] T 0f80b840ed954230bfbfe7e40127cd2f P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.272269 1167 ts_tablet_manager.cc:1434] T 0f80b840ed954230bfbfe7e40127cd2f P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251029 02:52:09.272527 1167 tablet_bootstrap.cc:492] T 13ee62fc2339410b9db70dfd3d3c6fa6 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:09.274209 1167 tablet_bootstrap.cc:654] T 13ee62fc2339410b9db70dfd3d3c6fa6 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:09.278795 1172 tablet_bootstrap.cc:492] T 2f198c3259864f7ab3625cb7bfae20b0 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:09.278927 1172 ts_tablet_manager.cc:1403] T 2f198c3259864f7ab3625cb7bfae20b0 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.001s
I20251029 02:52:09.279573 1172 raft_consensus.cc:359] T 2f198c3259864f7ab3625cb7bfae20b0 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:09.279748 1172 raft_consensus.cc:385] T 2f198c3259864f7ab3625cb7bfae20b0 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:09.279798 1172 raft_consensus.cc:740] T 2f198c3259864f7ab3625cb7bfae20b0 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:09.280000 1172 consensus_queue.cc:260] T 2f198c3259864f7ab3625cb7bfae20b0 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:09.280462 1172 ts_tablet_manager.cc:1434] T 2f198c3259864f7ab3625cb7bfae20b0 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20251029 02:52:09.280692 1172 tablet_bootstrap.cc:492] T 731c0acfbe204e8199ad246e38ae2ccd P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:09.282312 1172 tablet_bootstrap.cc:654] T 731c0acfbe204e8199ad246e38ae2ccd P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:09.282464 1166 tablet_bootstrap.cc:492] T af4860c6b72f4d988b206b5ef2f22906 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:09.282637 1166 ts_tablet_manager.cc:1403] T af4860c6b72f4d988b206b5ef2f22906 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.002s
I20251029 02:52:09.283390 1166 raft_consensus.cc:359] T af4860c6b72f4d988b206b5ef2f22906 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.283625 1166 raft_consensus.cc:385] T af4860c6b72f4d988b206b5ef2f22906 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:09.283685 1166 raft_consensus.cc:740] T af4860c6b72f4d988b206b5ef2f22906 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:09.283914 1166 consensus_queue.cc:260] T af4860c6b72f4d988b206b5ef2f22906 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.284930 1166 ts_tablet_manager.cc:1434] T af4860c6b72f4d988b206b5ef2f22906 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 02:52:09.285176 1166 tablet_bootstrap.cc:492] T 71b4b91071a54d6aa66160e7955bc915 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:09.287154 1172 tablet_bootstrap.cc:492] T 731c0acfbe204e8199ad246e38ae2ccd P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:09.287292 1172 ts_tablet_manager.cc:1403] T 731c0acfbe204e8199ad246e38ae2ccd P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20251029 02:52:09.287789 1166 tablet_bootstrap.cc:654] T 71b4b91071a54d6aa66160e7955bc915 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:09.288067 1172 raft_consensus.cc:359] T 731c0acfbe204e8199ad246e38ae2ccd P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:09.288236 1172 raft_consensus.cc:385] T 731c0acfbe204e8199ad246e38ae2ccd P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:09.288287 1172 raft_consensus.cc:740] T 731c0acfbe204e8199ad246e38ae2ccd P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:09.288457 1172 consensus_queue.cc:260] T 731c0acfbe204e8199ad246e38ae2ccd P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:09.289090 1172 ts_tablet_manager.cc:1434] T 731c0acfbe204e8199ad246e38ae2ccd P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 02:52:09.289367 1172 tablet_bootstrap.cc:492] T adf6e736cb9c4920807ee58682e134ef P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:09.290439 1176 tablet_replica.cc:333] T 5db519edc75d4bd0985b32322825ffae P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:09.290707 1176 raft_consensus.cc:2243] T 5db519edc75d4bd0985b32322825ffae P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:09.290850 1176 raft_consensus.cc:2272] T 5db519edc75d4bd0985b32322825ffae P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:09.291371 1176 ts_tablet_manager.cc:1916] T 5db519edc75d4bd0985b32322825ffae P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:09.292999 1175 ts_tablet_manager.cc:1929] T e7f846b294cc4eb687a26c8b5931ac19 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:09.293154 1175 log.cc:1199] T e7f846b294cc4eb687a26c8b5931ac19 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/e7f846b294cc4eb687a26c8b5931ac19
I20251029 02:52:09.293493 1175 ts_tablet_manager.cc:1950] T e7f846b294cc4eb687a26c8b5931ac19 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:09.297602 1060 raft_consensus.cc:2468] T 1bdb67125493430398ff5c80b73de2ac P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:09.298601 849 leader_election.cc:304] T 1bdb67125493430398ff5c80b73de2ac P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:09.298703 768 raft_consensus.cc:2468] T 1bdb67125493430398ff5c80b73de2ac P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:09.298930 2205 raft_consensus.cc:2804] T 1bdb67125493430398ff5c80b73de2ac P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:09.299038 2205 raft_consensus.cc:697] T 1bdb67125493430398ff5c80b73de2ac P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:52:09.299279 2205 consensus_queue.cc:237] T 1bdb67125493430398ff5c80b73de2ac P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.300158 2179 consensus_queue.cc:1048] T ccdca31db2834de4b6ed0f2d67a10535 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:09.302627 2179 raft_consensus.cc:493] T 6d830d30b26b4ba799785fec8a6f0769 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:09.302757 2179 raft_consensus.cc:515] T 6d830d30b26b4ba799785fec8a6f0769 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.303119 1172 tablet_bootstrap.cc:654] T adf6e736cb9c4920807ee58682e134ef P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:09.303274 2179 leader_election.cc:290] T 6d830d30b26b4ba799785fec8a6f0769 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:09.304193 1167 tablet_bootstrap.cc:492] T 13ee62fc2339410b9db70dfd3d3c6fa6 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:09.304366 1167 ts_tablet_manager.cc:1403] T 13ee62fc2339410b9db70dfd3d3c6fa6 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.032s user 0.001s sys 0.005s
I20251029 02:52:09.305938 521 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet ac162bce5b6e4080839095a045ef0cc0
I20251029 02:52:09.307335 1167 raft_consensus.cc:359] T 13ee62fc2339410b9db70dfd3d3c6fa6 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.307482 1167 raft_consensus.cc:385] T 13ee62fc2339410b9db70dfd3d3c6fa6 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:09.307523 1167 raft_consensus.cc:740] T 13ee62fc2339410b9db70dfd3d3c6fa6 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:09.307694 1167 consensus_queue.cc:260] T 13ee62fc2339410b9db70dfd3d3c6fa6 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.308076 745 tablet_service.cc:1552] Processing DeleteTablet for tablet ac162bce5b6e4080839095a045ef0cc0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:08 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:09.308144 1167 ts_tablet_manager.cc:1434] T 13ee62fc2339410b9db70dfd3d3c6fa6 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.004s user 0.000s sys 0.000s
I20251029 02:52:09.308415 1167 tablet_bootstrap.cc:492] T 6f35412a66b74051a475dce5dafa7792 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:09.310186 1167 tablet_bootstrap.cc:654] T 6f35412a66b74051a475dce5dafa7792 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:09.313302 914 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d830d30b26b4ba799785fec8a6f0769" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:09.313841 990 leader_election.cc:343] T 6d830d30b26b4ba799785fec8a6f0769 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:09.314604 879 tablet_service.cc:1552] Processing DeleteTablet for tablet ac162bce5b6e4080839095a045ef0cc0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:08 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:09.315310 1166 tablet_bootstrap.cc:492] T 71b4b91071a54d6aa66160e7955bc915 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:09.315466 1166 ts_tablet_manager.cc:1403] T 71b4b91071a54d6aa66160e7955bc915 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.030s user 0.002s sys 0.004s
I20251029 02:52:09.316303 1166 raft_consensus.cc:359] T 71b4b91071a54d6aa66160e7955bc915 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:09.316465 1166 raft_consensus.cc:385] T 71b4b91071a54d6aa66160e7955bc915 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:09.316519 1166 raft_consensus.cc:740] T 71b4b91071a54d6aa66160e7955bc915 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:09.316704 1166 consensus_queue.cc:260] T 71b4b91071a54d6aa66160e7955bc915 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:09.317337 1172 tablet_bootstrap.cc:492] T adf6e736cb9c4920807ee58682e134ef P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:09.317473 1172 ts_tablet_manager.cc:1403] T adf6e736cb9c4920807ee58682e134ef P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.028s user 0.003s sys 0.000s
I20251029 02:52:09.317953 1166 ts_tablet_manager.cc:1434] T 71b4b91071a54d6aa66160e7955bc915 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 02:52:09.318199 1172 raft_consensus.cc:359] T adf6e736cb9c4920807ee58682e134ef P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:09.318356 1172 raft_consensus.cc:385] T adf6e736cb9c4920807ee58682e134ef P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:09.318401 1172 raft_consensus.cc:740] T adf6e736cb9c4920807ee58682e134ef P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:09.318619 1172 consensus_queue.cc:260] T adf6e736cb9c4920807ee58682e134ef P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:09.319113 1172 ts_tablet_manager.cc:1434] T adf6e736cb9c4920807ee58682e134ef P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251029 02:52:09.319525 1172 tablet_bootstrap.cc:492] T c1222e30bc4141b3ae686d80053c4ef5 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:09.319751 1166 tablet_bootstrap.cc:492] T 217144957930495aa3e9dfcfde268e71 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:09.321237 1172 tablet_bootstrap.cc:654] T c1222e30bc4141b3ae686d80053c4ef5 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:09.321436 1166 tablet_bootstrap.cc:654] T 217144957930495aa3e9dfcfde268e71 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:09.326992 2205 raft_consensus.cc:493] T b6d4f7bf702f4834bc5f4ef2b50b92a7 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:09.327217 2205 raft_consensus.cc:515] T b6d4f7bf702f4834bc5f4ef2b50b92a7 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.327926 2205 leader_election.cc:290] T b6d4f7bf702f4834bc5f4ef2b50b92a7 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:09.328619 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b6d4f7bf702f4834bc5f4ef2b50b92a7" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:09.328872 1060 raft_consensus.cc:2468] T b6d4f7bf702f4834bc5f4ef2b50b92a7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:09.329752 849 leader_election.cc:304] T b6d4f7bf702f4834bc5f4ef2b50b92a7 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:09.329834 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b6d4f7bf702f4834bc5f4ef2b50b92a7" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:09.330034 2205 raft_consensus.cc:2804] T b6d4f7bf702f4834bc5f4ef2b50b92a7 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:09.330011 768 raft_consensus.cc:2468] T b6d4f7bf702f4834bc5f4ef2b50b92a7 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:09.330099 2205 raft_consensus.cc:493] T b6d4f7bf702f4834bc5f4ef2b50b92a7 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:09.330175 2205 raft_consensus.cc:3060] T b6d4f7bf702f4834bc5f4ef2b50b92a7 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:09.331509 1167 tablet_bootstrap.cc:492] T 6f35412a66b74051a475dce5dafa7792 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:09.331674 1167 ts_tablet_manager.cc:1403] T 6f35412a66b74051a475dce5dafa7792 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.000s
I20251029 02:52:09.332643 1167 raft_consensus.cc:359] T 6f35412a66b74051a475dce5dafa7792 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.332886 1167 raft_consensus.cc:385] T 6f35412a66b74051a475dce5dafa7792 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:09.332960 1167 raft_consensus.cc:740] T 6f35412a66b74051a475dce5dafa7792 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:09.333194 1167 consensus_queue.cc:260] T 6f35412a66b74051a475dce5dafa7792 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.333781 1167 ts_tablet_manager.cc:1434] T 6f35412a66b74051a475dce5dafa7792 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251029 02:52:09.334066 1167 tablet_bootstrap.cc:492] T 1c53fc1fff1a4bb988beb46ff91fd3be P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:09.335798 521 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet 82289e0f823c4d43bfd8c4e878572b2b (table default.Table_e7bb3e262a894c2f84b6edfc3917b30d [id=2cca3b7c50b243b5883b8ac1f353f253]). Expected version 1 got 0
I20251029 02:52:09.336040 521 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet d7abaf84ba5d445aa17433b3be891548 (table default.Table_c1c6933af1e046aa895e0ee915178f5c [id=9e33c530a35544a4ae22e2a1238592db]). Expected version 3 got 0
I20251029 02:52:09.337709 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 217144957930495aa3e9dfcfde268e71 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:01 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:09.339529 523 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 1497a93ed0e849e5935c362f014b7e28
I20251029 02:52:09.340075 2036 raft_consensus.cc:493] T 1275e192871440408a8345fef9b56260 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:09.340250 2036 raft_consensus.cc:515] T 1275e192871440408a8345fef9b56260 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.341049 2036 leader_election.cc:290] T 1275e192871440408a8345fef9b56260 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
W20251029 02:52:09.341549 524 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 41a171c7ba55422b8700c4f898dc7d9e has no consensus state
I20251029 02:52:09.343508 1167 tablet_bootstrap.cc:654] T 1c53fc1fff1a4bb988beb46ff91fd3be P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:09.345507 745 tablet_service.cc:1552] Processing DeleteTablet for tablet e65eaaae43ac435ebd665546de9d68dd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:01 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:09.346139 521 master_service.cc:954] ReplaceTablet: received request to replace tablet 23856678b7aa4f9ebf65fb4564a4bf91 from {username='slave'} at 127.0.0.1:35314
I20251029 02:52:09.347622 914 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1275e192871440408a8345fef9b56260" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:09.347851 914 raft_consensus.cc:2468] T 1275e192871440408a8345fef9b56260 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 0.
I20251029 02:52:09.348282 990 leader_election.cc:304] T 1275e192871440408a8345fef9b56260 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:09.348651 2036 raft_consensus.cc:2804] T 1275e192871440408a8345fef9b56260 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:09.348766 2036 raft_consensus.cc:493] T 1275e192871440408a8345fef9b56260 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:09.348842 2036 raft_consensus.cc:3060] T 1275e192871440408a8345fef9b56260 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:09.362663 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 794) took 2130 ms. Trace:
I20251029 02:52:09.362769 1175 rpcz_store.cc:276] 1029 02:52:07.232608 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:07.232684 (+ 76us) service_pool.cc:225] Handling call
1029 02:52:09.220983 (+1988299us) ts_tablet_manager.cc:1151] Deleting tablet e7f846b294cc4eb687a26c8b5931ac19
1029 02:52:09.220989 (+ 6us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:09.289580 (+ 68591us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:09.292905 (+ 3325us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:09.362634 (+ 69729us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1988002}
I20251029 02:52:09.362849 520 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_74a966e440094ced83dd557daa4c671a" } new_table_name: "default.Table_0f200bab93784e9d97c7d001330e303a" modify_external_catalogs: true
I20251029 02:52:09.363215 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 795) took 2130 ms. Trace:
I20251029 02:52:09.363308 1175 rpcz_store.cc:276] 1029 02:52:07.232761 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:07.232808 (+ 47us) service_pool.cc:225] Handling call
1029 02:52:09.363054 (+2130246us) ts_tablet_manager.cc:1151] Deleting tablet e5c88e03d9324a918020688c884eb4f6
1029 02:52:09.363067 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:09.363192 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2129489}
I20251029 02:52:09.363714 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 799) took 1752 ms. Trace:
I20251029 02:52:09.363574 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_07d2f4a3c8074ece8b4982e3819caabc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:09.363776 1175 rpcz_store.cc:276] 1029 02:52:07.611527 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:07.621991 (+ 10464us) service_pool.cc:225] Handling call
1029 02:52:09.363625 (+1741634us) ts_tablet_manager.cc:1151] Deleting tablet faac614af1814a2f91d66c9cf8ddc919
1029 02:52:09.363632 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:09.363700 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1741356}
I20251029 02:52:09.364030 1175 tablet_replica.cc:333] T 4bf06476c87a4549a73f5994314bb75f P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:09.364228 1175 raft_consensus.cc:2243] T 4bf06476c87a4549a73f5994314bb75f P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus shutting down.
W20251029 02:52:09.364259 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_07d2f4a3c8074ece8b4982e3819caabc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:09.364382 1175 raft_consensus.cc:2272] T 4bf06476c87a4549a73f5994314bb75f P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:09.364637 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_d518217e586b4a57a08855986cc0026b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:09.364859 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet e5c88e03d9324a918020688c884eb4f6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e5c88e03d9324a918020688c884eb4f6 already in progress: creating tablet
I20251029 02:52:09.364928 1175 ts_tablet_manager.cc:1916] T 4bf06476c87a4549a73f5994314bb75f P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 02:52:09.365131 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet faac614af1814a2f91d66c9cf8ddc919 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet faac614af1814a2f91d66c9cf8ddc919 already in progress: creating tablet
W20251029 02:52:09.365248 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d518217e586b4a57a08855986cc0026b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:09.365545 1166 tablet_bootstrap.cc:492] T 217144957930495aa3e9dfcfde268e71 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:09.365674 1166 ts_tablet_manager.cc:1403] T 217144957930495aa3e9dfcfde268e71 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.047s user 0.004s sys 0.004s
I20251029 02:52:09.363755 1172 tablet_bootstrap.cc:492] T c1222e30bc4141b3ae686d80053c4ef5 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:09.368353 1172 ts_tablet_manager.cc:1403] T c1222e30bc4141b3ae686d80053c4ef5 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.049s user 0.002s sys 0.008s
I20251029 02:52:09.366425 1166 raft_consensus.cc:359] T 217144957930495aa3e9dfcfde268e71 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:09.369235 1166 raft_consensus.cc:385] T 217144957930495aa3e9dfcfde268e71 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:09.369305 1166 raft_consensus.cc:740] T 217144957930495aa3e9dfcfde268e71 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:09.369370 1172 raft_consensus.cc:359] T c1222e30bc4141b3ae686d80053c4ef5 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:09.369592 1172 raft_consensus.cc:385] T c1222e30bc4141b3ae686d80053c4ef5 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:09.369557 1166 consensus_queue.cc:260] T 217144957930495aa3e9dfcfde268e71 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:09.369678 1172 raft_consensus.cc:740] T c1222e30bc4141b3ae686d80053c4ef5 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:09.370160 1172 consensus_queue.cc:260] T c1222e30bc4141b3ae686d80053c4ef5 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:09.370745 1172 ts_tablet_manager.cc:1434] T c1222e30bc4141b3ae686d80053c4ef5 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 02:52:09.372189 1172 tablet_bootstrap.cc:492] T 643e20d9974447daa78885b3401f3b77 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:09.374029 519 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet adf6e736cb9c4920807ee58682e134ef (table default.Table_0a039bfb2e1f458f89ffc3ee93300b8f [id=64650da66d464074828f2dc955045f68]). Expected version 3 got 0
I20251029 02:52:09.374585 1172 tablet_bootstrap.cc:654] T 643e20d9974447daa78885b3401f3b77 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:09.375768 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 2f198c3259864f7ab3625cb7bfae20b0 with delete_type TABLET_DATA_DELETED (replaced by tablet 877e278deb4a4b01ac66b06be0890baf) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:09.376117 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 44c0488eba0543078ff4f0b3d69366f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:59 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:09.376490 745 tablet_service.cc:1552] Processing DeleteTablet for tablet afb19ba39399410bb90b6225d7b5b5b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:57 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:09.377197 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_110cf0fc2d22469d93e57359178b6c2a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:09.377803 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_110cf0fc2d22469d93e57359178b6c2a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:09.380611 1166 ts_tablet_manager.cc:1434] T 217144957930495aa3e9dfcfde268e71 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20251029 02:52:09.381765 525 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet c1222e30bc4141b3ae686d80053c4ef5 (table default.Table_7125184a813c44d997ab9d948a1156bd [id=31801defa6db4ca7a08fbf7f029b65c1]). Expected version 3 got 0
I20251029 02:52:09.382035 1166 tablet_bootstrap.cc:492] T c32c28b0e74a4af3a0e69bbf24ab1a1f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:09.384934 525 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_87da878e5839481a97fa117e34501d10" } new_table_name: "default.Table_9a7d7f4f9073437097aca9f7d6dc665e" modify_external_catalogs: true
I20251029 02:52:09.388568 2036 raft_consensus.cc:515] T 1275e192871440408a8345fef9b56260 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.388667 1174 ts_tablet_manager.cc:1929] T 71068739b4384badbf305313d902f0f5 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:09.388839 1174 log.cc:1199] T 71068739b4384badbf305313d902f0f5 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/71068739b4384badbf305313d902f0f5
I20251029 02:52:09.389155 1174 ts_tablet_manager.cc:1950] T 71068739b4384badbf305313d902f0f5 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:09.389191 2036 leader_election.cc:290] T 1275e192871440408a8345fef9b56260 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Requested vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:09.389756 2201 raft_consensus.cc:493] T 6f35412a66b74051a475dce5dafa7792 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:09.389894 2201 raft_consensus.cc:515] T 6f35412a66b74051a475dce5dafa7792 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.390486 2201 leader_election.cc:290] T 6f35412a66b74051a475dce5dafa7792 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:09.392374 1175 ts_tablet_manager.cc:1929] T 4bf06476c87a4549a73f5994314bb75f P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20251029 02:52:09.392493 1175 log.cc:1199] T 4bf06476c87a4549a73f5994314bb75f P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/4bf06476c87a4549a73f5994314bb75f
I20251029 02:52:09.392907 2205 raft_consensus.cc:515] T b6d4f7bf702f4834bc5f4ef2b50b92a7 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.393203 1175 ts_tablet_manager.cc:1950] T 4bf06476c87a4549a73f5994314bb75f P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:09.393481 2205 leader_election.cc:290] T b6d4f7bf702f4834bc5f4ef2b50b92a7 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:09.393675 1166 tablet_bootstrap.cc:654] T c32c28b0e74a4af3a0e69bbf24ab1a1f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:09.394500 1057 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b6d4f7bf702f4834bc5f4ef2b50b92a7" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:52:09.394678 1057 raft_consensus.cc:3060] T b6d4f7bf702f4834bc5f4ef2b50b92a7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:09.394943 766 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b6d4f7bf702f4834bc5f4ef2b50b92a7" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:09.395098 766 raft_consensus.cc:3060] T b6d4f7bf702f4834bc5f4ef2b50b92a7 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:09.396729 1176 ts_tablet_manager.cc:1929] T 5db519edc75d4bd0985b32322825ffae P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:09.396826 1176 log.cc:1199] T 5db519edc75d4bd0985b32322825ffae P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/5db519edc75d4bd0985b32322825ffae
I20251029 02:52:09.394529 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f35412a66b74051a475dce5dafa7792" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:09.397881 1176 ts_tablet_manager.cc:1950] T 5db519edc75d4bd0985b32322825ffae P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:09.397902 1056 raft_consensus.cc:2468] T 6f35412a66b74051a475dce5dafa7792 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:09.399621 1172 tablet_bootstrap.cc:492] T 643e20d9974447daa78885b3401f3b77 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:09.399751 1172 ts_tablet_manager.cc:1403] T 643e20d9974447daa78885b3401f3b77 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.028s user 0.005s sys 0.002s
W20251029 02:52:09.400779 990 consensus_peers.cc:597] T 7a1866afc7564217ad777e7ce53786c5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 -> Peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Couldn't send request to peer c756528642a543eeacd93cef3e335a7d. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:09.400784 1172 raft_consensus.cc:359] T 643e20d9974447daa78885b3401f3b77 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:09.400995 1172 raft_consensus.cc:385] T 643e20d9974447daa78885b3401f3b77 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:09.401067 1172 raft_consensus.cc:740] T 643e20d9974447daa78885b3401f3b77 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:09.401316 849 leader_election.cc:304] T 6f35412a66b74051a475dce5dafa7792 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:09.401321 1172 consensus_queue.cc:260] T 643e20d9974447daa78885b3401f3b77 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:09.401618 2214 consensus_queue.cc:1048] T 785733da86ff425a90ccfab2933fc89e P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:09.402040 2214 raft_consensus.cc:2804] T 6f35412a66b74051a475dce5dafa7792 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:09.402145 2214 raft_consensus.cc:493] T 6f35412a66b74051a475dce5dafa7792 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:09.402240 2214 raft_consensus.cc:3060] T 6f35412a66b74051a475dce5dafa7792 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:09.402338 1172 ts_tablet_manager.cc:1434] T 643e20d9974447daa78885b3401f3b77 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 02:52:09.402904 919 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1275e192871440408a8345fef9b56260" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
I20251029 02:52:09.403076 919 raft_consensus.cc:3060] T 1275e192871440408a8345fef9b56260 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:09.403754 1167 tablet_bootstrap.cc:492] T 1c53fc1fff1a4bb988beb46ff91fd3be P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:09.403895 1167 ts_tablet_manager.cc:1403] T 1c53fc1fff1a4bb988beb46ff91fd3be P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.070s user 0.011s sys 0.015s
I20251029 02:52:09.404641 1167 raft_consensus.cc:359] T 1c53fc1fff1a4bb988beb46ff91fd3be P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.404809 1167 raft_consensus.cc:385] T 1c53fc1fff1a4bb988beb46ff91fd3be P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:09.404892 1167 raft_consensus.cc:740] T 1c53fc1fff1a4bb988beb46ff91fd3be P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:09.405141 1167 consensus_queue.cc:260] T 1c53fc1fff1a4bb988beb46ff91fd3be P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.405504 1172 tablet_bootstrap.cc:492] T d041e87ceb11441983d0e38b7929c3a4 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:09.408658 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f35412a66b74051a475dce5dafa7792" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:09.408883 768 raft_consensus.cc:2468] T 6f35412a66b74051a475dce5dafa7792 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:09.410630 1167 ts_tablet_manager.cc:1434] T 1c53fc1fff1a4bb988beb46ff91fd3be P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20251029 02:52:09.412074 526 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 591a6c86fc984bac9fb9eb62e35dc6c1
I20251029 02:52:09.414903 879 tablet_service.cc:1552] Processing DeleteTablet for tablet 591a6c86fc984bac9fb9eb62e35dc6c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:09 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:09.415597 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 591a6c86fc984bac9fb9eb62e35dc6c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:09 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:09.419554 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet d2990e9f626c40618c118266bc595684 (table default.Table_da047830460443a29c5269c98fc859e9 [id=f733948e92d8470da9d8050bd26a06b1]) successfully deleted
I20251029 02:52:09.419704 2113 raft_consensus.cc:493] T af4860c6b72f4d988b206b5ef2f22906 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:09.419853 2113 raft_consensus.cc:515] T af4860c6b72f4d988b206b5ef2f22906 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.420441 2113 leader_election.cc:290] T af4860c6b72f4d988b206b5ef2f22906 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:09.421195 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af4860c6b72f4d988b206b5ef2f22906" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:09.421634 711 leader_election.cc:343] T af4860c6b72f4d988b206b5ef2f22906 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:09.422708 1166 tablet_bootstrap.cc:492] T c32c28b0e74a4af3a0e69bbf24ab1a1f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:09.422865 1166 ts_tablet_manager.cc:1403] T c32c28b0e74a4af3a0e69bbf24ab1a1f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.041s user 0.003s sys 0.000s
I20251029 02:52:09.423555 1166 raft_consensus.cc:359] T c32c28b0e74a4af3a0e69bbf24ab1a1f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:09.423698 1166 raft_consensus.cc:385] T c32c28b0e74a4af3a0e69bbf24ab1a1f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:09.423756 1166 raft_consensus.cc:740] T c32c28b0e74a4af3a0e69bbf24ab1a1f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:09.423970 1166 consensus_queue.cc:260] T c32c28b0e74a4af3a0e69bbf24ab1a1f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:09.424482 1166 ts_tablet_manager.cc:1434] T c32c28b0e74a4af3a0e69bbf24ab1a1f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251029 02:52:09.424727 1166 tablet_bootstrap.cc:492] T 97b2ecae572a4b7ea1b9595074556708 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
W20251029 02:52:09.426270 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet dabb8d760aae4aa09eb83cdb586c9788 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dabb8d760aae4aa09eb83cdb586c9788 already in progress: creating tablet
I20251029 02:52:09.426318 1167 tablet_bootstrap.cc:492] T 44b518c3a0964a6fa863b2784d80c8a4 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:09.427819 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1bdb67125493430398ff5c80b73de2ac" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:09.428061 773 raft_consensus.cc:2393] T 1bdb67125493430398ff5c80b73de2ac P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in current term 1: Already voted for candidate c756528642a543eeacd93cef3e335a7d in this term.
I20251029 02:52:09.428130 1167 tablet_bootstrap.cc:654] T 44b518c3a0964a6fa863b2784d80c8a4 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:09.430879 1172 tablet_bootstrap.cc:654] T d041e87ceb11441983d0e38b7929c3a4 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:09.433879 522 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_2ce7a304a9184f1ebba440670ede298c" } new_table_name: "default.Table_6c1f042a9840467993a38a8e55a23cbb" modify_external_catalogs: true
I20251029 02:52:09.431914 1166 tablet_bootstrap.cc:654] T 97b2ecae572a4b7ea1b9595074556708 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:09.436053 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_443fde25ba1b481f9146d61ad842bc9b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:09.436892 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_443fde25ba1b481f9146d61ad842bc9b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:09.438719 524 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_f3da247a6675435a9c881f23dd8ff376" } new_table_name: "default.Table_aff419c3c59d4480b8c26d6bbb53f05b" modify_external_catalogs: true
W20251029 02:52:09.442705 847 consensus_peers.cc:597] T 785733da86ff425a90ccfab2933fc89e P c756528642a543eeacd93cef3e335a7d -> Peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Couldn't send request to peer b495a49349324f7d81908f2da700f0d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 785733da86ff425a90ccfab2933fc89e. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:09.443112 2113 raft_consensus.cc:493] T 8048506ced4540a5aee9f79bfc4599bf P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:09.443256 2113 raft_consensus.cc:515] T 8048506ced4540a5aee9f79bfc4599bf P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.443823 2113 leader_election.cc:290] T 8048506ced4540a5aee9f79bfc4599bf P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:09.444599 2036 raft_consensus.cc:493] T 061f3d38c4604fa4b57e74ff6f6fc7da P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:09.444737 2036 raft_consensus.cc:515] T 061f3d38c4604fa4b57e74ff6f6fc7da P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.445266 2036 leader_election.cc:290] T 061f3d38c4604fa4b57e74ff6f6fc7da P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:09.445897 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "13ee62fc2339410b9db70dfd3d3c6fa6" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:09.446064 773 raft_consensus.cc:2468] T 13ee62fc2339410b9db70dfd3d3c6fa6 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 0.
I20251029 02:52:09.446502 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "95f342d1cfb84c4eb31bd3d15ea3bdc2" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:09.446787 768 raft_consensus.cc:2468] T 95f342d1cfb84c4eb31bd3d15ea3bdc2 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 0.
I20251029 02:52:09.447923 914 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "061f3d38c4604fa4b57e74ff6f6fc7da" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:09.449386 914 raft_consensus.cc:3060] T 0f80b840ed954230bfbfe7e40127cd2f P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
W20251029 02:52:09.449487 990 leader_election.cc:343] T 061f3d38c4604fa4b57e74ff6f6fc7da P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:09.450057 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8048506ced4540a5aee9f79bfc4599bf" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:09.450495 711 leader_election.cc:343] T 8048506ced4540a5aee9f79bfc4599bf P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:09.453593 1167 tablet_bootstrap.cc:492] T 44b518c3a0964a6fa863b2784d80c8a4 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:09.456598 1167 ts_tablet_manager.cc:1403] T 44b518c3a0964a6fa863b2784d80c8a4 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.028s user 0.005s sys 0.000s
I20251029 02:52:09.459519 1172 tablet_bootstrap.cc:492] T d041e87ceb11441983d0e38b7929c3a4 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:09.459676 1172 ts_tablet_manager.cc:1403] T d041e87ceb11441983d0e38b7929c3a4 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.057s user 0.005s sys 0.001s
I20251029 02:52:09.460295 1172 raft_consensus.cc:359] T d041e87ceb11441983d0e38b7929c3a4 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:09.460451 1172 raft_consensus.cc:385] T d041e87ceb11441983d0e38b7929c3a4 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:09.460521 1172 raft_consensus.cc:740] T d041e87ceb11441983d0e38b7929c3a4 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:09.460737 1172 consensus_queue.cc:260] T d041e87ceb11441983d0e38b7929c3a4 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:09.461197 1172 ts_tablet_manager.cc:1434] T d041e87ceb11441983d0e38b7929c3a4 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251029 02:52:09.461475 1172 tablet_bootstrap.cc:492] T 59eee86f249e4d6a9d09d30276dcd659 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:09.463225 1167 raft_consensus.cc:359] T 44b518c3a0964a6fa863b2784d80c8a4 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.463433 1167 raft_consensus.cc:385] T 44b518c3a0964a6fa863b2784d80c8a4 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:09.463505 1167 raft_consensus.cc:740] T 44b518c3a0964a6fa863b2784d80c8a4 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:09.465620 1057 raft_consensus.cc:2468] T b6d4f7bf702f4834bc5f4ef2b50b92a7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:09.465917 1176 tablet_replica.cc:333] T ccdca31db2834de4b6ed0f2d67a10535 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:09.466143 1176 raft_consensus.cc:2243] T ccdca31db2834de4b6ed0f2d67a10535 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:09.466125 849 leader_election.cc:304] T b6d4f7bf702f4834bc5f4ef2b50b92a7 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:09.466425 2201 raft_consensus.cc:2804] T b6d4f7bf702f4834bc5f4ef2b50b92a7 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:09.466432 1176 raft_consensus.cc:2272] T ccdca31db2834de4b6ed0f2d67a10535 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:09.466610 2201 raft_consensus.cc:697] T b6d4f7bf702f4834bc5f4ef2b50b92a7 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:52:09.466890 2201 consensus_queue.cc:237] T b6d4f7bf702f4834bc5f4ef2b50b92a7 P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.467684 1176 ts_tablet_manager.cc:1916] T ccdca31db2834de4b6ed0f2d67a10535 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:09.471418 766 raft_consensus.cc:2468] T b6d4f7bf702f4834bc5f4ef2b50b92a7 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:09.472002 2214 raft_consensus.cc:515] T 6f35412a66b74051a475dce5dafa7792 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.472292 1167 consensus_queue.cc:260] T 44b518c3a0964a6fa863b2784d80c8a4 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.472851 2214 leader_election.cc:290] T 6f35412a66b74051a475dce5dafa7792 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:09.473481 1167 ts_tablet_manager.cc:1434] T 44b518c3a0964a6fa863b2784d80c8a4 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.017s user 0.002s sys 0.001s
I20251029 02:52:09.476476 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 802) took 1978 ms. Trace:
I20251029 02:52:09.476613 1174 rpcz_store.cc:276] 1029 02:52:07.498397 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:07.503767 (+ 5370us) service_pool.cc:225] Handling call
1029 02:52:09.262689 (+1758922us) ts_tablet_manager.cc:1151] Deleting tablet 71068739b4384badbf305313d902f0f5
1029 02:52:09.262699 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:09.295874 (+ 33175us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:09.388536 (+ 92662us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:09.476391 (+ 87855us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1746499}
I20251029 02:52:09.477146 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 810) took 1866 ms. Trace:
I20251029 02:52:09.477248 1174 rpcz_store.cc:276] 1029 02:52:07.610246 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:07.610314 (+ 68us) service_pool.cc:225] Handling call
1029 02:52:09.476954 (+1866640us) ts_tablet_manager.cc:1151] Deleting tablet faac614af1814a2f91d66c9cf8ddc919
1029 02:52:09.476966 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:09.477075 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1866452}
I20251029 02:52:09.477596 1174 tablet_replica.cc:333] T 4bf06476c87a4549a73f5994314bb75f P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:09.477824 1174 raft_consensus.cc:2243] T 4bf06476c87a4549a73f5994314bb75f P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:09.478011 1167 tablet_bootstrap.cc:492] T dabb8d760aae4aa09eb83cdb586c9788 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:09.478250 1174 raft_consensus.cc:2272] T 4bf06476c87a4549a73f5994314bb75f P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:09.478606 2113 raft_consensus.cc:493] T 44c0488eba0543078ff4f0b3d69366f2 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:09.478765 2113 raft_consensus.cc:515] T 44c0488eba0543078ff4f0b3d69366f2 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:09.479491 2113 leader_election.cc:290] T 44c0488eba0543078ff4f0b3d69366f2 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:09.480568 1167 tablet_bootstrap.cc:654] T dabb8d760aae4aa09eb83cdb586c9788 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:09.481882 1057 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f35412a66b74051a475dce5dafa7792" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:52:09.482100 1057 raft_consensus.cc:3060] T 6f35412a66b74051a475dce5dafa7792 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:09.482276 1172 tablet_bootstrap.cc:654] T 59eee86f249e4d6a9d09d30276dcd659 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:09.482620 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "44c0488eba0543078ff4f0b3d69366f2" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:09.483556 766 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f35412a66b74051a475dce5dafa7792" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:09.483733 766 raft_consensus.cc:3060] T 6f35412a66b74051a475dce5dafa7792 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
W20251029 02:52:09.487380 711 leader_election.cc:343] T 44c0488eba0543078ff4f0b3d69366f2 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:09.489045 2113 raft_consensus.cc:493] T 2f198c3259864f7ab3625cb7bfae20b0 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:09.489180 2113 raft_consensus.cc:515] T 2f198c3259864f7ab3625cb7bfae20b0 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:09.489800 2113 leader_election.cc:290] T 2f198c3259864f7ab3625cb7bfae20b0 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:09.490635 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2f198c3259864f7ab3625cb7bfae20b0" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:09.492434 711 leader_election.cc:343] T 2f198c3259864f7ab3625cb7bfae20b0 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:09.493139 1174 ts_tablet_manager.cc:1916] T 4bf06476c87a4549a73f5994314bb75f P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:09.494958 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d830d30b26b4ba799785fec8a6f0769" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:09.495107 1058 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af4860c6b72f4d988b206b5ef2f22906" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:09.495182 1056 raft_consensus.cc:2468] T 6d830d30b26b4ba799785fec8a6f0769 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b495a49349324f7d81908f2da700f0d9 in term 0.
I20251029 02:52:09.495216 1061 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8048506ced4540a5aee9f79bfc4599bf" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:09.495409 1061 raft_consensus.cc:2468] T 8048506ced4540a5aee9f79bfc4599bf P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b495a49349324f7d81908f2da700f0d9 in term 0.
I20251029 02:52:09.495429 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "28f7ceb91a34450cad4b5b45f5285b41" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:09.495540 1060 raft_consensus.cc:2468] T 28f7ceb91a34450cad4b5b45f5285b41 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b495a49349324f7d81908f2da700f0d9 in term 0.
I20251029 02:52:09.495223 1058 raft_consensus.cc:2468] T af4860c6b72f4d988b206b5ef2f22906 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b495a49349324f7d81908f2da700f0d9 in term 0.
I20251029 02:52:09.496062 1058 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "44c0488eba0543078ff4f0b3d69366f2" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:09.496606 1058 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2f198c3259864f7ab3625cb7bfae20b0" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:09.497872 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "44b518c3a0964a6fa863b2784d80c8a4" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:09.498108 768 raft_consensus.cc:2468] T 44b518c3a0964a6fa863b2784d80c8a4 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 0.
I20251029 02:52:09.498587 2113 raft_consensus.cc:493] T adf6e736cb9c4920807ee58682e134ef P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:09.498745 2113 raft_consensus.cc:515] T adf6e736cb9c4920807ee58682e134ef P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:09.499400 2113 leader_election.cc:290] T adf6e736cb9c4920807ee58682e134ef P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:09.501060 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "adf6e736cb9c4920807ee58682e134ef" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:09.501961 1167 tablet_bootstrap.cc:492] T dabb8d760aae4aa09eb83cdb586c9788 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
W20251029 02:52:09.502024 711 leader_election.cc:343] T adf6e736cb9c4920807ee58682e134ef P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:09.502099 1167 ts_tablet_manager.cc:1403] T dabb8d760aae4aa09eb83cdb586c9788 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.028s user 0.005s sys 0.001s
W20251029 02:52:09.502704 512 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.60:41233, user_credentials={real_user=slave}} blocked reactor thread for 46776.3us
I20251029 02:52:09.502925 1167 raft_consensus.cc:359] T dabb8d760aae4aa09eb83cdb586c9788 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.503085 1167 raft_consensus.cc:385] T dabb8d760aae4aa09eb83cdb586c9788 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:09.503136 1167 raft_consensus.cc:740] T dabb8d760aae4aa09eb83cdb586c9788 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:09.503125 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 71068739b4384badbf305313d902f0f5 (table default.Table_4e899dcc04fe4cb1b96ced1fa1b03585 [id=992716bbc1694bd6b85e4c8a3ab51e40]) successfully deleted
W20251029 02:52:09.503377 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet faac614af1814a2f91d66c9cf8ddc919 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet faac614af1814a2f91d66c9cf8ddc919 already in progress: creating tablet
I20251029 02:52:09.503338 1167 consensus_queue.cc:260] T dabb8d760aae4aa09eb83cdb586c9788 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.503886 1167 ts_tablet_manager.cc:1434] T dabb8d760aae4aa09eb83cdb586c9788 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251029 02:52:09.504148 1167 tablet_bootstrap.cc:492] T d61beda662d844cdaf67fd9a2e382ea2 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:09.504367 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 800) took 1872 ms. Trace:
W20251029 02:52:09.504406 847 consensus_peers.cc:597] T 777992acb1bd4765886b802940db74bb P c756528642a543eeacd93cef3e335a7d -> Peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Couldn't send request to peer b495a49349324f7d81908f2da700f0d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 777992acb1bd4765886b802940db74bb. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:09.504451 1175 rpcz_store.cc:276] 1029 02:52:07.632277 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:07.639110 (+ 6833us) service_pool.cc:225] Handling call
1029 02:52:09.363989 (+1724879us) ts_tablet_manager.cc:1151] Deleting tablet 4bf06476c87a4549a73f5994314bb75f
1029 02:52:09.363997 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:09.388526 (+ 24529us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:09.392253 (+ 3727us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:09.503893 (+111640us) spinlock_profiling.cc:243] Waited 48.3 ms on lock 0x60800130bcd0. stack: 00007f1174e9f9f6 00007f1174e9f7b0 00007f1173e7b3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f1180d9d608 00007f1180da0324 00007f1180d9fffd 0000000000a8867c 00007f1174dbf43e 00007f1180d77ed5 00007f1180d995b1 00007f1180d994f0 00007f1180d98d72 000000000097cc36
1029 02:52:09.504347 (+ 454us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":111060224,"tablet-delete.queue_time_us":1724665}
I20251029 02:52:09.504839 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 802) took 1798 ms. Trace:
I20251029 02:52:09.504904 1175 rpcz_store.cc:276] 1029 02:52:07.706240 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:07.706302 (+ 62us) service_pool.cc:225] Handling call
1029 02:52:09.504721 (+1798419us) ts_tablet_manager.cc:1151] Deleting tablet 35d9b8dfd61e4d8db441427a49fb9dca
1029 02:52:09.504730 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:09.504821 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1798222}
I20251029 02:52:09.505177 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 803) took 1798 ms. Trace:
I20251029 02:52:09.505236 1175 rpcz_store.cc:276] 1029 02:52:07.706339 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:07.706386 (+ 47us) service_pool.cc:225] Handling call
1029 02:52:09.505111 (+1798725us) ts_tablet_manager.cc:1151] Deleting tablet ba699862e6f646c6beaa39f080d0faba
1029 02:52:09.505120 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:09.505164 (+ 44us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":65,"tablet-delete.queue_time_us":1798613}
I20251029 02:52:09.505470 1175 tablet_replica.cc:333] T 9bb2166a590d4210b02dd779ef6654ad P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:09.505600 1167 tablet_bootstrap.cc:654] T d61beda662d844cdaf67fd9a2e382ea2 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:09.505651 1175 raft_consensus.cc:2243] T 9bb2166a590d4210b02dd779ef6654ad P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:09.505792 1175 raft_consensus.cc:2272] T 9bb2166a590d4210b02dd779ef6654ad P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:09.506466 1175 ts_tablet_manager.cc:1916] T 9bb2166a590d4210b02dd779ef6654ad P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:09.507324 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 4bf06476c87a4549a73f5994314bb75f (table default.Table_b708124baf534c96b3f4b8a7a4bb11cc [id=046010a03d774e91b0ad10df97848c56]) successfully deleted
W20251029 02:52:09.507649 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 35d9b8dfd61e4d8db441427a49fb9dca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 35d9b8dfd61e4d8db441427a49fb9dca already in progress: creating tablet
W20251029 02:52:09.507897 511 catalog_manager.cc:4952] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet ba699862e6f646c6beaa39f080d0faba because the tablet was not found. No further retry: Not found: Tablet not found: ba699862e6f646c6beaa39f080d0faba
W20251029 02:52:09.508394 520 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 786d9148fb1d417a8c2c8b8c53665899 consensus state has no leader
I20251029 02:52:09.513465 2113 raft_consensus.cc:493] T 731c0acfbe204e8199ad246e38ae2ccd P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:09.513607 2113 raft_consensus.cc:515] T 731c0acfbe204e8199ad246e38ae2ccd P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:09.514236 2113 leader_election.cc:290] T 731c0acfbe204e8199ad246e38ae2ccd P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:09.515205 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "731c0acfbe204e8199ad246e38ae2ccd" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:09.515851 711 leader_election.cc:343] T 731c0acfbe204e8199ad246e38ae2ccd P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
W20251029 02:52:09.521745 525 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6789bc16664e4b37b86702449ab27855 consensus state has no leader
W20251029 02:52:09.522403 522 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6c4f28b6f4a94d35bb036843aa3bc4bc has no consensus state
I20251029 02:52:09.523358 523 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_dd632ee443af4a59905aee8a89d0a4bb" } modify_external_catalogs: true
I20251029 02:52:09.523592 523 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_dd632ee443af4a59905aee8a89d0a4bb" } modify_external_catalogs: true
I20251029 02:52:09.524369 520 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_839c3ffd7768463da17f35d024d016a0" } modify_external_catalogs: true
I20251029 02:52:09.524612 520 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_839c3ffd7768463da17f35d024d016a0" } modify_external_catalogs: true
I20251029 02:52:09.525355 519 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_dd4934aabcd442039db111954d819f6d" } modify_external_catalogs: true
I20251029 02:52:09.525553 519 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_dd4934aabcd442039db111954d819f6d" } modify_external_catalogs: true
I20251029 02:52:09.535900 2036 raft_consensus.cc:493] T 59d03a190b784ab18a3e63a572675891 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:09.536085 2036 raft_consensus.cc:515] T 59d03a190b784ab18a3e63a572675891 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.536752 2036 leader_election.cc:290] T 59d03a190b784ab18a3e63a572675891 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:09.537184 517 master_service.cc:954] ReplaceTablet: received request to replace tablet c9e923724ca9402682f13b252dbc37aa from {username='slave'} at 127.0.0.1:35314
I20251029 02:52:09.537709 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59d03a190b784ab18a3e63a572675891" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:09.538422 990 leader_election.cc:343] T 59d03a190b784ab18a3e63a572675891 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:09.539145 1167 tablet_bootstrap.cc:492] T d61beda662d844cdaf67fd9a2e382ea2 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:09.539211 2113 raft_consensus.cc:493] T d7abaf84ba5d445aa17433b3be891548 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:09.539281 1167 ts_tablet_manager.cc:1403] T d61beda662d844cdaf67fd9a2e382ea2 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.035s user 0.002s sys 0.001s
I20251029 02:52:09.539343 2113 raft_consensus.cc:515] T d7abaf84ba5d445aa17433b3be891548 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:09.539470 2201 consensus_queue.cc:1048] T f529e17e97074dca91ec2f48954cfd32 P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:09.539952 2113 leader_election.cc:290] T d7abaf84ba5d445aa17433b3be891548 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:09.540016 1167 raft_consensus.cc:359] T d61beda662d844cdaf67fd9a2e382ea2 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.540151 1167 raft_consensus.cc:385] T d61beda662d844cdaf67fd9a2e382ea2 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:09.540205 1167 raft_consensus.cc:740] T d61beda662d844cdaf67fd9a2e382ea2 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:09.540395 1167 consensus_queue.cc:260] T d61beda662d844cdaf67fd9a2e382ea2 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.540609 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d7abaf84ba5d445aa17433b3be891548" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:09.540880 1167 ts_tablet_manager.cc:1434] T d61beda662d844cdaf67fd9a2e382ea2 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
W20251029 02:52:09.541049 711 leader_election.cc:343] T d7abaf84ba5d445aa17433b3be891548 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:09.544838 1167 tablet_bootstrap.cc:492] T 9269622ca92645d5aa8e73341201ccbb P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:09.552057 1167 tablet_bootstrap.cc:654] T 9269622ca92645d5aa8e73341201ccbb P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:09.554477 2201 consensus_queue.cc:1048] T 777992acb1bd4765886b802940db74bb P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:09.554575 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_ff774195536549b3b3a651c34438f014"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:09.555207 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ff774195536549b3b3a651c34438f014 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:09.556494 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_fa832508491b47ddbcd9e89058c75ad5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:09.557042 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fa832508491b47ddbcd9e89058c75ad5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:09.559492 1167 tablet_bootstrap.cc:492] T 9269622ca92645d5aa8e73341201ccbb P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:09.559625 1167 ts_tablet_manager.cc:1403] T 9269622ca92645d5aa8e73341201ccbb P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.004s
I20251029 02:52:09.560359 1167 raft_consensus.cc:359] T 9269622ca92645d5aa8e73341201ccbb P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.560518 1167 raft_consensus.cc:385] T 9269622ca92645d5aa8e73341201ccbb P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:09.560581 1167 raft_consensus.cc:740] T 9269622ca92645d5aa8e73341201ccbb P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:09.560774 1167 consensus_queue.cc:260] T 9269622ca92645d5aa8e73341201ccbb P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.566073 2036 raft_consensus.cc:493] T 0d84dc8dc465496f9adec798b2e03e13 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:09.566224 2036 raft_consensus.cc:515] T 0d84dc8dc465496f9adec798b2e03e13 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.566886 2036 leader_election.cc:290] T 0d84dc8dc465496f9adec798b2e03e13 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:09.567529 1167 ts_tablet_manager.cc:1434] T 9269622ca92645d5aa8e73341201ccbb P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.008s user 0.001s sys 0.002s
I20251029 02:52:09.567584 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d84dc8dc465496f9adec798b2e03e13" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:09.568241 919 raft_consensus.cc:2468] T 1275e192871440408a8345fef9b56260 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 1.
I20251029 02:52:09.568604 1057 raft_consensus.cc:2468] T 6f35412a66b74051a475dce5dafa7792 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:09.571674 1175 ts_tablet_manager.cc:1929] T 9bb2166a590d4210b02dd779ef6654ad P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:09.571782 1175 log.cc:1199] T 9bb2166a590d4210b02dd779ef6654ad P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/9bb2166a590d4210b02dd779ef6654ad
I20251029 02:52:09.572109 1175 ts_tablet_manager.cc:1950] T 9bb2166a590d4210b02dd779ef6654ad P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:09.572966 1176 ts_tablet_manager.cc:1929] T ccdca31db2834de4b6ed0f2d67a10535 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:09.573077 1176 log.cc:1199] T ccdca31db2834de4b6ed0f2d67a10535 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/ccdca31db2834de4b6ed0f2d67a10535
I20251029 02:52:09.573431 1176 ts_tablet_manager.cc:1950] T ccdca31db2834de4b6ed0f2d67a10535 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:09.574493 766 raft_consensus.cc:2468] T 6f35412a66b74051a475dce5dafa7792 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:09.575042 847 leader_election.cc:304] T 6f35412a66b74051a475dce5dafa7792 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters:
W20251029 02:52:09.575515 990 leader_election.cc:343] T 0d84dc8dc465496f9adec798b2e03e13 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:09.575848 1167 tablet_bootstrap.cc:492] T 0d84dc8dc465496f9adec798b2e03e13 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:09.576740 990 leader_election.cc:304] T 1275e192871440408a8345fef9b56260 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:09.577883 2103 raft_consensus.cc:2804] T 1275e192871440408a8345fef9b56260 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:09.578012 2103 raft_consensus.cc:697] T 1275e192871440408a8345fef9b56260 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Becoming Leader. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Running, Role: LEADER
I20251029 02:52:09.578240 2103 consensus_queue.cc:237] T 1275e192871440408a8345fef9b56260 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
W20251029 02:52:09.579223 524 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c92a3c77be8943fb929d321559e3a46c consensus state has no leader
I20251029 02:52:09.579171 2188 consensus_queue.cc:1048] T 0f80b840ed954230bfbfe7e40127cd2f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:09.580684 1167 tablet_bootstrap.cc:654] T 0d84dc8dc465496f9adec798b2e03e13 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:09.581032 2201 raft_consensus.cc:2804] T 6f35412a66b74051a475dce5dafa7792 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:09.581179 2201 raft_consensus.cc:697] T 6f35412a66b74051a475dce5dafa7792 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:52:09.581415 2201 consensus_queue.cc:237] T 6f35412a66b74051a475dce5dafa7792 P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.583932 919 raft_consensus.cc:3060] T 02e4588b03144ea2850e13a1f8f29f06 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:09.585263 524 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_331f901095f44ebfa644aa9cf8dde711" } modify_external_catalogs: true
I20251029 02:52:09.585469 524 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_331f901095f44ebfa644aa9cf8dde711" } modify_external_catalogs: true
W20251029 02:52:09.587666 847 consensus_peers.cc:597] T f529e17e97074dca91ec2f48954cfd32 P c756528642a543eeacd93cef3e335a7d -> Peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Couldn't send request to peer b495a49349324f7d81908f2da700f0d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f529e17e97074dca91ec2f48954cfd32. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:09.590497 2036 consensus_queue.cc:1048] T 02e4588b03144ea2850e13a1f8f29f06 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:09.590662 519 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 1c53fc1fff1a4bb988beb46ff91fd3be
I20251029 02:52:09.592059 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 1c53fc1fff1a4bb988beb46ff91fd3be with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:09 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:09.595202 879 tablet_service.cc:1552] Processing DeleteTablet for tablet 1c53fc1fff1a4bb988beb46ff91fd3be with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:09 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:09.600451 1167 tablet_bootstrap.cc:492] T 0d84dc8dc465496f9adec798b2e03e13 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:09.600584 1167 ts_tablet_manager.cc:1403] T 0d84dc8dc465496f9adec798b2e03e13 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.001s
I20251029 02:52:09.601281 1167 raft_consensus.cc:359] T 0d84dc8dc465496f9adec798b2e03e13 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.601445 1167 raft_consensus.cc:385] T 0d84dc8dc465496f9adec798b2e03e13 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:09.601487 1167 raft_consensus.cc:740] T 0d84dc8dc465496f9adec798b2e03e13 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:09.602375 1167 consensus_queue.cc:260] T 0d84dc8dc465496f9adec798b2e03e13 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.602727 2113 raft_consensus.cc:493] T 97b2ecae572a4b7ea1b9595074556708 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:09.602896 2113 raft_consensus.cc:515] T 97b2ecae572a4b7ea1b9595074556708 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.603653 2113 leader_election.cc:290] T 97b2ecae572a4b7ea1b9595074556708 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:09.603749 1167 ts_tablet_manager.cc:1434] T 0d84dc8dc465496f9adec798b2e03e13 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251029 02:52:09.604630 919 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97b2ecae572a4b7ea1b9595074556708" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:09.605154 1167 tablet_bootstrap.cc:492] T 28f7ceb91a34450cad4b5b45f5285b41 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
W20251029 02:52:09.605201 711 leader_election.cc:343] T 97b2ecae572a4b7ea1b9595074556708 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:09.607182 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet caef1c93e33a4390bcada59a41f43465 (table default.Table_b9f8911906c3473da4de53896081cb2c [id=8cccaacc256b40deabe999de1d75032e]) successfully deleted
I20251029 02:52:09.608021 766 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d830d30b26b4ba799785fec8a6f0769" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:09.608234 766 raft_consensus.cc:2468] T 6d830d30b26b4ba799785fec8a6f0769 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 0.
I20251029 02:52:09.608896 1167 tablet_bootstrap.cc:654] T 28f7ceb91a34450cad4b5b45f5285b41 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:09.609517 766 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1275e192871440408a8345fef9b56260" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:09.609711 766 raft_consensus.cc:2468] T 1275e192871440408a8345fef9b56260 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 0.
I20251029 02:52:09.610487 766 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1275e192871440408a8345fef9b56260" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:09.610669 766 raft_consensus.cc:3060] T 1275e192871440408a8345fef9b56260 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:09.616106 766 raft_consensus.cc:2468] T 1275e192871440408a8345fef9b56260 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 1.
I20251029 02:52:09.619881 2103 raft_consensus.cc:493] T 9bb2166a590d4210b02dd779ef6654ad P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:09.620078 2103 raft_consensus.cc:515] T 9bb2166a590d4210b02dd779ef6654ad P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.620700 2103 leader_election.cc:290] T 9bb2166a590d4210b02dd779ef6654ad P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:52:09.621358 914 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9bb2166a590d4210b02dd779ef6654ad" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:09.621579 914 raft_consensus.cc:2468] T 9bb2166a590d4210b02dd779ef6654ad P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 0.
I20251029 02:52:09.621868 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 804) took 1915 ms. Trace:
I20251029 02:52:09.621990 990 leader_election.cc:304] T 9bb2166a590d4210b02dd779ef6654ad P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:09.621989 1175 rpcz_store.cc:276] 1029 02:52:07.706566 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:07.706660 (+ 94us) service_pool.cc:225] Handling call
1029 02:52:09.505424 (+1798764us) ts_tablet_manager.cc:1151] Deleting tablet 9bb2166a590d4210b02dd779ef6654ad
1029 02:52:09.505433 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:09.568224 (+ 62791us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:09.571581 (+ 3357us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:09.621841 (+ 50260us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1798596}
I20251029 02:52:09.622236 2103 raft_consensus.cc:2804] T 9bb2166a590d4210b02dd779ef6654ad P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:09.622290 2103 raft_consensus.cc:493] T 9bb2166a590d4210b02dd779ef6654ad P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:09.622340 2103 raft_consensus.cc:3060] T 9bb2166a590d4210b02dd779ef6654ad P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:09.622483 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 801) took 1916 ms. Trace:
I20251029 02:52:09.622576 1175 rpcz_store.cc:276] 1029 02:52:07.706112 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:07.718431 (+ 12319us) service_pool.cc:225] Handling call
1029 02:52:09.622333 (+1903902us) ts_tablet_manager.cc:1151] Deleting tablet 050186302c5d4c7da393f28f4b9e4262
1029 02:52:09.622347 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:09.622467 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1903558}
I20251029 02:52:09.622900 1175 tablet_replica.cc:333] T 9e6eb099648549dbbf9969e59865f51f P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:09.623131 1175 raft_consensus.cc:2243] T 9e6eb099648549dbbf9969e59865f51f P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:09.623276 1175 raft_consensus.cc:2272] T 9e6eb099648549dbbf9969e59865f51f P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:09.623885 1175 ts_tablet_manager.cc:1916] T 9e6eb099648549dbbf9969e59865f51f P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:09.627377 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 9bb2166a590d4210b02dd779ef6654ad (table default.Table_50252b3b3cc4403da5e259f9f91be915 [id=ff64a323de354792997beb99bae4a52c]) successfully deleted
W20251029 02:52:09.627710 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 050186302c5d4c7da393f28f4b9e4262 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 050186302c5d4c7da393f28f4b9e4262 already in progress: creating tablet
I20251029 02:52:09.628284 1174 ts_tablet_manager.cc:1929] T 4bf06476c87a4549a73f5994314bb75f P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20251029 02:52:09.628417 1174 log.cc:1199] T 4bf06476c87a4549a73f5994314bb75f P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/4bf06476c87a4549a73f5994314bb75f
I20251029 02:52:09.628970 1174 ts_tablet_manager.cc:1950] T 4bf06476c87a4549a73f5994314bb75f P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:09.629914 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 811) took 1997 ms. Trace:
I20251029 02:52:09.629933 2201 raft_consensus.cc:493] T ba699862e6f646c6beaa39f080d0faba P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:09.630066 2201 raft_consensus.cc:515] T ba699862e6f646c6beaa39f080d0faba P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.630007 1174 rpcz_store.cc:276] 1029 02:52:07.632529 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:07.632591 (+ 62us) service_pool.cc:225] Handling call
1029 02:52:09.477502 (+1844911us) ts_tablet_manager.cc:1151] Deleting tablet 4bf06476c87a4549a73f5994314bb75f
1029 02:52:09.477511 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:09.571395 (+ 93884us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:09.628140 (+ 56745us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:09.629894 (+ 1754us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1844706}
I20251029 02:52:09.630440 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 812) took 1922 ms. Trace:
I20251029 02:52:09.630837 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 4bf06476c87a4549a73f5994314bb75f (table default.Table_b708124baf534c96b3f4b8a7a4bb11cc [id=046010a03d774e91b0ad10df97848c56]) successfully deleted
I20251029 02:52:09.630847 1174 rpcz_store.cc:276] 1029 02:52:07.707899 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:07.708276 (+ 377us) service_pool.cc:225] Handling call
1029 02:52:09.630324 (+1922048us) ts_tablet_manager.cc:1151] Deleting tablet 050186302c5d4c7da393f28f4b9e4262
1029 02:52:09.630334 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:09.630425 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1921876}
I20251029 02:52:09.631304 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 813) took 1923 ms. Trace:
I20251029 02:52:09.631414 1174 rpcz_store.cc:276] 1029 02:52:07.707983 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:07.708483 (+ 500us) service_pool.cc:225] Handling call
1029 02:52:09.631192 (+1922709us) ts_tablet_manager.cc:1151] Deleting tablet 35d9b8dfd61e4d8db441427a49fb9dca
1029 02:52:09.631202 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:09.631285 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1922575}
W20251029 02:52:09.631520 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 050186302c5d4c7da393f28f4b9e4262 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 050186302c5d4c7da393f28f4b9e4262 already in progress: creating tablet
W20251029 02:52:09.631857 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 35d9b8dfd61e4d8db441427a49fb9dca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 35d9b8dfd61e4d8db441427a49fb9dca already in progress: creating tablet
W20251029 02:52:09.632334 847 leader_election.cc:343] T ba699862e6f646c6beaa39f080d0faba P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: ba699862e6f646c6beaa39f080d0faba
I20251029 02:52:09.632321 1174 tablet_replica.cc:333] T 86fc22bb84794149b598b97df41eb881 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:09.632718 1174 raft_consensus.cc:2243] T 86fc22bb84794149b598b97df41eb881 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:09.632951 1174 raft_consensus.cc:2272] T 86fc22bb84794149b598b97df41eb881 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:09.632929 2201 leader_election.cc:290] T ba699862e6f646c6beaa39f080d0faba P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:09.635350 2103 raft_consensus.cc:515] T 9bb2166a590d4210b02dd779ef6654ad P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.635924 1174 ts_tablet_manager.cc:1916] T 86fc22bb84794149b598b97df41eb881 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:09.636327 2103 leader_election.cc:290] T 9bb2166a590d4210b02dd779ef6654ad P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Requested vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:52:09.636400 914 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9bb2166a590d4210b02dd779ef6654ad" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
I20251029 02:52:09.636572 914 raft_consensus.cc:3060] T 9bb2166a590d4210b02dd779ef6654ad P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:09.639074 914 raft_consensus.cc:2468] T 9bb2166a590d4210b02dd779ef6654ad P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 1.
I20251029 02:52:09.639439 990 leader_election.cc:304] T 9bb2166a590d4210b02dd779ef6654ad P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:09.639775 2188 raft_consensus.cc:2804] T 9bb2166a590d4210b02dd779ef6654ad P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:09.639892 2188 raft_consensus.cc:697] T 9bb2166a590d4210b02dd779ef6654ad P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Becoming Leader. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Running, Role: LEADER
I20251029 02:52:09.640125 2188 consensus_queue.cc:237] T 9bb2166a590d4210b02dd779ef6654ad P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.646723 1058 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba699862e6f646c6beaa39f080d0faba" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:09.646983 1058 raft_consensus.cc:2468] T ba699862e6f646c6beaa39f080d0faba P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:09.647439 849 leader_election.cc:304] T ba699862e6f646c6beaa39f080d0faba P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters: b495a49349324f7d81908f2da700f0d9
I20251029 02:52:09.647522 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 876) took 1118 ms. Trace:
I20251029 02:52:09.647593 1176 rpcz_store.cc:276] 1029 02:52:08.529429 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:08.529486 (+ 57us) service_pool.cc:225] Handling call
1029 02:52:09.465869 (+936383us) ts_tablet_manager.cc:1151] Deleting tablet ccdca31db2834de4b6ed0f2d67a10535
1029 02:52:09.465877 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:09.568203 (+102326us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:09.572871 (+ 4668us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:09.647507 (+ 74636us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":935973}
I20251029 02:52:09.647840 2214 raft_consensus.cc:2804] T ba699862e6f646c6beaa39f080d0faba P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:09.647917 2214 raft_consensus.cc:493] T ba699862e6f646c6beaa39f080d0faba P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:09.647939 1176 tablet_replica.cc:333] T 86fc22bb84794149b598b97df41eb881 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:09.647980 2214 raft_consensus.cc:3060] T ba699862e6f646c6beaa39f080d0faba P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:09.648125 1176 raft_consensus.cc:2243] T 86fc22bb84794149b598b97df41eb881 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:09.648387 1176 raft_consensus.cc:2272] T 86fc22bb84794149b598b97df41eb881 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:09.649365 1176 ts_tablet_manager.cc:1916] T 86fc22bb84794149b598b97df41eb881 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:09.650699 2214 raft_consensus.cc:515] T ba699862e6f646c6beaa39f080d0faba P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
W20251029 02:52:09.652801 847 leader_election.cc:343] T ba699862e6f646c6beaa39f080d0faba P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: ba699862e6f646c6beaa39f080d0faba
I20251029 02:52:09.653162 519 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_91ea489c97794537b7210ff0ce101dbf" } new_table_name: "default.Table_3c3bcfdfd5914e7b892a8af9aaaac175" modify_external_catalogs: true
I20251029 02:52:09.653640 2214 leader_election.cc:290] T ba699862e6f646c6beaa39f080d0faba P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:09.656514 2113 raft_consensus.cc:493] T 82289e0f823c4d43bfd8c4e878572b2b P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:09.656664 2113 raft_consensus.cc:515] T 82289e0f823c4d43bfd8c4e878572b2b P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:09.657294 2113 leader_election.cc:290] T 82289e0f823c4d43bfd8c4e878572b2b P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:09.657972 1058 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ba699862e6f646c6beaa39f080d0faba" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:52:09.658082 914 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82289e0f823c4d43bfd8c4e878572b2b" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:09.658156 1058 raft_consensus.cc:3060] T ba699862e6f646c6beaa39f080d0faba P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
W20251029 02:52:09.658823 711 leader_election.cc:343] T 82289e0f823c4d43bfd8c4e878572b2b P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:09.658874 1167 tablet_bootstrap.cc:492] T 28f7ceb91a34450cad4b5b45f5285b41 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:09.659003 1167 ts_tablet_manager.cc:1403] T 28f7ceb91a34450cad4b5b45f5285b41 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.054s user 0.005s sys 0.000s
I20251029 02:52:09.659744 1167 raft_consensus.cc:359] T 28f7ceb91a34450cad4b5b45f5285b41 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.659909 1167 raft_consensus.cc:385] T 28f7ceb91a34450cad4b5b45f5285b41 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:09.659960 1167 raft_consensus.cc:740] T 28f7ceb91a34450cad4b5b45f5285b41 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:09.660154 1167 consensus_queue.cc:260] T 28f7ceb91a34450cad4b5b45f5285b41 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.661545 1058 raft_consensus.cc:2468] T ba699862e6f646c6beaa39f080d0faba P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:09.662009 849 leader_election.cc:304] T ba699862e6f646c6beaa39f080d0faba P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters: b495a49349324f7d81908f2da700f0d9
I20251029 02:52:09.662474 2214 raft_consensus.cc:2804] T ba699862e6f646c6beaa39f080d0faba P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:09.662663 2214 raft_consensus.cc:697] T ba699862e6f646c6beaa39f080d0faba P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:52:09.662935 2214 consensus_queue.cc:237] T ba699862e6f646c6beaa39f080d0faba P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.667133 1167 ts_tablet_manager.cc:1434] T 28f7ceb91a34450cad4b5b45f5285b41 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20251029 02:52:09.668079 2103 raft_consensus.cc:493] T 9269622ca92645d5aa8e73341201ccbb P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:09.668277 2103 raft_consensus.cc:515] T 9269622ca92645d5aa8e73341201ccbb P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.668706 766 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "061f3d38c4604fa4b57e74ff6f6fc7da" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:09.669010 766 raft_consensus.cc:2468] T 061f3d38c4604fa4b57e74ff6f6fc7da P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 0.
I20251029 02:52:09.669389 914 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9269622ca92645d5aa8e73341201ccbb" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:09.669557 914 raft_consensus.cc:2468] T 9269622ca92645d5aa8e73341201ccbb P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 0.
I20251029 02:52:09.669930 990 leader_election.cc:304] T 9269622ca92645d5aa8e73341201ccbb P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:09.670166 2179 raft_consensus.cc:2804] T 9269622ca92645d5aa8e73341201ccbb P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:09.670224 2179 raft_consensus.cc:493] T 9269622ca92645d5aa8e73341201ccbb P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:09.670277 2179 raft_consensus.cc:3060] T 9269622ca92645d5aa8e73341201ccbb P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:09.673030 2179 raft_consensus.cc:515] T 9269622ca92645d5aa8e73341201ccbb P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.673534 2179 leader_election.cc:290] T 9269622ca92645d5aa8e73341201ccbb P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Requested vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:09.673744 2201 raft_consensus.cc:493] T 0d84dc8dc465496f9adec798b2e03e13 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:09.673864 2201 raft_consensus.cc:515] T 0d84dc8dc465496f9adec798b2e03e13 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.674368 2201 leader_election.cc:290] T 0d84dc8dc465496f9adec798b2e03e13 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:09.675562 1057 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d84dc8dc465496f9adec798b2e03e13" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:09.675738 1057 raft_consensus.cc:2468] T 0d84dc8dc465496f9adec798b2e03e13 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
W20251029 02:52:09.676229 847 leader_election.cc:343] T 0d84dc8dc465496f9adec798b2e03e13 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 0d84dc8dc465496f9adec798b2e03e13
I20251029 02:52:09.676537 849 leader_election.cc:304] T 0d84dc8dc465496f9adec798b2e03e13 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters: b495a49349324f7d81908f2da700f0d9
I20251029 02:52:09.677029 2201 raft_consensus.cc:2804] T 0d84dc8dc465496f9adec798b2e03e13 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:09.677099 2201 raft_consensus.cc:493] T 0d84dc8dc465496f9adec798b2e03e13 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:09.677157 2201 raft_consensus.cc:3060] T 0d84dc8dc465496f9adec798b2e03e13 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:09.679555 1175 ts_tablet_manager.cc:1929] T 9e6eb099648549dbbf9969e59865f51f P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:09.679724 1175 log.cc:1199] T 9e6eb099648549dbbf9969e59865f51f P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/9e6eb099648549dbbf9969e59865f51f
I20251029 02:52:09.679859 2201 raft_consensus.cc:515] T 0d84dc8dc465496f9adec798b2e03e13 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.680131 1175 ts_tablet_manager.cc:1950] T 9e6eb099648549dbbf9969e59865f51f P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
W20251029 02:52:09.681524 847 leader_election.cc:343] T 0d84dc8dc465496f9adec798b2e03e13 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 0d84dc8dc465496f9adec798b2e03e13
I20251029 02:52:09.682132 914 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9269622ca92645d5aa8e73341201ccbb" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
I20251029 02:52:09.682291 914 raft_consensus.cc:3060] T 9269622ca92645d5aa8e73341201ccbb P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:09.682602 1057 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "0d84dc8dc465496f9adec798b2e03e13" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:52:09.682809 1057 raft_consensus.cc:3060] T 0d84dc8dc465496f9adec798b2e03e13 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:09.685623 914 raft_consensus.cc:2468] T 9269622ca92645d5aa8e73341201ccbb P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 1.
I20251029 02:52:09.685865 1057 raft_consensus.cc:2468] T 0d84dc8dc465496f9adec798b2e03e13 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:09.686218 990 leader_election.cc:304] T 9269622ca92645d5aa8e73341201ccbb P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:09.686477 2179 raft_consensus.cc:2804] T 9269622ca92645d5aa8e73341201ccbb P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:09.686968 849 leader_election.cc:304] T 0d84dc8dc465496f9adec798b2e03e13 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters: b495a49349324f7d81908f2da700f0d9
I20251029 02:52:09.688567 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 5db519edc75d4bd0985b32322825ffae (table default.Table_b8ddec1d965444948171ef0cca0ff011 [id=60a528f028d24bbabd6f5a927fb7853b]) successfully deleted
I20251029 02:52:09.689956 2113 raft_consensus.cc:493] T 44ae2b07edca44afbe1b1fb3a5b1115d P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:09.690124 2113 raft_consensus.cc:515] T 44ae2b07edca44afbe1b1fb3a5b1115d P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:09.690609 2205 raft_consensus.cc:2804] T 0d84dc8dc465496f9adec798b2e03e13 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:09.690778 2113 leader_election.cc:290] T 44ae2b07edca44afbe1b1fb3a5b1115d P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:09.691612 914 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "44ae2b07edca44afbe1b1fb3a5b1115d" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:09.692173 711 leader_election.cc:343] T 44ae2b07edca44afbe1b1fb3a5b1115d P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:09.692521 712 leader_election.cc:304] T 6d830d30b26b4ba799785fec8a6f0769 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters: c756528642a543eeacd93cef3e335a7d
I20251029 02:52:09.693022 712 leader_election.cc:304] T 8048506ced4540a5aee9f79bfc4599bf P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters: c756528642a543eeacd93cef3e335a7d
I20251029 02:52:09.693311 2113 raft_consensus.cc:2804] T 6d830d30b26b4ba799785fec8a6f0769 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:09.693624 2113 raft_consensus.cc:493] T 6d830d30b26b4ba799785fec8a6f0769 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:09.693727 2113 raft_consensus.cc:3060] T 6d830d30b26b4ba799785fec8a6f0769 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:09.697961 2113 raft_consensus.cc:515] T 6d830d30b26b4ba799785fec8a6f0769 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.699086 914 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d830d30b26b4ba799785fec8a6f0769" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
W20251029 02:52:09.699797 711 leader_election.cc:343] T 6d830d30b26b4ba799785fec8a6f0769 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:09.700224 2113 leader_election.cc:290] T 6d830d30b26b4ba799785fec8a6f0769 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 election: Requested vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:09.700493 2113 raft_consensus.cc:2804] T 8048506ced4540a5aee9f79bfc4599bf P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:09.700632 2113 raft_consensus.cc:493] T 8048506ced4540a5aee9f79bfc4599bf P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:09.700887 2113 raft_consensus.cc:3060] T 8048506ced4540a5aee9f79bfc4599bf P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:09.705637 2201 leader_election.cc:290] T 0d84dc8dc465496f9adec798b2e03e13 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:09.706357 2201 raft_consensus.cc:493] T 28f7ceb91a34450cad4b5b45f5285b41 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:09.706579 2201 raft_consensus.cc:515] T 28f7ceb91a34450cad4b5b45f5285b41 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
W20251029 02:52:09.707533 989 leader_election.cc:343] T 5db519edc75d4bd0985b32322825ffae P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 5db519edc75d4bd0985b32322825ffae
I20251029 02:52:09.707736 914 raft_consensus.cc:3060] T 1c53fc1fff1a4bb988beb46ff91fd3be P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:09.707767 2201 leader_election.cc:290] T 28f7ceb91a34450cad4b5b45f5285b41 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:09.712822 1174 ts_tablet_manager.cc:1929] T 86fc22bb84794149b598b97df41eb881 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:09.713315 1174 log.cc:1199] T 86fc22bb84794149b598b97df41eb881 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/86fc22bb84794149b598b97df41eb881
I20251029 02:52:09.714628 2113 raft_consensus.cc:515] T 8048506ced4540a5aee9f79bfc4599bf P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.717002 1174 ts_tablet_manager.cc:1950] T 86fc22bb84794149b598b97df41eb881 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:09.717214 914 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8048506ced4540a5aee9f79bfc4599bf" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
I20251029 02:52:09.712641 2300 consensus_queue.cc:1048] T 1c53fc1fff1a4bb988beb46ff91fd3be P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:09.717478 989 leader_election.cc:304] T 5db519edc75d4bd0985b32322825ffae P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5; no voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d
W20251029 02:52:09.717689 989 raft_consensus.cc:2686] T 5db519edc75d4bd0985b32322825ffae P 0cfd4bee16f84b7c8b0ff3ce14db20d5: unable to run election callback: Service unavailable: Thread pool token was shut down
W20251029 02:52:09.717859 711 leader_election.cc:343] T 8048506ced4540a5aee9f79bfc4599bf P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
W20251029 02:52:09.717967 989 leader_election.cc:343] T d61beda662d844cdaf67fd9a2e382ea2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: d61beda662d844cdaf67fd9a2e382ea2
I20251029 02:52:09.718068 989 leader_election.cc:304] T d61beda662d844cdaf67fd9a2e382ea2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5; no voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:09.718284 2300 raft_consensus.cc:2749] T d61beda662d844cdaf67fd9a2e382ea2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251029 02:52:09.718669 989 leader_election.cc:304] T 1bdb67125493430398ff5c80b73de2ac P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5; no voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:09.719252 2300 raft_consensus.cc:2749] T 1bdb67125493430398ff5c80b73de2ac P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:09.719973 2113 leader_election.cc:290] T 8048506ced4540a5aee9f79bfc4599bf P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 election: Requested vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:09.719942 989 leader_election.cc:304] T 13ee62fc2339410b9db70dfd3d3c6fa6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters: c756528642a543eeacd93cef3e335a7d
I20251029 02:52:09.720589 2300 raft_consensus.cc:2804] T 13ee62fc2339410b9db70dfd3d3c6fa6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:09.720662 2300 raft_consensus.cc:493] T 13ee62fc2339410b9db70dfd3d3c6fa6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:09.720724 2300 raft_consensus.cc:3060] T 13ee62fc2339410b9db70dfd3d3c6fa6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:09.720973 989 leader_election.cc:304] T 95f342d1cfb84c4eb31bd3d15ea3bdc2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters: c756528642a543eeacd93cef3e335a7d
I20251029 02:52:09.721587 989 leader_election.cc:304] T 44b518c3a0964a6fa863b2784d80c8a4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters: c756528642a543eeacd93cef3e335a7d
I20251029 02:52:09.723639 2300 raft_consensus.cc:515] T 13ee62fc2339410b9db70dfd3d3c6fa6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.723870 2113 raft_consensus.cc:493] T c32c28b0e74a4af3a0e69bbf24ab1a1f P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:09.724090 2113 raft_consensus.cc:515] T c32c28b0e74a4af3a0e69bbf24ab1a1f P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:09.724696 914 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "13ee62fc2339410b9db70dfd3d3c6fa6" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
I20251029 02:52:09.724887 914 raft_consensus.cc:3060] T 13ee62fc2339410b9db70dfd3d3c6fa6 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:09.724895 2103 leader_election.cc:290] T 9269622ca92645d5aa8e73341201ccbb P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:09.724891 2036 raft_consensus.cc:2804] T 95f342d1cfb84c4eb31bd3d15ea3bdc2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:09.725041 2103 raft_consensus.cc:2804] T 44b518c3a0964a6fa863b2784d80c8a4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:09.725131 2103 raft_consensus.cc:493] T 44b518c3a0964a6fa863b2784d80c8a4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:09.725029 2036 raft_consensus.cc:493] T 95f342d1cfb84c4eb31bd3d15ea3bdc2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:09.725212 2103 raft_consensus.cc:3060] T 44b518c3a0964a6fa863b2784d80c8a4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:09.725265 2036 raft_consensus.cc:3060] T 95f342d1cfb84c4eb31bd3d15ea3bdc2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:09.727855 2036 raft_consensus.cc:515] T 95f342d1cfb84c4eb31bd3d15ea3bdc2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.727900 914 raft_consensus.cc:2468] T 13ee62fc2339410b9db70dfd3d3c6fa6 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 1.
I20251029 02:52:09.728423 990 leader_election.cc:304] T 13ee62fc2339410b9db70dfd3d3c6fa6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:09.729434 919 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "95f342d1cfb84c4eb31bd3d15ea3bdc2" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
I20251029 02:52:09.729674 2036 leader_election.cc:290] T 95f342d1cfb84c4eb31bd3d15ea3bdc2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Requested vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:09.729815 919 raft_consensus.cc:3060] T 95f342d1cfb84c4eb31bd3d15ea3bdc2 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:09.730223 2036 raft_consensus.cc:2804] T 13ee62fc2339410b9db70dfd3d3c6fa6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:09.730715 2036 raft_consensus.cc:697] T 13ee62fc2339410b9db70dfd3d3c6fa6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Becoming Leader. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Running, Role: LEADER
I20251029 02:52:09.731105 2036 consensus_queue.cc:237] T 13ee62fc2339410b9db70dfd3d3c6fa6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.732892 2300 leader_election.cc:290] T 13ee62fc2339410b9db70dfd3d3c6fa6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Requested vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:09.733537 919 raft_consensus.cc:2468] T 95f342d1cfb84c4eb31bd3d15ea3bdc2 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 1.
I20251029 02:52:09.737809 919 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c32c28b0e74a4af3a0e69bbf24ab1a1f" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:09.738597 711 leader_election.cc:343] T c32c28b0e74a4af3a0e69bbf24ab1a1f P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:09.739208 2113 leader_election.cc:290] T c32c28b0e74a4af3a0e69bbf24ab1a1f P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
W20251029 02:52:09.742944 712 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:45003, user_credentials={real_user=slave}} blocked reactor thread for 49963.2us
I20251029 02:52:09.743345 712 leader_election.cc:304] T 28f7ceb91a34450cad4b5b45f5285b41 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters: c756528642a543eeacd93cef3e335a7d
I20251029 02:52:09.743906 712 leader_election.cc:304] T af4860c6b72f4d988b206b5ef2f22906 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters: c756528642a543eeacd93cef3e335a7d
W20251029 02:52:09.744292 712 leader_election.cc:343] T 44c0488eba0543078ff4f0b3d69366f2 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:09.744436 712 leader_election.cc:304] T 44c0488eba0543078ff4f0b3d69366f2 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:09.744779 2113 raft_consensus.cc:2804] T 28f7ceb91a34450cad4b5b45f5285b41 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:09.745431 2113 raft_consensus.cc:493] T 28f7ceb91a34450cad4b5b45f5285b41 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:09.745595 2113 raft_consensus.cc:3060] T 28f7ceb91a34450cad4b5b45f5285b41 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:09.746624 2303 raft_consensus.cc:2804] T af4860c6b72f4d988b206b5ef2f22906 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:09.746819 2303 raft_consensus.cc:493] T af4860c6b72f4d988b206b5ef2f22906 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:09.746982 2303 raft_consensus.cc:3060] T af4860c6b72f4d988b206b5ef2f22906 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:09.750267 2113 raft_consensus.cc:515] T 28f7ceb91a34450cad4b5b45f5285b41 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.750968 2103 raft_consensus.cc:515] T 44b518c3a0964a6fa863b2784d80c8a4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.751646 2103 leader_election.cc:290] T 44b518c3a0964a6fa863b2784d80c8a4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Requested vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:09.755412 1058 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "28f7ceb91a34450cad4b5b45f5285b41" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:09.755667 1166 tablet_bootstrap.cc:492] T 97b2ecae572a4b7ea1b9595074556708 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:09.755671 1058 raft_consensus.cc:2468] T 28f7ceb91a34450cad4b5b45f5285b41 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:09.759512 1166 ts_tablet_manager.cc:1403] T 97b2ecae572a4b7ea1b9595074556708 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.332s user 0.028s sys 0.067s
I20251029 02:52:09.760259 919 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "28f7ceb91a34450cad4b5b45f5285b41" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
I20251029 02:52:09.760581 919 raft_consensus.cc:3060] T 28f7ceb91a34450cad4b5b45f5285b41 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:09.760594 1166 raft_consensus.cc:359] T 97b2ecae572a4b7ea1b9595074556708 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.761721 518 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet 1bdb67125493430398ff5c80b73de2ac (table default.Table_2846af457a4b4e27a739af50c7f8c7f7 [id=8c3d3d88395e4712af0f62c4ad14edea]). Expected version 3 got 0
I20251029 02:52:09.762403 526 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet 643e20d9974447daa78885b3401f3b77 (table default.Table_7c1cb64e05be4f6e8ac28286c982a3e7 [id=b4fc453bf54d45c4b8dddc3c49bf48f8]). Expected version 2 got 0
I20251029 02:52:09.763701 919 raft_consensus.cc:2468] T 28f7ceb91a34450cad4b5b45f5285b41 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b495a49349324f7d81908f2da700f0d9 in term 1.
I20251029 02:52:09.764206 711 leader_election.cc:304] T 28f7ceb91a34450cad4b5b45f5285b41 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:09.765661 914 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "44b518c3a0964a6fa863b2784d80c8a4" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
I20251029 02:52:09.765816 914 raft_consensus.cc:3060] T 44b518c3a0964a6fa863b2784d80c8a4 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:09.766232 1166 raft_consensus.cc:385] T 97b2ecae572a4b7ea1b9595074556708 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:09.766563 1166 raft_consensus.cc:740] T 97b2ecae572a4b7ea1b9595074556708 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:09.766786 1166 consensus_queue.cc:260] T 97b2ecae572a4b7ea1b9595074556708 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.767310 1166 ts_tablet_manager.cc:1434] T 97b2ecae572a4b7ea1b9595074556708 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20251029 02:52:09.766088 990 leader_election.cc:304] T 95f342d1cfb84c4eb31bd3d15ea3bdc2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:09.767908 2103 raft_consensus.cc:2804] T 95f342d1cfb84c4eb31bd3d15ea3bdc2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:09.768078 2103 raft_consensus.cc:697] T 95f342d1cfb84c4eb31bd3d15ea3bdc2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Becoming Leader. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Running, Role: LEADER
I20251029 02:52:09.768334 879 tablet_service.cc:1552] Processing DeleteTablet for tablet caef1c93e33a4390bcada59a41f43465 with delete_type TABLET_DATA_DELETED (replaced by tablet 49492d91b1e94e9ca086533dbd0440f6) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:09.768376 2103 consensus_queue.cc:237] T 95f342d1cfb84c4eb31bd3d15ea3bdc2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.768536 899 tablet_service.cc:1552] Processing DeleteTablet for tablet ba699862e6f646c6beaa39f080d0faba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:04 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:09.768707 892 tablet_service.cc:1552] Processing DeleteTablet for tablet 5db519edc75d4bd0985b32322825ffae with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:55 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:09.769083 900 tablet_service.cc:1505] Processing CreateTablet for tablet 3b8bb5025de24e8e95d13f04c62305e1 (DEFAULT_TABLE table=default.Table_f060d401cf984a96b90cf4735c9f34d2 [id=8c9724de88084668b15b37015c668435]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:09.769447 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b8bb5025de24e8e95d13f04c62305e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:09.772490 899 tablet_service.cc:1505] Processing CreateTablet for tablet 93440d660eac4986aeb17abc18144700 (DEFAULT_TABLE table=default.Table_126f4f49037f46628fa5e1d33b4fcf24 [id=41207bfd4192414598f5517feffe3351]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:09.772868 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93440d660eac4986aeb17abc18144700. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:09.774017 1166 tablet_bootstrap.cc:492] T 2f198c3259864f7ab3625cb7bfae20b0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:09.775122 892 tablet_service.cc:1505] Processing CreateTablet for tablet 03f59f9206024dfbaa4e67a5b6bfb902 (DEFAULT_TABLE table=default.Table_95f035fda47144eb9d79be23614b41b6 [id=38cb7bf11be54631ba227c9fcd3b1aa4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:09.775478 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03f59f9206024dfbaa4e67a5b6bfb902. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:09.775732 1166 tablet_bootstrap.cc:654] T 2f198c3259864f7ab3625cb7bfae20b0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:09.776890 914 raft_consensus.cc:2468] T 44b518c3a0964a6fa863b2784d80c8a4 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 1.
I20251029 02:52:09.779003 890 tablet_service.cc:1505] Processing CreateTablet for tablet b79327b8c1354810a8b1b0d9e62fe3c8 (DEFAULT_TABLE table=default.Table_ff347dc490d441baa678753aa33b710b [id=17e273128f4c47daa4faf2e7a71b532a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:09.779609 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b79327b8c1354810a8b1b0d9e62fe3c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:09.780023 885 tablet_service.cc:1505] Processing CreateTablet for tablet e1d00ea3dd3744ffa5258bbe0095851c (DEFAULT_TABLE table=default.Table_cfbb812339124da1b8e6e8a95d816a5e [id=b834fcfacd4e4384bb684d5f3ea671e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:09.780421 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1d00ea3dd3744ffa5258bbe0095851c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:09.782397 990 leader_election.cc:304] T 44b518c3a0964a6fa863b2784d80c8a4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:09.782788 2306 raft_consensus.cc:2804] T 44b518c3a0964a6fa863b2784d80c8a4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:09.782883 895 tablet_service.cc:1505] Processing CreateTablet for tablet cb9856e5fe244d3e91f30f5e1d276d18 (DEFAULT_TABLE table=default.Table_482767bf59bf442a99a8675f678afd06 [id=3c6e732a6c4347ada8cb3d4cb02835cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:09.783138 879 tablet_service.cc:1505] Processing CreateTablet for tablet 82f83631253a4b5785c814c993678d68 (DEFAULT_TABLE table=default.Table_47178701cd044f8fa5357cd607154988 [id=a0b7f2de597f4af3945a7d41a365acc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:09.783437 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb9856e5fe244d3e91f30f5e1d276d18. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:09.784307 2306 raft_consensus.cc:697] T 44b518c3a0964a6fa863b2784d80c8a4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Becoming Leader. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Running, Role: LEADER
I20251029 02:52:09.784729 2306 consensus_queue.cc:237] T 44b518c3a0964a6fa863b2784d80c8a4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.785097 766 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "28f7ceb91a34450cad4b5b45f5285b41" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:09.785374 766 raft_consensus.cc:2393] T 28f7ceb91a34450cad4b5b45f5285b41 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c756528642a543eeacd93cef3e335a7d in current term 1: Already voted for candidate b495a49349324f7d81908f2da700f0d9 in this term.
W20251029 02:52:09.786859 847 consensus_peers.cc:597] T dfa8b1b9a80e4e8689e0acb44e16346a P c756528642a543eeacd93cef3e335a7d -> Peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Couldn't send request to peer b495a49349324f7d81908f2da700f0d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dfa8b1b9a80e4e8689e0acb44e16346a. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:09.787230 849 leader_election.cc:304] T 28f7ceb91a34450cad4b5b45f5285b41 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters: b495a49349324f7d81908f2da700f0d9
I20251029 02:52:09.787848 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82f83631253a4b5785c814c993678d68. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:09.788116 2205 raft_consensus.cc:697] T 0d84dc8dc465496f9adec798b2e03e13 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:52:09.788403 2205 consensus_queue.cc:237] T 0d84dc8dc465496f9adec798b2e03e13 P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.789494 2179 raft_consensus.cc:697] T 9269622ca92645d5aa8e73341201ccbb P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Becoming Leader. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Running, Role: LEADER
I20251029 02:52:09.789808 2179 consensus_queue.cc:237] T 9269622ca92645d5aa8e73341201ccbb P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.792222 2302 raft_consensus.cc:2764] T 28f7ceb91a34450cad4b5b45f5285b41 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
W20251029 02:52:09.792625 849 consensus_peers.cc:597] T 9e6eb099648549dbbf9969e59865f51f P c756528642a543eeacd93cef3e335a7d -> Peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Couldn't send request to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9e6eb099648549dbbf9969e59865f51f. This is attempt 1: this message will repeat every 5th retry.
W20251029 02:52:09.793740 847 consensus_peers.cc:597] T 9e6eb099648549dbbf9969e59865f51f P c756528642a543eeacd93cef3e335a7d -> Peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Couldn't send request to peer b495a49349324f7d81908f2da700f0d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9e6eb099648549dbbf9969e59865f51f. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:09.797463 2311 raft_consensus.cc:2749] T 44c0488eba0543078ff4f0b3d69366f2 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:09.799362 1172 tablet_bootstrap.cc:492] T 59eee86f249e4d6a9d09d30276dcd659 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:09.799536 1172 ts_tablet_manager.cc:1403] T 59eee86f249e4d6a9d09d30276dcd659 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.338s user 0.019s sys 0.019s
I20251029 02:52:09.801352 1167 tablet_bootstrap.cc:492] T 30d03e1e42d34242bd354f9e9ec4f127 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:09.802531 2311 raft_consensus.cc:2804] T 28f7ceb91a34450cad4b5b45f5285b41 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:09.802825 2311 raft_consensus.cc:697] T 28f7ceb91a34450cad4b5b45f5285b41 P b495a49349324f7d81908f2da700f0d9 [term 1 LEADER]: Becoming Leader. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Running, Role: LEADER
I20251029 02:52:09.803719 885 tablet_service.cc:1505] Processing CreateTablet for tablet bc9fc1c90eaf4046a4022beab31f2902 (DEFAULT_TABLE table=default.Table_fe6f038b462644e7964ea2e48df2a331 [id=5d492f75aeeb4f4bbc9c075377531571]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:09.804103 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc9fc1c90eaf4046a4022beab31f2902. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:09.805240 2301 consensus_queue.cc:1048] T dfa8b1b9a80e4e8689e0acb44e16346a P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:09.808125 895 tablet_service.cc:1505] Processing CreateTablet for tablet aba37796661b4041adf7960a15ff52eb (DEFAULT_TABLE table=default.Table_8a944a4e396a4f7cb27c0b3c01ab3ef2 [id=6cf74eaf4e094647bc65dac51a43c434]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:09.808909 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aba37796661b4041adf7960a15ff52eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:09.811514 2311 consensus_queue.cc:237] T 28f7ceb91a34450cad4b5b45f5285b41 P b495a49349324f7d81908f2da700f0d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.822597 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_149358409d284922ad0f2c49410dca96"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:09.823230 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_149358409d284922ad0f2c49410dca96 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:09.826133 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_0f4514215d3042c5a46ac59072d2bbb2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:09.827220 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 805) took 2075 ms. Trace:
I20251029 02:52:09.827333 1175 rpcz_store.cc:276] 1029 02:52:07.751574 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:07.754918 (+ 3344us) service_pool.cc:225] Handling call
1029 02:52:09.622843 (+1867925us) ts_tablet_manager.cc:1151] Deleting tablet 9e6eb099648549dbbf9969e59865f51f
1029 02:52:09.622855 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:09.643943 (+ 21088us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:09.679421 (+ 35478us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:09.827151 (+147730us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1859825}
I20251029 02:52:09.827795 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 806) took 2069 ms. Trace:
I20251029 02:52:09.827908 1175 rpcz_store.cc:276] 1029 02:52:07.758144 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:07.758206 (+ 62us) service_pool.cc:225] Handling call
1029 02:52:09.827619 (+2069413us) ts_tablet_manager.cc:1151] Deleting tablet 9778f924d11a48b194ccf26124bd5509
1029 02:52:09.827632 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:09.827740 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2064165}
I20251029 02:52:09.828269 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 807) took 2068 ms. Trace:
I20251029 02:52:09.828369 1175 rpcz_store.cc:276] 1029 02:52:07.759970 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:07.760191 (+ 221us) service_pool.cc:225] Handling call
1029 02:52:09.828120 (+2067929us) ts_tablet_manager.cc:1151] Deleting tablet 80f54d8a486a4196aca3c28c84c0872f
1029 02:52:09.828130 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:09.828215 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2064615}
I20251029 02:52:09.829269 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 808) took 2069 ms. Trace:
I20251029 02:52:09.829367 1175 rpcz_store.cc:276] 1029 02:52:07.760135 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:07.761449 (+ 1314us) service_pool.cc:225] Handling call
1029 02:52:09.829126 (+2067677us) ts_tablet_manager.cc:1151] Deleting tablet ff3e9227b9674d31b874823cf4a62d9b
1029 02:52:09.829134 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:09.829207 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2065588}
I20251029 02:52:09.829661 1175 tablet_replica.cc:333] T 1275e192871440408a8345fef9b56260 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
W20251029 02:52:09.829696 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0f4514215d3042c5a46ac59072d2bbb2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:09.829818 2311 raft_consensus.cc:493] T e65eaaae43ac435ebd665546de9d68dd P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:09.829972 2311 raft_consensus.cc:515] T e65eaaae43ac435ebd665546de9d68dd P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.830614 2311 leader_election.cc:290] T e65eaaae43ac435ebd665546de9d68dd P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:09.752632 2303 raft_consensus.cc:515] T af4860c6b72f4d988b206b5ef2f22906 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
W20251029 02:52:09.833592 711 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:41701, user_credentials={real_user=slave}} blocked reactor thread for 69400.3us
I20251029 02:52:09.833889 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 9e6eb099648549dbbf9969e59865f51f (table default.Table_683d67689da74097b69f31b3d9461fe2 [id=0c5d3e4d34d045dc99ffc88e101cae1d]) successfully deleted
I20251029 02:52:09.834159 2303 leader_election.cc:290] T af4860c6b72f4d988b206b5ef2f22906 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 election: Requested vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
W20251029 02:52:09.834345 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 9778f924d11a48b194ccf26124bd5509 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9778f924d11a48b194ccf26124bd5509 already in progress: creating tablet
I20251029 02:52:09.834470 2113 leader_election.cc:290] T 28f7ceb91a34450cad4b5b45f5285b41 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 election: Requested vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
W20251029 02:52:09.834810 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 80f54d8a486a4196aca3c28c84c0872f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 80f54d8a486a4196aca3c28c84c0872f already in progress: creating tablet
W20251029 02:52:09.835176 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet ff3e9227b9674d31b874823cf4a62d9b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ff3e9227b9674d31b874823cf4a62d9b already in progress: creating tablet
I20251029 02:52:09.835793 919 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e65eaaae43ac435ebd665546de9d68dd" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:09.837193 745 tablet_service.cc:1505] Processing CreateTablet for tablet 3b8bb5025de24e8e95d13f04c62305e1 (DEFAULT_TABLE table=default.Table_f060d401cf984a96b90cf4735c9f34d2 [id=8c9724de88084668b15b37015c668435]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:09.837586 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b8bb5025de24e8e95d13f04c62305e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:09.843550 1167 tablet_bootstrap.cc:654] T 30d03e1e42d34242bd354f9e9ec4f127 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:09.844931 751 tablet_service.cc:1505] Processing CreateTablet for tablet 93440d660eac4986aeb17abc18144700 (DEFAULT_TABLE table=default.Table_126f4f49037f46628fa5e1d33b4fcf24 [id=41207bfd4192414598f5517feffe3351]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:09.845333 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93440d660eac4986aeb17abc18144700. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:09.845613 756 tablet_service.cc:1505] Processing CreateTablet for tablet bc9fc1c90eaf4046a4022beab31f2902 (DEFAULT_TABLE table=default.Table_fe6f038b462644e7964ea2e48df2a331 [id=5d492f75aeeb4f4bbc9c075377531571]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:09.845625 750 tablet_service.cc:1505] Processing CreateTablet for tablet aba37796661b4041adf7960a15ff52eb (DEFAULT_TABLE table=default.Table_8a944a4e396a4f7cb27c0b3c01ab3ef2 [id=6cf74eaf4e094647bc65dac51a43c434]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:09.846046 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc9fc1c90eaf4046a4022beab31f2902. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:09.846282 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aba37796661b4041adf7960a15ff52eb. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:09.846570 711 leader_election.cc:343] T e65eaaae43ac435ebd665546de9d68dd P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:09.822536 1172 raft_consensus.cc:359] T 59eee86f249e4d6a9d09d30276dcd659 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:09.846846 1172 raft_consensus.cc:385] T 59eee86f249e4d6a9d09d30276dcd659 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:09.846963 1172 raft_consensus.cc:740] T 59eee86f249e4d6a9d09d30276dcd659 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:09.847218 1172 consensus_queue.cc:260] T 59eee86f249e4d6a9d09d30276dcd659 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:09.848191 743 tablet_service.cc:1505] Processing CreateTablet for tablet 82f83631253a4b5785c814c993678d68 (DEFAULT_TABLE table=default.Table_47178701cd044f8fa5357cd607154988 [id=a0b7f2de597f4af3945a7d41a365acc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:09.848558 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82f83631253a4b5785c814c993678d68. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:09.850740 738 tablet_service.cc:1505] Processing CreateTablet for tablet b79327b8c1354810a8b1b0d9e62fe3c8 (DEFAULT_TABLE table=default.Table_ff347dc490d441baa678753aa33b710b [id=17e273128f4c47daa4faf2e7a71b532a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:09.851146 742 tablet_service.cc:1505] Processing CreateTablet for tablet e1d00ea3dd3744ffa5258bbe0095851c (DEFAULT_TABLE table=default.Table_cfbb812339124da1b8e6e8a95d816a5e [id=b834fcfacd4e4384bb684d5f3ea671e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:09.851351 1172 ts_tablet_manager.cc:1434] T 59eee86f249e4d6a9d09d30276dcd659 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.052s user 0.000s sys 0.003s
I20251029 02:52:09.851568 747 tablet_service.cc:1505] Processing CreateTablet for tablet cb9856e5fe244d3e91f30f5e1d276d18 (DEFAULT_TABLE table=default.Table_482767bf59bf442a99a8675f678afd06 [id=3c6e732a6c4347ada8cb3d4cb02835cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:09.854911 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b79327b8c1354810a8b1b0d9e62fe3c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:09.855800 741 tablet_service.cc:1505] Processing CreateTablet for tablet b3d275e975e64b8d9e026e0eab5b677d (DEFAULT_TABLE table=default.Table_1c456ca0e187431baa7d2054b49ac76c [id=addfa045c7c1492f8168232ee7204595]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:09.858749 749 tablet_service.cc:1505] Processing CreateTablet for tablet 0de19a308a534cdaa38f46ab43789322 (DEFAULT_TABLE table=default.Table_9c6ff90fbbd9431c9ae4dbf96e02b307 [id=917b51dc88174f0bab43a20302b49cc3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:09.859084 755 tablet_service.cc:1505] Processing CreateTablet for tablet f3ede523203545d18b09d535fdcabb6f (DEFAULT_TABLE table=default.Table_5d956728da054438a1bd400dac5dac1e [id=006ec0b27fd64c79af53e1d411e921aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:09.859297 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 814) took 2151 ms. Trace:
I20251029 02:52:09.859710 1174 rpcz_store.cc:276] 1029 02:52:07.708088 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:07.708680 (+ 592us) service_pool.cc:225] Handling call
1029 02:52:09.632223 (+1923543us) ts_tablet_manager.cc:1151] Deleting tablet 86fc22bb84794149b598b97df41eb881
1029 02:52:09.632234 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:09.640333 (+ 8099us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:09.712522 (+ 72189us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:09.859261 (+146739us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1923424}
I20251029 02:52:09.860338 892 tablet_service.cc:1505] Processing CreateTablet for tablet b3d275e975e64b8d9e026e0eab5b677d (DEFAULT_TABLE table=default.Table_1c456ca0e187431baa7d2054b49ac76c [id=addfa045c7c1492f8168232ee7204595]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:09.860622 895 tablet_service.cc:1505] Processing CreateTablet for tablet 0de19a308a534cdaa38f46ab43789322 (DEFAULT_TABLE table=default.Table_9c6ff90fbbd9431c9ae4dbf96e02b307 [id=917b51dc88174f0bab43a20302b49cc3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:09.860582 890 tablet_service.cc:1505] Processing CreateTablet for tablet f3ede523203545d18b09d535fdcabb6f (DEFAULT_TABLE table=default.Table_5d956728da054438a1bd400dac5dac1e [id=006ec0b27fd64c79af53e1d411e921aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:09.862164 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1d00ea3dd3744ffa5258bbe0095851c. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:09.862483 989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:38481, user_credentials={real_user=slave}} blocked reactor thread for 140255us
W20251029 02:52:09.862918 989 leader_election.cc:343] T 49d9ce32e4f04f11be2ecefc5d523a8b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 49d9ce32e4f04f11be2ecefc5d523a8b
I20251029 02:52:09.863063 989 leader_election.cc:304] T 49d9ce32e4f04f11be2ecefc5d523a8b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5; no voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:09.863551 989 leader_election.cc:304] T 6d830d30b26b4ba799785fec8a6f0769 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters: c756528642a543eeacd93cef3e335a7d
I20251029 02:52:09.864353 989 leader_election.cc:304] T 061f3d38c4604fa4b57e74ff6f6fc7da P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters: c756528642a543eeacd93cef3e335a7d
I20251029 02:52:09.864466 2188 raft_consensus.cc:2749] T 49d9ce32e4f04f11be2ecefc5d523a8b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:09.864866 2188 raft_consensus.cc:2804] T 6d830d30b26b4ba799785fec8a6f0769 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:09.864948 2188 raft_consensus.cc:493] T 6d830d30b26b4ba799785fec8a6f0769 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20251029 02:52:09.865007 989 consensus_peers.cc:597] T 0f80b840ed954230bfbfe7e40127cd2f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 -> Peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Couldn't send request to peer b495a49349324f7d81908f2da700f0d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0f80b840ed954230bfbfe7e40127cd2f. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:09.865237 2103 consensus_queue.cc:1048] T 1c53fc1fff1a4bb988beb46ff91fd3be P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:09.865569 2113 raft_consensus.cc:493] T 946f65014d97425aab3851009b742807 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:09.865593 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet 00ebd0c68c4441f9b63c86cdab62497d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:08 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:09.865677 2188 raft_consensus.cc:3060] T 6d830d30b26b4ba799785fec8a6f0769 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:09.865716 2113 raft_consensus.cc:515] T 946f65014d97425aab3851009b742807 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:09.866293 2113 leader_election.cc:290] T 946f65014d97425aab3851009b742807 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:09.866389 2179 raft_consensus.cc:2804] T 061f3d38c4604fa4b57e74ff6f6fc7da P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:09.866497 2179 raft_consensus.cc:493] T 061f3d38c4604fa4b57e74ff6f6fc7da P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:09.866691 2179 raft_consensus.cc:3060] T 061f3d38c4604fa4b57e74ff6f6fc7da P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:09.867069 1175 raft_consensus.cc:2243] T 1275e192871440408a8345fef9b56260 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:09.867269 1175 raft_consensus.cc:2272] T 1275e192871440408a8345fef9b56260 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:09.867929 1175 ts_tablet_manager.cc:1916] T 1275e192871440408a8345fef9b56260 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:09.868352 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 86fc22bb84794149b598b97df41eb881 (table default.Table_77a5f27bd0ae4eb68a30c883268d2f9c [id=d62f017ad2514f29867c9cde35996faf]) successfully deleted
I20251029 02:52:09.869916 2305 raft_consensus.cc:493] T dabb8d760aae4aa09eb83cdb586c9788 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:09.870050 2305 raft_consensus.cc:515] T dabb8d760aae4aa09eb83cdb586c9788 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.871250 2305 leader_election.cc:290] T dabb8d760aae4aa09eb83cdb586c9788 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:09.875229 2179 raft_consensus.cc:515] T 061f3d38c4604fa4b57e74ff6f6fc7da P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.876919 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb9856e5fe244d3e91f30f5e1d276d18. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:09.878333 2179 leader_election.cc:290] T 061f3d38c4604fa4b57e74ff6f6fc7da P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Requested vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:09.877488 2188 raft_consensus.cc:515] T 6d830d30b26b4ba799785fec8a6f0769 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.879051 2188 leader_election.cc:290] T 6d830d30b26b4ba799785fec8a6f0769 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Requested vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:09.880219 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3d275e975e64b8d9e026e0eab5b677d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:09.882611 748 tablet_service.cc:1552] Processing DeleteTablet for tablet d041e87ceb11441983d0e38b7929c3a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:01 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:09.886752 517 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet c9e923724ca9402682f13b252dbc37aa
I20251029 02:52:09.887409 517 catalog_manager.cc:6719] ReplaceTablet: tablet c9e923724ca9402682f13b252dbc37aa deleted and replaced by tablet 456b623500fd4d758e40004666b095a9
I20251029 02:52:09.888576 748 tablet_service.cc:1552] Processing DeleteTablet for tablet c9e923724ca9402682f13b252dbc37aa with delete_type TABLET_DATA_DELETED (replaced by tablet 456b623500fd4d758e40004666b095a9) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:09.892437 517 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_a3377614e4a54266a4457944d48b0060" } modify_external_catalogs: true
I20251029 02:52:09.892699 517 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_a3377614e4a54266a4457944d48b0060" } modify_external_catalogs: true
I20251029 02:52:09.892993 2113 raft_consensus.cc:493] T 217144957930495aa3e9dfcfde268e71 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:09.893131 2113 raft_consensus.cc:515] T 217144957930495aa3e9dfcfde268e71 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:09.893695 2113 leader_election.cc:290] T 217144957930495aa3e9dfcfde268e71 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:09.895362 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0de19a308a534cdaa38f46ab43789322. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:09.894430 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_28bb97b95f7942fb9da0cec248e6ac6d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:09.898478 900 tablet_service.cc:1505] Processing CreateTablet for tablet 6c4f28b6f4a94d35bb036843aa3bc4bc (DEFAULT_TABLE table=default.Table_6c1f042a9840467993a38a8e55a23cbb [id=29c45a9bc52e4cdabd41ab4220c2b40b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:09.899009 899 tablet_service.cc:1505] Processing CreateTablet for tablet 85bcb1fe0cb54b5aad3e57a812e6d802 (DEFAULT_TABLE table=default.Table_f8f86e0f1d8f4ba98ba26b268e26eee2 [id=76b0df830a7f41a9b6091ca2f1b4edbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:09.899147 898 tablet_service.cc:1505] Processing CreateTablet for tablet f4a8f0ad6faa4cf58f48156087392d3c (DEFAULT_TABLE table=default.Table_c6e9611c339e47a8b5021b69c449f141 [id=c19d4543d4ce43a780612d2d8653d496]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 02:52:09.899391 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_28bb97b95f7942fb9da0cec248e6ac6d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:09.899688 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3ede523203545d18b09d535fdcabb6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:09.900178 1172 tablet_bootstrap.cc:492] T a9e0fcc759a64679b3f59a6c964877b6 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:09.900605 879 tablet_service.cc:1552] Processing DeleteTablet for tablet c9e923724ca9402682f13b252dbc37aa with delete_type TABLET_DATA_DELETED (replaced by tablet 456b623500fd4d758e40004666b095a9) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:09.901842 1172 tablet_bootstrap.cc:654] T a9e0fcc759a64679b3f59a6c964877b6 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:09.906934 522 catalog_manager.cc:5649] T 1bdb67125493430398ff5c80b73de2ac P c756528642a543eeacd93cef3e335a7d reported cstate change: term changed from 0 to 1, leader changed from <none> to c756528642a543eeacd93cef3e335a7d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "c756528642a543eeacd93cef3e335a7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } health_report { overall_health: UNKNOWN } } }
I20251029 02:52:09.907606 522 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet 1bdb67125493430398ff5c80b73de2ac (table default.Table_2846af457a4b4e27a739af50c7f8c7f7 [id=8c3d3d88395e4712af0f62c4ad14edea]). Expected version 3 got 0
I20251029 02:52:09.907930 2113 raft_consensus.cc:493] T afb19ba39399410bb90b6225d7b5b5b1 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:09.908114 2113 raft_consensus.cc:515] T afb19ba39399410bb90b6225d7b5b5b1 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:09.908036 522 catalog_manager.cc:5649] T 6f35412a66b74051a475dce5dafa7792 P c756528642a543eeacd93cef3e335a7d reported cstate change: term changed from 0 to 1, leader changed from <none> to c756528642a543eeacd93cef3e335a7d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "c756528642a543eeacd93cef3e335a7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } health_report { overall_health: UNKNOWN } } }
I20251029 02:52:09.908677 522 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet 95f342d1cfb84c4eb31bd3d15ea3bdc2 (table default.Table_96ea715e50e74cc48759dda025a46318 [id=c63d52811db749d6badbed72fc133943]). Expected version 1 got 0
I20251029 02:52:09.908758 2113 leader_election.cc:290] T afb19ba39399410bb90b6225d7b5b5b1 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:09.908929 522 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet 9269622ca92645d5aa8e73341201ccbb (table default.Table_902ea0b243c7400c86250cfb5fae5b54 [id=d0ca7c93c8cf4dc19962a1957321c5b7]). Expected version 5 got 0
I20251029 02:52:09.909101 522 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet 02e4588b03144ea2850e13a1f8f29f06 (table default.Table_0772f9db5d6142d988f6ffca2e96bdd2 [id=fd6d31d379394857a35422211af00642]). Expected version 3 got 0
I20251029 02:52:09.909473 522 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet 44b518c3a0964a6fa863b2784d80c8a4 (table default.Table_9120016827e946cc8a145252d2569d2b [id=e0c0cf3d456444c191a32bb180e83798]). Expected version 1 got 0
I20251029 02:52:09.909694 522 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet d5741dde463a43d6a776ab92dcfb04bf (table default.Table_966779a0d37a4dc4ab3f89d2bf7c6b06 [id=08e3b2ebcf37415d9a2886ad5bbda068]). Expected version 1 got 0
I20251029 02:52:09.911885 881 tablet_service.cc:1505] Processing CreateTablet for tablet eb3c20822d684bea89eb358251e21e33 (DEFAULT_TABLE table=default.Table_52cba56ce9fb44bd84c35ef2f9219e56 [id=d7a96b1852fc4049b31ff576280980c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:09.912917 884 tablet_service.cc:1505] Processing CreateTablet for tablet 99d60ef5325e45059026596a756448de (DEFAULT_TABLE table=default.Table_aec07800b35948bba7db9168286fcf14 [id=7147b693c86e4c258fecdadfbe5c29f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:09.913539 879 tablet_service.cc:1505] Processing CreateTablet for tablet 0e8802cde9bf4784b587c93c5cf07957 (DEFAULT_TABLE table=default.Table_329849bbaa3a457096d8526f70636b96 [id=bcdff51837004f4aaa118266071b2887]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:09.913864 741 tablet_service.cc:1505] Processing CreateTablet for tablet eb3c20822d684bea89eb358251e21e33 (DEFAULT_TABLE table=default.Table_52cba56ce9fb44bd84c35ef2f9219e56 [id=d7a96b1852fc4049b31ff576280980c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:09.914381 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb3c20822d684bea89eb358251e21e33. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:09.918668 1172 tablet_bootstrap.cc:492] T a9e0fcc759a64679b3f59a6c964877b6 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:09.918991 920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "061f3d38c4604fa4b57e74ff6f6fc7da" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
I20251029 02:52:09.919036 914 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "946f65014d97425aab3851009b742807" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:09.919090 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d830d30b26b4ba799785fec8a6f0769" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
I20251029 02:52:09.919315 919 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af4860c6b72f4d988b206b5ef2f22906" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
I20251029 02:52:09.919394 918 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "afb19ba39399410bb90b6225d7b5b5b1" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:09.919341 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "217144957930495aa3e9dfcfde268e71" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:09.918923 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dabb8d760aae4aa09eb83cdb586c9788" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:09.919808 916 raft_consensus.cc:2468] T dabb8d760aae4aa09eb83cdb586c9788 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 0.
W20251029 02:52:09.919804 990 leader_election.cc:343] T 6d830d30b26b4ba799785fec8a6f0769 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
W20251029 02:52:09.920079 990 leader_election.cc:343] T 061f3d38c4604fa4b57e74ff6f6fc7da P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:09.920130 1174 tablet_replica.cc:333] T 9e6eb099648549dbbf9969e59865f51f P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:09.920290 990 leader_election.cc:304] T dabb8d760aae4aa09eb83cdb586c9788 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:09.920316 1174 raft_consensus.cc:2243] T 9e6eb099648549dbbf9969e59865f51f P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:09.920569 2188 raft_consensus.cc:2804] T dabb8d760aae4aa09eb83cdb586c9788 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:09.920603 1174 pending_rounds.cc:70] T 9e6eb099648549dbbf9969e59865f51f P c756528642a543eeacd93cef3e335a7d: Trying to abort 1 pending ops.
I20251029 02:52:09.920646 1174 pending_rounds.cc:77] T 9e6eb099648549dbbf9969e59865f51f P c756528642a543eeacd93cef3e335a7d: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215949123027505152 op_type: NO_OP noop_request { }
I20251029 02:52:09.920684 2188 raft_consensus.cc:493] T dabb8d760aae4aa09eb83cdb586c9788 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:09.920733 1174 raft_consensus.cc:2889] T 9e6eb099648549dbbf9969e59865f51f P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 02:52:09.920769 2188 raft_consensus.cc:3060] T dabb8d760aae4aa09eb83cdb586c9788 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:09.920833 1174 raft_consensus.cc:2272] T 9e6eb099648549dbbf9969e59865f51f P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:09.921386 1174 ts_tablet_manager.cc:1916] T 9e6eb099648549dbbf9969e59865f51f P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:09.921448 891 tablet_service.cc:1505] Processing CreateTablet for tablet 4b841723726e40bda0187fb84bf36a6a (DEFAULT_TABLE table=default.Table_8c898cd0e2f343a6ae5debb290c20b07 [id=3d94e6f551f5474c9816d976a72a43c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:09.921905 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0de19a308a534cdaa38f46ab43789322. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:09.922744 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3d275e975e64b8d9e026e0eab5b677d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:09.923610 2188 raft_consensus.cc:515] T dabb8d760aae4aa09eb83cdb586c9788 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.924188 2188 leader_election.cc:290] T dabb8d760aae4aa09eb83cdb586c9788 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Requested vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:09.924273 883 tablet_service.cc:1505] Processing CreateTablet for tablet d48d9178108b4eb59062d29b40ba0c11 (DEFAULT_TABLE table=default.Table_0c7029666c80452b8fae6945da658ec7 [id=d862dce1eb5b4ffc886fd70e0b9ad7bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:09.924621 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d48d9178108b4eb59062d29b40ba0c11. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:09.919946 712 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:45003, user_credentials={real_user=slave}} blocked reactor thread for 175639us
W20251029 02:52:09.920512 711 leader_election.cc:343] T 946f65014d97425aab3851009b742807 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:09.925478 1172 ts_tablet_manager.cc:1403] T a9e0fcc759a64679b3f59a6c964877b6 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.001s
W20251029 02:52:09.925530 711 leader_election.cc:343] T afb19ba39399410bb90b6225d7b5b5b1 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
W20251029 02:52:09.925755 711 leader_election.cc:343] T af4860c6b72f4d988b206b5ef2f22906 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
W20251029 02:52:09.925956 711 leader_election.cc:343] T 217144957930495aa3e9dfcfde268e71 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:09.926400 1172 raft_consensus.cc:359] T a9e0fcc759a64679b3f59a6c964877b6 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:09.926604 756 tablet_service.cc:1505] Processing CreateTablet for tablet 4b841723726e40bda0187fb84bf36a6a (DEFAULT_TABLE table=default.Table_8c898cd0e2f343a6ae5debb290c20b07 [id=3d94e6f551f5474c9816d976a72a43c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:09.926646 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dabb8d760aae4aa09eb83cdb586c9788" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
I20251029 02:52:09.926791 916 raft_consensus.cc:3060] T dabb8d760aae4aa09eb83cdb586c9788 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:09.926801 893 tablet_service.cc:1505] Processing CreateTablet for tablet ca71c4638a6b419b876e22af7e6cf00a (DEFAULT_TABLE table=default.Table_31f9807cb2934a47a4fb2f67f9507053 [id=c3b6a55d9eaa4248aadb99c5ae1a9be5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:09.926982 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b841723726e40bda0187fb84bf36a6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:09.927222 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca71c4638a6b419b876e22af7e6cf00a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:09.929139 1172 raft_consensus.cc:385] T a9e0fcc759a64679b3f59a6c964877b6 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:09.929244 1172 raft_consensus.cc:740] T a9e0fcc759a64679b3f59a6c964877b6 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:09.929438 916 raft_consensus.cc:2468] T dabb8d760aae4aa09eb83cdb586c9788 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 1.
I20251029 02:52:09.929498 1172 consensus_queue.cc:260] T a9e0fcc759a64679b3f59a6c964877b6 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:09.930094 1172 ts_tablet_manager.cc:1434] T a9e0fcc759a64679b3f59a6c964877b6 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20251029 02:52:09.930437 1172 tablet_bootstrap.cc:492] T 67651306f3b44e7eb14dcee9a2007a81 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:09.930822 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99d60ef5325e45059026596a756448de. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:09.930894 990 leader_election.cc:304] T dabb8d760aae4aa09eb83cdb586c9788 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:09.931234 2188 raft_consensus.cc:2804] T dabb8d760aae4aa09eb83cdb586c9788 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:09.931357 2188 raft_consensus.cc:697] T dabb8d760aae4aa09eb83cdb586c9788 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Becoming Leader. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Running, Role: LEADER
I20251029 02:52:09.931619 2188 consensus_queue.cc:237] T dabb8d760aae4aa09eb83cdb586c9788 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
W20251029 02:52:09.933905 712 leader_election.cc:343] T 2f198c3259864f7ab3625cb7bfae20b0 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:09.934046 712 leader_election.cc:304] T 2f198c3259864f7ab3625cb7bfae20b0 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:09.934412 2113 raft_consensus.cc:2749] T 2f198c3259864f7ab3625cb7bfae20b0 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:09.934787 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb3c20822d684bea89eb358251e21e33. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:09.930123 751 tablet_service.cc:1505] Processing CreateTablet for tablet 0e8802cde9bf4784b587c93c5cf07957 (DEFAULT_TABLE table=default.Table_329849bbaa3a457096d8526f70636b96 [id=bcdff51837004f4aaa118266071b2887]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:09.937129 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e8802cde9bf4784b587c93c5cf07957. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:09.937783 878 tablet_service.cc:1505] Processing CreateTablet for tablet aacc72b167954d4fbb14ee2bfa293e6f (DEFAULT_TABLE table=default.Table_5681078bd9b54f948bd2d98511003cba [id=8c3b50bd1e434dfebe679141ceb18be8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:09.938227 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aacc72b167954d4fbb14ee2bfa293e6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:09.940660 896 tablet_service.cc:1505] Processing CreateTablet for tablet 03ab7f2c2f4343f18a7bfccb96274c64 (DEFAULT_TABLE table=default.Table_e9a962d143c94976af9285475ea40631 [id=57a3a3968dec4a7181291fa5efd51530]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:09.940984 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03ab7f2c2f4343f18a7bfccb96274c64. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:09.944118 1172 tablet_bootstrap.cc:654] T 67651306f3b44e7eb14dcee9a2007a81 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:09.948112 2302 raft_consensus.cc:493] T d61beda662d844cdaf67fd9a2e382ea2 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:09.948240 2302 raft_consensus.cc:515] T d61beda662d844cdaf67fd9a2e382ea2 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:09.948858 2302 leader_election.cc:290] T d61beda662d844cdaf67fd9a2e382ea2 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:09.949581 1058 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d61beda662d844cdaf67fd9a2e382ea2" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:09.949820 1058 raft_consensus.cc:2393] T d61beda662d844cdaf67fd9a2e382ea2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c756528642a543eeacd93cef3e335a7d in current term 1: Already voted for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in this term.
W20251029 02:52:09.951174 847 leader_election.cc:343] T d61beda662d844cdaf67fd9a2e382ea2 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: d61beda662d844cdaf67fd9a2e382ea2
I20251029 02:52:09.951295 847 leader_election.cc:304] T d61beda662d844cdaf67fd9a2e382ea2 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9
I20251029 02:52:09.951504 750 tablet_service.cc:1552] Processing DeleteTablet for tablet 59eee86f249e4d6a9d09d30276dcd659 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:56 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:09.951678 2302 raft_consensus.cc:3060] T d61beda662d844cdaf67fd9a2e382ea2 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:09.952111 747 tablet_service.cc:1505] Processing CreateTablet for tablet 6c4f28b6f4a94d35bb036843aa3bc4bc (DEFAULT_TABLE table=default.Table_6c1f042a9840467993a38a8e55a23cbb [id=29c45a9bc52e4cdabd41ab4220c2b40b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:09.952458 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c4f28b6f4a94d35bb036843aa3bc4bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:09.954165 2302 raft_consensus.cc:2749] T d61beda662d844cdaf67fd9a2e382ea2 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:09.954909 748 tablet_service.cc:1505] Processing CreateTablet for tablet aacc72b167954d4fbb14ee2bfa293e6f (DEFAULT_TABLE table=default.Table_5681078bd9b54f948bd2d98511003cba [id=8c3b50bd1e434dfebe679141ceb18be8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:09.955268 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aacc72b167954d4fbb14ee2bfa293e6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:09.955415 756 tablet_service.cc:1505] Processing CreateTablet for tablet 62b1c6d828664f02acc4a3e6dfb5a578 (DEFAULT_TABLE table=default.Table_dbb86af86c304f02ae3a9ece7bb4998b [id=00f1239c6da24a7abd15e121a8ff838d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:09.957386 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62b1c6d828664f02acc4a3e6dfb5a578. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:09.961760 749 tablet_service.cc:1505] Processing CreateTablet for tablet 99d60ef5325e45059026596a756448de (DEFAULT_TABLE table=default.Table_aec07800b35948bba7db9168286fcf14 [id=7147b693c86e4c258fecdadfbe5c29f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:09.962186 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99d60ef5325e45059026596a756448de. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:09.965632 756 tablet_service.cc:1505] Processing CreateTablet for tablet 03ab7f2c2f4343f18a7bfccb96274c64 (DEFAULT_TABLE table=default.Table_e9a962d143c94976af9285475ea40631 [id=57a3a3968dec4a7181291fa5efd51530]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:09.966133 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03ab7f2c2f4343f18a7bfccb96274c64. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:09.966800 883 tablet_service.cc:1505] Processing CreateTablet for tablet 62b1c6d828664f02acc4a3e6dfb5a578 (DEFAULT_TABLE table=default.Table_dbb86af86c304f02ae3a9ece7bb4998b [id=00f1239c6da24a7abd15e121a8ff838d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:09.967209 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62b1c6d828664f02acc4a3e6dfb5a578. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:09.969336 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4a8f0ad6faa4cf58f48156087392d3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:09.970029 878 tablet_service.cc:1505] Processing CreateTablet for tablet 96f7876e5d6d4a0db236df5e7f266fd2 (DEFAULT_TABLE table=default.Table_aaecc1c9efd04549a2484850ff8efc60 [id=00da2e21da8f4d54ae86834777691781]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:09.970369 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96f7876e5d6d4a0db236df5e7f266fd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:09.970786 750 tablet_service.cc:1505] Processing CreateTablet for tablet ca71c4638a6b419b876e22af7e6cf00a (DEFAULT_TABLE table=default.Table_31f9807cb2934a47a4fb2f67f9507053 [id=c3b6a55d9eaa4248aadb99c5ae1a9be5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:09.971160 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca71c4638a6b419b876e22af7e6cf00a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:09.974834 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c4f28b6f4a94d35bb036843aa3bc4bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:09.977465 742 tablet_service.cc:1505] Processing CreateTablet for tablet f4a8f0ad6faa4cf58f48156087392d3c (DEFAULT_TABLE table=default.Table_c6e9611c339e47a8b5021b69c449f141 [id=c19d4543d4ce43a780612d2d8653d496]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:09.977882 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4a8f0ad6faa4cf58f48156087392d3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:09.981420 743 tablet_service.cc:1505] Processing CreateTablet for tablet 85bcb1fe0cb54b5aad3e57a812e6d802 (DEFAULT_TABLE table=default.Table_f8f86e0f1d8f4ba98ba26b268e26eee2 [id=76b0df830a7f41a9b6091ca2f1b4edbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:09.982306 892 tablet_service.cc:1505] Processing CreateTablet for tablet 51a6e45378344f86847a44cf4950307b (DEFAULT_TABLE table=default.Table_13f9fb5682214cc28f12dce88109d831 [id=0816dc14afa8435e80fdbaa4959c985a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:09.982133 754 tablet_service.cc:1505] Processing CreateTablet for tablet 51a6e45378344f86847a44cf4950307b (DEFAULT_TABLE table=default.Table_13f9fb5682214cc28f12dce88109d831 [id=0816dc14afa8435e80fdbaa4959c985a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:09.982980 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51a6e45378344f86847a44cf4950307b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:09.986066 893 tablet_service.cc:1505] Processing CreateTablet for tablet 971b95ab982141e184a314e7eb5e4a6a (DEFAULT_TABLE table=default.Table_df78a989ef8e4c48b1716f5698c6ecf9 [id=5bd6c6f6062a42438261b3d26ea313e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:09.982537 740 tablet_service.cc:1505] Processing CreateTablet for tablet 971b95ab982141e184a314e7eb5e4a6a (DEFAULT_TABLE table=default.Table_df78a989ef8e4c48b1716f5698c6ecf9 [id=5bd6c6f6062a42438261b3d26ea313e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:09.986778 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 971b95ab982141e184a314e7eb5e4a6a. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:09.988916 519 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8c972c88f64e486bb2b75306549bb23f consensus state has no leader
I20251029 02:52:09.990805 878 tablet_service.cc:1505] Processing CreateTablet for tablet 9efc87a35b96414fa38d7e849185278c (DEFAULT_TABLE table=default.Table_957971e954a244309b62d05bb172dc9d [id=2ab2cb2eabc545ec84049542f7e19eb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:09.992689 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_9791da4fa0ac47409f6cb424408a5f3b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:09.993364 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9791da4fa0ac47409f6cb424408a5f3b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:09.996093 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b841723726e40bda0187fb84bf36a6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:09.996683 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85bcb1fe0cb54b5aad3e57a812e6d802. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:09.997144 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3ede523203545d18b09d535fdcabb6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:09.997552 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e8802cde9bf4784b587c93c5cf07957. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:09.998071 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51a6e45378344f86847a44cf4950307b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:09.998621 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 971b95ab982141e184a314e7eb5e4a6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.002482 748 tablet_service.cc:1505] Processing CreateTablet for tablet 96f7876e5d6d4a0db236df5e7f266fd2 (DEFAULT_TABLE table=default.Table_aaecc1c9efd04549a2484850ff8efc60 [id=00da2e21da8f4d54ae86834777691781]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.002957 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96f7876e5d6d4a0db236df5e7f266fd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.004657 881 tablet_service.cc:1505] Processing CreateTablet for tablet 5950a2dcdcd3440b873cd53d070ca708 (DEFAULT_TABLE table=default.Table_9af081e1d784401792ad8964bdad2e73 [id=8a9e4bde637c48738015985b4461165a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.006531 740 tablet_service.cc:1505] Processing CreateTablet for tablet 5950a2dcdcd3440b873cd53d070ca708 (DEFAULT_TABLE table=default.Table_9af081e1d784401792ad8964bdad2e73 [id=8a9e4bde637c48738015985b4461165a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.006531 756 tablet_service.cc:1505] Processing CreateTablet for tablet 9efc87a35b96414fa38d7e849185278c (DEFAULT_TABLE table=default.Table_957971e954a244309b62d05bb172dc9d [id=2ab2cb2eabc545ec84049542f7e19eb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.007010 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5950a2dcdcd3440b873cd53d070ca708. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.007328 755 tablet_service.cc:1505] Processing CreateTablet for tablet d48d9178108b4eb59062d29b40ba0c11 (DEFAULT_TABLE table=default.Table_0c7029666c80452b8fae6945da658ec7 [id=d862dce1eb5b4ffc886fd70e0b9ad7bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.007676 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d48d9178108b4eb59062d29b40ba0c11. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.010751 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85bcb1fe0cb54b5aad3e57a812e6d802. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.010841 1176 ts_tablet_manager.cc:1929] T 86fc22bb84794149b598b97df41eb881 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:10.011123 1176 log.cc:1199] T 86fc22bb84794149b598b97df41eb881 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/86fc22bb84794149b598b97df41eb881
I20251029 02:52:10.011709 1176 ts_tablet_manager.cc:1950] T 86fc22bb84794149b598b97df41eb881 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:10.012214 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 878) took 1482 ms. Trace:
I20251029 02:52:10.012413 1176 rpcz_store.cc:276] 1029 02:52:08.529799 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:08.529843 (+ 44us) service_pool.cc:225] Handling call
1029 02:52:09.647891 (+1118048us) ts_tablet_manager.cc:1151] Deleting tablet 86fc22bb84794149b598b97df41eb881
1029 02:52:09.647901 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:09.731583 (+ 83682us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:10.010707 (+279124us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:10.012147 (+ 1440us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1117874}
I20251029 02:52:10.012992 1176 tablet_replica.cc:333] T 7a1866afc7564217ad777e7ce53786c5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:10.013278 1176 raft_consensus.cc:2243] T 7a1866afc7564217ad777e7ce53786c5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:09.895376 745 tablet_service.cc:1505] Processing CreateTablet for tablet 03f59f9206024dfbaa4e67a5b6bfb902 (DEFAULT_TABLE table=default.Table_95f035fda47144eb9d79be23614b41b6 [id=38cb7bf11be54631ba227c9fcd3b1aa4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.013897 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03f59f9206024dfbaa4e67a5b6bfb902. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.013651 1176 pending_rounds.cc:70] T 7a1866afc7564217ad777e7ce53786c5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Trying to abort 1 pending ops.
I20251029 02:52:10.014019 1176 pending_rounds.cc:77] T 7a1866afc7564217ad777e7ce53786c5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215949121667424256 op_type: NO_OP noop_request { }
I20251029 02:52:10.014195 1176 raft_consensus.cc:2889] T 7a1866afc7564217ad777e7ce53786c5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 02:52:10.014323 1176 raft_consensus.cc:2272] T 7a1866afc7564217ad777e7ce53786c5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:10.014906 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9efc87a35b96414fa38d7e849185278c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.015089 1176 ts_tablet_manager.cc:1916] T 7a1866afc7564217ad777e7ce53786c5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:10.021711 754 tablet_service.cc:1505] Processing CreateTablet for tablet 8378d67f1da743cca91803e37ea54281 (DEFAULT_TABLE table=default.Table_fdb0a130609a45878b3a6f07cf79947c [id=c46a973298994dc4a994905220615830]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.022157 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8378d67f1da743cca91803e37ea54281. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.024919 521 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 23856678b7aa4f9ebf65fb4564a4bf91
I20251029 02:52:10.025493 521 catalog_manager.cc:6719] ReplaceTablet: tablet 23856678b7aa4f9ebf65fb4564a4bf91 deleted and replaced by tablet 7fcb1aac63c84edba0a8b8c964a8b957
W20251029 02:52:10.026968 513 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 8c972c88f64e486bb2b75306549bb23f (table default.Table_3c3bcfdfd5914e7b892a8af9aaaac175 [id=c4e7234e2c3f43e6ab5f972e2cbd7785], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 8c972c88f64e486bb2b75306549bb23f consensus state has no leader
I20251029 02:52:10.028455 896 tablet_service.cc:1505] Processing CreateTablet for tablet 6acef3dd9c1b464ba33e11cdd66f1196 (DEFAULT_TABLE table=default.Table_954aab60929348dcbdbd69aa987112ac [id=a91ae536f3274f96bd6a78cb1dbedea4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.029167 521 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet a9e0fcc759a64679b3f59a6c964877b6 (table default.Table_9dce7f70d51d4a3c8d7f98a2435cbce3 [id=657afc87656d4250b4ca87b473fbb752]). Expected version 3 got 0
I20251029 02:52:10.030138 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_b68e55dfa3d04c8d95c29f27537f59ad"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:10.030759 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b68e55dfa3d04c8d95c29f27537f59ad in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:10.031158 2302 raft_consensus.cc:493] T 49d9ce32e4f04f11be2ecefc5d523a8b P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:10.031288 2302 raft_consensus.cc:515] T 49d9ce32e4f04f11be2ecefc5d523a8b P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:10.031844 2302 leader_election.cc:290] T 49d9ce32e4f04f11be2ecefc5d523a8b P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:10.032366 898 tablet_service.cc:1552] Processing DeleteTablet for tablet 23856678b7aa4f9ebf65fb4564a4bf91 with delete_type TABLET_DATA_DELETED (replaced by tablet 7fcb1aac63c84edba0a8b8c964a8b957) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:10.032666 1058 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "49d9ce32e4f04f11be2ecefc5d523a8b" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:10.032888 1058 raft_consensus.cc:2468] T 49d9ce32e4f04f11be2ecefc5d523a8b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:10.033551 849 leader_election.cc:304] T 49d9ce32e4f04f11be2ecefc5d523a8b P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:10.033876 520 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 43a76b984da3414da701221f87349337
W20251029 02:52:10.034024 847 leader_election.cc:343] T 49d9ce32e4f04f11be2ecefc5d523a8b P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 49d9ce32e4f04f11be2ecefc5d523a8b
I20251029 02:52:10.034325 2302 raft_consensus.cc:2804] T 49d9ce32e4f04f11be2ecefc5d523a8b P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:10.034402 2302 raft_consensus.cc:493] T 49d9ce32e4f04f11be2ecefc5d523a8b P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:10.034469 2302 raft_consensus.cc:3060] T 49d9ce32e4f04f11be2ecefc5d523a8b P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:10.034772 520 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_90a0cc2204ac4322861cf1f52d9f2a5b" } new_table_name: "default.Table_1fe2eb342d984f98ae67e78d577827e8" modify_external_catalogs: true
I20251029 02:52:10.037357 2302 raft_consensus.cc:515] T 49d9ce32e4f04f11be2ecefc5d523a8b P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:10.037439 884 tablet_service.cc:1505] Processing CreateTablet for tablet 47348ffa21574e4084f0b725a78e2a70 (DEFAULT_TABLE table=default.Table_2aba26f0828343a6981aad6b2e3ac97f [id=a53e996da85d45b7a90ef1a33a74c094]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.038192 898 tablet_service.cc:1552] Processing DeleteTablet for tablet 43a76b984da3414da701221f87349337 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:09 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:10.038488 2306 raft_consensus.cc:493] T e7f846b294cc4eb687a26c8b5931ac19 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:10.038698 2306 raft_consensus.cc:515] T e7f846b294cc4eb687a26c8b5931ac19 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:10.039202 885 tablet_service.cc:1505] Processing CreateTablet for tablet fa525da8037f427592b5d0858ba9a815 (DEFAULT_TABLE table=default.Table_4d99c72b75124b20bf00ff0d75e7aada [id=80f9cf1d5452497cbf5fcd9d8981a8fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.039204 1058 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "49d9ce32e4f04f11be2ecefc5d523a8b" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:52:10.039366 2306 leader_election.cc:290] T e7f846b294cc4eb687a26c8b5931ac19 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:10.039409 1058 raft_consensus.cc:3060] T 49d9ce32e4f04f11be2ecefc5d523a8b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:10.039901 1167 tablet_bootstrap.cc:492] T 30d03e1e42d34242bd354f9e9ec4f127 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:10.040010 1167 ts_tablet_manager.cc:1403] T 30d03e1e42d34242bd354f9e9ec4f127 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.239s user 0.010s sys 0.010s
I20251029 02:52:10.040117 1166 tablet_bootstrap.cc:492] T 2f198c3259864f7ab3625cb7bfae20b0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:10.040313 1166 ts_tablet_manager.cc:1403] T 2f198c3259864f7ab3625cb7bfae20b0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.266s user 0.019s sys 0.047s
I20251029 02:52:10.040760 1167 raft_consensus.cc:359] T 30d03e1e42d34242bd354f9e9ec4f127 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:10.040922 1167 raft_consensus.cc:385] T 30d03e1e42d34242bd354f9e9ec4f127 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:10.040970 1167 raft_consensus.cc:740] T 30d03e1e42d34242bd354f9e9ec4f127 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:10.041148 1167 consensus_queue.cc:260] T 30d03e1e42d34242bd354f9e9ec4f127 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:10.041262 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet dfa8b1b9a80e4e8689e0acb44e16346a with delete_type TABLET_DATA_DELETED (replaced by tablet 97b2ecae572a4b7ea1b9595074556708) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:10.041226 1166 raft_consensus.cc:359] T 2f198c3259864f7ab3625cb7bfae20b0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:10.041422 1166 raft_consensus.cc:385] T 2f198c3259864f7ab3625cb7bfae20b0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:10.041497 1166 raft_consensus.cc:740] T 2f198c3259864f7ab3625cb7bfae20b0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:10.041661 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet 0d84dc8dc465496f9adec798b2e03e13 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:56 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:10.041752 1166 consensus_queue.cc:260] T 2f198c3259864f7ab3625cb7bfae20b0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:10.042099 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet 59d03a190b784ab18a3e63a572675891 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:57 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:10.042276 1166 ts_tablet_manager.cc:1434] T 2f198c3259864f7ab3625cb7bfae20b0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 02:52:10.042377 1058 raft_consensus.cc:2468] T 49d9ce32e4f04f11be2ecefc5d523a8b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:10.042423 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet 49d9ce32e4f04f11be2ecefc5d523a8b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:57 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:10.042757 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet 5815626b05204b578cc4a110f246bf10 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:08 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:10.043031 1166 tablet_bootstrap.cc:492] T adf6e736cb9c4920807ee58682e134ef P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
W20251029 02:52:10.043418 847 leader_election.cc:343] T 49d9ce32e4f04f11be2ecefc5d523a8b P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 49d9ce32e4f04f11be2ecefc5d523a8b
I20251029 02:52:10.043581 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet ac162bce5b6e4080839095a045ef0cc0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:08 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:10.043651 2302 leader_election.cc:290] T 49d9ce32e4f04f11be2ecefc5d523a8b P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:10.044157 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet 591a6c86fc984bac9fb9eb62e35dc6c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:09 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:10.044497 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet 1c53fc1fff1a4bb988beb46ff91fd3be with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:09 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:10.044991 737 tablet_service.cc:1552] Processing DeleteTablet for tablet 23856678b7aa4f9ebf65fb4564a4bf91 with delete_type TABLET_DATA_DELETED (replaced by tablet 7fcb1aac63c84edba0a8b8c964a8b957) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:10.045470 744 tablet_service.cc:1505] Processing CreateTablet for tablet 6acef3dd9c1b464ba33e11cdd66f1196 (DEFAULT_TABLE table=default.Table_954aab60929348dcbdbd69aa987112ac [id=a91ae536f3274f96bd6a78cb1dbedea4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.045677 1029 tablet_service.cc:1505] Processing CreateTablet for tablet 3b8bb5025de24e8e95d13f04c62305e1 (DEFAULT_TABLE table=default.Table_f060d401cf984a96b90cf4735c9f34d2 [id=8c9724de88084668b15b37015c668435]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.045886 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6acef3dd9c1b464ba33e11cdd66f1196. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.046548 1032 tablet_service.cc:1505] Processing CreateTablet for tablet 03f59f9206024dfbaa4e67a5b6bfb902 (DEFAULT_TABLE table=default.Table_95f035fda47144eb9d79be23614b41b6 [id=38cb7bf11be54631ba227c9fcd3b1aa4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.050691 739 tablet_service.cc:1552] Processing DeleteTablet for tablet 43a76b984da3414da701221f87349337 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:09 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:10.051364 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e7f846b294cc4eb687a26c8b5931ac19" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:10.051988 990 leader_election.cc:343] T e7f846b294cc4eb687a26c8b5931ac19 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:10.052104 849 leader_election.cc:304] T 49d9ce32e4f04f11be2ecefc5d523a8b P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters: b495a49349324f7d81908f2da700f0d9
I20251029 02:52:10.052767 2302 raft_consensus.cc:2804] T 49d9ce32e4f04f11be2ecefc5d523a8b P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:10.052947 2302 raft_consensus.cc:697] T 49d9ce32e4f04f11be2ecefc5d523a8b P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:52:10.053989 2302 consensus_queue.cc:237] T 49d9ce32e4f04f11be2ecefc5d523a8b P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:10.053767 1167 ts_tablet_manager.cc:1434] T 30d03e1e42d34242bd354f9e9ec4f127 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.014s user 0.001s sys 0.001s
I20251029 02:52:10.055142 1167 tablet_bootstrap.cc:492] T 59d03a190b784ab18a3e63a572675891 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:10.057789 1167 tablet_bootstrap.cc:654] T 59d03a190b784ab18a3e63a572675891 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:10.058965 740 tablet_service.cc:1505] Processing CreateTablet for tablet fa525da8037f427592b5d0858ba9a815 (DEFAULT_TABLE table=default.Table_4d99c72b75124b20bf00ff0d75e7aada [id=80f9cf1d5452497cbf5fcd9d8981a8fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.059767 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa525da8037f427592b5d0858ba9a815. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.068851 2301 raft_consensus.cc:493] T 5db519edc75d4bd0985b32322825ffae P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:10.069089 2301 raft_consensus.cc:515] T 5db519edc75d4bd0985b32322825ffae P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:10.069934 2301 leader_election.cc:290] T 5db519edc75d4bd0985b32322825ffae P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
W20251029 02:52:10.071653 847 leader_election.cc:343] T 5db519edc75d4bd0985b32322825ffae P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 5db519edc75d4bd0985b32322825ffae
W20251029 02:52:10.072369 849 leader_election.cc:343] T 5db519edc75d4bd0985b32322825ffae P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Not found: Tablet not found: 5db519edc75d4bd0985b32322825ffae
I20251029 02:52:10.072546 849 leader_election.cc:304] T 5db519edc75d4bd0985b32322825ffae P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9
I20251029 02:52:10.073029 2301 raft_consensus.cc:2749] T 5db519edc75d4bd0985b32322825ffae P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:10.074139 888 tablet_service.cc:1505] Processing CreateTablet for tablet a9e356c6846f47e68ba5fdfa7befc1cf (DEFAULT_TABLE table=default.Table_2577a8048e3345ce82eafa3657784d8b [id=98af8d8aca43483f88bf88d9962862ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.074705 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9e356c6846f47e68ba5fdfa7befc1cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.074891 895 tablet_service.cc:1505] Processing CreateTablet for tablet dab6ed2266ed42508738399d388c4da3 (DEFAULT_TABLE table=default.Table_4e9d94609b4a4d1f94615dcc9830d717 [id=761ebdd0095c414488aa221cbc99d3bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.075454 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dab6ed2266ed42508738399d388c4da3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.080112 739 tablet_service.cc:1505] Processing CreateTablet for tablet dab6ed2266ed42508738399d388c4da3 (DEFAULT_TABLE table=default.Table_4e9d94609b4a4d1f94615dcc9830d717 [id=761ebdd0095c414488aa221cbc99d3bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.080413 756 tablet_service.cc:1505] Processing CreateTablet for tablet 47348ffa21574e4084f0b725a78e2a70 (DEFAULT_TABLE table=default.Table_2aba26f0828343a6981aad6b2e3ac97f [id=a53e996da85d45b7a90ef1a33a74c094]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.080529 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dab6ed2266ed42508738399d388c4da3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.080744 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47348ffa21574e4084f0b725a78e2a70. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.084213 894 tablet_service.cc:1505] Processing CreateTablet for tablet 13670ea867e349b2b5a25021469c32d3 (DEFAULT_TABLE table=default.Table_da047830460443a29c5269c98fc859e9 [id=f733948e92d8470da9d8050bd26a06b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.084645 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13670ea867e349b2b5a25021469c32d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.080112 744 tablet_service.cc:1505] Processing CreateTablet for tablet 41cfa2d722034618a05de717b28a778d (DEFAULT_TABLE table=default.Table_ffc17029e0734c73a3735b9f99cf1332 [id=0c8e5e5ec31c43bdbc827be824b65780]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.085415 898 tablet_service.cc:1505] Processing CreateTablet for tablet 41cfa2d722034618a05de717b28a778d (DEFAULT_TABLE table=default.Table_ffc17029e0734c73a3735b9f99cf1332 [id=0c8e5e5ec31c43bdbc827be824b65780]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.085687 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41cfa2d722034618a05de717b28a778d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.086145 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41cfa2d722034618a05de717b28a778d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.089967 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6acef3dd9c1b464ba33e11cdd66f1196. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.092396 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5950a2dcdcd3440b873cd53d070ca708. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.093736 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47348ffa21574e4084f0b725a78e2a70. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.099380 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa525da8037f427592b5d0858ba9a815. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.104668 739 tablet_service.cc:1505] Processing CreateTablet for tablet a9e356c6846f47e68ba5fdfa7befc1cf (DEFAULT_TABLE table=default.Table_2577a8048e3345ce82eafa3657784d8b [id=98af8d8aca43483f88bf88d9962862ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.105139 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9e356c6846f47e68ba5fdfa7befc1cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.106387 882 tablet_service.cc:1505] Processing CreateTablet for tablet eacad0b8458242efb48cc17351fb3f2f (DEFAULT_TABLE table=default.Table_3bc0c202b81d42c5a788393fe07d02a1 [id=72e5e134d7a54f5fb76630982ae948c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.106793 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eacad0b8458242efb48cc17351fb3f2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.112967 1167 tablet_bootstrap.cc:492] T 59d03a190b784ab18a3e63a572675891 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:10.113236 1167 ts_tablet_manager.cc:1403] T 59d03a190b784ab18a3e63a572675891 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.058s user 0.004s sys 0.000s
I20251029 02:52:10.114676 1167 raft_consensus.cc:359] T 59d03a190b784ab18a3e63a572675891 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:10.114892 1167 raft_consensus.cc:385] T 59d03a190b784ab18a3e63a572675891 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:10.114995 1167 raft_consensus.cc:740] T 59d03a190b784ab18a3e63a572675891 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:10.106359 2309 raft_consensus.cc:493] T 71b4b91071a54d6aa66160e7955bc915 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:10.117233 2306 raft_consensus.cc:493] T 30d03e1e42d34242bd354f9e9ec4f127 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:10.117383 2306 raft_consensus.cc:515] T 30d03e1e42d34242bd354f9e9ec4f127 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:10.118023 2306 leader_election.cc:290] T 30d03e1e42d34242bd354f9e9ec4f127 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:10.118707 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30d03e1e42d34242bd354f9e9ec4f127" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:10.118892 917 raft_consensus.cc:2468] T 30d03e1e42d34242bd354f9e9ec4f127 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 0.
I20251029 02:52:10.116684 2309 raft_consensus.cc:515] T 71b4b91071a54d6aa66160e7955bc915 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:10.123760 990 leader_election.cc:304] T 30d03e1e42d34242bd354f9e9ec4f127 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:10.124300 2309 leader_election.cc:290] T 71b4b91071a54d6aa66160e7955bc915 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:10.124423 2309 raft_consensus.cc:493] T d041e87ceb11441983d0e38b7929c3a4 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:10.124533 2309 raft_consensus.cc:515] T d041e87ceb11441983d0e38b7929c3a4 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:10.124616 2306 raft_consensus.cc:2804] T 30d03e1e42d34242bd354f9e9ec4f127 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:10.124724 2306 raft_consensus.cc:493] T 30d03e1e42d34242bd354f9e9ec4f127 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:10.124815 2306 raft_consensus.cc:3060] T 30d03e1e42d34242bd354f9e9ec4f127 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:10.125070 2309 leader_election.cc:290] T d041e87ceb11441983d0e38b7929c3a4 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:10.127635 2306 raft_consensus.cc:515] T 30d03e1e42d34242bd354f9e9ec4f127 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:10.128156 2306 leader_election.cc:290] T 30d03e1e42d34242bd354f9e9ec4f127 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Requested vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:10.129657 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30d03e1e42d34242bd354f9e9ec4f127" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
I20251029 02:52:10.129837 917 raft_consensus.cc:3060] T 30d03e1e42d34242bd354f9e9ec4f127 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:10.132261 2301 consensus_queue.cc:1048] T 1bdb67125493430398ff5c80b73de2ac P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:10.139570 919 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "71b4b91071a54d6aa66160e7955bc915" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:10.139624 918 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d041e87ceb11441983d0e38b7929c3a4" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:10.142906 917 raft_consensus.cc:2468] T 30d03e1e42d34242bd354f9e9ec4f127 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 1.
I20251029 02:52:10.143734 990 leader_election.cc:304] T 30d03e1e42d34242bd354f9e9ec4f127 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:10.148741 523 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 4660fd0534df49e0af43c36fb1122d84
I20251029 02:52:10.150646 2306 raft_consensus.cc:2804] T 30d03e1e42d34242bd354f9e9ec4f127 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:10.150815 2306 raft_consensus.cc:697] T 30d03e1e42d34242bd354f9e9ec4f127 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Becoming Leader. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Running, Role: LEADER
I20251029 02:52:10.153071 2306 consensus_queue.cc:237] T 30d03e1e42d34242bd354f9e9ec4f127 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:10.154399 882 tablet_service.cc:1552] Processing DeleteTablet for tablet 4660fd0534df49e0af43c36fb1122d84 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:09 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:10.155808 1167 consensus_queue.cc:260] T 59d03a190b784ab18a3e63a572675891 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:10.156015 2188 raft_consensus.cc:493] T 217144957930495aa3e9dfcfde268e71 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:10.156199 2188 raft_consensus.cc:515] T 217144957930495aa3e9dfcfde268e71 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:10.156450 1167 ts_tablet_manager.cc:1434] T 59d03a190b784ab18a3e63a572675891 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.043s user 0.001s sys 0.001s
I20251029 02:52:10.156746 2188 leader_election.cc:290] T 217144957930495aa3e9dfcfde268e71 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:10.158679 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "217144957930495aa3e9dfcfde268e71" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:10.159116 990 leader_election.cc:343] T 217144957930495aa3e9dfcfde268e71 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:10.163345 754 tablet_service.cc:1505] Processing CreateTablet for tablet eacad0b8458242efb48cc17351fb3f2f (DEFAULT_TABLE table=default.Table_3bc0c202b81d42c5a788393fe07d02a1 [id=72e5e134d7a54f5fb76630982ae948c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.164219 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_4a6906297ac74f45b5f153339e2713c3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:10.165017 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4a6906297ac74f45b5f153339e2713c3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:10.142999 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9efc87a35b96414fa38d7e849185278c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.166949 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eacad0b8458242efb48cc17351fb3f2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.168877 1025 tablet_service.cc:1505] Processing CreateTablet for tablet 93440d660eac4986aeb17abc18144700 (DEFAULT_TABLE table=default.Table_126f4f49037f46628fa5e1d33b4fcf24 [id=41207bfd4192414598f5517feffe3351]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.170147 1039 tablet_service.cc:1505] Processing CreateTablet for tablet 82f83631253a4b5785c814c993678d68 (DEFAULT_TABLE table=default.Table_47178701cd044f8fa5357cd607154988 [id=a0b7f2de597f4af3945a7d41a365acc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.171432 1028 tablet_service.cc:1505] Processing CreateTablet for tablet b79327b8c1354810a8b1b0d9e62fe3c8 (DEFAULT_TABLE table=default.Table_ff347dc490d441baa678753aa33b710b [id=17e273128f4c47daa4faf2e7a71b532a]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 02:52:10.167585 711 leader_election.cc:343] T 71b4b91071a54d6aa66160e7955bc915 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
W20251029 02:52:10.171969 711 leader_election.cc:343] T d041e87ceb11441983d0e38b7929c3a4 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:10.172629 2302 consensus_queue.cc:1048] T 1bdb67125493430398ff5c80b73de2ac P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:10.172772 1037 tablet_service.cc:1505] Processing CreateTablet for tablet e1d00ea3dd3744ffa5258bbe0095851c (DEFAULT_TABLE table=default.Table_cfbb812339124da1b8e6e8a95d816a5e [id=b834fcfacd4e4384bb684d5f3ea671e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.173985 1031 tablet_service.cc:1505] Processing CreateTablet for tablet cb9856e5fe244d3e91f30f5e1d276d18 (DEFAULT_TABLE table=default.Table_482767bf59bf442a99a8675f678afd06 [id=3c6e732a6c4347ada8cb3d4cb02835cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.176209 2302 raft_consensus.cc:493] T caef1c93e33a4390bcada59a41f43465 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:10.176422 2302 raft_consensus.cc:515] T caef1c93e33a4390bcada59a41f43465 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:10.175887 900 tablet_service.cc:1505] Processing CreateTablet for tablet 8378d67f1da743cca91803e37ea54281 (DEFAULT_TABLE table=default.Table_fdb0a130609a45878b3a6f07cf79947c [id=c46a973298994dc4a994905220615830]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.178071 2302 leader_election.cc:290] T caef1c93e33a4390bcada59a41f43465 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
W20251029 02:52:10.178469 849 leader_election.cc:343] T caef1c93e33a4390bcada59a41f43465 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Not found: Tablet not found: caef1c93e33a4390bcada59a41f43465
W20251029 02:52:10.179550 847 leader_election.cc:343] T caef1c93e33a4390bcada59a41f43465 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: caef1c93e33a4390bcada59a41f43465
I20251029 02:52:10.179677 847 leader_election.cc:304] T caef1c93e33a4390bcada59a41f43465 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9
I20251029 02:52:10.180164 2302 raft_consensus.cc:2749] T caef1c93e33a4390bcada59a41f43465 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:10.180856 743 tablet_service.cc:1552] Processing DeleteTablet for tablet 4660fd0534df49e0af43c36fb1122d84 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:09 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:10.186882 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cb9856e5fe244d3e91f30f5e1d276d18. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.187132 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b79327b8c1354810a8b1b0d9e62fe3c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.190723 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93440d660eac4986aeb17abc18144700. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.192601 1167 tablet_bootstrap.cc:492] T 738046ab57334f2dab9f4aab979c5b42 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:10.192616 1166 tablet_bootstrap.cc:654] T adf6e736cb9c4920807ee58682e134ef P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:10.194728 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82f83631253a4b5785c814c993678d68. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.194861 1167 tablet_bootstrap.cc:654] T 738046ab57334f2dab9f4aab979c5b42 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:10.195937 749 tablet_service.cc:1505] Processing CreateTablet for tablet 13670ea867e349b2b5a25021469c32d3 (DEFAULT_TABLE table=default.Table_da047830460443a29c5269c98fc859e9 [id=f733948e92d8470da9d8050bd26a06b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.198800 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1d00ea3dd3744ffa5258bbe0095851c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.199891 740 tablet_service.cc:1505] Processing CreateTablet for tablet 41a171c7ba55422b8700c4f898dc7d9e (DEFAULT_TABLE table=default.Table_6442d40b6bad4ac9a2dbd42a8a3cb096 [id=e530a85da85d4d8a96be6097e31fde58]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.201081 1172 tablet_bootstrap.cc:492] T 67651306f3b44e7eb14dcee9a2007a81 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:10.201221 1172 ts_tablet_manager.cc:1403] T 67651306f3b44e7eb14dcee9a2007a81 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.271s user 0.035s sys 0.056s
I20251029 02:52:10.203352 1172 raft_consensus.cc:359] T 67651306f3b44e7eb14dcee9a2007a81 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:10.203526 1172 raft_consensus.cc:385] T 67651306f3b44e7eb14dcee9a2007a81 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:10.203581 1172 raft_consensus.cc:740] T 67651306f3b44e7eb14dcee9a2007a81 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:10.203779 1172 consensus_queue.cc:260] T 67651306f3b44e7eb14dcee9a2007a81 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:10.204311 1172 ts_tablet_manager.cc:1434] T 67651306f3b44e7eb14dcee9a2007a81 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251029 02:52:10.213657 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b8bb5025de24e8e95d13f04c62305e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.218729 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03f59f9206024dfbaa4e67a5b6bfb902. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.223217 1172 tablet_bootstrap.cc:492] T f281191e7c6143f983f11d87f1c9b64e P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:10.224604 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13670ea867e349b2b5a25021469c32d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.224927 1172 tablet_bootstrap.cc:654] T f281191e7c6143f983f11d87f1c9b64e P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:10.231647 1172 tablet_bootstrap.cc:492] T f281191e7c6143f983f11d87f1c9b64e P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:10.231794 1172 ts_tablet_manager.cc:1403] T f281191e7c6143f983f11d87f1c9b64e P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.002s
I20251029 02:52:10.232648 1172 raft_consensus.cc:359] T f281191e7c6143f983f11d87f1c9b64e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:10.232837 1172 raft_consensus.cc:385] T f281191e7c6143f983f11d87f1c9b64e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:10.232899 1172 raft_consensus.cc:740] T f281191e7c6143f983f11d87f1c9b64e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:10.233103 1172 consensus_queue.cc:260] T f281191e7c6143f983f11d87f1c9b64e P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:10.233541 1172 ts_tablet_manager.cc:1434] T f281191e7c6143f983f11d87f1c9b64e P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251029 02:52:10.235383 754 tablet_service.cc:1505] Processing CreateTablet for tablet 0d788aeac52047c6bd9edcaf1369c35c (DEFAULT_TABLE table=default.Table_f7fa4c862f364b6eb8510a09d693001b [id=2568391dc7c646eaaa5adcc7627d785e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.237569 884 tablet_service.cc:1505] Processing CreateTablet for tablet 0d788aeac52047c6bd9edcaf1369c35c (DEFAULT_TABLE table=default.Table_f7fa4c862f364b6eb8510a09d693001b [id=2568391dc7c646eaaa5adcc7627d785e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.238282 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d788aeac52047c6bd9edcaf1369c35c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.250082 524 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet eeb75fc46e2a41f78252868ad5256966
I20251029 02:52:10.250787 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_af97a1e008a94a86970089216d02b3d4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:10.251003 744 tablet_service.cc:1552] Processing DeleteTablet for tablet eeb75fc46e2a41f78252868ad5256966 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:09 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:10.251807 526 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_d497c064fe99434faf6566e6c5b6836c" } new_table_name: "default.Table_fb8f72b1a3064359b44394739cfd3bcb" modify_external_catalogs: true
W20251029 02:52:10.253497 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_af97a1e008a94a86970089216d02b3d4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:10.253736 881 tablet_service.cc:1552] Processing DeleteTablet for tablet eeb75fc46e2a41f78252868ad5256966 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:09 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:10.255050 1172 tablet_bootstrap.cc:492] T 378f549ff7364d018d8b9a5ef01ac878 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:10.256469 744 tablet_service.cc:1505] Processing CreateTablet for tablet 08ca0c889f2949408657fa60ce087608 (DEFAULT_TABLE table=default.Table_b708124baf534c96b3f4b8a7a4bb11cc [id=046010a03d774e91b0ad10df97848c56]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.256940 1172 tablet_bootstrap.cc:654] T 378f549ff7364d018d8b9a5ef01ac878 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:10.257076 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08ca0c889f2949408657fa60ce087608. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.258922 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d788aeac52047c6bd9edcaf1369c35c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.260003 2302 consensus_queue.cc:1048] T b6d4f7bf702f4834bc5f4ef2b50b92a7 P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:10.260622 893 tablet_service.cc:1505] Processing CreateTablet for tablet 08ca0c889f2949408657fa60ce087608 (DEFAULT_TABLE table=default.Table_b708124baf534c96b3f4b8a7a4bb11cc [id=046010a03d774e91b0ad10df97848c56]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.261008 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08ca0c889f2949408657fa60ce087608. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.262629 756 tablet_service.cc:1505] Processing CreateTablet for tablet 70850519135644d1ac3d596ecf67221f (DEFAULT_TABLE table=default.Table_99567b9d8c154303bae22fda47db1d0a [id=2b8bd2d2c2664edaad536b1d7706096b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.262929 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70850519135644d1ac3d596ecf67221f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.263005 2309 raft_consensus.cc:493] T 643e20d9974447daa78885b3401f3b77 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:10.263163 2309 raft_consensus.cc:515] T 643e20d9974447daa78885b3401f3b77 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:10.263916 2309 leader_election.cc:290] T 643e20d9974447daa78885b3401f3b77 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:10.264341 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "643e20d9974447daa78885b3401f3b77" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:10.264854 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41a171c7ba55422b8700c4f898dc7d9e. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:10.265190 711 leader_election.cc:343] T 643e20d9974447daa78885b3401f3b77 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:10.273625 879 tablet_service.cc:1505] Processing CreateTablet for tablet 41a171c7ba55422b8700c4f898dc7d9e (DEFAULT_TABLE table=default.Table_6442d40b6bad4ac9a2dbd42a8a3cb096 [id=e530a85da85d4d8a96be6097e31fde58]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.274474 1167 tablet_bootstrap.cc:492] T 738046ab57334f2dab9f4aab979c5b42 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:10.274474 881 tablet_service.cc:1505] Processing CreateTablet for tablet 70850519135644d1ac3d596ecf67221f (DEFAULT_TABLE table=default.Table_99567b9d8c154303bae22fda47db1d0a [id=2b8bd2d2c2664edaad536b1d7706096b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.274780 1167 ts_tablet_manager.cc:1403] T 738046ab57334f2dab9f4aab979c5b42 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.082s user 0.003s sys 0.001s
I20251029 02:52:10.275740 1167 raft_consensus.cc:359] T 738046ab57334f2dab9f4aab979c5b42 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:10.275979 1167 raft_consensus.cc:385] T 738046ab57334f2dab9f4aab979c5b42 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:10.276064 1167 raft_consensus.cc:740] T 738046ab57334f2dab9f4aab979c5b42 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:10.276310 1167 consensus_queue.cc:260] T 738046ab57334f2dab9f4aab979c5b42 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:10.276809 1167 ts_tablet_manager.cc:1434] T 738046ab57334f2dab9f4aab979c5b42 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251029 02:52:10.277153 1167 tablet_bootstrap.cc:492] T 7a1866afc7564217ad777e7ce53786c5 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:10.278362 1042 tablet_service.cc:1505] Processing CreateTablet for tablet bc9fc1c90eaf4046a4022beab31f2902 (DEFAULT_TABLE table=default.Table_fe6f038b462644e7964ea2e48df2a331 [id=5d492f75aeeb4f4bbc9c075377531571]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.278759 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc9fc1c90eaf4046a4022beab31f2902. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.278959 1025 tablet_service.cc:1505] Processing CreateTablet for tablet f3ede523203545d18b09d535fdcabb6f (DEFAULT_TABLE table=default.Table_5d956728da054438a1bd400dac5dac1e [id=006ec0b27fd64c79af53e1d411e921aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.279271 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3ede523203545d18b09d535fdcabb6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.282737 1039 tablet_service.cc:1505] Processing CreateTablet for tablet b3d275e975e64b8d9e026e0eab5b677d (DEFAULT_TABLE table=default.Table_1c456ca0e187431baa7d2054b49ac76c [id=addfa045c7c1492f8168232ee7204595]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.283042 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3d275e975e64b8d9e026e0eab5b677d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.278739 1028 tablet_service.cc:1505] Processing CreateTablet for tablet aba37796661b4041adf7960a15ff52eb (DEFAULT_TABLE table=default.Table_8a944a4e396a4f7cb27c0b3c01ab3ef2 [id=6cf74eaf4e094647bc65dac51a43c434]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.285374 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aba37796661b4041adf7960a15ff52eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.286926 1037 tablet_service.cc:1505] Processing CreateTablet for tablet 0de19a308a534cdaa38f46ab43789322 (DEFAULT_TABLE table=default.Table_9c6ff90fbbd9431c9ae4dbf96e02b307 [id=917b51dc88174f0bab43a20302b49cc3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.286959 2306 raft_consensus.cc:493] T 2f198c3259864f7ab3625cb7bfae20b0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:10.287173 2306 raft_consensus.cc:515] T 2f198c3259864f7ab3625cb7bfae20b0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:10.287312 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0de19a308a534cdaa38f46ab43789322. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.287575 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41a171c7ba55422b8700c4f898dc7d9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.292524 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70850519135644d1ac3d596ecf67221f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.293020 2309 raft_consensus.cc:493] T da2a21c3236a4bd99872b89b12a36508 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0cfd4bee16f84b7c8b0ff3ce14db20d5)
I20251029 02:52:10.293292 2309 raft_consensus.cc:515] T da2a21c3236a4bd99872b89b12a36508 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:10.293988 2309 leader_election.cc:290] T da2a21c3236a4bd99872b89b12a36508 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:10.294142 1172 tablet_bootstrap.cc:492] T 378f549ff7364d018d8b9a5ef01ac878 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:10.294241 1167 tablet_bootstrap.cc:654] T 7a1866afc7564217ad777e7ce53786c5 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:10.294255 1172 ts_tablet_manager.cc:1403] T 378f549ff7364d018d8b9a5ef01ac878 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.039s user 0.001s sys 0.005s
I20251029 02:52:10.294821 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da2a21c3236a4bd99872b89b12a36508" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:10.295011 1172 raft_consensus.cc:359] T 378f549ff7364d018d8b9a5ef01ac878 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:10.295346 1172 raft_consensus.cc:385] T 378f549ff7364d018d8b9a5ef01ac878 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:10.295398 1172 raft_consensus.cc:740] T 378f549ff7364d018d8b9a5ef01ac878 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:10.295603 1172 consensus_queue.cc:260] T 378f549ff7364d018d8b9a5ef01ac878 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:10.296720 1172 ts_tablet_manager.cc:1434] T 378f549ff7364d018d8b9a5ef01ac878 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 02:52:10.297546 1172 tablet_bootstrap.cc:492] T 7bb85a12f573494cbb3118d1947eda27 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:10.298000 2306 leader_election.cc:290] T 2f198c3259864f7ab3625cb7bfae20b0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:10.299031 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2f198c3259864f7ab3625cb7bfae20b0" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:10.300375 1172 tablet_bootstrap.cc:654] T 7bb85a12f573494cbb3118d1947eda27 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:10.303038 1167 tablet_bootstrap.cc:492] T 7a1866afc7564217ad777e7ce53786c5 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:10.303165 1167 ts_tablet_manager.cc:1403] T 7a1866afc7564217ad777e7ce53786c5 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.026s user 0.001s sys 0.005s
I20251029 02:52:10.304100 1167 raft_consensus.cc:359] T 7a1866afc7564217ad777e7ce53786c5 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:10.304299 1167 raft_consensus.cc:385] T 7a1866afc7564217ad777e7ce53786c5 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:10.304363 1167 raft_consensus.cc:740] T 7a1866afc7564217ad777e7ce53786c5 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:10.304982 1166 tablet_bootstrap.cc:492] T adf6e736cb9c4920807ee58682e134ef P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:10.305115 1166 ts_tablet_manager.cc:1403] T adf6e736cb9c4920807ee58682e134ef P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.263s user 0.022s sys 0.015s
I20251029 02:52:10.305745 1166 raft_consensus.cc:359] T adf6e736cb9c4920807ee58682e134ef P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:10.305909 1166 raft_consensus.cc:385] T adf6e736cb9c4920807ee58682e134ef P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:10.305958 1166 raft_consensus.cc:740] T adf6e736cb9c4920807ee58682e134ef P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:10.306169 1166 consensus_queue.cc:260] T adf6e736cb9c4920807ee58682e134ef P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:10.306610 1166 ts_tablet_manager.cc:1434] T adf6e736cb9c4920807ee58682e134ef P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20251029 02:52:10.306871 1166 tablet_bootstrap.cc:492] T 82289e0f823c4d43bfd8c4e878572b2b P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:10.308405 1166 tablet_bootstrap.cc:654] T 82289e0f823c4d43bfd8c4e878572b2b P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:10.308280 1167 consensus_queue.cc:260] T 7a1866afc7564217ad777e7ce53786c5 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:10.308871 1167 ts_tablet_manager.cc:1434] T 7a1866afc7564217ad777e7ce53786c5 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20251029 02:52:10.309163 1167 tablet_bootstrap.cc:492] T e7f846b294cc4eb687a26c8b5931ac19 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:10.312683 2306 raft_consensus.cc:493] T 97b2ecae572a4b7ea1b9595074556708 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:10.312868 2306 raft_consensus.cc:515] T 97b2ecae572a4b7ea1b9595074556708 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:10.313347 881 tablet_service.cc:1505] Processing CreateTablet for tablet 0700b93cc398403d8b3602e84182a3ce (DEFAULT_TABLE table=default.Table_e64c529c8e2d48b487caaa29ed3d17d8 [id=a436307fe09d4ac08c4b98d3d2cfce63]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.313588 2306 leader_election.cc:290] T 97b2ecae572a4b7ea1b9595074556708 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:52:10.320677 1172 tablet_bootstrap.cc:492] T 7bb85a12f573494cbb3118d1947eda27 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:10.320878 1172 ts_tablet_manager.cc:1403] T 7bb85a12f573494cbb3118d1947eda27 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.024s user 0.001s sys 0.004s
I20251029 02:52:10.313907 918 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97b2ecae572a4b7ea1b9595074556708" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:10.321691 990 leader_election.cc:343] T 97b2ecae572a4b7ea1b9595074556708 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:10.321904 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0700b93cc398403d8b3602e84182a3ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.322890 1172 raft_consensus.cc:359] T 7bb85a12f573494cbb3118d1947eda27 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:10.323067 1172 raft_consensus.cc:385] T 7bb85a12f573494cbb3118d1947eda27 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:10.323114 1172 raft_consensus.cc:740] T 7bb85a12f573494cbb3118d1947eda27 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:10.323589 2309 raft_consensus.cc:493] T f281191e7c6143f983f11d87f1c9b64e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:10.323729 2309 raft_consensus.cc:515] T f281191e7c6143f983f11d87f1c9b64e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:10.324237 1031 tablet_service.cc:1552] Processing DeleteTablet for tablet c9e923724ca9402682f13b252dbc37aa with delete_type TABLET_DATA_DELETED (replaced by tablet 456b623500fd4d758e40004666b095a9) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:10.324326 2309 leader_election.cc:290] T f281191e7c6143f983f11d87f1c9b64e P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:10.323832 2302 raft_consensus.cc:493] T da2a21c3236a4bd99872b89b12a36508 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0cfd4bee16f84b7c8b0ff3ce14db20d5)
I20251029 02:52:10.324608 2302 raft_consensus.cc:515] T da2a21c3236a4bd99872b89b12a36508 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:10.324913 740 tablet_service.cc:1505] Processing CreateTablet for tablet 0700b93cc398403d8b3602e84182a3ce (DEFAULT_TABLE table=default.Table_e64c529c8e2d48b487caaa29ed3d17d8 [id=a436307fe09d4ac08c4b98d3d2cfce63]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.325203 2302 leader_election.cc:290] T da2a21c3236a4bd99872b89b12a36508 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
W20251029 02:52:10.326288 849 leader_election.cc:343] T da2a21c3236a4bd99872b89b12a36508 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Not found: Tablet not found: da2a21c3236a4bd99872b89b12a36508
I20251029 02:52:10.326414 1167 tablet_bootstrap.cc:654] T e7f846b294cc4eb687a26c8b5931ac19 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:10.328047 918 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f281191e7c6143f983f11d87f1c9b64e" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:10.328773 2337 tablet.cc:1722] T cedfb53078d94c81b5e294ce7438325e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 02:52:10.327198 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da2a21c3236a4bd99872b89b12a36508" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:10.329532 768 raft_consensus.cc:2468] T da2a21c3236a4bd99872b89b12a36508 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:10.329716 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0700b93cc398403d8b3602e84182a3ce. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:10.328667 711 leader_election.cc:343] T f281191e7c6143f983f11d87f1c9b64e P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:10.330542 847 leader_election.cc:304] T da2a21c3236a4bd99872b89b12a36508 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5
I20251029 02:52:10.331054 2302 raft_consensus.cc:2804] T da2a21c3236a4bd99872b89b12a36508 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251029 02:52:10.331127 2302 raft_consensus.cc:493] T da2a21c3236a4bd99872b89b12a36508 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0cfd4bee16f84b7c8b0ff3ce14db20d5)
I20251029 02:52:10.331187 2302 raft_consensus.cc:3060] T da2a21c3236a4bd99872b89b12a36508 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Advancing to term 2
I20251029 02:52:10.334209 2302 raft_consensus.cc:515] T da2a21c3236a4bd99872b89b12a36508 P c756528642a543eeacd93cef3e335a7d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:10.335740 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da2a21c3236a4bd99872b89b12a36508" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:10.335975 768 raft_consensus.cc:3060] T da2a21c3236a4bd99872b89b12a36508 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Advancing to term 2
I20251029 02:52:10.337479 1166 tablet_bootstrap.cc:492] T 82289e0f823c4d43bfd8c4e878572b2b P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:10.337616 1166 ts_tablet_manager.cc:1403] T 82289e0f823c4d43bfd8c4e878572b2b P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.031s user 0.004s sys 0.000s
I20251029 02:52:10.337967 756 tablet_service.cc:1505] Processing CreateTablet for tablet 48a367a3d4404a7898316a3c4384fd63 (DEFAULT_TABLE table=default.Table_9e7a20ed2df8447d92cc51920fd8177e [id=747a5663894d4eb1b12c9fa8b27d476d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.338433 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48a367a3d4404a7898316a3c4384fd63. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.339435 1172 consensus_queue.cc:260] T 7bb85a12f573494cbb3118d1947eda27 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:10.340058 768 raft_consensus.cc:2468] T da2a21c3236a4bd99872b89b12a36508 P b495a49349324f7d81908f2da700f0d9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 2.
W20251029 02:52:10.341228 849 leader_election.cc:343] T da2a21c3236a4bd99872b89b12a36508 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Not found: Tablet not found: da2a21c3236a4bd99872b89b12a36508
I20251029 02:52:10.341427 2302 leader_election.cc:290] T da2a21c3236a4bd99872b89b12a36508 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 election: Requested vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:10.341847 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8378d67f1da743cca91803e37ea54281. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.342876 1166 raft_consensus.cc:359] T 82289e0f823c4d43bfd8c4e878572b2b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:10.343043 1166 raft_consensus.cc:385] T 82289e0f823c4d43bfd8c4e878572b2b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:10.343096 1166 raft_consensus.cc:740] T 82289e0f823c4d43bfd8c4e878572b2b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:10.343540 1031 tablet_service.cc:1505] Processing CreateTablet for tablet 85bcb1fe0cb54b5aad3e57a812e6d802 (DEFAULT_TABLE table=default.Table_f8f86e0f1d8f4ba98ba26b268e26eee2 [id=76b0df830a7f41a9b6091ca2f1b4edbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.344347 847 leader_election.cc:304] T da2a21c3236a4bd99872b89b12a36508 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5
I20251029 02:52:10.344720 1025 tablet_service.cc:1505] Processing CreateTablet for tablet 0e8802cde9bf4784b587c93c5cf07957 (DEFAULT_TABLE table=default.Table_329849bbaa3a457096d8526f70636b96 [id=bcdff51837004f4aaa118266071b2887]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.344987 2309 raft_consensus.cc:493] T c1222e30bc4141b3ae686d80053c4ef5 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:10.345059 2302 raft_consensus.cc:2804] T da2a21c3236a4bd99872b89b12a36508 P c756528642a543eeacd93cef3e335a7d [term 2 FOLLOWER]: Leader election won for term 2
I20251029 02:52:10.345111 2309 raft_consensus.cc:515] T c1222e30bc4141b3ae686d80053c4ef5 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:10.345425 2302 raft_consensus.cc:697] T da2a21c3236a4bd99872b89b12a36508 P c756528642a543eeacd93cef3e335a7d [term 2 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:52:10.345352 1039 tablet_service.cc:1505] Processing CreateTablet for tablet f4a8f0ad6faa4cf58f48156087392d3c (DEFAULT_TABLE table=default.Table_c6e9611c339e47a8b5021b69c449f141 [id=c19d4543d4ce43a780612d2d8653d496]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.345677 2309 leader_election.cc:290] T c1222e30bc4141b3ae686d80053c4ef5 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:10.346340 1172 ts_tablet_manager.cc:1434] T 7bb85a12f573494cbb3118d1947eda27 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.025s user 0.001s sys 0.001s
I20251029 02:52:10.346302 918 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c1222e30bc4141b3ae686d80053c4ef5" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:10.346128 1037 tablet_service.cc:1505] Processing CreateTablet for tablet 6c4f28b6f4a94d35bb036843aa3bc4bc (DEFAULT_TABLE table=default.Table_6c1f042a9840467993a38a8e55a23cbb [id=29c45a9bc52e4cdabd41ab4220c2b40b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.346890 1166 consensus_queue.cc:260] T 82289e0f823c4d43bfd8c4e878572b2b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
W20251029 02:52:10.347110 711 leader_election.cc:343] T c1222e30bc4141b3ae686d80053c4ef5 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:10.347384 1166 ts_tablet_manager.cc:1434] T 82289e0f823c4d43bfd8c4e878572b2b P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20251029 02:52:10.347651 1166 tablet_bootstrap.cc:492] T 44c0488eba0543078ff4f0b3d69366f2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:10.348321 878 tablet_service.cc:1505] Processing CreateTablet for tablet 48a367a3d4404a7898316a3c4384fd63 (DEFAULT_TABLE table=default.Table_9e7a20ed2df8447d92cc51920fd8177e [id=747a5663894d4eb1b12c9fa8b27d476d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.348909 738 tablet_service.cc:1505] Processing CreateTablet for tablet 416e7c48daa849cd9635656529488922 (DEFAULT_TABLE table=default.Table_0a146ce6c31f4dbf994a2442f2386d93 [id=559a939c5ba5402ebd5881cf88766be7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.349047 1166 tablet_bootstrap.cc:654] T 44c0488eba0543078ff4f0b3d69366f2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:10.349247 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 416e7c48daa849cd9635656529488922. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.349260 2302 consensus_queue.cc:237] T da2a21c3236a4bd99872b89b12a36508 P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:10.351640 1172 tablet_bootstrap.cc:492] T 58dde9397b434b089ade9968f369006a P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:10.355933 2201 raft_consensus.cc:493] T 41a6139fd9b84707a8c7eebc0381e383 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0cfd4bee16f84b7c8b0ff3ce14db20d5)
I20251029 02:52:10.356122 2201 raft_consensus.cc:515] T 41a6139fd9b84707a8c7eebc0381e383 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:10.356788 2201 leader_election.cc:290] T 41a6139fd9b84707a8c7eebc0381e383 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:10.358171 1172 tablet_bootstrap.cc:654] T 58dde9397b434b089ade9968f369006a P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
W20251029 02:52:10.358191 849 leader_election.cc:343] T 41a6139fd9b84707a8c7eebc0381e383 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Not found: Tablet not found: 41a6139fd9b84707a8c7eebc0381e383
I20251029 02:52:10.358661 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48a367a3d4404a7898316a3c4384fd63. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.359140 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "41a6139fd9b84707a8c7eebc0381e383" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:10.359808 847 leader_election.cc:304] T 41a6139fd9b84707a8c7eebc0381e383 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9
I20251029 02:52:10.360291 2201 raft_consensus.cc:2749] T 41a6139fd9b84707a8c7eebc0381e383 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251029 02:52:10.361537 2309 raft_consensus.cc:493] T 59eee86f249e4d6a9d09d30276dcd659 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:10.361693 2309 raft_consensus.cc:515] T 59eee86f249e4d6a9d09d30276dcd659 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:10.362367 2309 leader_election.cc:290] T 59eee86f249e4d6a9d09d30276dcd659 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:10.364732 918 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59eee86f249e4d6a9d09d30276dcd659" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:10.367497 517 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 877e278deb4a4b01ac66b06be0890baf
I20251029 02:52:10.368273 756 tablet_service.cc:1552] Processing DeleteTablet for tablet 877e278deb4a4b01ac66b06be0890baf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:09 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:10.368279 517 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_ec7e5a0085cd43e28618d7b9d3d30c65" } new_table_name: "default.Table_2c589dbe63b741eeb175624a88baf0a9" modify_external_catalogs: true
I20251029 02:52:10.369410 756 tablet_service.cc:1505] Processing CreateTablet for tablet 749f380eafae44ddbe2a244757f37909 (DEFAULT_TABLE table=default.Table_1f3338455b1a4eef85ecea36443bc329 [id=39683553c6fd44bca1cc3de2d0b9d089]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.369477 744 tablet_service.cc:1505] Processing CreateTablet for tablet 7fba607c566c4fa1a803cba8d86ca646 (DEFAULT_TABLE table=default.Table_9bc6d7e931a34e00b8a1c2233fa53c3f [id=c8105da2dd8540aa977bf54b09d874ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.369819 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 749f380eafae44ddbe2a244757f37909. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.370734 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7fba607c566c4fa1a803cba8d86ca646. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.370873 884 tablet_service.cc:1505] Processing CreateTablet for tablet 416e7c48daa849cd9635656529488922 (DEFAULT_TABLE table=default.Table_0a146ce6c31f4dbf994a2442f2386d93 [id=559a939c5ba5402ebd5881cf88766be7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.370749 754 tablet_service.cc:1505] Processing CreateTablet for tablet 858ef21620344d909d085f22e62081ac (DEFAULT_TABLE table=default.Table_d76042aac8cb46c38a05fdf710a406cc [id=bca89cc60e344f71a51878dcf10b2ab7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.371230 739 tablet_service.cc:1505] Processing CreateTablet for tablet 8de7380e93dd4a9abdb84522a18abda6 (DEFAULT_TABLE table=default.Table_af62893fe44b437ba6c5753df4b31da9 [id=dd34a215f0cb45b7929a1b86576027f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.371870 2338 tablet.cc:1722] T cedfb53078d94c81b5e294ce7438325e P c756528642a543eeacd93cef3e335a7d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 02:52:10.373008 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85bcb1fe0cb54b5aad3e57a812e6d802. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.374217 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 858ef21620344d909d085f22e62081ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.374662 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8de7380e93dd4a9abdb84522a18abda6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.374636 749 tablet_service.cc:1505] Processing CreateTablet for tablet 4f004ea546d64681bd29b2e7ee3bb225 (DEFAULT_TABLE table=default.Table_d2ac974941f9428eaa9605606209ed8d [id=62013562b0b2496cb82c301377d317ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.367627 740 tablet_service.cc:1505] Processing CreateTablet for tablet eeb75fc46e2a41f78252868ad5256966 (DEFAULT_TABLE table=default.Table_331f901095f44ebfa644aa9cf8dde711 [id=d58b97f75daa4d8083b61b23aa87e1e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.375172 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eeb75fc46e2a41f78252868ad5256966. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.375386 896 tablet_service.cc:1552] Processing DeleteTablet for tablet 877e278deb4a4b01ac66b06be0890baf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:09 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:10.375897 894 tablet_service.cc:1505] Processing CreateTablet for tablet 749f380eafae44ddbe2a244757f37909 (DEFAULT_TABLE table=default.Table_1f3338455b1a4eef85ecea36443bc329 [id=39683553c6fd44bca1cc3de2d0b9d089]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.375866 896 tablet_service.cc:1505] Processing CreateTablet for tablet 858ef21620344d909d085f22e62081ac (DEFAULT_TABLE table=default.Table_d76042aac8cb46c38a05fdf710a406cc [id=bca89cc60e344f71a51878dcf10b2ab7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.376081 882 tablet_service.cc:1505] Processing CreateTablet for tablet 4f004ea546d64681bd29b2e7ee3bb225 (DEFAULT_TABLE table=default.Table_d2ac974941f9428eaa9605606209ed8d [id=62013562b0b2496cb82c301377d317ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.376443 891 tablet_service.cc:1505] Processing CreateTablet for tablet 7fba607c566c4fa1a803cba8d86ca646 (DEFAULT_TABLE table=default.Table_9bc6d7e931a34e00b8a1c2233fa53c3f [id=c8105da2dd8540aa977bf54b09d874ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.376845 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e8802cde9bf4784b587c93c5cf07957. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.377302 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f4a8f0ad6faa4cf58f48156087392d3c. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:10.380139 520 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0415b52fadf948238c4e83dc09000f96 consensus state has no leader
I20251029 02:52:10.383772 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f004ea546d64681bd29b2e7ee3bb225. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.384624 2309 raft_consensus.cc:493] T a9e0fcc759a64679b3f59a6c964877b6 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:10.384629 2201 raft_consensus.cc:493] T 59d03a190b784ab18a3e63a572675891 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:10.384747 2309 raft_consensus.cc:515] T a9e0fcc759a64679b3f59a6c964877b6 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:10.384761 2201 raft_consensus.cc:515] T 59d03a190b784ab18a3e63a572675891 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:10.385317 2201 leader_election.cc:290] T 59d03a190b784ab18a3e63a572675891 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:10.385313 2309 leader_election.cc:290] T a9e0fcc759a64679b3f59a6c964877b6 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
W20251029 02:52:10.385411 711 leader_election.cc:343] T 59eee86f249e4d6a9d09d30276dcd659 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:10.386085 1058 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59d03a190b784ab18a3e63a572675891" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:10.386286 1058 raft_consensus.cc:2468] T 59d03a190b784ab18a3e63a572675891 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:10.386828 849 leader_election.cc:304] T 59d03a190b784ab18a3e63a572675891 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:10.387161 2201 raft_consensus.cc:2804] T 59d03a190b784ab18a3e63a572675891 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:10.387238 2201 raft_consensus.cc:493] T 59d03a190b784ab18a3e63a572675891 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:10.387312 2201 raft_consensus.cc:3060] T 59d03a190b784ab18a3e63a572675891 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:10.387548 918 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a9e0fcc759a64679b3f59a6c964877b6" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:10.387985 711 leader_election.cc:343] T a9e0fcc759a64679b3f59a6c964877b6 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:10.388350 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6c4f28b6f4a94d35bb036843aa3bc4bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.390117 2201 raft_consensus.cc:515] T 59d03a190b784ab18a3e63a572675891 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:10.391892 1167 tablet_bootstrap.cc:492] T e7f846b294cc4eb687a26c8b5931ac19 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:10.392048 1167 ts_tablet_manager.cc:1403] T e7f846b294cc4eb687a26c8b5931ac19 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.083s user 0.009s sys 0.006s
I20251029 02:52:10.392148 1058 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59d03a190b784ab18a3e63a572675891" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:52:10.392295 1058 raft_consensus.cc:3060] T 59d03a190b784ab18a3e63a572675891 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:10.392817 1167 raft_consensus.cc:359] T e7f846b294cc4eb687a26c8b5931ac19 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:10.392968 1167 raft_consensus.cc:385] T e7f846b294cc4eb687a26c8b5931ac19 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:10.393028 1167 raft_consensus.cc:740] T e7f846b294cc4eb687a26c8b5931ac19 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:10.393208 1167 consensus_queue.cc:260] T e7f846b294cc4eb687a26c8b5931ac19 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:10.393643 1167 ts_tablet_manager.cc:1434] T e7f846b294cc4eb687a26c8b5931ac19 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251029 02:52:10.393908 1167 tablet_bootstrap.cc:492] T 6d830d30b26b4ba799785fec8a6f0769 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:10.394731 2201 leader_election.cc:290] T 59d03a190b784ab18a3e63a572675891 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
W20251029 02:52:10.396080 847 leader_election.cc:343] T 59d03a190b784ab18a3e63a572675891 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 59d03a190b784ab18a3e63a572675891
I20251029 02:52:10.397042 1058 raft_consensus.cc:2468] T 59d03a190b784ab18a3e63a572675891 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:10.397660 849 leader_election.cc:304] T 59d03a190b784ab18a3e63a572675891 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters: b495a49349324f7d81908f2da700f0d9
I20251029 02:52:10.398043 2205 raft_consensus.cc:2804] T 59d03a190b784ab18a3e63a572675891 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:10.398159 2205 raft_consensus.cc:697] T 59d03a190b784ab18a3e63a572675891 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:52:10.398391 2205 consensus_queue.cc:237] T 59d03a190b784ab18a3e63a572675891 P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
W20251029 02:52:10.402909 847 leader_election.cc:343] T 59d03a190b784ab18a3e63a572675891 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 59d03a190b784ab18a3e63a572675891
I20251029 02:52:10.406023 2188 raft_consensus.cc:493] T c32c28b0e74a4af3a0e69bbf24ab1a1f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:10.406193 2188 raft_consensus.cc:515] T c32c28b0e74a4af3a0e69bbf24ab1a1f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:10.406792 2188 leader_election.cc:290] T c32c28b0e74a4af3a0e69bbf24ab1a1f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:10.407328 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c32c28b0e74a4af3a0e69bbf24ab1a1f" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:10.407871 990 leader_election.cc:343] T c32c28b0e74a4af3a0e69bbf24ab1a1f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:10.407972 2201 consensus_queue.cc:1048] T 6f35412a66b74051a475dce5dafa7792 P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:10.415031 1029 tablet_service.cc:1505] Processing CreateTablet for tablet eb3c20822d684bea89eb358251e21e33 (DEFAULT_TABLE table=default.Table_52cba56ce9fb44bd84c35ef2f9219e56 [id=d7a96b1852fc4049b31ff576280980c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.415486 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb3c20822d684bea89eb358251e21e33. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.416834 2214 consensus_queue.cc:1048] T 6f35412a66b74051a475dce5dafa7792 P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:10.420994 2309 raft_consensus.cc:493] T 41a6139fd9b84707a8c7eebc0381e383 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0cfd4bee16f84b7c8b0ff3ce14db20d5)
I20251029 02:52:10.421182 2309 raft_consensus.cc:515] T 41a6139fd9b84707a8c7eebc0381e383 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:10.421891 2309 leader_election.cc:290] T 41a6139fd9b84707a8c7eebc0381e383 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:10.425144 893 tablet_service.cc:1505] Processing CreateTablet for tablet eeb75fc46e2a41f78252868ad5256966 (DEFAULT_TABLE table=default.Table_331f901095f44ebfa644aa9cf8dde711 [id=d58b97f75daa4d8083b61b23aa87e1e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.426385 2188 consensus_queue.cc:1048] T 1275e192871440408a8345fef9b56260 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:10.425819 918 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "41a6139fd9b84707a8c7eebc0381e383" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:10.426968 918 raft_consensus.cc:2468] T 41a6139fd9b84707a8c7eebc0381e383 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b495a49349324f7d81908f2da700f0d9 in term 1.
I20251029 02:52:10.427345 711 leader_election.cc:304] T 41a6139fd9b84707a8c7eebc0381e383 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:10.427667 2309 raft_consensus.cc:2804] T 41a6139fd9b84707a8c7eebc0381e383 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251029 02:52:10.427764 2309 raft_consensus.cc:493] T 41a6139fd9b84707a8c7eebc0381e383 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0cfd4bee16f84b7c8b0ff3ce14db20d5)
I20251029 02:52:10.427884 2309 raft_consensus.cc:3060] T 41a6139fd9b84707a8c7eebc0381e383 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Advancing to term 2
I20251029 02:52:10.431852 2309 raft_consensus.cc:515] T 41a6139fd9b84707a8c7eebc0381e383 P b495a49349324f7d81908f2da700f0d9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:10.432996 2309 leader_election.cc:290] T 41a6139fd9b84707a8c7eebc0381e383 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 2 election: Requested vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:10.433769 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eeb75fc46e2a41f78252868ad5256966. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.433737 918 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "41a6139fd9b84707a8c7eebc0381e383" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
I20251029 02:52:10.433943 918 raft_consensus.cc:3060] T 41a6139fd9b84707a8c7eebc0381e383 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Advancing to term 2
I20251029 02:52:10.437080 918 raft_consensus.cc:2468] T 41a6139fd9b84707a8c7eebc0381e383 P c756528642a543eeacd93cef3e335a7d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b495a49349324f7d81908f2da700f0d9 in term 2.
I20251029 02:52:10.437441 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f004ea546d64681bd29b2e7ee3bb225. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.437505 711 leader_election.cc:304] T 41a6139fd9b84707a8c7eebc0381e383 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:10.437923 2309 raft_consensus.cc:2804] T 41a6139fd9b84707a8c7eebc0381e383 P b495a49349324f7d81908f2da700f0d9 [term 2 FOLLOWER]: Leader election won for term 2
I20251029 02:52:10.438809 2309 raft_consensus.cc:697] T 41a6139fd9b84707a8c7eebc0381e383 P b495a49349324f7d81908f2da700f0d9 [term 2 LEADER]: Becoming Leader. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Running, Role: LEADER
I20251029 02:52:10.439373 2309 consensus_queue.cc:237] T 41a6139fd9b84707a8c7eebc0381e383 P b495a49349324f7d81908f2da700f0d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:10.444136 1167 tablet_bootstrap.cc:654] T 6d830d30b26b4ba799785fec8a6f0769 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:10.445272 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7fba607c566c4fa1a803cba8d86ca646. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.447170 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 416e7c48daa849cd9635656529488922. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.449661 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 749f380eafae44ddbe2a244757f37909. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.451879 1042 tablet_service.cc:1505] Processing CreateTablet for tablet aacc72b167954d4fbb14ee2bfa293e6f (DEFAULT_TABLE table=default.Table_5681078bd9b54f948bd2d98511003cba [id=8c3b50bd1e434dfebe679141ceb18be8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.451954 1036 tablet_service.cc:1505] Processing CreateTablet for tablet 96f7876e5d6d4a0db236df5e7f266fd2 (DEFAULT_TABLE table=default.Table_aaecc1c9efd04549a2484850ff8efc60 [id=00da2e21da8f4d54ae86834777691781]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.452131 1026 tablet_service.cc:1505] Processing CreateTablet for tablet 51a6e45378344f86847a44cf4950307b (DEFAULT_TABLE table=default.Table_13f9fb5682214cc28f12dce88109d831 [id=0816dc14afa8435e80fdbaa4959c985a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.452237 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aacc72b167954d4fbb14ee2bfa293e6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.452265 1025 tablet_service.cc:1505] Processing CreateTablet for tablet 62b1c6d828664f02acc4a3e6dfb5a578 (DEFAULT_TABLE table=default.Table_dbb86af86c304f02ae3a9ece7bb4998b [id=00f1239c6da24a7abd15e121a8ff838d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.452392 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51a6e45378344f86847a44cf4950307b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.452550 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96f7876e5d6d4a0db236df5e7f266fd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.453552 1041 tablet_service.cc:1505] Processing CreateTablet for tablet 03ab7f2c2f4343f18a7bfccb96274c64 (DEFAULT_TABLE table=default.Table_e9a962d143c94976af9285475ea40631 [id=57a3a3968dec4a7181291fa5efd51530]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.454977 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 858ef21620344d909d085f22e62081ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.459228 1029 tablet_service.cc:1505] Processing CreateTablet for tablet 4b841723726e40bda0187fb84bf36a6a (DEFAULT_TABLE table=default.Table_8c898cd0e2f343a6ae5debb290c20b07 [id=3d94e6f551f5474c9816d976a72a43c9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.459371 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03ab7f2c2f4343f18a7bfccb96274c64. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.459957 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62b1c6d828664f02acc4a3e6dfb5a578. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.465857 1039 tablet_service.cc:1505] Processing CreateTablet for tablet ca71c4638a6b419b876e22af7e6cf00a (DEFAULT_TABLE table=default.Table_31f9807cb2934a47a4fb2f67f9507053 [id=c3b6a55d9eaa4248aadb99c5ae1a9be5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.466382 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca71c4638a6b419b876e22af7e6cf00a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.466768 1031 tablet_service.cc:1505] Processing CreateTablet for tablet d48d9178108b4eb59062d29b40ba0c11 (DEFAULT_TABLE table=default.Table_0c7029666c80452b8fae6945da658ec7 [id=d862dce1eb5b4ffc886fd70e0b9ad7bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.467489 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4b841723726e40bda0187fb84bf36a6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.470628 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d48d9178108b4eb59062d29b40ba0c11. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.473870 2201 consensus_queue.cc:1048] T b6d4f7bf702f4834bc5f4ef2b50b92a7 P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:10.486579 1174 ts_tablet_manager.cc:1929] T 9e6eb099648549dbbf9969e59865f51f P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:10.486835 1174 log.cc:1199] T 9e6eb099648549dbbf9969e59865f51f P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/9e6eb099648549dbbf9969e59865f51f
I20251029 02:52:10.487360 1174 ts_tablet_manager.cc:1950] T 9e6eb099648549dbbf9969e59865f51f P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:10.488451 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 815) took 2736 ms. Trace:
I20251029 02:52:10.482332 1037 tablet_service.cc:1505] Processing CreateTablet for tablet 99d60ef5325e45059026596a756448de (DEFAULT_TABLE table=default.Table_aec07800b35948bba7db9168286fcf14 [id=7147b693c86e4c258fecdadfbe5c29f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.492991 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99d60ef5325e45059026596a756448de. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.506446 749 tablet_service.cc:1505] Processing CreateTablet for tablet 56ed1b1d2ef144e79e42e708326b2480 (DEFAULT_TABLE table=default.Table_d61d4be2068f4a1985e207d3413141f9 [id=a20067c639e94ad091a0db8d17af2f87]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.506983 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56ed1b1d2ef144e79e42e708326b2480. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.512966 1175 ts_tablet_manager.cc:1929] T 1275e192871440408a8345fef9b56260 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:10.513111 1175 log.cc:1199] T 1275e192871440408a8345fef9b56260 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/1275e192871440408a8345fef9b56260
I20251029 02:52:10.513463 1175 ts_tablet_manager.cc:1950] T 1275e192871440408a8345fef9b56260 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:10.514374 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 809) took 2744 ms. Trace:
I20251029 02:52:10.522749 518 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet 1bdb67125493430398ff5c80b73de2ac (table default.Table_2846af457a4b4e27a739af50c7f8c7f7 [id=8c3d3d88395e4712af0f62c4ad14edea]). Expected version 3 got 0
I20251029 02:52:10.524760 1029 tablet_service.cc:1505] Processing CreateTablet for tablet 971b95ab982141e184a314e7eb5e4a6a (DEFAULT_TABLE table=default.Table_df78a989ef8e4c48b1716f5698c6ecf9 [id=5bd6c6f6062a42438261b3d26ea313e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.525156 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 971b95ab982141e184a314e7eb5e4a6a. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:10.525542 526 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6364601618314251b5601f30971ac61f consensus state has no leader
I20251029 02:52:10.527882 525 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_1c932fc09555407cbd8c9c2d3ef20f7b" } new_table_name: "default.Table_eb665fea357646b1bfc96cd81dc7cbcc" modify_external_catalogs: true
W20251029 02:52:10.529850 517 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a8cef3074aef43b2981eaffdc5a5b5bb consensus state has no leader
I20251029 02:52:10.529917 1041 tablet_service.cc:1505] Processing CreateTablet for tablet 9efc87a35b96414fa38d7e849185278c (DEFAULT_TABLE table=default.Table_957971e954a244309b62d05bb172dc9d [id=2ab2cb2eabc545ec84049542f7e19eb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.530278 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9efc87a35b96414fa38d7e849185278c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.531649 520 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_e64c529c8e2d48b487caaa29ed3d17d8" } modify_external_catalogs: true
I20251029 02:52:10.531809 520 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_e64c529c8e2d48b487caaa29ed3d17d8" } modify_external_catalogs: true
I20251029 02:52:10.533074 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_79052b7cce26442aa1454f10280e7e7a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:10.533151 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_2050c80c7f844f4b9fc3aeffb54f8d6e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:10.533607 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_79052b7cce26442aa1454f10280e7e7a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:52:10.533666 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2050c80c7f844f4b9fc3aeffb54f8d6e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:10.534963 524 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_ddd7ee9b37df4bb4be7957aef7e82cb8" } modify_external_catalogs: true
I20251029 02:52:10.535091 524 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_ddd7ee9b37df4bb4be7957aef7e82cb8" } modify_external_catalogs: true
I20251029 02:52:10.536262 526 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_d7cbd7b4c4bf4886a4ccf4c89da3cbce" } modify_external_catalogs: true
I20251029 02:52:10.536401 526 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_d7cbd7b4c4bf4886a4ccf4c89da3cbce" } modify_external_catalogs: true
I20251029 02:52:10.536702 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_a0c3ca13b0804fa399653c68e8bf4335"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:10.537261 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a0c3ca13b0804fa399653c68e8bf4335 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:10.540674 517 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet 1bdb67125493430398ff5c80b73de2ac (table default.Table_2846af457a4b4e27a739af50c7f8c7f7 [id=8c3d3d88395e4712af0f62c4ad14edea]). Expected version 3 got 0
I20251029 02:52:10.541210 517 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet 378f549ff7364d018d8b9a5ef01ac878 (table default.Table_8c5d559a2d7f427094b719d1c0b6cff4 [id=0e6e6f5dc35241c5a04c9d6cf4c39b26]). Expected version 1 got 0
I20251029 02:52:10.541548 517 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet f281191e7c6143f983f11d87f1c9b64e (table default.Table_e5ebaa623385480b9cd7be9694ff024f [id=cc3e1f3858f04176acb8c39c65396a5e]). Expected version 2 got 0
I20251029 02:52:10.543501 518 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_757229f7dede492d8da98328eaabc0dd" } new_table_name: "default.Table_b96dfc0b5a574a6ab3f16335d08a7f50" modify_external_catalogs: true
W20251029 02:52:10.551631 988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251029 02:52:10.546043 738 tablet_service.cc:1552] Processing DeleteTablet for tablet 67651306f3b44e7eb14dcee9a2007a81 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:03 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:10.546675 740 tablet_service.cc:1552] Processing DeleteTablet for tablet 7bb85a12f573494cbb3118d1947eda27 with delete_type TABLET_DATA_DELETED (replaced by tablet a9f78f61ee2841dea71bbbfceacc3ea6) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:10.550400 1172 tablet_bootstrap.cc:492] T 58dde9397b434b089ade9968f369006a P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:10.554643 1172 ts_tablet_manager.cc:1403] T 58dde9397b434b089ade9968f369006a P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.203s user 0.005s sys 0.009s
I20251029 02:52:10.555516 1172 raft_consensus.cc:359] T 58dde9397b434b089ade9968f369006a P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:10.555757 1172 raft_consensus.cc:385] T 58dde9397b434b089ade9968f369006a P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:10.555846 1172 raft_consensus.cc:740] T 58dde9397b434b089ade9968f369006a P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:10.556087 1172 consensus_queue.cc:260] T 58dde9397b434b089ade9968f369006a P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:10.556779 1172 ts_tablet_manager.cc:1434] T 58dde9397b434b089ade9968f369006a P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 02:52:10.557087 1172 tablet_bootstrap.cc:492] T 40e844fd7ad74680bcbe7bef936c5b9d P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:10.558313 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_5db0b0e77f584976b6253ec87f386bb8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:10.558848 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5db0b0e77f584976b6253ec87f386bb8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:10.514492 1175 rpcz_store.cc:276] 1029 02:52:07.769748 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:07.769806 (+ 58us) service_pool.cc:225] Handling call
1029 02:52:09.829578 (+2059772us) ts_tablet_manager.cc:1151] Deleting tablet 1275e192871440408a8345fef9b56260
1029 02:52:09.829586 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:10.056252 (+226666us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:10.512849 (+456597us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:10.514350 (+ 1501us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2059566}
I20251029 02:52:10.560854 520 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 0700b93cc398403d8b3602e84182a3ce
I20251029 02:52:10.561509 520 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_eb24c80a26b04dab8cc909a42d7184f3" } modify_external_catalogs: true
I20251029 02:52:10.561671 520 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_eb24c80a26b04dab8cc909a42d7184f3" } modify_external_catalogs: true
I20251029 02:52:10.565721 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 1275e192871440408a8345fef9b56260 (table default.Table_5265805d6c3746209f9466df88b164b9 [id=5c4f8f38dfc748dda3abf03f7882cf35]) successfully deleted
I20251029 02:52:10.566339 2306 consensus_queue.cc:1048] T 9bb2166a590d4210b02dd779ef6654ad P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:10.566799 517 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet 1bdb67125493430398ff5c80b73de2ac (table default.Table_2846af457a4b4e27a739af50c7f8c7f7 [id=8c3d3d88395e4712af0f62c4ad14edea]). Expected version 3 got 0
I20251029 02:52:10.566861 740 tablet_service.cc:1552] Processing DeleteTablet for tablet 0700b93cc398403d8b3602e84182a3ce with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:10 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:10.570398 2179 consensus_queue.cc:1048] T 95f342d1cfb84c4eb31bd3d15ea3bdc2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.025s
W20251029 02:52:10.570988 525 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e74eda48dca647efacabe525ef316944 consensus state has no leader
I20251029 02:52:10.571290 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_7e276e1364a8470a853a68d2d977a4b4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:10.571702 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7e276e1364a8470a853a68d2d977a4b4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:10.572119 524 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet bfb9115f324f44c98f876a2d56bb23a1
I20251029 02:52:10.572969 526 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet ef2e9dd8f90a47389d9062d1f99a1da1
I20251029 02:52:10.577984 756 tablet_service.cc:1552] Processing DeleteTablet for tablet 58dde9397b434b089ade9968f369006a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:59 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:10.577984 738 tablet_service.cc:1552] Processing DeleteTablet for tablet ef2e9dd8f90a47389d9062d1f99a1da1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:10 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:10.578040 740 tablet_service.cc:1552] Processing DeleteTablet for tablet bfb9115f324f44c98f876a2d56bb23a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:10 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:10.579650 521 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_11c3e33dc6ae4f9e8852d7a31b078288" } modify_external_catalogs: true
I20251029 02:52:10.579864 521 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_11c3e33dc6ae4f9e8852d7a31b078288" } modify_external_catalogs: true
I20251029 02:52:10.583910 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_7fdd50180d294d44bc4c64f2cc5dc57a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:10.584069 523 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_9bc6d7e931a34e00b8a1c2233fa53c3f" } new_table_name: "default.Table_9c149fba4ea94e5dabe5d6584e7c471a" modify_external_catalogs: true
W20251029 02:52:10.584559 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7fdd50180d294d44bc4c64f2cc5dc57a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:10.586078 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_f08f8ab20ebf4c9a99d5b88cf17d3b7d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:10.586696 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f08f8ab20ebf4c9a99d5b88cf17d3b7d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:10.589013 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_a27ea57c89b0433d8b21e21e88948bde"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:10.589700 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a27ea57c89b0433d8b21e21e88948bde in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:10.591578 1031 tablet_service.cc:1505] Processing CreateTablet for tablet fa525da8037f427592b5d0858ba9a815 (DEFAULT_TABLE table=default.Table_4d99c72b75124b20bf00ff0d75e7aada [id=80f9cf1d5452497cbf5fcd9d8981a8fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.591904 1039 tablet_service.cc:1505] Processing CreateTablet for tablet 8378d67f1da743cca91803e37ea54281 (DEFAULT_TABLE table=default.Table_fdb0a130609a45878b3a6f07cf79947c [id=c46a973298994dc4a994905220615830]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.591984 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa525da8037f427592b5d0858ba9a815. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.592200 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8378d67f1da743cca91803e37ea54281. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.593263 1174 rpcz_store.cc:276] 1029 02:52:07.752299 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:07.752407 (+ 108us) service_pool.cc:225] Handling call
1029 02:52:09.860200 (+2107793us) ts_tablet_manager.cc:1151] Deleting tablet 9e6eb099648549dbbf9969e59865f51f
1029 02:52:09.860216 (+ 16us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:10.128226 (+268010us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:10.486432 (+358206us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:10.488422 (+ 1990us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2061499}
I20251029 02:52:10.593458 1029 tablet_service.cc:1505] Processing CreateTablet for tablet 5950a2dcdcd3440b873cd53d070ca708 (DEFAULT_TABLE table=default.Table_9af081e1d784401792ad8964bdad2e73 [id=8a9e4bde637c48738015985b4461165a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.491511 1167 tablet_bootstrap.cc:492] T 6d830d30b26b4ba799785fec8a6f0769 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:10.593720 1167 ts_tablet_manager.cc:1403] T 6d830d30b26b4ba799785fec8a6f0769 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.200s user 0.009s sys 0.013s
I20251029 02:52:10.593791 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5950a2dcdcd3440b873cd53d070ca708. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.497378 895 tablet_service.cc:1505] Processing CreateTablet for tablet 8de7380e93dd4a9abdb84522a18abda6 (DEFAULT_TABLE table=default.Table_af62893fe44b437ba6c5753df4b31da9 [id=dd34a215f0cb45b7929a1b86576027f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.594651 1172 tablet_bootstrap.cc:654] T 40e844fd7ad74680bcbe7bef936c5b9d P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:10.594885 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8de7380e93dd4a9abdb84522a18abda6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.595047 1175 tablet_replica.cc:333] T 13ee62fc2339410b9db70dfd3d3c6fa6 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:10.595218 1175 raft_consensus.cc:2243] T 13ee62fc2339410b9db70dfd3d3c6fa6 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:10.595350 1175 raft_consensus.cc:2272] T 13ee62fc2339410b9db70dfd3d3c6fa6 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:10.595842 1175 ts_tablet_manager.cc:1916] T 13ee62fc2339410b9db70dfd3d3c6fa6 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:10.504639 878 tablet_service.cc:1505] Processing CreateTablet for tablet 56ed1b1d2ef144e79e42e708326b2480 (DEFAULT_TABLE table=default.Table_d61d4be2068f4a1985e207d3413141f9 [id=a20067c639e94ad091a0db8d17af2f87]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.598024 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 817) took 2838 ms. Trace:
I20251029 02:52:10.598100 1174 rpcz_store.cc:276] 1029 02:52:07.759225 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:07.759277 (+ 52us) service_pool.cc:225] Handling call
1029 02:52:10.597875 (+2838598us) ts_tablet_manager.cc:1151] Deleting tablet 80f54d8a486a4196aca3c28c84c0872f
1029 02:52:10.597887 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:10.598004 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2799039}
I20251029 02:52:10.598304 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56ed1b1d2ef144e79e42e708326b2480. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.600929 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 816) took 2841 ms. Trace:
I20251029 02:52:10.601034 1174 rpcz_store.cc:276] 1029 02:52:07.759065 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:07.759464 (+ 399us) service_pool.cc:225] Handling call
1029 02:52:10.600776 (+2841312us) ts_tablet_manager.cc:1151] Deleting tablet 9778f924d11a48b194ccf26124bd5509
1029 02:52:10.600788 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:10.600905 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2801798}
I20251029 02:52:10.523279 884 tablet_service.cc:1552] Processing DeleteTablet for tablet 86fc22bb84794149b598b97df41eb881 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:58 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:10.527282 893 tablet_service.cc:1552] Processing DeleteTablet for tablet 49d9ce32e4f04f11be2ecefc5d523a8b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:57 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:10.548808 882 tablet_service.cc:1552] Processing DeleteTablet for tablet 13ee62fc2339410b9db70dfd3d3c6fa6 with delete_type TABLET_DATA_DELETED (replaced by tablet 35d9b8dfd61e4d8db441427a49fb9dca) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:10.548979 891 tablet_service.cc:1552] Processing DeleteTablet for tablet 0f80b840ed954230bfbfe7e40127cd2f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:56 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:10.550001 892 tablet_service.cc:1552] Processing DeleteTablet for tablet d61beda662d844cdaf67fd9a2e382ea2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:55 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:10.550266 898 tablet_service.cc:1552] Processing DeleteTablet for tablet 28f7ceb91a34450cad4b5b45f5285b41 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:59 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:10.551019 885 tablet_service.cc:1552] Processing DeleteTablet for tablet 0d84dc8dc465496f9adec798b2e03e13 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:56 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:10.562673 896 tablet_service.cc:1552] Processing DeleteTablet for tablet 0700b93cc398403d8b3602e84182a3ce with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:10 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:10.577337 894 tablet_service.cc:1552] Processing DeleteTablet for tablet bfb9115f324f44c98f876a2d56bb23a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:10 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:10.581693 881 tablet_service.cc:1552] Processing DeleteTablet for tablet ef2e9dd8f90a47389d9062d1f99a1da1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:10 UTC) from {username='slave'} at 127.0.0.1:35702
W20251029 02:52:10.592849 847 consensus_peers.cc:597] T ba699862e6f646c6beaa39f080d0faba P c756528642a543eeacd93cef3e335a7d -> Peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Couldn't send request to peer b495a49349324f7d81908f2da700f0d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ba699862e6f646c6beaa39f080d0faba. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:10.602152 1167 raft_consensus.cc:359] T 6d830d30b26b4ba799785fec8a6f0769 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:10.602303 1167 raft_consensus.cc:385] T 6d830d30b26b4ba799785fec8a6f0769 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:10.602352 1167 raft_consensus.cc:740] T 6d830d30b26b4ba799785fec8a6f0769 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:10.602730 1167 consensus_queue.cc:260] T 6d830d30b26b4ba799785fec8a6f0769 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:10.603209 1167 ts_tablet_manager.cc:1434] T 6d830d30b26b4ba799785fec8a6f0769 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20251029 02:52:10.603478 1167 tablet_bootstrap.cc:492] T 2e97d1bc06a24e8d8ba11acb6fd280d8 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:10.605297 1167 tablet_bootstrap.cc:654] T 2e97d1bc06a24e8d8ba11acb6fd280d8 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:10.610610 1028 tablet_service.cc:1505] Processing CreateTablet for tablet 6acef3dd9c1b464ba33e11cdd66f1196 (DEFAULT_TABLE table=default.Table_954aab60929348dcbdbd69aa987112ac [id=a91ae536f3274f96bd6a78cb1dbedea4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.611003 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6acef3dd9c1b464ba33e11cdd66f1196. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.613802 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 9e6eb099648549dbbf9969e59865f51f (table default.Table_683d67689da74097b69f31b3d9461fe2 [id=0c5d3e4d34d045dc99ffc88e101cae1d]) successfully deleted
W20251029 02:52:10.614096 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 80f54d8a486a4196aca3c28c84c0872f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 80f54d8a486a4196aca3c28c84c0872f already in progress: creating tablet
W20251029 02:52:10.614312 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 9778f924d11a48b194ccf26124bd5509 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9778f924d11a48b194ccf26124bd5509 already in progress: creating tablet
I20251029 02:52:10.614789 1167 tablet_bootstrap.cc:492] T 2e97d1bc06a24e8d8ba11acb6fd280d8 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:10.615129 1167 ts_tablet_manager.cc:1403] T 2e97d1bc06a24e8d8ba11acb6fd280d8 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20251029 02:52:10.615132 1037 tablet_service.cc:1552] Processing DeleteTablet for tablet 23856678b7aa4f9ebf65fb4564a4bf91 with delete_type TABLET_DATA_DELETED (replaced by tablet 7fcb1aac63c84edba0a8b8c964a8b957) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:10.615576 1041 tablet_service.cc:1505] Processing CreateTablet for tablet 47348ffa21574e4084f0b725a78e2a70 (DEFAULT_TABLE table=default.Table_2aba26f0828343a6981aad6b2e3ac97f [id=a53e996da85d45b7a90ef1a33a74c094]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.615922 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47348ffa21574e4084f0b725a78e2a70. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.616752 520 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 862d268a526545ba8f25bb91c2992321
I20251029 02:52:10.618481 749 tablet_service.cc:1552] Processing DeleteTablet for tablet 862d268a526545ba8f25bb91c2992321 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:10 UTC) from {username='slave'} at 127.0.0.1:44580
W20251029 02:52:10.618670 518 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 055184c641ba45a588bca611fb24d130 consensus state has no leader
I20251029 02:52:10.618963 517 catalog_manager.cc:6194] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: default.Table_e75db5cf8f5f49ea837cb6d63f4b7ebf [id=86195563fb4a4d7abd746f4ac7cd241c] alter complete (version 1)
I20251029 02:52:10.620380 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet 43a76b984da3414da701221f87349337 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:09 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:10.622491 1029 tablet_service.cc:1505] Processing CreateTablet for tablet 41cfa2d722034618a05de717b28a778d (DEFAULT_TABLE table=default.Table_ffc17029e0734c73a3735b9f99cf1332 [id=0c8e5e5ec31c43bdbc827be824b65780]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.622877 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41cfa2d722034618a05de717b28a778d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.623456 1176 ts_tablet_manager.cc:1929] T 7a1866afc7564217ad777e7ce53786c5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:10.623559 1176 log.cc:1199] T 7a1866afc7564217ad777e7ce53786c5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/7a1866afc7564217ad777e7ce53786c5
I20251029 02:52:10.627624 1037 tablet_service.cc:1505] Processing CreateTablet for tablet dab6ed2266ed42508738399d388c4da3 (DEFAULT_TABLE table=default.Table_4e9d94609b4a4d1f94615dcc9830d717 [id=761ebdd0095c414488aa221cbc99d3bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.627712 1032 tablet_service.cc:1505] Processing CreateTablet for tablet a9e356c6846f47e68ba5fdfa7befc1cf (DEFAULT_TABLE table=default.Table_2577a8048e3345ce82eafa3657784d8b [id=98af8d8aca43483f88bf88d9962862ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.627977 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dab6ed2266ed42508738399d388c4da3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.630822 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a9e356c6846f47e68ba5fdfa7befc1cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.631057 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 818) took 2868 ms. Trace:
I20251029 02:52:10.631151 1174 rpcz_store.cc:276] 1029 02:52:07.762318 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:07.762368 (+ 50us) service_pool.cc:225] Handling call
1029 02:52:10.618317 (+2855949us) ts_tablet_manager.cc:1151] Deleting tablet ff3e9227b9674d31b874823cf4a62d9b
1029 02:52:10.618332 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:10.631031 (+ 12699us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2819257}
I20251029 02:52:10.631524 1174 tablet_replica.cc:333] T 56d05e4a500043c995bb51a204cb295d P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:10.631732 1174 raft_consensus.cc:2243] T 56d05e4a500043c995bb51a204cb295d P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:10.631891 1174 raft_consensus.cc:2272] T 56d05e4a500043c995bb51a204cb295d P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:10.632398 1174 ts_tablet_manager.cc:1916] T 56d05e4a500043c995bb51a204cb295d P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 02:52:10.633033 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet ff3e9227b9674d31b874823cf4a62d9b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ff3e9227b9674d31b874823cf4a62d9b already in progress: creating tablet
I20251029 02:52:10.633411 1025 tablet_service.cc:1505] Processing CreateTablet for tablet 13670ea867e349b2b5a25021469c32d3 (DEFAULT_TABLE table=default.Table_da047830460443a29c5269c98fc859e9 [id=f733948e92d8470da9d8050bd26a06b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.633514 1042 tablet_service.cc:1505] Processing CreateTablet for tablet eacad0b8458242efb48cc17351fb3f2f (DEFAULT_TABLE table=default.Table_3bc0c202b81d42c5a788393fe07d02a1 [id=72e5e134d7a54f5fb76630982ae948c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.633725 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 13670ea867e349b2b5a25021469c32d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.634002 1036 tablet_service.cc:1505] Processing CreateTablet for tablet 41a171c7ba55422b8700c4f898dc7d9e (DEFAULT_TABLE table=default.Table_6442d40b6bad4ac9a2dbd42a8a3cb096 [id=e530a85da85d4d8a96be6097e31fde58]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.633985 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eacad0b8458242efb48cc17351fb3f2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.633740 1176 ts_tablet_manager.cc:1950] T 7a1866afc7564217ad777e7ce53786c5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:10.634867 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 874) took 2105 ms. Trace:
I20251029 02:52:10.634972 1018 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:37622 (request call id 140) took 1151 ms. Trace:
I20251029 02:52:10.635078 1018 rpcz_store.cc:276] 1029 02:52:09.483309 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:09.483361 (+ 52us) service_pool.cc:225] Handling call
1029 02:52:09.573751 (+ 90390us) spinlock_profiling.cc:243] Waited 82.2 ms on lock 0x6160000e2054. stack: 00007f34717829f6 00007f34717827b0 00007f347075e3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f347d61c200 00007f34814cc66f 00007f347b8f2c36 00007f347b8f2965 00007f347872127e 00007f347872015e 00007f3478726395 00007f3478726cbf 00007f3478726ae8 0000000000a8867c
1029 02:52:09.649607 (+ 75856us) spinlock_profiling.cc:243] Waited 74.2 ms on lock 0x61800008248c. stack: 00007f34717829f6 00007f34717827b0 00007f347075e3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f347eba944e 00007f347e9a4ac1 00007f34814cc478 00007f347b8f2c36 00007f347b8f2965 00007f347872127e 00007f347872015e 00007f3478726395 00007f3478726cbf 00007f3478726ae8
1029 02:52:10.634951 (+985344us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":359793280}
I20251029 02:52:10.635401 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41a171c7ba55422b8700c4f898dc7d9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.634933 1176 rpcz_store.cc:276] 1029 02:52:08.529279 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:08.539454 (+ 10175us) service_pool.cc:225] Handling call
1029 02:52:10.012894 (+1473440us) ts_tablet_manager.cc:1151] Deleting tablet 7a1866afc7564217ad777e7ce53786c5
1029 02:52:10.012906 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:10.511193 (+498287us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:10.530453 (+ 19260us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:10.634852 (+104399us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1472974}
I20251029 02:52:10.636869 1176 tablet_replica.cc:333] T 0f80b840ed954230bfbfe7e40127cd2f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:10.637050 1176 raft_consensus.cc:2243] T 0f80b840ed954230bfbfe7e40127cd2f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:10.637362 1176 raft_consensus.cc:2272] T 0f80b840ed954230bfbfe7e40127cd2f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:10.638643 1026 tablet_service.cc:1552] Processing DeleteTablet for tablet 4660fd0534df49e0af43c36fb1122d84 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:09 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:10.639654 1167 raft_consensus.cc:359] T 2e97d1bc06a24e8d8ba11acb6fd280d8 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:10.639843 1167 raft_consensus.cc:385] T 2e97d1bc06a24e8d8ba11acb6fd280d8 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:10.639889 1167 raft_consensus.cc:740] T 2e97d1bc06a24e8d8ba11acb6fd280d8 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:10.640575 895 tablet_service.cc:1552] Processing DeleteTablet for tablet 738046ab57334f2dab9f4aab979c5b42 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:04 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:10.642294 521 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 34d141bee474452693dba1b9d7f2b023
W20251029 02:52:10.644325 523 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7fba607c566c4fa1a803cba8d86ca646 consensus state has no leader
I20251029 02:52:10.647034 749 tablet_service.cc:1552] Processing DeleteTablet for tablet 34d141bee474452693dba1b9d7f2b023 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:10 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:10.647534 895 tablet_service.cc:1552] Processing DeleteTablet for tablet e7f846b294cc4eb687a26c8b5931ac19 with delete_type TABLET_DATA_DELETED (replaced by tablet cdc5fb2b13e042d680fa3f2eea2d62fa) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:10.640056 1167 consensus_queue.cc:260] T 2e97d1bc06a24e8d8ba11acb6fd280d8 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:10.638425 1176 ts_tablet_manager.cc:1916] T 0f80b840ed954230bfbfe7e40127cd2f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:10.653856 879 tablet_service.cc:1552] Processing DeleteTablet for tablet 30d03e1e42d34242bd354f9e9ec4f127 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:58 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:10.655140 895 tablet_service.cc:1552] Processing DeleteTablet for tablet 59d03a190b784ab18a3e63a572675891 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:57 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:10.655340 878 tablet_service.cc:1552] Processing DeleteTablet for tablet 34d141bee474452693dba1b9d7f2b023 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:10 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:10.655727 879 tablet_service.cc:1552] Processing DeleteTablet for tablet 7a1866afc7564217ad777e7ce53786c5 with delete_type TABLET_DATA_DELETED (replaced by tablet dc33fd3404ec41c3935b02c84a12971b) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:10.656944 1167 ts_tablet_manager.cc:1434] T 2e97d1bc06a24e8d8ba11acb6fd280d8 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.042s user 0.002s sys 0.000s
I20251029 02:52:10.657264 1167 tablet_bootstrap.cc:492] T 061f3d38c4604fa4b57e74ff6f6fc7da P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:10.658420 1172 tablet_bootstrap.cc:492] T 40e844fd7ad74680bcbe7bef936c5b9d P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:10.658610 900 tablet_service.cc:1552] Processing DeleteTablet for tablet 862d268a526545ba8f25bb91c2992321 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:10 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:10.663664 1167 tablet_bootstrap.cc:654] T 061f3d38c4604fa4b57e74ff6f6fc7da P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:10.664325 1172 ts_tablet_manager.cc:1403] T 40e844fd7ad74680bcbe7bef936c5b9d P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.107s user 0.000s sys 0.004s
I20251029 02:52:10.665207 2179 raft_consensus.cc:493] T 82289e0f823c4d43bfd8c4e878572b2b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:10.665395 2179 raft_consensus.cc:515] T 82289e0f823c4d43bfd8c4e878572b2b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:10.665591 1172 raft_consensus.cc:359] T 40e844fd7ad74680bcbe7bef936c5b9d P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:10.665758 1172 raft_consensus.cc:385] T 40e844fd7ad74680bcbe7bef936c5b9d P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:10.665802 1172 raft_consensus.cc:740] T 40e844fd7ad74680bcbe7bef936c5b9d P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:10.666172 2179 leader_election.cc:290] T 82289e0f823c4d43bfd8c4e878572b2b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:10.668514 1172 consensus_queue.cc:260] T 40e844fd7ad74680bcbe7bef936c5b9d P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:10.669466 1172 ts_tablet_manager.cc:1434] T 40e844fd7ad74680bcbe7bef936c5b9d P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20251029 02:52:10.671010 1172 tablet_bootstrap.cc:492] T 20a762344c5f46f88481c4ac7ad6e43b P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:10.675525 1175 ts_tablet_manager.cc:1929] T 13ee62fc2339410b9db70dfd3d3c6fa6 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:10.675555 1174 ts_tablet_manager.cc:1929] T 56d05e4a500043c995bb51a204cb295d P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:10.675657 1174 log.cc:1199] T 56d05e4a500043c995bb51a204cb295d P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/56d05e4a500043c995bb51a204cb295d
I20251029 02:52:10.679461 1174 ts_tablet_manager.cc:1950] T 56d05e4a500043c995bb51a204cb295d P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:10.680733 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 819) took 2915 ms. Trace:
I20251029 02:52:10.685038 1174 rpcz_store.cc:276] 1029 02:52:07.765766 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:07.765833 (+ 67us) service_pool.cc:225] Handling call
1029 02:52:10.631478 (+2865645us) ts_tablet_manager.cc:1151] Deleting tablet 56d05e4a500043c995bb51a204cb295d
1029 02:52:10.631489 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:10.651406 (+ 19917us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:10.675466 (+ 24060us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:10.680711 (+ 5245us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2832376}
I20251029 02:52:10.688886 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82289e0f823c4d43bfd8c4e878572b2b" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:10.688917 2205 raft_consensus.cc:493] T 7a1866afc7564217ad777e7ce53786c5 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:10.689416 2205 raft_consensus.cc:515] T 7a1866afc7564217ad777e7ce53786c5 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:10.685792 1174 tablet_replica.cc:333] T dd6ad641fbda472fab07aa35c5e4523a P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:10.689698 1174 raft_consensus.cc:2243] T dd6ad641fbda472fab07aa35c5e4523a P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:10.689831 1174 raft_consensus.cc:2272] T dd6ad641fbda472fab07aa35c5e4523a P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus is shut down!
W20251029 02:52:10.689812 990 leader_election.cc:343] T 2f198c3259864f7ab3625cb7bfae20b0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:10.690009 2205 leader_election.cc:290] T 7a1866afc7564217ad777e7ce53786c5 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:10.690342 1174 ts_tablet_manager.cc:1916] T dd6ad641fbda472fab07aa35c5e4523a P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 02:52:10.690501 990 leader_election.cc:343] T 82289e0f823c4d43bfd8c4e878572b2b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
W20251029 02:52:10.691089 847 leader_election.cc:343] T 7a1866afc7564217ad777e7ce53786c5 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 7a1866afc7564217ad777e7ce53786c5
W20251029 02:52:10.695065 849 leader_election.cc:343] T 7a1866afc7564217ad777e7ce53786c5 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Not found: Tablet not found: 7a1866afc7564217ad777e7ce53786c5
I20251029 02:52:10.695431 849 leader_election.cc:304] T 7a1866afc7564217ad777e7ce53786c5 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9
I20251029 02:52:10.695780 2205 raft_consensus.cc:2749] T 7a1866afc7564217ad777e7ce53786c5 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:10.678776 749 tablet_service.cc:1552] Processing DeleteTablet for tablet 40e844fd7ad74680bcbe7bef936c5b9d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:59 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:10.682286 1172 tablet_bootstrap.cc:654] T 20a762344c5f46f88481c4ac7ad6e43b P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:10.700798 1175 log.cc:1199] T 13ee62fc2339410b9db70dfd3d3c6fa6 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/13ee62fc2339410b9db70dfd3d3c6fa6
I20251029 02:52:10.701049 2179 consensus_queue.cc:1048] T 13ee62fc2339410b9db70dfd3d3c6fa6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:10.701282 1175 ts_tablet_manager.cc:1950] T 13ee62fc2339410b9db70dfd3d3c6fa6 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:10.701900 1167 tablet_bootstrap.cc:492] T 061f3d38c4604fa4b57e74ff6f6fc7da P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:10.702044 1167 ts_tablet_manager.cc:1403] T 061f3d38c4604fa4b57e74ff6f6fc7da P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.045s user 0.002s sys 0.006s
I20251029 02:52:10.702064 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 810) took 2931 ms. Trace:
I20251029 02:52:10.702136 1175 rpcz_store.cc:276] 1029 02:52:07.770575 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:07.770629 (+ 54us) service_pool.cc:225] Handling call
1029 02:52:10.594999 (+2824370us) ts_tablet_manager.cc:1151] Deleting tablet 13ee62fc2339410b9db70dfd3d3c6fa6
1029 02:52:10.595010 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:10.625358 (+ 30348us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:10.675402 (+ 50044us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:10.702040 (+ 26638us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2824166}
I20251029 02:52:10.702464 1175 tablet_replica.cc:333] T b6d4f7bf702f4834bc5f4ef2b50b92a7 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:10.702685 1175 raft_consensus.cc:2243] T b6d4f7bf702f4834bc5f4ef2b50b92a7 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:10.702826 1167 raft_consensus.cc:359] T 061f3d38c4604fa4b57e74ff6f6fc7da P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:10.703006 1167 raft_consensus.cc:385] T 061f3d38c4604fa4b57e74ff6f6fc7da P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:10.702944 1175 raft_consensus.cc:2272] T b6d4f7bf702f4834bc5f4ef2b50b92a7 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:10.703080 1167 raft_consensus.cc:740] T 061f3d38c4604fa4b57e74ff6f6fc7da P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:10.702994 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 13ee62fc2339410b9db70dfd3d3c6fa6 (table default.Table_cb4697efc8e049be929f4e7a004879d1 [id=b54d7930e083434680f3934fc4023653]) successfully deleted
I20251029 02:52:10.703317 1167 consensus_queue.cc:260] T 061f3d38c4604fa4b57e74ff6f6fc7da P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:10.703920 1167 ts_tablet_manager.cc:1434] T 061f3d38c4604fa4b57e74ff6f6fc7da P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 02:52:10.704209 1167 tablet_bootstrap.cc:492] T 8048506ced4540a5aee9f79bfc4599bf P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:10.704486 1175 ts_tablet_manager.cc:1916] T b6d4f7bf702f4834bc5f4ef2b50b92a7 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:10.705767 1167 tablet_bootstrap.cc:654] T 8048506ced4540a5aee9f79bfc4599bf P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:10.709792 2309 consensus_queue.cc:1048] T 28f7ceb91a34450cad4b5b45f5285b41 P b495a49349324f7d81908f2da700f0d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:10.714713 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 56d05e4a500043c995bb51a204cb295d (table default.Table_0c1681c9cd0347f2ba2eab73b8cf11a9 [id=fdb365438df14f12a382aa046aef41a4]) successfully deleted
I20251029 02:52:10.714952 1172 tablet_bootstrap.cc:492] T 20a762344c5f46f88481c4ac7ad6e43b P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:10.715144 1172 ts_tablet_manager.cc:1403] T 20a762344c5f46f88481c4ac7ad6e43b P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.044s user 0.004s sys 0.004s
I20251029 02:52:10.716054 1172 raft_consensus.cc:359] T 20a762344c5f46f88481c4ac7ad6e43b P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:10.716322 1172 raft_consensus.cc:385] T 20a762344c5f46f88481c4ac7ad6e43b P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:10.716418 1172 raft_consensus.cc:740] T 20a762344c5f46f88481c4ac7ad6e43b P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:10.716691 1172 consensus_queue.cc:260] T 20a762344c5f46f88481c4ac7ad6e43b P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:10.717403 1172 ts_tablet_manager.cc:1434] T 20a762344c5f46f88481c4ac7ad6e43b P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251029 02:52:10.717743 1172 tablet_bootstrap.cc:492] T 27b08f88b99643d28ac609e45bf8104e P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:10.720005 1172 tablet_bootstrap.cc:654] T 27b08f88b99643d28ac609e45bf8104e P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:10.720855 1174 ts_tablet_manager.cc:1929] T dd6ad641fbda472fab07aa35c5e4523a P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:10.724993 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_75f0822a7de44b8ea84235f6790a2140"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:10.725658 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_75f0822a7de44b8ea84235f6790a2140 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:10.725652 1041 tablet_service.cc:1505] Processing CreateTablet for tablet 0d788aeac52047c6bd9edcaf1369c35c (DEFAULT_TABLE table=default.Table_f7fa4c862f364b6eb8510a09d693001b [id=2568391dc7c646eaaa5adcc7627d785e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.726094 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d788aeac52047c6bd9edcaf1369c35c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.726575 2179 raft_consensus.cc:493] T adf6e736cb9c4920807ee58682e134ef P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:10.726686 2179 raft_consensus.cc:515] T adf6e736cb9c4920807ee58682e134ef P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:10.727212 2179 leader_election.cc:290] T adf6e736cb9c4920807ee58682e134ef P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:10.736076 519 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_a4f394d1b04146848a9a3dafc0919f0e" } new_table_name: "default.Table_3d4aa5bec3c749139b0692e6cc06b9db" modify_external_catalogs: true
I20251029 02:52:10.738183 524 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_bdc0e5608cc44945849e3a3744ca6d9d" } new_table_name: "default.Table_fd38cb7a06ca47d9b65efd17ae9c5d28" modify_external_catalogs: true
I20251029 02:52:10.727785 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "adf6e736cb9c4920807ee58682e134ef" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:10.738668 1032 tablet_service.cc:1505] Processing CreateTablet for tablet 08ca0c889f2949408657fa60ce087608 (DEFAULT_TABLE table=default.Table_b708124baf534c96b3f4b8a7a4bb11cc [id=046010a03d774e91b0ad10df97848c56]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.739060 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08ca0c889f2949408657fa60ce087608. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:10.739194 990 leader_election.cc:343] T adf6e736cb9c4920807ee58682e134ef P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:10.739970 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_f0423becd1a243ff9b81ce1b985f083e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:10.740401 847 consensus_peers.cc:597] T 0d84dc8dc465496f9adec798b2e03e13 P c756528642a543eeacd93cef3e335a7d -> Peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Couldn't send request to peer b495a49349324f7d81908f2da700f0d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0d84dc8dc465496f9adec798b2e03e13. This is attempt 1: this message will repeat every 5th retry.
W20251029 02:52:10.740541 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f0423becd1a243ff9b81ce1b985f083e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:10.743192 1037 tablet_service.cc:1505] Processing CreateTablet for tablet 70850519135644d1ac3d596ecf67221f (DEFAULT_TABLE table=default.Table_99567b9d8c154303bae22fda47db1d0a [id=2b8bd2d2c2664edaad536b1d7706096b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.743606 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70850519135644d1ac3d596ecf67221f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.744956 1041 tablet_service.cc:1552] Processing DeleteTablet for tablet eeb75fc46e2a41f78252868ad5256966 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:09 UTC) from {username='slave'} at 127.0.0.1:37626
W20251029 02:52:10.749622 519 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9099fb43dbec46cd83f2cb3f4796ecc6 consensus state has no leader
I20251029 02:52:10.754581 2205 raft_consensus.cc:493] T 2e97d1bc06a24e8d8ba11acb6fd280d8 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:10.754773 2205 raft_consensus.cc:515] T 2e97d1bc06a24e8d8ba11acb6fd280d8 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:10.755565 2205 leader_election.cc:290] T 2e97d1bc06a24e8d8ba11acb6fd280d8 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:10.756393 1057 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e97d1bc06a24e8d8ba11acb6fd280d8" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:10.756575 1057 raft_consensus.cc:2468] T 2e97d1bc06a24e8d8ba11acb6fd280d8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:10.756876 766 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e97d1bc06a24e8d8ba11acb6fd280d8" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:10.757138 766 raft_consensus.cc:2468] T 2e97d1bc06a24e8d8ba11acb6fd280d8 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:10.757905 849 leader_election.cc:304] T 2e97d1bc06a24e8d8ba11acb6fd280d8 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:10.758651 2205 raft_consensus.cc:2804] T 2e97d1bc06a24e8d8ba11acb6fd280d8 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:10.758754 2205 raft_consensus.cc:493] T 2e97d1bc06a24e8d8ba11acb6fd280d8 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:10.758836 2205 raft_consensus.cc:3060] T 2e97d1bc06a24e8d8ba11acb6fd280d8 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
W20251029 02:52:10.758948 524 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 04528b4b700744aab24f6b1d5cb2787a consensus state has no leader
I20251029 02:52:10.762423 1175 ts_tablet_manager.cc:1929] T b6d4f7bf702f4834bc5f4ef2b50b92a7 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:10.762583 1175 log.cc:1199] T b6d4f7bf702f4834bc5f4ef2b50b92a7 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/b6d4f7bf702f4834bc5f4ef2b50b92a7
I20251029 02:52:10.763681 1175 ts_tablet_manager.cc:1950] T b6d4f7bf702f4834bc5f4ef2b50b92a7 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:10.763448 523 catalog_manager.cc:5649] T 95f342d1cfb84c4eb31bd3d15ea3bdc2 P c756528642a543eeacd93cef3e335a7d reported cstate change: term changed from 0 to 1, leader changed from <none> to 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } }
I20251029 02:52:10.764039 523 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet 95f342d1cfb84c4eb31bd3d15ea3bdc2 (table default.Table_96ea715e50e74cc48759dda025a46318 [id=c63d52811db749d6badbed72fc133943]). Expected version 1 got 0
I20251029 02:52:10.764953 1174 log.cc:1199] T dd6ad641fbda472fab07aa35c5e4523a P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/dd6ad641fbda472fab07aa35c5e4523a
I20251029 02:52:10.765331 1174 ts_tablet_manager.cc:1950] T dd6ad641fbda472fab07aa35c5e4523a P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:10.766899 1176 ts_tablet_manager.cc:1929] T 0f80b840ed954230bfbfe7e40127cd2f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:10.767127 1176 log.cc:1199] T 0f80b840ed954230bfbfe7e40127cd2f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/0f80b840ed954230bfbfe7e40127cd2f
I20251029 02:52:10.767391 524 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_17afc4bb94e7410d8a45c9430e402e8c" } modify_external_catalogs: true
I20251029 02:52:10.767607 524 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_17afc4bb94e7410d8a45c9430e402e8c" } modify_external_catalogs: true
I20251029 02:52:10.765398 749 tablet_service.cc:1505] Processing CreateTablet for tablet 1150d2210a234f1aa554f486cdcabc4c (DEFAULT_TABLE table=default.Table_4a6906297ac74f45b5f153339e2713c3 [id=cc511ee9649842ba818d94c8831d9b2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.768321 1176 ts_tablet_manager.cc:1950] T 0f80b840ed954230bfbfe7e40127cd2f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:10.768692 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 873) took 2239 ms. Trace:
I20251029 02:52:10.768810 1176 rpcz_store.cc:276] 1029 02:52:08.529181 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:08.541730 (+ 12549us) service_pool.cc:225] Handling call
1029 02:52:10.636831 (+2095101us) ts_tablet_manager.cc:1151] Deleting tablet 0f80b840ed954230bfbfe7e40127cd2f
1029 02:52:10.636839 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:10.739790 (+102951us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:10.766748 (+ 26958us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:10.768673 (+ 1925us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2094880}
I20251029 02:52:10.770138 1028 tablet_service.cc:1505] Processing CreateTablet for tablet 0700b93cc398403d8b3602e84182a3ce (DEFAULT_TABLE table=default.Table_e64c529c8e2d48b487caaa29ed3d17d8 [id=a436307fe09d4ac08c4b98d3d2cfce63]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.777876 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0700b93cc398403d8b3602e84182a3ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.779093 519 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_fdb0a130609a45878b3a6f07cf79947c" } modify_external_catalogs: true
I20251029 02:52:10.779112 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_0f172f8e7eb649ffa6f1723cab301cc5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:10.779291 519 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_fdb0a130609a45878b3a6f07cf79947c" } modify_external_catalogs: true
W20251029 02:52:10.779670 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0f172f8e7eb649ffa6f1723cab301cc5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:10.780354 1176 tablet_replica.cc:333] T 777992acb1bd4765886b802940db74bb P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:10.780532 1176 raft_consensus.cc:2243] T 777992acb1bd4765886b802940db74bb P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:10.780551 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_a1ba26e950644781ba8d825e5472d758"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:10.780663 1176 raft_consensus.cc:2272] T 777992acb1bd4765886b802940db74bb P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:10.781054 524 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 5a1aa131cf4e4edca5bb2515edd7556a
I20251029 02:52:10.781191 1176 ts_tablet_manager.cc:1916] T 777992acb1bd4765886b802940db74bb P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 02:52:10.781054 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a1ba26e950644781ba8d825e5472d758 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:10.782109 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1150d2210a234f1aa554f486cdcabc4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.783525 740 tablet_service.cc:1552] Processing DeleteTablet for tablet 5a1aa131cf4e4edca5bb2515edd7556a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:10 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:10.783919 517 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_73edd4fe45f94ebab19f2a834a265b63" } modify_external_catalogs: true
I20251029 02:52:10.784098 517 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_73edd4fe45f94ebab19f2a834a265b63" } modify_external_catalogs: true
I20251029 02:52:10.785944 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_11dd5d796fa041058d7cbc7ce4ef730d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:10.786499 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_11dd5d796fa041058d7cbc7ce4ef730d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:10.788877 522 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_a27ea57c89b0433d8b21e21e88948bde" } new_table_name: "default.Table_4873cb28e4c84935ab481f06ad654716" modify_external_catalogs: true
I20251029 02:52:10.792929 740 tablet_service.cc:1505] Processing CreateTablet for tablet 3e6c048371984e36a44613294eed534f (DEFAULT_TABLE table=default.Table_b68e55dfa3d04c8d95c29f27537f59ad [id=53952ef589954f1c94fa49b5fa0fb623]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.793306 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e6c048371984e36a44613294eed534f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.765775 900 tablet_service.cc:1505] Processing CreateTablet for tablet 1150d2210a234f1aa554f486cdcabc4c (DEFAULT_TABLE table=default.Table_4a6906297ac74f45b5f153339e2713c3 [id=cc511ee9649842ba818d94c8831d9b2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.794785 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1150d2210a234f1aa554f486cdcabc4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.769384 2205 raft_consensus.cc:515] T 2e97d1bc06a24e8d8ba11acb6fd280d8 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:10.777387 2302 raft_consensus.cc:493] T 738046ab57334f2dab9f4aab979c5b42 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:10.798630 2302 raft_consensus.cc:515] T 738046ab57334f2dab9f4aab979c5b42 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:10.799008 519 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 8378d67f1da743cca91803e37ea54281
I20251029 02:52:10.799317 2302 leader_election.cc:290] T 738046ab57334f2dab9f4aab979c5b42 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:10.788383 878 tablet_service.cc:1552] Processing DeleteTablet for tablet 6d830d30b26b4ba799785fec8a6f0769 with delete_type TABLET_DATA_DELETED (replaced by tablet abf18e06df9b482a8b25c078b9495272) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:10.788501 879 tablet_service.cc:1552] Processing DeleteTablet for tablet 5a1aa131cf4e4edca5bb2515edd7556a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:10 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:10.791837 895 tablet_service.cc:1505] Processing CreateTablet for tablet 3e6c048371984e36a44613294eed534f (DEFAULT_TABLE table=default.Table_b68e55dfa3d04c8d95c29f27537f59ad [id=53952ef589954f1c94fa49b5fa0fb623]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.800112 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e6c048371984e36a44613294eed534f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.801404 2205 leader_election.cc:290] T 2e97d1bc06a24e8d8ba11acb6fd280d8 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:10.802968 879 tablet_service.cc:1552] Processing DeleteTablet for tablet 8378d67f1da743cca91803e37ea54281 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:10 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:10.803886 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e97d1bc06a24e8d8ba11acb6fd280d8" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:10.804066 773 raft_consensus.cc:3060] T 2e97d1bc06a24e8d8ba11acb6fd280d8 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:10.804981 879 tablet_service.cc:1505] Processing CreateTablet for tablet 68883d80d9744f7bac4d7f4f72283d85 (DEFAULT_TABLE table=default.Table_28bb97b95f7942fb9da0cec248e6ac6d [id=77f674208cb649a59af0aa91e5e1780f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.805176 749 tablet_service.cc:1552] Processing DeleteTablet for tablet 8378d67f1da743cca91803e37ea54281 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:10 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:10.805349 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68883d80d9744f7bac4d7f4f72283d85. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.807312 526 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet 95f342d1cfb84c4eb31bd3d15ea3bdc2 (table default.Table_96ea715e50e74cc48759dda025a46318 [id=c63d52811db749d6badbed72fc133943]). Expected version 1 got 0
I20251029 02:52:10.807794 766 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "738046ab57334f2dab9f4aab979c5b42" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:10.808017 766 raft_consensus.cc:2468] T 738046ab57334f2dab9f4aab979c5b42 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:10.808604 847 leader_election.cc:304] T 738046ab57334f2dab9f4aab979c5b42 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:10.809343 878 tablet_service.cc:1552] Processing DeleteTablet for tablet 2e97d1bc06a24e8d8ba11acb6fd280d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:55 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:10.809414 773 raft_consensus.cc:2468] T 2e97d1bc06a24e8d8ba11acb6fd280d8 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:10.809459 885 tablet_service.cc:1552] Processing DeleteTablet for tablet 061f3d38c4604fa4b57e74ff6f6fc7da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:58 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:10.809868 847 leader_election.cc:304] T 2e97d1bc06a24e8d8ba11acb6fd280d8 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:10.810163 2302 raft_consensus.cc:2804] T 738046ab57334f2dab9f4aab979c5b42 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:10.810258 2302 raft_consensus.cc:493] T 738046ab57334f2dab9f4aab979c5b42 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:10.810333 2302 raft_consensus.cc:3060] T 738046ab57334f2dab9f4aab979c5b42 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:10.810684 885 tablet_service.cc:1505] Processing CreateTablet for tablet e2a739ebb4aa446fba435a9aecd28ed5 (DEFAULT_TABLE table=default.Table_ff774195536549b3b3a651c34438f014 [id=a610d1dc5b3e468c86c1f196e2ccdc54]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.811000 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2a739ebb4aa446fba435a9aecd28ed5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.815316 2302 raft_consensus.cc:515] T 738046ab57334f2dab9f4aab979c5b42 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:10.817327 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "738046ab57334f2dab9f4aab979c5b42" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:10.817586 773 raft_consensus.cc:3060] T 738046ab57334f2dab9f4aab979c5b42 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:10.817899 740 tablet_service.cc:1505] Processing CreateTablet for tablet e2a739ebb4aa446fba435a9aecd28ed5 (DEFAULT_TABLE table=default.Table_ff774195536549b3b3a651c34438f014 [id=a610d1dc5b3e468c86c1f196e2ccdc54]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.818341 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2a739ebb4aa446fba435a9aecd28ed5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.794397 1167 tablet_bootstrap.cc:492] T 8048506ced4540a5aee9f79bfc4599bf P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:10.819694 526 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_eb665fea357646b1bfc96cd81dc7cbcc" } modify_external_catalogs: true
I20251029 02:52:10.819912 1167 ts_tablet_manager.cc:1403] T 8048506ced4540a5aee9f79bfc4599bf P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.116s user 0.004s sys 0.000s
I20251029 02:52:10.820106 526 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_eb665fea357646b1bfc96cd81dc7cbcc" } modify_external_catalogs: true
I20251029 02:52:10.820673 1167 raft_consensus.cc:359] T 8048506ced4540a5aee9f79bfc4599bf P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:10.820842 1167 raft_consensus.cc:385] T 8048506ced4540a5aee9f79bfc4599bf P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:10.820896 1167 raft_consensus.cc:740] T 8048506ced4540a5aee9f79bfc4599bf P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:10.821171 2302 leader_election.cc:290] T 738046ab57334f2dab9f4aab979c5b42 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:10.822317 519 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_330b705cd45340578758a7fb532d14fa" } new_table_name: "default.Table_fdb7650da3b84059a8aa5d2dc28fb8f5" modify_external_catalogs: true
I20251029 02:52:10.823679 2302 raft_consensus.cc:2804] T 2e97d1bc06a24e8d8ba11acb6fd280d8 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:10.823879 2302 raft_consensus.cc:697] T 2e97d1bc06a24e8d8ba11acb6fd280d8 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:52:10.824220 2302 consensus_queue.cc:237] T 2e97d1bc06a24e8d8ba11acb6fd280d8 P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:10.825876 773 raft_consensus.cc:2468] T 738046ab57334f2dab9f4aab979c5b42 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:10.826221 770 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "95f342d1cfb84c4eb31bd3d15ea3bdc2" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:10.826393 770 raft_consensus.cc:3060] T 95f342d1cfb84c4eb31bd3d15ea3bdc2 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:10.826752 847 leader_election.cc:304] T 738046ab57334f2dab9f4aab979c5b42 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:10.827441 2205 raft_consensus.cc:2804] T 738046ab57334f2dab9f4aab979c5b42 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:10.827446 767 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "44b518c3a0964a6fa863b2784d80c8a4" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:10.827607 767 raft_consensus.cc:3060] T 44b518c3a0964a6fa863b2784d80c8a4 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:10.827857 2205 raft_consensus.cc:697] T 738046ab57334f2dab9f4aab979c5b42 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:52:10.828145 2205 consensus_queue.cc:237] T 738046ab57334f2dab9f4aab979c5b42 P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:10.829145 770 raft_consensus.cc:2468] T 95f342d1cfb84c4eb31bd3d15ea3bdc2 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 1.
I20251029 02:52:10.830024 2113 raft_consensus.cc:493] T 67651306f3b44e7eb14dcee9a2007a81 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:10.830142 2113 raft_consensus.cc:515] T 67651306f3b44e7eb14dcee9a2007a81 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:10.830416 767 raft_consensus.cc:2468] T 44b518c3a0964a6fa863b2784d80c8a4 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 1.
I20251029 02:52:10.830806 2113 leader_election.cc:290] T 67651306f3b44e7eb14dcee9a2007a81 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:10.831301 775 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dabb8d760aae4aa09eb83cdb586c9788" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:10.831468 775 raft_consensus.cc:2468] T dabb8d760aae4aa09eb83cdb586c9788 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 0.
I20251029 02:52:10.831715 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "061f3d38c4604fa4b57e74ff6f6fc7da" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:10.831707 918 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "67651306f3b44e7eb14dcee9a2007a81" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:10.831873 772 raft_consensus.cc:3060] T 061f3d38c4604fa4b57e74ff6f6fc7da P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
W20251029 02:52:10.832209 711 leader_election.cc:343] T 67651306f3b44e7eb14dcee9a2007a81 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:10.834731 765 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d830d30b26b4ba799785fec8a6f0769" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:10.835052 765 raft_consensus.cc:2393] T 6d830d30b26b4ba799785fec8a6f0769 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in current term 1: Already voted for candidate b495a49349324f7d81908f2da700f0d9 in this term.
I20251029 02:52:10.835422 764 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dabb8d760aae4aa09eb83cdb586c9788" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:10.835595 764 raft_consensus.cc:3060] T dabb8d760aae4aa09eb83cdb586c9788 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:10.836432 738 tablet_service.cc:1505] Processing CreateTablet for tablet 9b06a5d8a32341d19807ae0a99819916 (DEFAULT_TABLE table=default.Table_443fde25ba1b481f9146d61ad842bc9b [id=21b310f695e24d089cb6f90ed70fad5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.836800 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b06a5d8a32341d19807ae0a99819916. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.838138 2309 raft_consensus.cc:493] T 58dde9397b434b089ade9968f369006a P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:10.838259 2309 raft_consensus.cc:515] T 58dde9397b434b089ade9968f369006a P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:10.838891 2309 leader_election.cc:290] T 58dde9397b434b089ade9968f369006a P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:10.839493 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9269622ca92645d5aa8e73341201ccbb" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:10.839596 918 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "58dde9397b434b089ade9968f369006a" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:10.839665 776 raft_consensus.cc:3060] T 9269622ca92645d5aa8e73341201ccbb P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:10.840013 885 tablet_service.cc:1505] Processing CreateTablet for tablet 7751599f5dfc43baa20dc3487c6eddd9 (DEFAULT_TABLE table=default.Table_07d2f4a3c8074ece8b4982e3819caabc [id=8ad6d1d4fbe9404f8984dc8dc98916b7]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 02:52:10.840113 711 leader_election.cc:343] T 58dde9397b434b089ade9968f369006a P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:10.840067 762 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30d03e1e42d34242bd354f9e9ec4f127" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:10.840276 762 raft_consensus.cc:2468] T 30d03e1e42d34242bd354f9e9ec4f127 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 0.
I20251029 02:52:10.840404 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7751599f5dfc43baa20dc3487c6eddd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.840955 521 catalog_manager.cc:5649] T 44b518c3a0964a6fa863b2784d80c8a4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } health_report { overall_health: HEALTHY } } }
I20251029 02:52:10.841153 768 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9269622ca92645d5aa8e73341201ccbb" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:10.841503 521 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet 44b518c3a0964a6fa863b2784d80c8a4 (table default.Table_9120016827e946cc8a145252d2569d2b [id=e0c0cf3d456444c191a32bb180e83798]). Expected version 1 got 0
I20251029 02:52:10.841559 763 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "217144957930495aa3e9dfcfde268e71" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:10.841727 763 raft_consensus.cc:2468] T 217144957930495aa3e9dfcfde268e71 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 0.
I20251029 02:52:10.842031 521 catalog_manager.cc:5649] T 9269622ca92645d5aa8e73341201ccbb P 0cfd4bee16f84b7c8b0ff3ce14db20d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } health_report { overall_health: HEALTHY } } }
I20251029 02:52:10.842834 762 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "30d03e1e42d34242bd354f9e9ec4f127" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:10.843041 762 raft_consensus.cc:3060] T 30d03e1e42d34242bd354f9e9ec4f127 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:10.849495 879 tablet_service.cc:1505] Processing CreateTablet for tablet 9b06a5d8a32341d19807ae0a99819916 (DEFAULT_TABLE table=default.Table_443fde25ba1b481f9146d61ad842bc9b [id=21b310f695e24d089cb6f90ed70fad5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.849951 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b06a5d8a32341d19807ae0a99819916. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.853812 1167 consensus_queue.cc:260] T 8048506ced4540a5aee9f79bfc4599bf P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:10.860376 1167 ts_tablet_manager.cc:1434] T 8048506ced4540a5aee9f79bfc4599bf P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.040s user 0.002s sys 0.000s
I20251029 02:52:10.860834 1167 tablet_bootstrap.cc:492] T af4860c6b72f4d988b206b5ef2f22906 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:10.862957 1167 tablet_bootstrap.cc:654] T af4860c6b72f4d988b206b5ef2f22906 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:10.863209 766 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2f198c3259864f7ab3625cb7bfae20b0" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:10.863472 766 raft_consensus.cc:2468] T 2f198c3259864f7ab3625cb7bfae20b0 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 0.
I20251029 02:52:10.864246 756 tablet_service.cc:1505] Processing CreateTablet for tablet 7751599f5dfc43baa20dc3487c6eddd9 (DEFAULT_TABLE table=default.Table_07d2f4a3c8074ece8b4982e3819caabc [id=8ad6d1d4fbe9404f8984dc8dc98916b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.864689 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7751599f5dfc43baa20dc3487c6eddd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.865226 763 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97b2ecae572a4b7ea1b9595074556708" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:10.865418 763 raft_consensus.cc:2468] T 97b2ecae572a4b7ea1b9595074556708 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 0.
I20251029 02:52:10.867342 740 tablet_service.cc:1505] Processing CreateTablet for tablet ea1401ae51524d4380fba75d762031f3 (DEFAULT_TABLE table=default.Table_d518217e586b4a57a08855986cc0026b [id=23ac8ca52fc64ba4aa2bc2c3515628eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.867668 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea1401ae51524d4380fba75d762031f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.869544 776 raft_consensus.cc:2468] T 9269622ca92645d5aa8e73341201ccbb P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 1.
I20251029 02:52:10.871721 900 tablet_service.cc:1505] Processing CreateTablet for tablet ea1401ae51524d4380fba75d762031f3 (DEFAULT_TABLE table=default.Table_d518217e586b4a57a08855986cc0026b [id=23ac8ca52fc64ba4aa2bc2c3515628eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.872186 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea1401ae51524d4380fba75d762031f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.873302 762 raft_consensus.cc:2468] T 30d03e1e42d34242bd354f9e9ec4f127 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 1.
I20251029 02:52:10.874318 772 raft_consensus.cc:2468] T 061f3d38c4604fa4b57e74ff6f6fc7da P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 1.
I20251029 02:52:10.874768 764 raft_consensus.cc:2468] T dabb8d760aae4aa09eb83cdb586c9788 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 1.
I20251029 02:52:10.876405 764 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c32c28b0e74a4af3a0e69bbf24ab1a1f" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:10.876608 764 raft_consensus.cc:2468] T c32c28b0e74a4af3a0e69bbf24ab1a1f P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 0.
I20251029 02:52:10.879639 885 tablet_service.cc:1505] Processing CreateTablet for tablet 60f0eadc074a4872b6638314fed98e0e (DEFAULT_TABLE table=default.Table_6d9b81aa4e3a4880ac256ecf0fe9f010 [id=c65df2dc54134ab5afafee609470e144]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.880033 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60f0eadc074a4872b6638314fed98e0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.882663 738 tablet_service.cc:1505] Processing CreateTablet for tablet 60f0eadc074a4872b6638314fed98e0e (DEFAULT_TABLE table=default.Table_6d9b81aa4e3a4880ac256ecf0fe9f010 [id=c65df2dc54134ab5afafee609470e144]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.883147 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60f0eadc074a4872b6638314fed98e0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.884101 749 tablet_service.cc:1505] Processing CreateTablet for tablet 68883d80d9744f7bac4d7f4f72283d85 (DEFAULT_TABLE table=default.Table_28bb97b95f7942fb9da0cec248e6ac6d [id=77f674208cb649a59af0aa91e5e1780f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.884516 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68883d80d9744f7bac4d7f4f72283d85. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.885432 2338 tablet.cc:1722] T 02e4588b03144ea2850e13a1f8f29f06 P c756528642a543eeacd93cef3e335a7d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251029 02:52:10.886376 2309 raft_consensus.cc:493] T 20a762344c5f46f88481c4ac7ad6e43b P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:10.886502 2309 raft_consensus.cc:515] T 20a762344c5f46f88481c4ac7ad6e43b P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:10.886811 895 tablet_service.cc:1505] Processing CreateTablet for tablet 96427e60edeb45bdab9666a5055b5a99 (DEFAULT_TABLE table=default.Table_41dc74911c3747a7a2763f31d7ca40e1 [id=6ae2337c7870453fb2f1f4cbb8b5c164]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.887140 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96427e60edeb45bdab9666a5055b5a99. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.887126 2309 leader_election.cc:290] T 20a762344c5f46f88481c4ac7ad6e43b P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:10.889840 918 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "20a762344c5f46f88481c4ac7ad6e43b" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:10.890456 711 leader_election.cc:343] T 20a762344c5f46f88481c4ac7ad6e43b P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:10.891451 754 tablet_service.cc:1505] Processing CreateTablet for tablet 96427e60edeb45bdab9666a5055b5a99 (DEFAULT_TABLE table=default.Table_41dc74911c3747a7a2763f31d7ca40e1 [id=6ae2337c7870453fb2f1f4cbb8b5c164]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.891820 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96427e60edeb45bdab9666a5055b5a99. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.891845 878 tablet_service.cc:1505] Processing CreateTablet for tablet 5b44578ba7cc480199492195aca27112 (DEFAULT_TABLE table=default.Table_715191bdf4ad4b2fbf349552331f0f1e [id=359072c1e7f54a3499d6a5d98fceaf77]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.892163 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b44578ba7cc480199492195aca27112. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.844727 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet ccdca31db2834de4b6ed0f2d67a10535 (table default.Table_75341a6f40d84ad3938442923cf3131e [id=72e9fc37abc14519b6b87ebfc83c292c]) successfully deleted
I20251029 02:52:10.850391 517 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 12432c84ca104b0e99926a247318906a
I20251029 02:52:10.898758 521 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet 9269622ca92645d5aa8e73341201ccbb (table default.Table_902ea0b243c7400c86250cfb5fae5b54 [id=d0ca7c93c8cf4dc19962a1957321c5b7]). Expected version 5 got 0
I20251029 02:52:10.899124 521 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet 95f342d1cfb84c4eb31bd3d15ea3bdc2 (table default.Table_96ea715e50e74cc48759dda025a46318 [id=c63d52811db749d6badbed72fc133943]). Expected version 1 got 0
I20251029 02:52:10.899430 878 tablet_service.cc:1552] Processing DeleteTablet for tablet 12432c84ca104b0e99926a247318906a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:10 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:10.899536 521 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet 02e4588b03144ea2850e13a1f8f29f06 (table default.Table_0772f9db5d6142d988f6ffca2e96bdd2 [id=fd6d31d379394857a35422211af00642]). Expected version 3 got 0
W20251029 02:52:10.851007 522 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d557a4750fee4c2b92eaa834619a678c has no consensus state
W20251029 02:52:10.861683 519 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 07316861d2da4f55bd41111e400677fe consensus state has no leader
I20251029 02:52:10.862249 526 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet e74eda48dca647efacabe525ef316944
I20251029 02:52:10.899967 739 tablet_service.cc:1552] Processing DeleteTablet for tablet 12432c84ca104b0e99926a247318906a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:10 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:10.900504 881 tablet_service.cc:1552] Processing DeleteTablet for tablet e74eda48dca647efacabe525ef316944 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:10 UTC) from {username='slave'} at 127.0.0.1:35702
W20251029 02:52:10.900775 513 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.3:45003, user_credentials={real_user=slave}} blocked reactor thread for 56077.4us
I20251029 02:52:10.900894 739 tablet_service.cc:1552] Processing DeleteTablet for tablet e74eda48dca647efacabe525ef316944 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:10 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:10.901010 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 86fc22bb84794149b598b97df41eb881 (table default.Table_77a5f27bd0ae4eb68a30c883268d2f9c [id=d62f017ad2514f29867c9cde35996faf]) successfully deleted
I20251029 02:52:10.901248 754 tablet_service.cc:1505] Processing CreateTablet for tablet 5b44578ba7cc480199492195aca27112 (DEFAULT_TABLE table=default.Table_715191bdf4ad4b2fbf349552331f0f1e [id=359072c1e7f54a3499d6a5d98fceaf77]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.901594 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b44578ba7cc480199492195aca27112. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.901733 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 820) took 3128 ms. Trace:
I20251029 02:52:10.901816 1174 rpcz_store.cc:276] 1029 02:52:07.773475 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:07.773542 (+ 67us) service_pool.cc:225] Handling call
1029 02:52:10.685729 (+2912187us) ts_tablet_manager.cc:1151] Deleting tablet dd6ad641fbda472fab07aa35c5e4523a
1029 02:52:10.685745 (+ 16us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:10.710665 (+ 24920us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:10.720746 (+ 10081us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:10.901646 (+180900us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2886585}
I20251029 02:52:10.902138 1174 tablet_replica.cc:333] T bfec87f690a640c39c58c64e16cc67e7 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:10.902330 1174 raft_consensus.cc:2243] T bfec87f690a640c39c58c64e16cc67e7 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:10.902330 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet dd6ad641fbda472fab07aa35c5e4523a (table default.Table_29766c136dbc44e486d4256e7dbba851 [id=1d39d6caa9e24fe38b5cf9a20a6c66b9]) successfully deleted
I20251029 02:52:10.902467 1174 raft_consensus.cc:2272] T bfec87f690a640c39c58c64e16cc67e7 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:10.902815 878 tablet_service.cc:1505] Processing CreateTablet for tablet e426b37e4f304fe28e6136f32f1236ad (DEFAULT_TABLE table=default.Table_345b52c4609e43c78920c2f13d1e7b3f [id=4251ac20e9ab4f87bd5a8b88269ac78f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.903340 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e426b37e4f304fe28e6136f32f1236ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.904863 2205 consensus_queue.cc:1048] T ba699862e6f646c6beaa39f080d0faba P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:10.910506 739 tablet_service.cc:1505] Processing CreateTablet for tablet e426b37e4f304fe28e6136f32f1236ad (DEFAULT_TABLE table=default.Table_345b52c4609e43c78920c2f13d1e7b3f [id=4251ac20e9ab4f87bd5a8b88269ac78f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.910868 881 tablet_service.cc:1505] Processing CreateTablet for tablet e08101541bbf4de78f7ca92b41d0c875 (DEFAULT_TABLE table=default.Table_8d301ab34d9e48b69964f88339f88120 [id=f542f284ce8d46fcab2fc719fc45de78]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.910956 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e426b37e4f304fe28e6136f32f1236ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.911194 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e08101541bbf4de78f7ca92b41d0c875. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.917819 526 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_8c898cd0e2f343a6ae5debb290c20b07" } modify_external_catalogs: true
I20251029 02:52:10.918025 526 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_8c898cd0e2f343a6ae5debb290c20b07" } modify_external_catalogs: true
I20251029 02:52:10.919327 744 tablet_service.cc:1505] Processing CreateTablet for tablet e08101541bbf4de78f7ca92b41d0c875 (DEFAULT_TABLE table=default.Table_8d301ab34d9e48b69964f88339f88120 [id=f542f284ce8d46fcab2fc719fc45de78]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.919723 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e08101541bbf4de78f7ca92b41d0c875. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.902983 1174 ts_tablet_manager.cc:1916] T bfec87f690a640c39c58c64e16cc67e7 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:10.923008 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_3fd1224741e74537ac50e1e7ee2f9719"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:10.923200 881 tablet_service.cc:1505] Processing CreateTablet for tablet 2c4e78fbd7e645eb8b3be8ac4bfbb425 (DEFAULT_TABLE table=default.Table_17c5072bc19c40b0ac7348fdf90a1e8d [id=cb0b7302d94749adb36f5c10ff5b3562]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.923521 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c4e78fbd7e645eb8b3be8ac4bfbb425. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:10.923555 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3fd1224741e74537ac50e1e7ee2f9719 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:52:10.927166 847 consensus_peers.cc:597] T 49d9ce32e4f04f11be2ecefc5d523a8b P c756528642a543eeacd93cef3e335a7d -> Peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Couldn't send request to peer b495a49349324f7d81908f2da700f0d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 49d9ce32e4f04f11be2ecefc5d523a8b. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:10.927721 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_3db438617ecf44dcb0dbdca8f8b0b1fe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:10.929020 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3db438617ecf44dcb0dbdca8f8b0b1fe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:10.930181 2205 raft_consensus.cc:493] T e7f846b294cc4eb687a26c8b5931ac19 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:10.930312 2205 raft_consensus.cc:515] T e7f846b294cc4eb687a26c8b5931ac19 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:10.930940 2205 leader_election.cc:290] T e7f846b294cc4eb687a26c8b5931ac19 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:10.932343 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_5e8222bfdcd948849865e2ce542ea6a5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:10.933019 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5e8222bfdcd948849865e2ce542ea6a5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:52:10.933929 847 leader_election.cc:343] T e7f846b294cc4eb687a26c8b5931ac19 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: e7f846b294cc4eb687a26c8b5931ac19
I20251029 02:52:10.940057 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_85df974f5259462298e155ee87b26acb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:10.940130 754 tablet_service.cc:1505] Processing CreateTablet for tablet 2c4e78fbd7e645eb8b3be8ac4bfbb425 (DEFAULT_TABLE table=default.Table_17c5072bc19c40b0ac7348fdf90a1e8d [id=cb0b7302d94749adb36f5c10ff5b3562]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.940513 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c4e78fbd7e645eb8b3be8ac4bfbb425. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:10.940603 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_85df974f5259462298e155ee87b26acb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:10.942595 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 811) took 3150 ms. Trace:
I20251029 02:52:10.946008 523 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_c6def3b3356a4994a768adbbdf45fb9b" } new_table_name: "default.Table_c3cd24cd38b5440eb38f94d9ded9bf42" modify_external_catalogs: true
I20251029 02:52:10.788827 1032 tablet_service.cc:1505] Processing CreateTablet for tablet 416e7c48daa849cd9635656529488922 (DEFAULT_TABLE table=default.Table_0a146ce6c31f4dbf994a2442f2386d93 [id=559a939c5ba5402ebd5881cf88766be7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.800508 1057 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "738046ab57334f2dab9f4aab979c5b42" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:10.947388 1057 raft_consensus.cc:2468] T 738046ab57334f2dab9f4aab979c5b42 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:10.811317 1058 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2e97d1bc06a24e8d8ba11acb6fd280d8" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:52:10.947734 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 416e7c48daa849cd9635656529488922. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.812285 1036 tablet_service.cc:1505] Processing CreateTablet for tablet 4f004ea546d64681bd29b2e7ee3bb225 (DEFAULT_TABLE table=default.Table_d2ac974941f9428eaa9605606209ed8d [id=62013562b0b2496cb82c301377d317ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.948179 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f004ea546d64681bd29b2e7ee3bb225. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.949736 878 tablet_service.cc:1505] Processing CreateTablet for tablet af74d277abb149e09900c46baf2ea1d3 (DEFAULT_TABLE table=default.Table_0f4514215d3042c5a46ac59072d2bbb2 [id=19d51187854b458ebf8af447277ea1cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.950073 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af74d277abb149e09900c46baf2ea1d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.812443 1031 tablet_service.cc:1552] Processing DeleteTablet for tablet 877e278deb4a4b01ac66b06be0890baf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:09 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:10.950857 1031 tablet_service.cc:1505] Processing CreateTablet for tablet 68883d80d9744f7bac4d7f4f72283d85 (DEFAULT_TABLE table=default.Table_28bb97b95f7942fb9da0cec248e6ac6d [id=77f674208cb649a59af0aa91e5e1780f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.951154 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68883d80d9744f7bac4d7f4f72283d85. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.812695 1042 tablet_service.cc:1505] Processing CreateTablet for tablet 7fba607c566c4fa1a803cba8d86ca646 (DEFAULT_TABLE table=default.Table_9bc6d7e931a34e00b8a1c2233fa53c3f [id=c8105da2dd8540aa977bf54b09d874ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.812979 1039 tablet_service.cc:1505] Processing CreateTablet for tablet 858ef21620344d909d085f22e62081ac (DEFAULT_TABLE table=default.Table_d76042aac8cb46c38a05fdf710a406cc [id=bca89cc60e344f71a51878dcf10b2ab7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.813123 1027 tablet_service.cc:1505] Processing CreateTablet for tablet 8de7380e93dd4a9abdb84522a18abda6 (DEFAULT_TABLE table=default.Table_af62893fe44b437ba6c5753df4b31da9 [id=dd34a215f0cb45b7929a1b86576027f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.813473 1033 tablet_service.cc:1505] Processing CreateTablet for tablet 56ed1b1d2ef144e79e42e708326b2480 (DEFAULT_TABLE table=default.Table_d61d4be2068f4a1985e207d3413141f9 [id=a20067c639e94ad091a0db8d17af2f87]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.813764 1166 tablet_bootstrap.cc:492] T 44c0488eba0543078ff4f0b3d69366f2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:10.954578 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_b2aa73be4c2a45c6ac352144abf10116"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:10.954824 1166 ts_tablet_manager.cc:1403] T 44c0488eba0543078ff4f0b3d69366f2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.607s user 0.064s sys 0.081s
W20251029 02:52:10.955142 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b2aa73be4c2a45c6ac352144abf10116 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:10.955576 1174 ts_tablet_manager.cc:1929] T bfec87f690a640c39c58c64e16cc67e7 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:10.955545 1166 raft_consensus.cc:359] T 44c0488eba0543078ff4f0b3d69366f2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:10.955685 1174 log.cc:1199] T bfec87f690a640c39c58c64e16cc67e7 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/bfec87f690a640c39c58c64e16cc67e7
I20251029 02:52:10.955682 1166 raft_consensus.cc:385] T 44c0488eba0543078ff4f0b3d69366f2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:10.955724 1166 raft_consensus.cc:740] T 44c0488eba0543078ff4f0b3d69366f2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:10.955906 1166 consensus_queue.cc:260] T 44c0488eba0543078ff4f0b3d69366f2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:10.955964 1174 ts_tablet_manager.cc:1950] T bfec87f690a640c39c58c64e16cc67e7 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:10.956338 1166 ts_tablet_manager.cc:1434] T 44c0488eba0543078ff4f0b3d69366f2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20251029 02:52:10.814160 1030 tablet_service.cc:1552] Processing DeleteTablet for tablet 0700b93cc398403d8b3602e84182a3ce with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:10 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:10.956853 1030 tablet_service.cc:1505] Processing CreateTablet for tablet e2a739ebb4aa446fba435a9aecd28ed5 (DEFAULT_TABLE table=default.Table_ff774195536549b3b3a651c34438f014 [id=a610d1dc5b3e468c86c1f196e2ccdc54]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.814247 1038 tablet_service.cc:1552] Processing DeleteTablet for tablet bfb9115f324f44c98f876a2d56bb23a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:10 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:10.814764 1023 tablet_service.cc:1552] Processing DeleteTablet for tablet ef2e9dd8f90a47389d9062d1f99a1da1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:10 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:10.816673 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "738046ab57334f2dab9f4aab979c5b42" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:52:10.957472 1060 raft_consensus.cc:3060] T 738046ab57334f2dab9f4aab979c5b42 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:10.959064 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_fb178c25d3c1496f8ed5db2fe26d9fb1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:10.959632 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fb178c25d3c1496f8ed5db2fe26d9fb1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:10.960157 1060 raft_consensus.cc:2468] T 738046ab57334f2dab9f4aab979c5b42 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:10.960875 2179 raft_consensus.cc:493] T 44c0488eba0543078ff4f0b3d69366f2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:10.960985 2179 raft_consensus.cc:515] T 44c0488eba0543078ff4f0b3d69366f2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:10.961583 2179 leader_election.cc:290] T 44c0488eba0543078ff4f0b3d69366f2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:10.961823 520 master_service.cc:954] ReplaceTablet: received request to replace tablet d041e87ceb11441983d0e38b7929c3a4 from {username='slave'} at 127.0.0.1:35314
I20251029 02:52:10.962661 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 7a1866afc7564217ad777e7ce53786c5 (table default.Table_46e2f96551d3497c919e1cab3521d63b [id=09ee821707fa4082889bfccb8667b0b7]) successfully deleted
I20251029 02:52:10.963562 918 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "44c0488eba0543078ff4f0b3d69366f2" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:10.963682 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 0f80b840ed954230bfbfe7e40127cd2f (table default.Table_5902a11255a049be9dd07acf8b50adda [id=5bd71ce398e84db28f54262cd3c5528b]) successfully deleted
I20251029 02:52:10.962390 895 tablet_service.cc:1505] Processing CreateTablet for tablet 7c6b9031ad71444d95df0c33135636b3 (DEFAULT_TABLE table=default.Table_a3886404687640698bd3302e39ee6854 [id=7390a7db2be349b09df554aa92d4068c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.964104 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c6b9031ad71444d95df0c33135636b3. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:10.964211 990 leader_election.cc:343] T 44c0488eba0543078ff4f0b3d69366f2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:10.962435 520 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_13f9fb5682214cc28f12dce88109d831" } new_table_name: "default.Table_c243191d6825474288c6942a1efeb0dd" modify_external_catalogs: true
I20251029 02:52:10.967375 526 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 4b841723726e40bda0187fb84bf36a6a
I20251029 02:52:10.969581 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 4b841723726e40bda0187fb84bf36a6a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:10 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:10.819089 1020 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:37622 (request call id 139) took 1365 ms. Trace:
I20251029 02:52:10.969808 1020 rpcz_store.cc:276] 1029 02:52:09.454021 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:09.455171 (+ 1150us) service_pool.cc:225] Handling call
1029 02:52:09.474717 (+ 19546us) spinlock_profiling.cc:243] Waited 13.7 ms on lock 0x616000023b54. stack: 00007f34717829f6 00007f34717827b0 00007f347075e3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f347d61c200 00007f34814cc66f 00007f347b8f2c36 00007f347b8f2965 00007f347872127e 00007f347872015e 00007f3478726395 00007f3478726cbf 00007f3478726ae8 0000000000a8867c
1029 02:52:09.652200 (+177483us) spinlock_profiling.cc:243] Waited 85.5 ms on lock 0x6160002a0a54. stack: 00007f34717829f6 00007f34717827b0 00007f347075e3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f347d61c200 00007f34814cc66f 00007f347b8f2c36 00007f347b8f2965 00007f347872127e 00007f347872015e 00007f3478726395 00007f3478726cbf 00007f3478726ae8 0000000000a8867c
1029 02:52:10.819066 (+1166866us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":228075136}
I20251029 02:52:10.819411 1026 tablet_service.cc:1552] Processing DeleteTablet for tablet 34d141bee474452693dba1b9d7f2b023 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:10 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:10.820827 1035 tablet_service.cc:1505] Processing CreateTablet for tablet 3e6c048371984e36a44613294eed534f (DEFAULT_TABLE table=default.Table_b68e55dfa3d04c8d95c29f27537f59ad [id=53952ef589954f1c94fa49b5fa0fb623]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.821205 1040 tablet_service.cc:1552] Processing DeleteTablet for tablet 5a1aa131cf4e4edca5bb2515edd7556a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:10 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:10.821489 1024 tablet_service.cc:1505] Processing CreateTablet for tablet 1150d2210a234f1aa554f486cdcabc4c (DEFAULT_TABLE table=default.Table_4a6906297ac74f45b5f153339e2713c3 [id=cc511ee9649842ba818d94c8831d9b2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.821648 1034 tablet_service.cc:1552] Processing DeleteTablet for tablet 862d268a526545ba8f25bb91c2992321 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:10 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:10.865665 2188 consensus_queue.cc:1048] T 9269622ca92645d5aa8e73341201ccbb P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:10.884542 2337 tablet.cc:1722] T 02e4588b03144ea2850e13a1f8f29f06 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251029 02:52:10.970814 900 tablet_service.cc:1505] Processing CreateTablet for tablet 921074e69f9d49a3bdad3d2791ab037a (DEFAULT_TABLE table=default.Table_110cf0fc2d22469d93e57359178b6c2a [id=88c3c42a3ece45f98490fd86ea2d7314]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.971668 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 921074e69f9d49a3bdad3d2791ab037a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.971892 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_3ddda212ceb444adae605530b0985d6a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:10.972519 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3ddda212ceb444adae605530b0985d6a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:10.974860 517 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_18581b0529cb451d9bda8bf2c84a8144" } new_table_name: "default.Table_05649dba28434c1a832deacbffb54eb1" modify_external_catalogs: true
I20251029 02:52:10.975096 2306 consensus_queue.cc:1048] T 30d03e1e42d34242bd354f9e9ec4f127 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:10.976480 1175 rpcz_store.cc:276] 1029 02:52:07.792231 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:07.792280 (+ 49us) service_pool.cc:225] Handling call
1029 02:52:10.702413 (+2910133us) ts_tablet_manager.cc:1151] Deleting tablet b6d4f7bf702f4834bc5f4ef2b50b92a7
1029 02:52:10.702424 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:10.728874 (+ 26450us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:10.762293 (+ 33419us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:10.942576 (+180283us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2909920}
I20251029 02:52:10.977530 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet b6d4f7bf702f4834bc5f4ef2b50b92a7 (table default.Table_c70eb9f284cd49109fd09e430a1cea6c [id=be817eb6ec33406eb316d36575c134fb]) successfully deleted
I20251029 02:52:10.925534 1025 tablet_service.cc:1505] Processing CreateTablet for tablet 48a367a3d4404a7898316a3c4384fd63 (DEFAULT_TABLE table=default.Table_9e7a20ed2df8447d92cc51920fd8177e [id=747a5663894d4eb1b12c9fa8b27d476d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.931861 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e7f846b294cc4eb687a26c8b5931ac19" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:10.978294 1056 raft_consensus.cc:2468] T e7f846b294cc4eb687a26c8b5931ac19 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:10.978927 1175 tablet_replica.cc:333] T 2e97d1bc06a24e8d8ba11acb6fd280d8 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:10.978893 849 leader_election.cc:304] T e7f846b294cc4eb687a26c8b5931ac19 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters: b495a49349324f7d81908f2da700f0d9
I20251029 02:52:10.979157 1175 raft_consensus.cc:2243] T 2e97d1bc06a24e8d8ba11acb6fd280d8 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:10.979312 1175 raft_consensus.cc:2272] T 2e97d1bc06a24e8d8ba11acb6fd280d8 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:10.979597 2205 raft_consensus.cc:2804] T e7f846b294cc4eb687a26c8b5931ac19 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:10.979671 2205 raft_consensus.cc:493] T e7f846b294cc4eb687a26c8b5931ac19 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:10.979730 2205 raft_consensus.cc:3060] T e7f846b294cc4eb687a26c8b5931ac19 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:10.979836 1175 ts_tablet_manager.cc:1916] T 2e97d1bc06a24e8d8ba11acb6fd280d8 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:10.983366 2205 raft_consensus.cc:515] T e7f846b294cc4eb687a26c8b5931ac19 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:10.984645 2205 leader_election.cc:290] T e7f846b294cc4eb687a26c8b5931ac19 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:10.984730 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "adf6e736cb9c4920807ee58682e134ef" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:10.935607 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet 8378d67f1da743cca91803e37ea54281 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:10 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:10.947695 1058 raft_consensus.cc:3060] T 2e97d1bc06a24e8d8ba11acb6fd280d8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
W20251029 02:52:10.984943 847 leader_election.cc:343] T e7f846b294cc4eb687a26c8b5931ac19 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: e7f846b294cc4eb687a26c8b5931ac19
I20251029 02:52:10.811951 1041 tablet_service.cc:1505] Processing CreateTablet for tablet 749f380eafae44ddbe2a244757f37909 (DEFAULT_TABLE table=default.Table_1f3338455b1a4eef85ecea36443bc329 [id=39683553c6fd44bca1cc3de2d0b9d089]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.984984 776 raft_consensus.cc:2468] T adf6e736cb9c4920807ee58682e134ef P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 0.
I20251029 02:52:10.985492 762 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82289e0f823c4d43bfd8c4e878572b2b" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:10.985631 762 raft_consensus.cc:2468] T 82289e0f823c4d43bfd8c4e878572b2b P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 0.
I20251029 02:52:10.987936 2306 raft_consensus.cc:493] T 71b4b91071a54d6aa66160e7955bc915 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:10.988118 2306 raft_consensus.cc:515] T 71b4b91071a54d6aa66160e7955bc915 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:10.988484 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e7f846b294cc4eb687a26c8b5931ac19" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:52:10.988617 1056 raft_consensus.cc:3060] T e7f846b294cc4eb687a26c8b5931ac19 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:10.988757 2306 leader_election.cc:290] T 71b4b91071a54d6aa66160e7955bc915 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:10.989166 2205 consensus_queue.cc:1048] T 0d84dc8dc465496f9adec798b2e03e13 P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:10.991390 1056 raft_consensus.cc:2468] T e7f846b294cc4eb687a26c8b5931ac19 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:10.991771 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "71b4b91071a54d6aa66160e7955bc915" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:10.992774 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 858ef21620344d909d085f22e62081ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.949273 738 tablet_service.cc:1505] Processing CreateTablet for tablet 921074e69f9d49a3bdad3d2791ab037a (DEFAULT_TABLE table=default.Table_110cf0fc2d22469d93e57359178b6c2a [id=88c3c42a3ece45f98490fd86ea2d7314]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.998230 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 921074e69f9d49a3bdad3d2791ab037a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:10.998942 1058 raft_consensus.cc:2468] T 2e97d1bc06a24e8d8ba11acb6fd280d8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:11.003628 849 leader_election.cc:304] T e7f846b294cc4eb687a26c8b5931ac19 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters: b495a49349324f7d81908f2da700f0d9
W20251029 02:52:11.004705 990 leader_election.cc:343] T 71b4b91071a54d6aa66160e7955bc915 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:11.005270 2205 raft_consensus.cc:2804] T e7f846b294cc4eb687a26c8b5931ac19 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:11.005510 2205 raft_consensus.cc:697] T e7f846b294cc4eb687a26c8b5931ac19 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:52:11.005787 2205 consensus_queue.cc:237] T e7f846b294cc4eb687a26c8b5931ac19 P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:11.008526 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56ed1b1d2ef144e79e42e708326b2480. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:11.013103 1058 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "adf6e736cb9c4920807ee58682e134ef" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:11.013098 745 tablet_service.cc:1505] Processing CreateTablet for tablet 7fcb1aac63c84edba0a8b8c964a8b957 (DEFAULT_TABLE table=default.Table_5ed09727711e4eb6a60deb327d188aea [id=d256ddd14762490f8fbfe401d9802d4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:11.013295 881 tablet_service.cc:1505] Processing CreateTablet for tablet 7fcb1aac63c84edba0a8b8c964a8b957 (DEFAULT_TABLE table=default.Table_5ed09727711e4eb6a60deb327d188aea [id=d256ddd14762490f8fbfe401d9802d4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:11.013326 1058 raft_consensus.cc:2468] T adf6e736cb9c4920807ee58682e134ef P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b495a49349324f7d81908f2da700f0d9 in term 0.
I20251029 02:52:11.013571 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7fcb1aac63c84edba0a8b8c964a8b957. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:11.013633 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7fcb1aac63c84edba0a8b8c964a8b957. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:11.014164 1058 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "731c0acfbe204e8199ad246e38ae2ccd" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:11.014820 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1150d2210a234f1aa554f486cdcabc4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:11.015169 1058 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d7abaf84ba5d445aa17433b3be891548" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:10.960248 740 tablet_service.cc:1505] Processing CreateTablet for tablet af74d277abb149e09900c46baf2ea1d3 (DEFAULT_TABLE table=default.Table_0f4514215d3042c5a46ac59072d2bbb2 [id=19d51187854b458ebf8af447277ea1cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.963056 739 tablet_service.cc:1505] Processing CreateTablet for tablet 7c6b9031ad71444d95df0c33135636b3 (DEFAULT_TABLE table=default.Table_a3886404687640698bd3302e39ee6854 [id=7390a7db2be349b09df554aa92d4068c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.963639 744 tablet_service.cc:1505] Processing CreateTablet for tablet 456b623500fd4d758e40004666b095a9 (DEFAULT_TABLE table=default.Table_443f33587b8f428484b99e07355d69ca [id=03a25420ce6a46fc93a9fb753246bc7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.965731 743 tablet_service.cc:1505] Processing CreateTablet for tablet 2849e856dd714da88fca5a9fafe74c22 (DEFAULT_TABLE table=default.Table_fa832508491b47ddbcd9e89058c75ad5 [id=958e0005a6bc408188e53fc33d6509dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:10.974174 747 tablet_service.cc:1552] Processing DeleteTablet for tablet 4b841723726e40bda0187fb84bf36a6a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:10 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:10.944653 2309 raft_consensus.cc:493] T 378f549ff7364d018d8b9a5ef01ac878 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:11.016420 2309 raft_consensus.cc:515] T 378f549ff7364d018d8b9a5ef01ac878 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:11.017074 2309 leader_election.cc:290] T 378f549ff7364d018d8b9a5ef01ac878 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:11.017596 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "378f549ff7364d018d8b9a5ef01ac878" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:11.018134 711 leader_election.cc:343] T 378f549ff7364d018d8b9a5ef01ac878 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:10.970814 879 tablet_service.cc:1505] Processing CreateTablet for tablet 456b623500fd4d758e40004666b095a9 (DEFAULT_TABLE table=default.Table_443f33587b8f428484b99e07355d69ca [id=03a25420ce6a46fc93a9fb753246bc7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:11.018700 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 456b623500fd4d758e40004666b095a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:11.018806 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e6c048371984e36a44613294eed534f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:11.021091 1058 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97b2ecae572a4b7ea1b9595074556708" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:11.021274 1058 raft_consensus.cc:2468] T 97b2ecae572a4b7ea1b9595074556708 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b495a49349324f7d81908f2da700f0d9 in term 0.
I20251029 02:52:11.021744 1058 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82289e0f823c4d43bfd8c4e878572b2b" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:11.021878 1058 raft_consensus.cc:2468] T 82289e0f823c4d43bfd8c4e878572b2b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b495a49349324f7d81908f2da700f0d9 in term 0.
I20251029 02:52:11.022392 1061 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "44ae2b07edca44afbe1b1fb3a5b1115d" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:11.022941 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c6b9031ad71444d95df0c33135636b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:11.023550 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 456b623500fd4d758e40004666b095a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:11.023985 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2849e856dd714da88fca5a9fafe74c22. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:11.029587 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 749f380eafae44ddbe2a244757f37909. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:11.030469 511 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet d557a4750fee4c2b92eaa834619a678c (table default.Table_4873cb28e4c84935ab481f06ad654716 [id=9bc3968738ca426f8ac8dec3b513b085], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet d557a4750fee4c2b92eaa834619a678c has no consensus state
I20251029 02:52:11.030745 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8de7380e93dd4a9abdb84522a18abda6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:11.030781 894 tablet_service.cc:1505] Processing CreateTablet for tablet 2cea2f2fe68545db804a732d0c01a876 (DEFAULT_TABLE table=default.Table_149358409d284922ad0f2c49410dca96 [id=195d147c89f240bea546a83693a6c7dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:11.031257 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2cea2f2fe68545db804a732d0c01a876. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:11.034742 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2a739ebb4aa446fba435a9aecd28ed5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:11.037086 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af74d277abb149e09900c46baf2ea1d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:11.037635 885 tablet_service.cc:1505] Processing CreateTablet for tablet 2849e856dd714da88fca5a9fafe74c22 (DEFAULT_TABLE table=default.Table_fa832508491b47ddbcd9e89058c75ad5 [id=958e0005a6bc408188e53fc33d6509dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:11.038019 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2849e856dd714da88fca5a9fafe74c22. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:11.041498 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7fba607c566c4fa1a803cba8d86ca646. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:11.044994 1058 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8048506ced4540a5aee9f79bfc4599bf" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:52:11.045174 1058 raft_consensus.cc:3060] T 8048506ced4540a5aee9f79bfc4599bf P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:11.046218 1166 tablet_bootstrap.cc:492] T c1222e30bc4141b3ae686d80053c4ef5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:11.046806 1028 tablet_service.cc:1505] Processing CreateTablet for tablet eeb75fc46e2a41f78252868ad5256966 (DEFAULT_TABLE table=default.Table_331f901095f44ebfa644aa9cf8dde711 [id=d58b97f75daa4d8083b61b23aa87e1e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:11.048348 1166 tablet_bootstrap.cc:654] T c1222e30bc4141b3ae686d80053c4ef5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:11.050534 1167 tablet_bootstrap.cc:492] T af4860c6b72f4d988b206b5ef2f22906 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:11.050676 1167 ts_tablet_manager.cc:1403] T af4860c6b72f4d988b206b5ef2f22906 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.190s user 0.003s sys 0.003s
I20251029 02:52:11.050660 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d830d30b26b4ba799785fec8a6f0769" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:52:11.050911 1060 raft_consensus.cc:2393] T 6d830d30b26b4ba799785fec8a6f0769 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b495a49349324f7d81908f2da700f0d9 in current term 1: Already voted for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in this term.
I20251029 02:52:11.051469 1167 raft_consensus.cc:359] T af4860c6b72f4d988b206b5ef2f22906 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:11.051614 1167 raft_consensus.cc:385] T af4860c6b72f4d988b206b5ef2f22906 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:11.051656 1167 raft_consensus.cc:740] T af4860c6b72f4d988b206b5ef2f22906 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:11.053992 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 821) took 3211 ms. Trace:
W20251029 02:52:11.053072 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.3:45003, user_credentials={real_user=slave}} blocked reactor thread for 160036us
I20251029 02:52:11.054078 1174 rpcz_store.cc:276] 1029 02:52:07.842644 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:07.842725 (+ 81us) service_pool.cc:225] Handling call
1029 02:52:10.902092 (+3059367us) ts_tablet_manager.cc:1151] Deleting tablet bfec87f690a640c39c58c64e16cc67e7
1029 02:52:10.902102 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:10.952619 (+ 50517us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:10.955495 (+ 2876us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:11.053973 (+ 98478us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3043256}
I20251029 02:52:11.054787 1174 tablet_replica.cc:333] T f246f6c202cf44d48c9f5e4f335202c5 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:11.055050 1174 raft_consensus.cc:2243] T f246f6c202cf44d48c9f5e4f335202c5 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:11.055194 1174 raft_consensus.cc:2272] T f246f6c202cf44d48c9f5e4f335202c5 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:11.055402 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet bfec87f690a640c39c58c64e16cc67e7 (table default.Table_11de4e6789214efca8b43cbcdae81a93 [id=6506de17cec5443b8a4628bd46a39b42]) successfully deleted
I20251029 02:52:11.055413 2188 consensus_queue.cc:1048] T dabb8d760aae4aa09eb83cdb586c9788 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:11.055675 1174 ts_tablet_manager.cc:1916] T f246f6c202cf44d48c9f5e4f335202c5 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:11.058375 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af4860c6b72f4d988b206b5ef2f22906" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:52:11.058554 1056 raft_consensus.cc:3060] T af4860c6b72f4d988b206b5ef2f22906 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:11.058846 747 tablet_service.cc:1505] Processing CreateTablet for tablet 2cea2f2fe68545db804a732d0c01a876 (DEFAULT_TABLE table=default.Table_149358409d284922ad0f2c49410dca96 [id=195d147c89f240bea546a83693a6c7dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:11.059247 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2cea2f2fe68545db804a732d0c01a876. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:11.061352 1056 raft_consensus.cc:2468] T af4860c6b72f4d988b206b5ef2f22906 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b495a49349324f7d81908f2da700f0d9 in term 1.
I20251029 02:52:11.063014 894 tablet_service.cc:1505] Processing CreateTablet for tablet fcc2d413e3b54b2da9b5b44f8b859816 (DEFAULT_TABLE table=default.Table_9791da4fa0ac47409f6cb424408a5f3b [id=60ef33b133314f1d9007a46fbf3a5338]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:11.063222 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c32c28b0e74a4af3a0e69bbf24ab1a1f" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:11.063413 1060 raft_consensus.cc:2468] T c32c28b0e74a4af3a0e69bbf24ab1a1f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b495a49349324f7d81908f2da700f0d9 in term 0.
I20251029 02:52:11.063491 748 tablet_service.cc:1505] Processing CreateTablet for tablet 44ef8aa6597a49d6bd27ece880589522 (DEFAULT_TABLE table=default.Table_af97a1e008a94a86970089216d02b3d4 [id=cc32193d11ed47aeb9f70ce8151dff1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:11.063679 751 tablet_service.cc:1505] Processing CreateTablet for tablet fcc2d413e3b54b2da9b5b44f8b859816 (DEFAULT_TABLE table=default.Table_9791da4fa0ac47409f6cb424408a5f3b [id=60ef33b133314f1d9007a46fbf3a5338]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:11.063881 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44ef8aa6597a49d6bd27ece880589522. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:11.064064 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fcc2d413e3b54b2da9b5b44f8b859816. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:11.063493 742 tablet_service.cc:1505] Processing CreateTablet for tablet 854bb36d1d3340a9bab2ad2eb4ee8cb5 (DEFAULT_TABLE table=default.Table_7fd7b56a9c09475a80cce932d6e15cc1 [id=d5d80eda41084397972a6b36cfff433b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:11.066933 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 854bb36d1d3340a9bab2ad2eb4ee8cb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:11.063009 896 tablet_service.cc:1505] Processing CreateTablet for tablet 44ef8aa6597a49d6bd27ece880589522 (DEFAULT_TABLE table=default.Table_af97a1e008a94a86970089216d02b3d4 [id=cc32193d11ed47aeb9f70ce8151dff1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:11.071084 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fcc2d413e3b54b2da9b5b44f8b859816. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:11.071312 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44ef8aa6597a49d6bd27ece880589522. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:11.075399 1167 consensus_queue.cc:260] T af4860c6b72f4d988b206b5ef2f22906 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:11.076038 1167 ts_tablet_manager.cc:1434] T af4860c6b72f4d988b206b5ef2f22906 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.025s user 0.002s sys 0.000s
I20251029 02:52:11.076340 1167 tablet_bootstrap.cc:492] T 71b4b91071a54d6aa66160e7955bc915 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
W20251029 02:52:11.077457 523 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f17b76f5dd4c4dd78d1fd45cc490763c consensus state has no leader
I20251029 02:52:11.077979 1167 tablet_bootstrap.cc:654] T 71b4b91071a54d6aa66160e7955bc915 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:11.079764 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_492f50b2191a46ee8e6bb05e69377006"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:11.080003 524 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_6081d34b2c5742caa7681b275bb8e49f" } modify_external_catalogs: true
I20251029 02:52:11.080178 524 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_6081d34b2c5742caa7681b275bb8e49f" } modify_external_catalogs: true
W20251029 02:52:11.080488 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_492f50b2191a46ee8e6bb05e69377006 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:11.081671 2309 raft_consensus.cc:493] T 7bb85a12f573494cbb3118d1947eda27 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:11.081811 2309 raft_consensus.cc:515] T 7bb85a12f573494cbb3118d1947eda27 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:11.082468 2309 leader_election.cc:290] T 7bb85a12f573494cbb3118d1947eda27 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:11.082834 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_1b4647439c30401a9285f7281255f7dd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:11.083453 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1b4647439c30401a9285f7281255f7dd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:11.083791 878 tablet_service.cc:1505] Processing CreateTablet for tablet 854bb36d1d3340a9bab2ad2eb4ee8cb5 (DEFAULT_TABLE table=default.Table_7fd7b56a9c09475a80cce932d6e15cc1 [id=d5d80eda41084397972a6b36cfff433b]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 02:52:11.084028 520 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 51a6e45378344f86847a44cf4950307b consensus state has no leader
I20251029 02:52:11.084206 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 854bb36d1d3340a9bab2ad2eb4ee8cb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:11.084564 1176 ts_tablet_manager.cc:1929] T 777992acb1bd4765886b802940db74bb P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:11.084671 1176 log.cc:1199] T 777992acb1bd4765886b802940db74bb P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/777992acb1bd4765886b802940db74bb
I20251029 02:52:11.085412 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7bb85a12f573494cbb3118d1947eda27" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:11.086092 523 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_293c6e92af8a4707be8e2596399fd6a2" } modify_external_catalogs: true
I20251029 02:52:11.086297 523 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_293c6e92af8a4707be8e2596399fd6a2" } modify_external_catalogs: true
W20251029 02:52:11.089092 711 leader_election.cc:343] T 7bb85a12f573494cbb3118d1947eda27 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:11.092397 1058 raft_consensus.cc:2468] T 8048506ced4540a5aee9f79bfc4599bf P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b495a49349324f7d81908f2da700f0d9 in term 1.
I20251029 02:52:11.058104 1061 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e65eaaae43ac435ebd665546de9d68dd" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:11.101696 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eeb75fc46e2a41f78252868ad5256966. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:11.102110 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48a367a3d4404a7898316a3c4384fd63. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:11.102326 1057 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "28f7ceb91a34450cad4b5b45f5285b41" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:52:11.102489 1057 raft_consensus.cc:3060] T 28f7ceb91a34450cad4b5b45f5285b41 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:11.105464 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_27d80833eb694669a044be7eb64e4aa4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:11.106278 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_27d80833eb694669a044be7eb64e4aa4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:11.106653 522 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_3297bb18a32a4454a6458e189b9985d1" } modify_external_catalogs: true
I20251029 02:52:11.109849 1172 tablet_bootstrap.cc:492] T 27b08f88b99643d28ac609e45bf8104e P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:11.110225 1172 ts_tablet_manager.cc:1403] T 27b08f88b99643d28ac609e45bf8104e P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.393s user 0.007s sys 0.002s
I20251029 02:52:11.112180 1172 raft_consensus.cc:359] T 27b08f88b99643d28ac609e45bf8104e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:11.112754 1172 raft_consensus.cc:385] T 27b08f88b99643d28ac609e45bf8104e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:11.113521 1172 raft_consensus.cc:740] T 27b08f88b99643d28ac609e45bf8104e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:11.114110 1172 consensus_queue.cc:260] T 27b08f88b99643d28ac609e45bf8104e P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:11.135375 1176 ts_tablet_manager.cc:1950] T 777992acb1bd4765886b802940db74bb P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
W20251029 02:52:11.144455 517 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f66a7af499b446c7a33965c9e37ce460 consensus state has no leader
I20251029 02:52:11.144941 517 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_47a306977f08434993da629ff2ed490a" } new_table_name: "default.Table_054598fd8c4b4a87b9b5944a738dec88" modify_external_catalogs: true
I20251029 02:52:11.145133 526 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_877abcb0d221493c8875368e1c75382b" } new_table_name: "default.Table_e36498ba27074ef9938ea3a76ece24bc" modify_external_catalogs: true
I20251029 02:52:11.146410 1174 ts_tablet_manager.cc:1929] T f246f6c202cf44d48c9f5e4f335202c5 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:11.148483 519 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_abdff89e238347a69b825355bd66f2cd" } modify_external_catalogs: true
I20251029 02:52:11.148749 519 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_abdff89e238347a69b825355bd66f2cd" } modify_external_catalogs: true
I20251029 02:52:11.150285 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 875) took 2620 ms. Trace:
I20251029 02:52:11.152735 522 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_3297bb18a32a4454a6458e189b9985d1" } modify_external_catalogs: true
I20251029 02:52:11.153872 1166 tablet_bootstrap.cc:492] T c1222e30bc4141b3ae686d80053c4ef5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:11.155812 1057 raft_consensus.cc:2468] T 28f7ceb91a34450cad4b5b45f5285b41 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b495a49349324f7d81908f2da700f0d9 in term 1.
I20251029 02:52:11.158812 1166 ts_tablet_manager.cc:1403] T c1222e30bc4141b3ae686d80053c4ef5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.113s user 0.018s sys 0.006s
I20251029 02:52:11.160069 1051 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "67651306f3b44e7eb14dcee9a2007a81" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:11.156101 1061 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "946f65014d97425aab3851009b742807" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:11.156481 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "71b4b91071a54d6aa66160e7955bc915" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:11.156572 1059 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d041e87ceb11441983d0e38b7929c3a4" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:11.162194 1056 raft_consensus.cc:2468] T 71b4b91071a54d6aa66160e7955bc915 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b495a49349324f7d81908f2da700f0d9 in term 0.
I20251029 02:52:11.156783 1062 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "643e20d9974447daa78885b3401f3b77" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:11.157676 1055 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59eee86f249e4d6a9d09d30276dcd659" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:11.158030 1054 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f281191e7c6143f983f11d87f1c9b64e" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:11.158140 1058 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "217144957930495aa3e9dfcfde268e71" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:11.162493 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "afb19ba39399410bb90b6225d7b5b5b1" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:11.162638 1058 raft_consensus.cc:2468] T 217144957930495aa3e9dfcfde268e71 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b495a49349324f7d81908f2da700f0d9 in term 0.
I20251029 02:52:11.150378 1176 rpcz_store.cc:276] 1029 02:52:08.529351 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:08.547812 (+ 18461us) service_pool.cc:225] Handling call
1029 02:52:10.780310 (+2232498us) ts_tablet_manager.cc:1151] Deleting tablet 777992acb1bd4765886b802940db74bb
1029 02:52:10.780318 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:11.010108 (+229790us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:11.084449 (+ 74341us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:11.150264 (+ 65815us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2232218}
I20251029 02:52:11.163084 1176 tablet_replica.cc:333] T c9dff4dff82f49e7ad49caad5e404f09 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:11.163259 1176 raft_consensus.cc:2243] T c9dff4dff82f49e7ad49caad5e404f09 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:11.163488 1176 raft_consensus.cc:2272] T c9dff4dff82f49e7ad49caad5e404f09 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:11.164013 1176 ts_tablet_manager.cc:1916] T c9dff4dff82f49e7ad49caad5e404f09 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:11.171922 1174 log.cc:1199] T f246f6c202cf44d48c9f5e4f335202c5 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/f246f6c202cf44d48c9f5e4f335202c5
I20251029 02:52:11.172367 1174 ts_tablet_manager.cc:1950] T f246f6c202cf44d48c9f5e4f335202c5 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:11.180823 1172 ts_tablet_manager.cc:1434] T 27b08f88b99643d28ac609e45bf8104e P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.070s user 0.001s sys 0.001s
I20251029 02:52:11.181813 1172 tablet_bootstrap.cc:492] T 6789bc16664e4b37b86702449ab27855 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:11.176664 1167 tablet_bootstrap.cc:492] T 71b4b91071a54d6aa66160e7955bc915 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:11.184602 2302 consensus_queue.cc:1048] T 49d9ce32e4f04f11be2ecefc5d523a8b P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:11.186813 1167 ts_tablet_manager.cc:1403] T 71b4b91071a54d6aa66160e7955bc915 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.111s user 0.004s sys 0.000s
I20251029 02:52:11.187570 1167 raft_consensus.cc:359] T 71b4b91071a54d6aa66160e7955bc915 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:11.187755 1167 raft_consensus.cc:385] T 71b4b91071a54d6aa66160e7955bc915 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:11.187826 1167 raft_consensus.cc:740] T 71b4b91071a54d6aa66160e7955bc915 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:11.188042 1167 consensus_queue.cc:260] T 71b4b91071a54d6aa66160e7955bc915 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:11.188671 1167 ts_tablet_manager.cc:1434] T 71b4b91071a54d6aa66160e7955bc915 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251029 02:52:11.189124 1167 tablet_bootstrap.cc:492] T c32c28b0e74a4af3a0e69bbf24ab1a1f P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:11.190739 1167 tablet_bootstrap.cc:654] T c32c28b0e74a4af3a0e69bbf24ab1a1f P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:11.194813 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 822) took 3226 ms. Trace:
I20251029 02:52:11.194963 1174 rpcz_store.cc:276] 1029 02:52:07.968199 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:07.968256 (+ 57us) service_pool.cc:225] Handling call
1029 02:52:11.054733 (+3086477us) ts_tablet_manager.cc:1151] Deleting tablet f246f6c202cf44d48c9f5e4f335202c5
1029 02:52:11.054740 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:11.068184 (+ 13444us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:11.146306 (+ 78122us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:11.194748 (+ 48442us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3086276}
I20251029 02:52:11.195489 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet f246f6c202cf44d48c9f5e4f335202c5 (table default.Table_e4de02c7c1f04704bc39e1f0efd2c9d9 [id=40738627473f4a55adc5ed250af29d6c]) successfully deleted
I20251029 02:52:11.195554 1166 raft_consensus.cc:359] T c1222e30bc4141b3ae686d80053c4ef5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:11.196414 1174 tablet_replica.cc:333] T da2a21c3236a4bd99872b89b12a36508 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:11.196657 1174 raft_consensus.cc:2243] T da2a21c3236a4bd99872b89b12a36508 P c756528642a543eeacd93cef3e335a7d [term 2 LEADER]: Raft consensus shutting down.
I20251029 02:52:11.168038 1052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a9e0fcc759a64679b3f59a6c964877b6" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:11.199199 1176 ts_tablet_manager.cc:1929] T c9dff4dff82f49e7ad49caad5e404f09 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:11.199373 1176 log.cc:1199] T c9dff4dff82f49e7ad49caad5e404f09 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/c9dff4dff82f49e7ad49caad5e404f09
I20251029 02:52:11.168185 1053 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c1222e30bc4141b3ae686d80053c4ef5" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:11.200861 1052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "58dde9397b434b089ade9968f369006a" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:11.202656 1174 pending_rounds.cc:70] T da2a21c3236a4bd99872b89b12a36508 P c756528642a543eeacd93cef3e335a7d: Trying to abort 1 pending ops.
I20251029 02:52:11.202749 1174 pending_rounds.cc:77] T da2a21c3236a4bd99872b89b12a36508 P c756528642a543eeacd93cef3e335a7d: Aborting op as it isn't in flight: id { term: 2 index: 3 } timestamp: 7215949129114021888 op_type: NO_OP noop_request { }
I20251029 02:52:11.203028 1174 raft_consensus.cc:2889] T da2a21c3236a4bd99872b89b12a36508 P c756528642a543eeacd93cef3e335a7d [term 2 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 02:52:11.203054 1054 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7bb85a12f573494cbb3118d1947eda27" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:11.203176 1174 raft_consensus.cc:2272] T da2a21c3236a4bd99872b89b12a36508 P c756528642a543eeacd93cef3e335a7d [term 2 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:11.203594 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "378f549ff7364d018d8b9a5ef01ac878" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:11.204035 524 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 509288c033d54e4b9da3f26a30dc46a0
I20251029 02:52:11.204903 756 tablet_service.cc:1552] Processing DeleteTablet for tablet 509288c033d54e4b9da3f26a30dc46a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:11 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:11.206609 1052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "20a762344c5f46f88481c4ac7ad6e43b" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:11.206948 523 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet c172417d1d444c439e9cda611a2ed551
I20251029 02:52:11.207226 1174 ts_tablet_manager.cc:1916] T da2a21c3236a4bd99872b89b12a36508 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:11.207926 756 tablet_service.cc:1552] Processing DeleteTablet for tablet c172417d1d444c439e9cda611a2ed551 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:11 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:11.207898 712 leader_election.cc:304] T adf6e736cb9c4920807ee58682e134ef P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters: c756528642a543eeacd93cef3e335a7d
W20251029 02:52:11.208376 712 leader_election.cc:343] T 731c0acfbe204e8199ad246e38ae2ccd P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:11.208493 712 leader_election.cc:304] T 731c0acfbe204e8199ad246e38ae2ccd P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:11.208746 2113 raft_consensus.cc:2804] T adf6e736cb9c4920807ee58682e134ef P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:11.208818 2113 raft_consensus.cc:493] T adf6e736cb9c4920807ee58682e134ef P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:11.208879 2113 raft_consensus.cc:3060] T adf6e736cb9c4920807ee58682e134ef P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:11.209486 2309 raft_consensus.cc:2749] T 731c0acfbe204e8199ad246e38ae2ccd P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:11.210875 878 tablet_service.cc:1552] Processing DeleteTablet for tablet 509288c033d54e4b9da3f26a30dc46a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:11 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:11.211031 894 tablet_service.cc:1552] Processing DeleteTablet for tablet c172417d1d444c439e9cda611a2ed551 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:11 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:11.211967 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "44c0488eba0543078ff4f0b3d69366f2" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:11.212136 776 raft_consensus.cc:2468] T 44c0488eba0543078ff4f0b3d69366f2 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 0.
I20251029 02:52:11.212536 764 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "71b4b91071a54d6aa66160e7955bc915" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:11.212677 764 raft_consensus.cc:2468] T 71b4b91071a54d6aa66160e7955bc915 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 0.
I20251029 02:52:11.214161 2302 raft_consensus.cc:493] T 6d830d30b26b4ba799785fec8a6f0769 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:11.214293 2302 raft_consensus.cc:515] T 6d830d30b26b4ba799785fec8a6f0769 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:11.214368 1175 ts_tablet_manager.cc:1929] T 2e97d1bc06a24e8d8ba11acb6fd280d8 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:11.214469 1175 log.cc:1199] T 2e97d1bc06a24e8d8ba11acb6fd280d8 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/2e97d1bc06a24e8d8ba11acb6fd280d8
I20251029 02:52:11.214798 1175 ts_tablet_manager.cc:1950] T 2e97d1bc06a24e8d8ba11acb6fd280d8 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:11.214926 2302 leader_election.cc:290] T 6d830d30b26b4ba799785fec8a6f0769 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:11.215978 1052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d830d30b26b4ba799785fec8a6f0769" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:11.216167 1052 raft_consensus.cc:2393] T 6d830d30b26b4ba799785fec8a6f0769 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c756528642a543eeacd93cef3e335a7d in current term 1: Already voted for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in this term.
I20251029 02:52:11.218775 769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6d830d30b26b4ba799785fec8a6f0769" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:11.218972 769 raft_consensus.cc:2393] T 6d830d30b26b4ba799785fec8a6f0769 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c756528642a543eeacd93cef3e335a7d in current term 1: Already voted for candidate b495a49349324f7d81908f2da700f0d9 in this term.
I20251029 02:52:11.219377 847 leader_election.cc:304] T 6d830d30b26b4ba799785fec8a6f0769 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9
I20251029 02:52:11.219782 2302 raft_consensus.cc:3060] T 6d830d30b26b4ba799785fec8a6f0769 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:11.220052 1172 tablet_bootstrap.cc:654] T 6789bc16664e4b37b86702449ab27855 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:11.221077 2113 raft_consensus.cc:515] T adf6e736cb9c4920807ee58682e134ef P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:11.221597 2113 leader_election.cc:290] T adf6e736cb9c4920807ee58682e134ef P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 election: Requested vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:11.221930 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "adf6e736cb9c4920807ee58682e134ef" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
I20251029 02:52:11.222358 2302 raft_consensus.cc:2749] T 6d830d30b26b4ba799785fec8a6f0769 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 02:52:11.222419 711 leader_election.cc:343] T adf6e736cb9c4920807ee58682e134ef P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:11.224553 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 812) took 3392 ms. Trace:
I20251029 02:52:11.224684 1175 rpcz_store.cc:276] 1029 02:52:07.832055 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:07.832112 (+ 57us) service_pool.cc:225] Handling call
1029 02:52:10.978865 (+3146753us) ts_tablet_manager.cc:1151] Deleting tablet 2e97d1bc06a24e8d8ba11acb6fd280d8
1029 02:52:10.978878 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:11.049467 (+ 70589us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:11.214252 (+164785us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:11.224532 (+ 10280us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3146543}
I20251029 02:52:11.225181 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 2e97d1bc06a24e8d8ba11acb6fd280d8 (table default.Table_24be2b894899463a810d2fe7511d4d6b [id=43e922b1efe44ad9882b5084b22d1560]) successfully deleted
I20251029 02:52:11.225248 1167 tablet_bootstrap.cc:492] T c32c28b0e74a4af3a0e69bbf24ab1a1f P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:11.225387 1167 ts_tablet_manager.cc:1403] T c32c28b0e74a4af3a0e69bbf24ab1a1f P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.036s user 0.003s sys 0.000s
I20251029 02:52:11.226055 1167 raft_consensus.cc:359] T c32c28b0e74a4af3a0e69bbf24ab1a1f P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:11.226217 1167 raft_consensus.cc:385] T c32c28b0e74a4af3a0e69bbf24ab1a1f P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:11.226344 1167 raft_consensus.cc:740] T c32c28b0e74a4af3a0e69bbf24ab1a1f P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:11.226580 1167 consensus_queue.cc:260] T c32c28b0e74a4af3a0e69bbf24ab1a1f P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:11.230813 1167 ts_tablet_manager.cc:1434] T c32c28b0e74a4af3a0e69bbf24ab1a1f P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251029 02:52:11.233347 1167 tablet_bootstrap.cc:492] T 946f65014d97425aab3851009b742807 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:11.234158 1175 tablet_replica.cc:333] T da2a21c3236a4bd99872b89b12a36508 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:11.234684 1175 raft_consensus.cc:2243] T da2a21c3236a4bd99872b89b12a36508 P b495a49349324f7d81908f2da700f0d9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:11.234908 1175 raft_consensus.cc:2272] T da2a21c3236a4bd99872b89b12a36508 P b495a49349324f7d81908f2da700f0d9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:11.235648 1175 ts_tablet_manager.cc:1916] T da2a21c3236a4bd99872b89b12a36508 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:11.234484 1172 tablet_bootstrap.cc:492] T 6789bc16664e4b37b86702449ab27855 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:11.237752 1172 ts_tablet_manager.cc:1403] T 6789bc16664e4b37b86702449ab27855 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.056s user 0.002s sys 0.001s
I20251029 02:52:11.238484 1172 raft_consensus.cc:359] T 6789bc16664e4b37b86702449ab27855 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:11.238739 1172 raft_consensus.cc:385] T 6789bc16664e4b37b86702449ab27855 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:11.238834 1172 raft_consensus.cc:740] T 6789bc16664e4b37b86702449ab27855 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:11.239092 1172 consensus_queue.cc:260] T 6789bc16664e4b37b86702449ab27855 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
W20251029 02:52:11.246248 526 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 10e0ad0693a1432091e1a590fb76a720 consensus state has no leader
I20251029 02:52:11.246573 521 catalog_manager.cc:6194] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: default.Table_0772f9db5d6142d988f6ffca2e96bdd2 [id=fd6d31d379394857a35422211af00642] alter complete (version 3)
W20251029 02:52:11.246713 517 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 595b4073cc0446cea041838097af7471 consensus state has no leader
I20251029 02:52:11.247064 519 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet c7a3e6d8e3fb45898d8a5e094dbf0fd7
I20251029 02:52:11.247691 522 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 7dbc9c8b110a48f0a801ad468738c3a3
I20251029 02:52:11.248844 756 tablet_service.cc:1552] Processing DeleteTablet for tablet 7dbc9c8b110a48f0a801ad468738c3a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:11 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:11.249539 878 tablet_service.cc:1552] Processing DeleteTablet for tablet c7a3e6d8e3fb45898d8a5e094dbf0fd7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:11 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:11.249593 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 8048506ced4540a5aee9f79bfc4599bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:06 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:11.249727 885 tablet_service.cc:1552] Processing DeleteTablet for tablet 7dbc9c8b110a48f0a801ad468738c3a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:11 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:11.254856 740 tablet_service.cc:1552] Processing DeleteTablet for tablet c7a3e6d8e3fb45898d8a5e094dbf0fd7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:11 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:11.257581 1172 ts_tablet_manager.cc:1434] T 6789bc16664e4b37b86702449ab27855 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.020s user 0.000s sys 0.001s
I20251029 02:52:11.258550 1166 raft_consensus.cc:385] T c1222e30bc4141b3ae686d80053c4ef5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:11.258716 1166 raft_consensus.cc:740] T c1222e30bc4141b3ae686d80053c4ef5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:11.258726 522 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet 9269622ca92645d5aa8e73341201ccbb (table default.Table_902ea0b243c7400c86250cfb5fae5b54 [id=d0ca7c93c8cf4dc19962a1957321c5b7]). Expected version 5 got 0
I20251029 02:52:11.259033 1166 consensus_queue.cc:260] T c1222e30bc4141b3ae686d80053c4ef5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:11.259166 522 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet 1bdb67125493430398ff5c80b73de2ac (table default.Table_2846af457a4b4e27a739af50c7f8c7f7 [id=8c3d3d88395e4712af0f62c4ad14edea]). Expected version 3 got 0
I20251029 02:52:11.259390 1053 raft_consensus.cc:2468] T c1222e30bc4141b3ae686d80053c4ef5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b495a49349324f7d81908f2da700f0d9 in term 0.
I20251029 02:52:11.259900 2113 raft_consensus.cc:493] T d2990e9f626c40618c118266bc595684 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0cfd4bee16f84b7c8b0ff3ce14db20d5)
I20251029 02:52:11.261263 1172 tablet_bootstrap.cc:492] T 372d93914abc41b38a702742b7db0bc2 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:11.261363 2113 raft_consensus.cc:515] T d2990e9f626c40618c118266bc595684 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:11.262032 878 tablet_service.cc:1552] Processing DeleteTablet for tablet c32c28b0e74a4af3a0e69bbf24ab1a1f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:57 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:11.262780 878 tablet_service.cc:1552] Processing DeleteTablet for tablet af4860c6b72f4d988b206b5ef2f22906 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:56 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:11.262823 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2990e9f626c40618c118266bc595684" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:11.263031 1172 tablet_bootstrap.cc:654] T 372d93914abc41b38a702742b7db0bc2 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:11.264933 2302 consensus_queue.cc:1048] T 59d03a190b784ab18a3e63a572675891 P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:11.266289 2113 leader_election.cc:290] T d2990e9f626c40618c118266bc595684 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:52:11.271353 1167 tablet_bootstrap.cc:654] T 946f65014d97425aab3851009b742807 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:11.277190 522 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_aaecc1c9efd04549a2484850ff8efc60" } modify_external_catalogs: true
I20251029 02:52:11.277441 522 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_aaecc1c9efd04549a2484850ff8efc60" } modify_external_catalogs: true
I20251029 02:52:11.278030 1176 ts_tablet_manager.cc:1950] T c9dff4dff82f49e7ad49caad5e404f09 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:11.278352 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 881) took 2714 ms. Trace:
I20251029 02:52:11.278438 1176 rpcz_store.cc:276] 1029 02:52:08.563675 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:08.563825 (+ 150us) service_pool.cc:225] Handling call
1029 02:52:11.163038 (+2599213us) ts_tablet_manager.cc:1151] Deleting tablet c9dff4dff82f49e7ad49caad5e404f09
1029 02:52:11.163047 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:11.171918 (+ 8871us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:11.176312 (+ 4394us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:11.278334 (+102022us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2599003}
I20251029 02:52:11.279740 1166 ts_tablet_manager.cc:1434] T c1222e30bc4141b3ae686d80053c4ef5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.121s user 0.001s sys 0.001s
I20251029 02:52:11.281365 1166 tablet_bootstrap.cc:492] T 946f65014d97425aab3851009b742807 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:11.282697 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_f565624917764d89a8a93c2e498f1105"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:11.282855 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_4bae416d7e1144d1bb614a134d8cccba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:11.283272 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 880) took 2719 ms. Trace:
W20251029 02:52:11.283315 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4bae416d7e1144d1bb614a134d8cccba in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:11.283347 1176 rpcz_store.cc:276] 1029 02:52:08.563517 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:08.564078 (+ 561us) service_pool.cc:225] Handling call
1029 02:52:11.278711 (+2714633us) ts_tablet_manager.cc:1151] Deleting tablet d3faa2d4df584c6e8da426a3830a6454
1029 02:52:11.278720 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:11.283253 (+ 4533us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2714491}
I20251029 02:52:11.283627 1176 tablet_replica.cc:333] T 71b4b91071a54d6aa66160e7955bc915 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:11.283819 1176 raft_consensus.cc:2243] T 71b4b91071a54d6aa66160e7955bc915 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:11.283973 1176 raft_consensus.cc:2272] T 71b4b91071a54d6aa66160e7955bc915 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:11.284463 1176 ts_tablet_manager.cc:1916] T 71b4b91071a54d6aa66160e7955bc915 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:11.284541 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_1963d5301e1a4a8ea388fbedfa53c6df"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:11.285184 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_3dd2917370f84ddd901c89c68f34c97d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:11.285642 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3dd2917370f84ddd901c89c68f34c97d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:52:11.283318 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f565624917764d89a8a93c2e498f1105 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:52:11.287660 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1963d5301e1a4a8ea388fbedfa53c6df in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:11.288079 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_1a2a463f983a4f408912990b262bc2a6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:11.288470 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1a2a463f983a4f408912990b262bc2a6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:11.289661 526 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_d2ac974941f9428eaa9605606209ed8d" } new_table_name: "default.Table_dd7b41306ba3479a9e7a9c305e9dff62" modify_external_catalogs: true
I20251029 02:52:11.289628 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_ec03963009c14e09b1fecad8b28ec9bb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:11.289861 522 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 96f7876e5d6d4a0db236df5e7f266fd2
W20251029 02:52:11.290104 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ec03963009c14e09b1fecad8b28ec9bb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:11.291054 878 tablet_service.cc:1552] Processing DeleteTablet for tablet 96f7876e5d6d4a0db236df5e7f266fd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:11 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:11.291117 1166 tablet_bootstrap.cc:654] T 946f65014d97425aab3851009b742807 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:11.291082 518 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_9791da4fa0ac47409f6cb424408a5f3b" } new_table_name: "default.Table_7ed13b6519054480839e84ccc5b0b77d" modify_external_catalogs: true
I20251029 02:52:11.296770 740 tablet_service.cc:1552] Processing DeleteTablet for tablet 96f7876e5d6d4a0db236df5e7f266fd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:11 UTC) from {username='slave'} at 127.0.0.1:44580
W20251029 02:52:11.297467 847 consensus_peers.cc:597] T 59d03a190b784ab18a3e63a572675891 P c756528642a543eeacd93cef3e335a7d -> Peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Couldn't send request to peer b495a49349324f7d81908f2da700f0d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 59d03a190b784ab18a3e63a572675891. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:11.300348 1172 tablet_bootstrap.cc:492] T 372d93914abc41b38a702742b7db0bc2 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:11.300486 1172 ts_tablet_manager.cc:1403] T 372d93914abc41b38a702742b7db0bc2 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.039s user 0.002s sys 0.000s
I20251029 02:52:11.300520 2201 raft_consensus.cc:493] T d2990e9f626c40618c118266bc595684 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0cfd4bee16f84b7c8b0ff3ce14db20d5)
I20251029 02:52:11.300679 2201 raft_consensus.cc:515] T d2990e9f626c40618c118266bc595684 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:11.301180 1172 raft_consensus.cc:359] T 372d93914abc41b38a702742b7db0bc2 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:11.301309 2201 leader_election.cc:290] T d2990e9f626c40618c118266bc595684 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:11.301343 1172 raft_consensus.cc:385] T 372d93914abc41b38a702742b7db0bc2 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:11.301393 1172 raft_consensus.cc:740] T 372d93914abc41b38a702742b7db0bc2 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:11.301584 1172 consensus_queue.cc:260] T 372d93914abc41b38a702742b7db0bc2 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:11.302222 769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2990e9f626c40618c118266bc595684" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
W20251029 02:52:11.302296 849 leader_election.cc:343] T d2990e9f626c40618c118266bc595684 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Not found: Tablet not found: d2990e9f626c40618c118266bc595684
I20251029 02:52:11.302410 769 raft_consensus.cc:2468] T d2990e9f626c40618c118266bc595684 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:11.302907 847 leader_election.cc:304] T d2990e9f626c40618c118266bc595684 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5
I20251029 02:52:11.303562 2201 raft_consensus.cc:2804] T d2990e9f626c40618c118266bc595684 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251029 02:52:11.303651 2201 raft_consensus.cc:493] T d2990e9f626c40618c118266bc595684 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0cfd4bee16f84b7c8b0ff3ce14db20d5)
I20251029 02:52:11.303759 2201 raft_consensus.cc:3060] T d2990e9f626c40618c118266bc595684 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Advancing to term 2
I20251029 02:52:11.305461 1172 ts_tablet_manager.cc:1434] T 372d93914abc41b38a702742b7db0bc2 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251029 02:52:11.305938 1172 tablet_bootstrap.cc:492] T 32519fbffcef44a890161e5f3043806c P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:11.307186 2201 raft_consensus.cc:515] T d2990e9f626c40618c118266bc595684 P c756528642a543eeacd93cef3e335a7d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
W20251029 02:52:11.308387 849 leader_election.cc:343] T d2990e9f626c40618c118266bc595684 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Not found: Tablet not found: d2990e9f626c40618c118266bc595684
I20251029 02:52:11.309084 769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d2990e9f626c40618c118266bc595684" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:11.309249 769 raft_consensus.cc:3060] T d2990e9f626c40618c118266bc595684 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Advancing to term 2
W20251029 02:52:11.310158 526 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4f004ea546d64681bd29b2e7ee3bb225 consensus state has no leader
I20251029 02:52:11.283865 1175 ts_tablet_manager.cc:1929] T da2a21c3236a4bd99872b89b12a36508 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251029 02:52:11.311371 1175 log.cc:1199] T da2a21c3236a4bd99872b89b12a36508 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/da2a21c3236a4bd99872b89b12a36508
I20251029 02:52:11.311934 769 raft_consensus.cc:2468] T d2990e9f626c40618c118266bc595684 P b495a49349324f7d81908f2da700f0d9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 2.
I20251029 02:52:11.312686 847 leader_election.cc:304] T d2990e9f626c40618c118266bc595684 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5
I20251029 02:52:11.313038 2302 raft_consensus.cc:2804] T d2990e9f626c40618c118266bc595684 P c756528642a543eeacd93cef3e335a7d [term 2 FOLLOWER]: Leader election won for term 2
I20251029 02:52:11.313190 2302 raft_consensus.cc:697] T d2990e9f626c40618c118266bc595684 P c756528642a543eeacd93cef3e335a7d [term 2 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:52:11.313474 2302 consensus_queue.cc:237] T d2990e9f626c40618c118266bc595684 P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:11.315685 2201 leader_election.cc:290] T d2990e9f626c40618c118266bc595684 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 election: Requested vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:11.316573 519 master_service.cc:954] ReplaceTablet: received request to replace tablet ecf3b900206b4c86a7adbc83a769d90e from {username='slave'} at 127.0.0.1:35314
W20251029 02:52:11.318305 518 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fcc2d413e3b54b2da9b5b44f8b859816 consensus state has no leader
I20251029 02:52:11.323572 917 raft_consensus.cc:1275] T 41a6139fd9b84707a8c7eebc0381e383 P c756528642a543eeacd93cef3e335a7d [term 2 FOLLOWER]: Refusing update from remote peer b495a49349324f7d81908f2da700f0d9: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20251029 02:52:11.324234 2113 consensus_queue.cc:1048] T 41a6139fd9b84707a8c7eebc0381e383 P b495a49349324f7d81908f2da700f0d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20251029 02:52:11.326458 519 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet ecf3b900206b4c86a7adbc83a769d90e
I20251029 02:52:11.327018 519 catalog_manager.cc:6719] ReplaceTablet: tablet ecf3b900206b4c86a7adbc83a769d90e deleted and replaced by tablet 2e96a398bfd74cf68cf0377a813dc1dd
I20251029 02:52:11.328161 878 tablet_service.cc:1552] Processing DeleteTablet for tablet ecf3b900206b4c86a7adbc83a769d90e with delete_type TABLET_DATA_DELETED (replaced by tablet 2e96a398bfd74cf68cf0377a813dc1dd) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:11.328904 740 tablet_service.cc:1552] Processing DeleteTablet for tablet ecf3b900206b4c86a7adbc83a769d90e with delete_type TABLET_DATA_DELETED (replaced by tablet 2e96a398bfd74cf68cf0377a813dc1dd) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:11.333670 1174 ts_tablet_manager.cc:1929] T da2a21c3236a4bd99872b89b12a36508 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.3
I20251029 02:52:11.338464 1175 ts_tablet_manager.cc:1950] T da2a21c3236a4bd99872b89b12a36508 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:11.334571 1167 tablet_bootstrap.cc:492] T 946f65014d97425aab3851009b742807 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:11.338827 1167 ts_tablet_manager.cc:1403] T 946f65014d97425aab3851009b742807 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.107s user 0.004s sys 0.000s
I20251029 02:52:11.338976 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 813) took 3372 ms. Trace:
I20251029 02:52:11.339082 1175 rpcz_store.cc:276] 1029 02:52:07.967050 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:07.968344 (+ 1294us) service_pool.cc:225] Handling call
1029 02:52:11.234106 (+3265762us) ts_tablet_manager.cc:1151] Deleting tablet da2a21c3236a4bd99872b89b12a36508
1029 02:52:11.234115 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:11.241706 (+ 7591us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:11.275993 (+ 34287us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:11.338953 (+ 62960us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3265562}
I20251029 02:52:11.339545 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet da2a21c3236a4bd99872b89b12a36508 (table default.Table_2a63af6894694f16b45f4bf48f73a68f [id=fb42af08ec1f48498ee5e91cce62333e]) successfully deleted
W20251029 02:52:11.339866 510 consensus_peers.cc:597] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0 -> Peer e9672114a6a742e2864c16ff60ba246e (127.0.104.62:42343): Couldn't send request to peer e9672114a6a742e2864c16ff60ba246e. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.62:42343: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20251029 02:52:11.339985 1167 raft_consensus.cc:359] T 946f65014d97425aab3851009b742807 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:11.340183 1167 raft_consensus.cc:385] T 946f65014d97425aab3851009b742807 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:11.340255 1167 raft_consensus.cc:740] T 946f65014d97425aab3851009b742807 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:11.340513 1167 consensus_queue.cc:260] T 946f65014d97425aab3851009b742807 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:11.340811 1172 tablet_bootstrap.cc:654] T 32519fbffcef44a890161e5f3043806c P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:11.341195 1167 ts_tablet_manager.cc:1434] T 946f65014d97425aab3851009b742807 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 02:52:11.341488 1167 tablet_bootstrap.cc:492] T 97b2ecae572a4b7ea1b9595074556708 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:11.345435 1175 tablet_replica.cc:333] T 41a6139fd9b84707a8c7eebc0381e383 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:11.345638 1175 raft_consensus.cc:2243] T 41a6139fd9b84707a8c7eebc0381e383 P b495a49349324f7d81908f2da700f0d9 [term 2 LEADER]: Raft consensus shutting down.
I20251029 02:52:11.345855 1175 pending_rounds.cc:70] T 41a6139fd9b84707a8c7eebc0381e383 P b495a49349324f7d81908f2da700f0d9: Trying to abort 1 pending ops.
I20251029 02:52:11.345903 1175 pending_rounds.cc:77] T 41a6139fd9b84707a8c7eebc0381e383 P b495a49349324f7d81908f2da700f0d9: Aborting op as it isn't in flight: id { term: 2 index: 3 } timestamp: 7215949129482465280 op_type: NO_OP noop_request { }
I20251029 02:52:11.345999 1175 raft_consensus.cc:2889] T 41a6139fd9b84707a8c7eebc0381e383 P b495a49349324f7d81908f2da700f0d9 [term 2 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 02:52:11.346076 1175 raft_consensus.cc:2272] T 41a6139fd9b84707a8c7eebc0381e383 P b495a49349324f7d81908f2da700f0d9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:11.346700 2113 raft_consensus.cc:493] T 40e844fd7ad74680bcbe7bef936c5b9d P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:11.346879 2113 raft_consensus.cc:515] T 40e844fd7ad74680bcbe7bef936c5b9d P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:11.347723 2113 leader_election.cc:290] T 40e844fd7ad74680bcbe7bef936c5b9d P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:11.347811 918 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40e844fd7ad74680bcbe7bef936c5b9d" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:11.348294 711 leader_election.cc:343] T 40e844fd7ad74680bcbe7bef936c5b9d P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:11.349253 1175 ts_tablet_manager.cc:1916] T 41a6139fd9b84707a8c7eebc0381e383 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:11.350554 1167 tablet_bootstrap.cc:654] T 97b2ecae572a4b7ea1b9595074556708 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:11.357652 519 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_ac619c8d8e8a4b8bba9247c4b0b1c65e" } new_table_name: "default.Table_c4873397293c4864b9ed6fa23ebcbb38" modify_external_catalogs: true
I20251029 02:52:11.359707 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_ad87203491b8455d98e85df6212fe26f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:11.360440 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ad87203491b8455d98e85df6212fe26f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:52:11.371230 519 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet be23b3227a16415ab2fd4c6196dbcad4 consensus state has no leader
I20251029 02:52:11.379290 1166 tablet_bootstrap.cc:492] T 946f65014d97425aab3851009b742807 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:11.379515 1166 ts_tablet_manager.cc:1403] T 946f65014d97425aab3851009b742807 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.100s user 0.002s sys 0.002s
I20251029 02:52:11.380303 1166 raft_consensus.cc:359] T 946f65014d97425aab3851009b742807 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:11.381942 1166 raft_consensus.cc:385] T 946f65014d97425aab3851009b742807 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:11.382120 1166 raft_consensus.cc:740] T 946f65014d97425aab3851009b742807 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:11.382426 1166 consensus_queue.cc:260] T 946f65014d97425aab3851009b742807 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:11.387056 1166 ts_tablet_manager.cc:1434] T 946f65014d97425aab3851009b742807 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20251029 02:52:11.387413 1166 tablet_bootstrap.cc:492] T afb19ba39399410bb90b6225d7b5b5b1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:11.397059 1166 tablet_bootstrap.cc:654] T afb19ba39399410bb90b6225d7b5b5b1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:11.397665 1172 tablet_bootstrap.cc:492] T 32519fbffcef44a890161e5f3043806c P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:11.398253 1176 ts_tablet_manager.cc:1929] T 71b4b91071a54d6aa66160e7955bc915 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:11.398425 1176 log.cc:1199] T 71b4b91071a54d6aa66160e7955bc915 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/71b4b91071a54d6aa66160e7955bc915
I20251029 02:52:11.398928 1176 ts_tablet_manager.cc:1950] T 71b4b91071a54d6aa66160e7955bc915 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:11.399441 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 879) took 2836 ms. Trace:
I20251029 02:52:11.399586 1176 rpcz_store.cc:276] 1029 02:52:08.563396 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:08.578399 (+ 15003us) service_pool.cc:225] Handling call
1029 02:52:11.283569 (+2705170us) ts_tablet_manager.cc:1151] Deleting tablet 71b4b91071a54d6aa66160e7955bc915
1029 02:52:11.283581 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:11.374944 (+ 91363us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:11.398135 (+ 23191us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:11.399422 (+ 1287us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2704875}
I20251029 02:52:11.399983 1176 tablet_replica.cc:333] T d61beda662d844cdaf67fd9a2e382ea2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:11.400207 1176 raft_consensus.cc:2243] T d61beda662d844cdaf67fd9a2e382ea2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:11.400377 1167 tablet_bootstrap.cc:492] T 97b2ecae572a4b7ea1b9595074556708 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:11.400398 1176 raft_consensus.cc:2272] T d61beda662d844cdaf67fd9a2e382ea2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:11.400535 1167 ts_tablet_manager.cc:1403] T 97b2ecae572a4b7ea1b9595074556708 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.059s user 0.000s sys 0.003s
I20251029 02:52:11.401055 1176 ts_tablet_manager.cc:1916] T d61beda662d844cdaf67fd9a2e382ea2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:11.401998 1172 ts_tablet_manager.cc:1403] T 32519fbffcef44a890161e5f3043806c P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.096s user 0.002s sys 0.002s
I20251029 02:52:11.402928 1172 raft_consensus.cc:359] T 32519fbffcef44a890161e5f3043806c P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:11.403136 1172 raft_consensus.cc:385] T 32519fbffcef44a890161e5f3043806c P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:11.403192 1172 raft_consensus.cc:740] T 32519fbffcef44a890161e5f3043806c P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:11.403422 1172 consensus_queue.cc:260] T 32519fbffcef44a890161e5f3043806c P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:11.407541 524 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet 27b08f88b99643d28ac609e45bf8104e (table default.Table_a9e50760813844af85a85c5f8d22f8e0 [id=74a2a41c908146ec8e16c6a3227b94f2]). Expected version 1 got 0
I20251029 02:52:11.410025 1167 raft_consensus.cc:359] T 97b2ecae572a4b7ea1b9595074556708 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:11.410239 1167 raft_consensus.cc:385] T 97b2ecae572a4b7ea1b9595074556708 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:11.410305 1167 raft_consensus.cc:740] T 97b2ecae572a4b7ea1b9595074556708 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:11.413780 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_35f77c51187d489fad31db01129e5e98"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:11.414047 518 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet 27b08f88b99643d28ac609e45bf8104e (table default.Table_a9e50760813844af85a85c5f8d22f8e0 [id=74a2a41c908146ec8e16c6a3227b94f2]). Expected version 1 got 0
I20251029 02:52:11.414266 518 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet 6789bc16664e4b37b86702449ab27855 (table default.Table_9a7d7f4f9073437097aca9f7d6dc665e [id=ab8aeaf90f6c4bc2ad5a98780945c40a]). Expected version 2 got 0
I20251029 02:52:11.414461 518 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet 32519fbffcef44a890161e5f3043806c (table default.Table_d64a9ac717a94c40a98f826136089824 [id=b95fe1c7d6ff4519b90f993f1fa961fd]). Expected version 2 got 0
I20251029 02:52:11.414577 1167 consensus_queue.cc:260] T 97b2ecae572a4b7ea1b9595074556708 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:11.414682 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_ad45e164be6a481c871f036851e127f2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:11.415251 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ad45e164be6a481c871f036851e127f2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:11.415580 2205 raft_consensus.cc:493] T 8048506ced4540a5aee9f79bfc4599bf P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:11.415720 2205 raft_consensus.cc:515] T 8048506ced4540a5aee9f79bfc4599bf P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:11.416424 2205 leader_election.cc:290] T 8048506ced4540a5aee9f79bfc4599bf P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
W20251029 02:52:11.416999 712 leader_election.cc:343] T d7abaf84ba5d445aa17433b3be891548 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:11.417110 769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8048506ced4540a5aee9f79bfc4599bf" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:11.417151 1053 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8048506ced4540a5aee9f79bfc4599bf" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:11.417317 769 raft_consensus.cc:2393] T 8048506ced4540a5aee9f79bfc4599bf P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c756528642a543eeacd93cef3e335a7d in current term 1: Already voted for candidate b495a49349324f7d81908f2da700f0d9 in this term.
I20251029 02:52:11.417398 1053 raft_consensus.cc:2393] T 8048506ced4540a5aee9f79bfc4599bf P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c756528642a543eeacd93cef3e335a7d in current term 1: Already voted for candidate b495a49349324f7d81908f2da700f0d9 in this term.
I20251029 02:52:11.417114 712 leader_election.cc:304] T d7abaf84ba5d445aa17433b3be891548 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:11.417892 712 leader_election.cc:304] T 97b2ecae572a4b7ea1b9595074556708 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters: c756528642a543eeacd93cef3e335a7d
I20251029 02:52:11.418236 712 leader_election.cc:304] T 82289e0f823c4d43bfd8c4e878572b2b P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters: c756528642a543eeacd93cef3e335a7d
I20251029 02:52:11.418267 1167 ts_tablet_manager.cc:1434] T 97b2ecae572a4b7ea1b9595074556708 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.018s user 0.003s sys 0.000s
I20251029 02:52:11.418563 2309 raft_consensus.cc:2749] T d7abaf84ba5d445aa17433b3be891548 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:11.418743 2309 raft_consensus.cc:2804] T 97b2ecae572a4b7ea1b9595074556708 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:11.418807 2309 raft_consensus.cc:493] T 97b2ecae572a4b7ea1b9595074556708 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:11.418867 2309 raft_consensus.cc:3060] T 97b2ecae572a4b7ea1b9595074556708 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:11.420288 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 777992acb1bd4765886b802940db74bb (table default.Table_d9fe74a49f4843b089aae7fca9e4a826 [id=152882c742b2459ba863fd41083284c1]) successfully deleted
I20251029 02:52:11.421826 2309 raft_consensus.cc:515] T 97b2ecae572a4b7ea1b9595074556708 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:11.422551 2113 raft_consensus.cc:2804] T 82289e0f823c4d43bfd8c4e878572b2b P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:11.422622 2113 raft_consensus.cc:493] T 82289e0f823c4d43bfd8c4e878572b2b P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:11.422680 2113 raft_consensus.cc:3060] T 82289e0f823c4d43bfd8c4e878572b2b P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:11.423890 878 tablet_service.cc:1505] Processing CreateTablet for tablet 89750e7fa7414914ad9f0f0d146544e1 (DEFAULT_TABLE table=default.Table_fb178c25d3c1496f8ed5db2fe26d9fb1 [id=3edaa0265abc45f18102db2e21c8c302]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:11.425472 2113 raft_consensus.cc:515] T 82289e0f823c4d43bfd8c4e878572b2b P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:11.425861 2302 raft_consensus.cc:493] T 946f65014d97425aab3851009b742807 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:11.426010 2302 raft_consensus.cc:515] T 946f65014d97425aab3851009b742807 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:11.427071 2302 leader_election.cc:290] T 946f65014d97425aab3851009b742807 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:11.427332 2113 leader_election.cc:290] T 82289e0f823c4d43bfd8c4e878572b2b P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 election: Requested vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:11.427345 1053 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "946f65014d97425aab3851009b742807" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:11.427525 1053 raft_consensus.cc:2468] T 946f65014d97425aab3851009b742807 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:11.427908 849 leader_election.cc:304] T 946f65014d97425aab3851009b742807 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:11.427971 769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "946f65014d97425aab3851009b742807" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:11.428117 769 raft_consensus.cc:2468] T 946f65014d97425aab3851009b742807 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:11.428220 2302 raft_consensus.cc:2804] T 946f65014d97425aab3851009b742807 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:11.428287 2302 raft_consensus.cc:493] T 946f65014d97425aab3851009b742807 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:11.428346 2302 raft_consensus.cc:3060] T 946f65014d97425aab3851009b742807 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:11.428426 847 leader_election.cc:304] T 8048506ced4540a5aee9f79bfc4599bf P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9
I20251029 02:52:11.428783 2205 raft_consensus.cc:3060] T 8048506ced4540a5aee9f79bfc4599bf P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:11.431236 2205 raft_consensus.cc:2749] T 8048506ced4540a5aee9f79bfc4599bf P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:11.431305 2302 raft_consensus.cc:515] T 946f65014d97425aab3851009b742807 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
W20251029 02:52:11.414465 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_35f77c51187d489fad31db01129e5e98 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:11.432219 1053 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "946f65014d97425aab3851009b742807" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:52:11.432394 1053 raft_consensus.cc:3060] T 946f65014d97425aab3851009b742807 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:11.414475 520 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_2846af457a4b4e27a739af50c7f8c7f7" } modify_external_catalogs: true
I20251029 02:52:11.433106 520 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_2846af457a4b4e27a739af50c7f8c7f7" } modify_external_catalogs: true
I20251029 02:52:11.434546 524 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_114cbdf1ef0f493f8f8c834735271053" } modify_external_catalogs: true
I20251029 02:52:11.434697 2302 leader_election.cc:290] T 946f65014d97425aab3851009b742807 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:11.434700 524 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_114cbdf1ef0f493f8f8c834735271053" } modify_external_catalogs: true
I20251029 02:52:11.435496 1053 raft_consensus.cc:2468] T 946f65014d97425aab3851009b742807 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:11.435910 849 leader_election.cc:304] T 946f65014d97425aab3851009b742807 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:11.436256 2302 raft_consensus.cc:2804] T 946f65014d97425aab3851009b742807 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:11.436403 2302 raft_consensus.cc:697] T 946f65014d97425aab3851009b742807 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:52:11.436661 2302 consensus_queue.cc:237] T 946f65014d97425aab3851009b742807 P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:11.437041 918 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82289e0f823c4d43bfd8c4e878572b2b" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
I20251029 02:52:11.437124 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97b2ecae572a4b7ea1b9595074556708" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
I20251029 02:52:11.437271 917 raft_consensus.cc:3060] T 97b2ecae572a4b7ea1b9595074556708 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:11.440502 917 raft_consensus.cc:2468] T 97b2ecae572a4b7ea1b9595074556708 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b495a49349324f7d81908f2da700f0d9 in term 1.
I20251029 02:52:11.441160 1174 log.cc:1199] T da2a21c3236a4bd99872b89b12a36508 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/da2a21c3236a4bd99872b89b12a36508
I20251029 02:52:11.441813 1174 ts_tablet_manager.cc:1950] T da2a21c3236a4bd99872b89b12a36508 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:11.441815 769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "946f65014d97425aab3851009b742807" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:11.442041 769 raft_consensus.cc:3060] T 946f65014d97425aab3851009b742807 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:11.443971 1025 tablet_service.cc:1505] Processing CreateTablet for tablet 9b06a5d8a32341d19807ae0a99819916 (DEFAULT_TABLE table=default.Table_443fde25ba1b481f9146d61ad842bc9b [id=21b310f695e24d089cb6f90ed70fad5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:11.444401 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b06a5d8a32341d19807ae0a99819916. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:11.444710 769 raft_consensus.cc:2468] T 946f65014d97425aab3851009b742807 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:11.445603 740 tablet_service.cc:1505] Processing CreateTablet for tablet 89750e7fa7414914ad9f0f0d146544e1 (DEFAULT_TABLE table=default.Table_fb178c25d3c1496f8ed5db2fe26d9fb1 [id=3edaa0265abc45f18102db2e21c8c302]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 02:52:11.446979 711 leader_election.cc:343] T 82289e0f823c4d43bfd8c4e878572b2b P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:11.447170 711 leader_election.cc:304] T 97b2ecae572a4b7ea1b9595074556708 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:11.449947 2309 leader_election.cc:290] T 97b2ecae572a4b7ea1b9595074556708 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 election: Requested vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:11.450259 2309 raft_consensus.cc:2804] T 97b2ecae572a4b7ea1b9595074556708 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:11.450462 2309 raft_consensus.cc:697] T 97b2ecae572a4b7ea1b9595074556708 P b495a49349324f7d81908f2da700f0d9 [term 1 LEADER]: Becoming Leader. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Running, Role: LEADER
I20251029 02:52:11.451578 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_eb6ed8b2aee44b00a5ac4f8d2a357f28"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:11.452142 523 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_d44327be26be4ff1b62a7be7dada45c0" } new_table_name: "default.Table_a20cf185a2d94a80a1877ecaf2d276f3" modify_external_catalogs: true
W20251029 02:52:11.452293 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eb6ed8b2aee44b00a5ac4f8d2a357f28 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:11.452656 1028 tablet_service.cc:1505] Processing CreateTablet for tablet ea1401ae51524d4380fba75d762031f3 (DEFAULT_TABLE table=default.Table_d518217e586b4a57a08855986cc0026b [id=23ac8ca52fc64ba4aa2bc2c3515628eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:11.452931 2309 consensus_queue.cc:237] T 97b2ecae572a4b7ea1b9595074556708 P b495a49349324f7d81908f2da700f0d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:11.453073 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea1401ae51524d4380fba75d762031f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:11.453413 526 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_aae2119a32f548aa829d7f408c573777" } modify_external_catalogs: true
I20251029 02:52:11.453612 526 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_aae2119a32f548aa829d7f408c573777" } modify_external_catalogs: true
I20251029 02:52:11.453428 521 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_59a5648999114c6f8b2a58e938c8e49a" } modify_external_catalogs: true
I20251029 02:52:11.453886 521 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_59a5648999114c6f8b2a58e938c8e49a" } modify_external_catalogs: true
I20251029 02:52:11.455111 518 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_755825b17834457cb94b05163ae3e26d" } new_table_name: "default.Table_6563934f48114b0987d298bfad3e9bf7" modify_external_catalogs: true
I20251029 02:52:11.460906 1025 tablet_service.cc:1505] Processing CreateTablet for tablet 7751599f5dfc43baa20dc3487c6eddd9 (DEFAULT_TABLE table=default.Table_07d2f4a3c8074ece8b4982e3819caabc [id=8ad6d1d4fbe9404f8984dc8dc98916b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:11.461443 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7751599f5dfc43baa20dc3487c6eddd9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:11.462973 2205 raft_consensus.cc:493] T 71b4b91071a54d6aa66160e7955bc915 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:11.463132 2205 raft_consensus.cc:515] T 71b4b91071a54d6aa66160e7955bc915 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:11.463867 2205 leader_election.cc:290] T 71b4b91071a54d6aa66160e7955bc915 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:11.464277 769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "71b4b91071a54d6aa66160e7955bc915" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:11.464464 769 raft_consensus.cc:2468] T 71b4b91071a54d6aa66160e7955bc915 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
W20251029 02:52:11.464466 849 leader_election.cc:343] T 71b4b91071a54d6aa66160e7955bc915 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Not found: Tablet not found: 71b4b91071a54d6aa66160e7955bc915
I20251029 02:52:11.464854 847 leader_election.cc:304] T 71b4b91071a54d6aa66160e7955bc915 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5
I20251029 02:52:11.465210 2205 raft_consensus.cc:2804] T 71b4b91071a54d6aa66160e7955bc915 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:11.465283 2205 raft_consensus.cc:493] T 71b4b91071a54d6aa66160e7955bc915 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:11.465351 2205 raft_consensus.cc:3060] T 71b4b91071a54d6aa66160e7955bc915 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:11.467999 2205 raft_consensus.cc:515] T 71b4b91071a54d6aa66160e7955bc915 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:11.469035 769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "71b4b91071a54d6aa66160e7955bc915" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:11.469220 769 raft_consensus.cc:3060] T 71b4b91071a54d6aa66160e7955bc915 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
W20251029 02:52:11.469334 849 leader_election.cc:343] T 71b4b91071a54d6aa66160e7955bc915 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Not found: Tablet not found: 71b4b91071a54d6aa66160e7955bc915
I20251029 02:52:11.469982 1028 tablet_service.cc:1505] Processing CreateTablet for tablet 60f0eadc074a4872b6638314fed98e0e (DEFAULT_TABLE table=default.Table_6d9b81aa4e3a4880ac256ecf0fe9f010 [id=c65df2dc54134ab5afafee609470e144]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:11.470391 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60f0eadc074a4872b6638314fed98e0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:11.472265 769 raft_consensus.cc:2468] T 71b4b91071a54d6aa66160e7955bc915 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:11.472678 847 leader_election.cc:304] T 71b4b91071a54d6aa66160e7955bc915 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5
I20251029 02:52:11.473008 2214 raft_consensus.cc:2804] T 71b4b91071a54d6aa66160e7955bc915 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:11.473124 2214 raft_consensus.cc:697] T 71b4b91071a54d6aa66160e7955bc915 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:52:11.473384 2214 consensus_queue.cc:237] T 71b4b91071a54d6aa66160e7955bc915 P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:11.483227 1037 tablet_service.cc:1505] Processing CreateTablet for tablet 5b44578ba7cc480199492195aca27112 (DEFAULT_TABLE table=default.Table_715191bdf4ad4b2fbf349552331f0f1e [id=359072c1e7f54a3499d6a5d98fceaf77]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:11.483654 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b44578ba7cc480199492195aca27112. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:11.487445 2205 leader_election.cc:290] T 71b4b91071a54d6aa66160e7955bc915 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:11.494040 522 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_5e8222bfdcd948849865e2ce542ea6a5" } modify_external_catalogs: true
I20251029 02:52:11.494364 522 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_5e8222bfdcd948849865e2ce542ea6a5" } modify_external_catalogs: true
I20251029 02:52:11.498907 1167 tablet_bootstrap.cc:492] T 82289e0f823c4d43bfd8c4e878572b2b P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:11.500325 1167 tablet_bootstrap.cc:654] T 82289e0f823c4d43bfd8c4e878572b2b P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:11.500691 756 tablet_service.cc:1505] Processing CreateTablet for tablet 99f81f12002141e1a595a08cbc220743 (DEFAULT_TABLE table=default.Table_3db438617ecf44dcb0dbdca8f8b0b1fe [id=c623013a2a0d4c769264716bf77f9b42]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:11.500885 1037 tablet_service.cc:1505] Processing CreateTablet for tablet e426b37e4f304fe28e6136f32f1236ad (DEFAULT_TABLE table=default.Table_345b52c4609e43c78920c2f13d1e7b3f [id=4251ac20e9ab4f87bd5a8b88269ac78f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:11.501258 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e426b37e4f304fe28e6136f32f1236ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:11.501747 1172 ts_tablet_manager.cc:1434] T 32519fbffcef44a890161e5f3043806c P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.100s user 0.002s sys 0.000s
I20251029 02:52:11.502882 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89750e7fa7414914ad9f0f0d146544e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:11.506404 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89750e7fa7414914ad9f0f0d146544e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:11.508828 1041 tablet_service.cc:1552] Processing DeleteTablet for tablet 12432c84ca104b0e99926a247318906a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:10 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:11.509152 1041 tablet_service.cc:1552] Processing DeleteTablet for tablet e74eda48dca647efacabe525ef316944 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:10 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:11.509579 1041 tablet_service.cc:1505] Processing CreateTablet for tablet e08101541bbf4de78f7ca92b41d0c875 (DEFAULT_TABLE table=default.Table_8d301ab34d9e48b69964f88339f88120 [id=f542f284ce8d46fcab2fc719fc45de78]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:11.509938 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e08101541bbf4de78f7ca92b41d0c875. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:11.510694 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99f81f12002141e1a595a08cbc220743. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:11.511217 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 823) took 3542 ms. Trace:
I20251029 02:52:11.511302 1174 rpcz_store.cc:276] 1029 02:52:07.968373 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:07.968491 (+ 118us) service_pool.cc:225] Handling call
1029 02:52:11.195790 (+3227299us) ts_tablet_manager.cc:1151] Deleting tablet da2a21c3236a4bd99872b89b12a36508
1029 02:52:11.195800 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:11.320389 (+124589us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:11.333540 (+ 13151us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:11.511192 (+177652us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3227162}
I20251029 02:52:11.511597 1174 tablet_replica.cc:333] T 41a6139fd9b84707a8c7eebc0381e383 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:11.511822 1174 raft_consensus.cc:2243] T 41a6139fd9b84707a8c7eebc0381e383 P c756528642a543eeacd93cef3e335a7d [term 2 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:11.511957 1174 pending_rounds.cc:70] T 41a6139fd9b84707a8c7eebc0381e383 P c756528642a543eeacd93cef3e335a7d: Trying to abort 1 pending ops.
I20251029 02:52:11.512001 1174 pending_rounds.cc:77] T 41a6139fd9b84707a8c7eebc0381e383 P c756528642a543eeacd93cef3e335a7d: Aborting op as it isn't in flight: id { term: 2 index: 3 } timestamp: 7215949129482465280 op_type: NO_OP noop_request { }
I20251029 02:52:11.512086 1174 raft_consensus.cc:2889] T 41a6139fd9b84707a8c7eebc0381e383 P c756528642a543eeacd93cef3e335a7d [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20251029 02:52:11.512163 1174 raft_consensus.cc:2272] T 41a6139fd9b84707a8c7eebc0381e383 P c756528642a543eeacd93cef3e335a7d [term 2 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:11.514791 2201 raft_consensus.cc:493] T 061f3d38c4604fa4b57e74ff6f6fc7da P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:11.514915 2201 raft_consensus.cc:515] T 061f3d38c4604fa4b57e74ff6f6fc7da P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:11.515563 2201 leader_election.cc:290] T 061f3d38c4604fa4b57e74ff6f6fc7da P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:11.516537 519 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_ced93e731339479fa9ca51ef65dea8a6" } new_table_name: "default.Table_ae6101d366dc45db81fc1af337fb9d31" modify_external_catalogs: true
I20251029 02:52:11.517357 1174 ts_tablet_manager.cc:1916] T 41a6139fd9b84707a8c7eebc0381e383 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:11.518370 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_697672643d0d4986badc08e39aa0bb14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:11.518977 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_697672643d0d4986badc08e39aa0bb14 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:11.523797 1053 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "061f3d38c4604fa4b57e74ff6f6fc7da" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:11.524108 1053 raft_consensus.cc:2393] T 061f3d38c4604fa4b57e74ff6f6fc7da P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c756528642a543eeacd93cef3e335a7d in current term 1: Already voted for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in this term.
I20251029 02:52:11.524753 769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "061f3d38c4604fa4b57e74ff6f6fc7da" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:11.525239 769 raft_consensus.cc:2393] T 061f3d38c4604fa4b57e74ff6f6fc7da P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c756528642a543eeacd93cef3e335a7d in current term 1: Already voted for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in this term.
I20251029 02:52:11.525648 847 leader_election.cc:304] T 061f3d38c4604fa4b57e74ff6f6fc7da P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9
I20251029 02:52:11.526017 1176 ts_tablet_manager.cc:1929] T d61beda662d844cdaf67fd9a2e382ea2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:11.526217 2205 raft_consensus.cc:3060] T 061f3d38c4604fa4b57e74ff6f6fc7da P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:11.529222 2205 raft_consensus.cc:2749] T 061f3d38c4604fa4b57e74ff6f6fc7da P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:11.532496 740 tablet_service.cc:1505] Processing CreateTablet for tablet 81f9503e1a344602bedffd9b451451cf (DEFAULT_TABLE table=default.Table_3fd1224741e74537ac50e1e7ee2f9719 [id=f0ea7ef1dc5d42f5ab9b1e1047457716]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:11.532985 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81f9503e1a344602bedffd9b451451cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:11.533259 756 tablet_service.cc:1505] Processing CreateTablet for tablet 9280503662894544a1f227a62a9b8a01 (DEFAULT_TABLE table=default.Table_a1ba26e950644781ba8d825e5472d758 [id=82aacc471bc241c58b676fe6a1568a78]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:11.533624 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9280503662894544a1f227a62a9b8a01. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:11.536463 1174 ts_tablet_manager.cc:1929] T 41a6139fd9b84707a8c7eebc0381e383 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.3
I20251029 02:52:11.536845 1174 log.cc:1199] T 41a6139fd9b84707a8c7eebc0381e383 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/41a6139fd9b84707a8c7eebc0381e383
I20251029 02:52:11.537549 1174 ts_tablet_manager.cc:1950] T 41a6139fd9b84707a8c7eebc0381e383 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:11.538640 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 824) took 3531 ms. Trace:
W20251029 02:52:11.542713 512 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.60:41233, user_credentials={real_user=slave}} blocked reactor thread for 44869.2us
I20251029 02:52:11.543123 1174 rpcz_store.cc:276] 1029 02:52:08.007127 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:08.007186 (+ 59us) service_pool.cc:225] Handling call
1029 02:52:11.511554 (+3504368us) ts_tablet_manager.cc:1151] Deleting tablet 41a6139fd9b84707a8c7eebc0381e383
1029 02:52:11.511562 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:11.520488 (+ 8926us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:11.536355 (+ 15867us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:11.538617 (+ 2262us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3504139}
I20251029 02:52:11.543409 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet da2a21c3236a4bd99872b89b12a36508 (table default.Table_2a63af6894694f16b45f4bf48f73a68f [id=fb42af08ec1f48498ee5e91cce62333e]) successfully deleted
I20251029 02:52:11.543464 1174 tablet_replica.cc:333] T 1bc45078a6a942c38ff3350c15885eff P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:11.543643 1174 raft_consensus.cc:2243] T 1bc45078a6a942c38ff3350c15885eff P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:11.543798 1174 raft_consensus.cc:2272] T 1bc45078a6a942c38ff3350c15885eff P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:11.544302 1174 ts_tablet_manager.cc:1916] T 1bc45078a6a942c38ff3350c15885eff P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:11.545261 885 tablet_service.cc:1505] Processing CreateTablet for tablet 81f9503e1a344602bedffd9b451451cf (DEFAULT_TABLE table=default.Table_3fd1224741e74537ac50e1e7ee2f9719 [id=f0ea7ef1dc5d42f5ab9b1e1047457716]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:11.545503 894 tablet_service.cc:1505] Processing CreateTablet for tablet 9280503662894544a1f227a62a9b8a01 (DEFAULT_TABLE table=default.Table_a1ba26e950644781ba8d825e5472d758 [id=82aacc471bc241c58b676fe6a1568a78]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:11.545680 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81f9503e1a344602bedffd9b451451cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:11.546109 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9280503662894544a1f227a62a9b8a01. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:11.548820 739 tablet_service.cc:1505] Processing CreateTablet for tablet ffe664eab28c4aa88ac299c5b5e66410 (DEFAULT_TABLE table=default.Table_75f0822a7de44b8ea84235f6790a2140 [id=67ad707678dc4b98b380fdc8a9699b80]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:11.549212 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ffe664eab28c4aa88ac299c5b5e66410. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:11.551920 745 tablet_service.cc:1505] Processing CreateTablet for tablet d557a4750fee4c2b92eaa834619a678c (DEFAULT_TABLE table=default.Table_4873cb28e4c84935ab481f06ad654716 [id=9bc3968738ca426f8ac8dec3b513b085]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:11.552292 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d557a4750fee4c2b92eaa834619a678c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:11.554786 738 tablet_service.cc:1505] Processing CreateTablet for tablet 209671f4c6284613a2d77b17688c2263 (DEFAULT_TABLE table=default.Table_f0423becd1a243ff9b81ce1b985f083e [id=aedc704e0d604bf4a113882bd22a4317]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:11.555117 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 209671f4c6284613a2d77b17688c2263. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:11.556738 1167 tablet_bootstrap.cc:492] T 82289e0f823c4d43bfd8c4e878572b2b P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:11.556898 1167 ts_tablet_manager.cc:1403] T 82289e0f823c4d43bfd8c4e878572b2b P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.058s user 0.003s sys 0.006s
I20251029 02:52:11.557600 1167 raft_consensus.cc:359] T 82289e0f823c4d43bfd8c4e878572b2b P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:11.557754 1167 raft_consensus.cc:385] T 82289e0f823c4d43bfd8c4e878572b2b P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:11.557812 1167 raft_consensus.cc:740] T 82289e0f823c4d43bfd8c4e878572b2b P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:11.558053 1167 consensus_queue.cc:260] T 82289e0f823c4d43bfd8c4e878572b2b P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:11.526712 1041 tablet_service.cc:1505] Processing CreateTablet for tablet 2c4e78fbd7e645eb8b3be8ac4bfbb425 (DEFAULT_TABLE table=default.Table_17c5072bc19c40b0ac7348fdf90a1e8d [id=cb0b7302d94749adb36f5c10ff5b3562]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:11.558558 1167 ts_tablet_manager.cc:1434] T 82289e0f823c4d43bfd8c4e878572b2b P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 02:52:11.558832 1041 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c4e78fbd7e645eb8b3be8ac4bfbb425. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:11.558842 1167 tablet_bootstrap.cc:492] T 44ae2b07edca44afbe1b1fb3a5b1115d P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:11.558845 747 tablet_service.cc:1505] Processing CreateTablet for tablet 404ae4213500460386d1edd6e66f04c3 (DEFAULT_TABLE table=default.Table_5db0b0e77f584976b6253ec87f386bb8 [id=59ae93647c29498ca382f850a9d42ac3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:11.559170 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 404ae4213500460386d1edd6e66f04c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:11.560452 1167 tablet_bootstrap.cc:654] T 44ae2b07edca44afbe1b1fb3a5b1115d P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:11.527232 1027 tablet_service.cc:1505] Processing CreateTablet for tablet 921074e69f9d49a3bdad3d2791ab037a (DEFAULT_TABLE table=default.Table_110cf0fc2d22469d93e57359178b6c2a [id=88c3c42a3ece45f98490fd86ea2d7314]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:11.562811 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 921074e69f9d49a3bdad3d2791ab037a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:11.562763 751 tablet_service.cc:1505] Processing CreateTablet for tablet 6408901d1e5f4ac8a7bff0de708660eb (DEFAULT_TABLE table=default.Table_7fdd50180d294d44bc4c64f2cc5dc57a [id=b7cab31fff3e439d980f98446e240547]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:11.563091 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6408901d1e5f4ac8a7bff0de708660eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:11.566596 742 tablet_service.cc:1505] Processing CreateTablet for tablet b11bd7c31cbb4b5cb79b4257399b46e8 (DEFAULT_TABLE table=default.Table_7e276e1364a8470a853a68d2d977a4b4 [id=44cc9872095b4aa29f052c012a4f11f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:11.566931 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b11bd7c31cbb4b5cb79b4257399b46e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:11.527632 1030 tablet_service.cc:1505] Processing CreateTablet for tablet af74d277abb149e09900c46baf2ea1d3 (DEFAULT_TABLE table=default.Table_0f4514215d3042c5a46ac59072d2bbb2 [id=19d51187854b458ebf8af447277ea1cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:11.567822 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af74d277abb149e09900c46baf2ea1d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:11.527921 1042 tablet_service.cc:1552] Processing DeleteTablet for tablet 217144957930495aa3e9dfcfde268e71 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:01 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:11.570922 1042 tablet_service.cc:1505] Processing CreateTablet for tablet 854bb36d1d3340a9bab2ad2eb4ee8cb5 (DEFAULT_TABLE table=default.Table_7fd7b56a9c09475a80cce932d6e15cc1 [id=d5d80eda41084397972a6b36cfff433b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:11.571303 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 854bb36d1d3340a9bab2ad2eb4ee8cb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:11.571432 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 41a6139fd9b84707a8c7eebc0381e383 (table default.Table_e48aadc66a0e48658d45d0812c29cbdf [id=f45315b686594867aae0114f4ce2f26c]) successfully deleted
I20251029 02:52:11.571633 879 tablet_service.cc:1505] Processing CreateTablet for tablet ffe664eab28c4aa88ac299c5b5e66410 (DEFAULT_TABLE table=default.Table_75f0822a7de44b8ea84235f6790a2140 [id=67ad707678dc4b98b380fdc8a9699b80]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:11.572007 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ffe664eab28c4aa88ac299c5b5e66410. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:11.571964 894 tablet_service.cc:1505] Processing CreateTablet for tablet 209671f4c6284613a2d77b17688c2263 (DEFAULT_TABLE table=default.Table_f0423becd1a243ff9b81ce1b985f083e [id=aedc704e0d604bf4a113882bd22a4317]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:11.572304 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 209671f4c6284613a2d77b17688c2263. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:11.572494 895 tablet_service.cc:1505] Processing CreateTablet for tablet 404ae4213500460386d1edd6e66f04c3 (DEFAULT_TABLE table=default.Table_5db0b0e77f584976b6253ec87f386bb8 [id=59ae93647c29498ca382f850a9d42ac3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:11.572790 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 404ae4213500460386d1edd6e66f04c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:11.528158 1033 tablet_service.cc:1552] Processing DeleteTablet for tablet 97b2ecae572a4b7ea1b9595074556708 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:00 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:11.574707 1033 tablet_service.cc:1552] Processing DeleteTablet for tablet 28f7ceb91a34450cad4b5b45f5285b41 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:59 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:11.574884 1033 tablet_service.cc:1552] Processing DeleteTablet for tablet 6d830d30b26b4ba799785fec8a6f0769 with delete_type TABLET_DATA_DELETED (replaced by tablet abf18e06df9b482a8b25c078b9495272) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:11.575019 1033 tablet_service.cc:1552] Processing DeleteTablet for tablet b6d4f7bf702f4834bc5f4ef2b50b92a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:55 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:11.575309 1033 tablet_service.cc:1505] Processing CreateTablet for tablet 7c6b9031ad71444d95df0c33135636b3 (DEFAULT_TABLE table=default.Table_a3886404687640698bd3302e39ee6854 [id=7390a7db2be349b09df554aa92d4068c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:11.575613 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c6b9031ad71444d95df0c33135636b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:11.576336 900 tablet_service.cc:1505] Processing CreateTablet for tablet d557a4750fee4c2b92eaa834619a678c (DEFAULT_TABLE table=default.Table_4873cb28e4c84935ab481f06ad654716 [id=9bc3968738ca426f8ac8dec3b513b085]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:11.576623 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d557a4750fee4c2b92eaa834619a678c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:11.528383 1031 tablet_service.cc:1552] Processing DeleteTablet for tablet e7f846b294cc4eb687a26c8b5931ac19 with delete_type TABLET_DATA_DELETED (replaced by tablet cdc5fb2b13e042d680fa3f2eea2d62fa) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:11.579196 1031 tablet_service.cc:1505] Processing CreateTablet for tablet 456b623500fd4d758e40004666b095a9 (DEFAULT_TABLE table=default.Table_443f33587b8f428484b99e07355d69ca [id=03a25420ce6a46fc93a9fb753246bc7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:11.579497 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 456b623500fd4d758e40004666b095a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:11.581547 881 tablet_service.cc:1505] Processing CreateTablet for tablet 6408901d1e5f4ac8a7bff0de708660eb (DEFAULT_TABLE table=default.Table_7fdd50180d294d44bc4c64f2cc5dc57a [id=b7cab31fff3e439d980f98446e240547]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:11.581836 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6408901d1e5f4ac8a7bff0de708660eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:11.528577 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet 061f3d38c4604fa4b57e74ff6f6fc7da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:58 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:11.583767 1035 tablet_service.cc:1505] Processing CreateTablet for tablet 2849e856dd714da88fca5a9fafe74c22 (DEFAULT_TABLE table=default.Table_fa832508491b47ddbcd9e89058c75ad5 [id=958e0005a6bc408188e53fc33d6509dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:11.584117 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2849e856dd714da88fca5a9fafe74c22. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:11.586602 1174 ts_tablet_manager.cc:1929] T 1bc45078a6a942c38ff3350c15885eff P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:11.586704 1174 log.cc:1199] T 1bc45078a6a942c38ff3350c15885eff P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/1bc45078a6a942c38ff3350c15885eff
I20251029 02:52:11.528821 1024 tablet_service.cc:1552] Processing DeleteTablet for tablet 30d03e1e42d34242bd354f9e9ec4f127 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:58 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:11.587333 896 tablet_service.cc:1505] Processing CreateTablet for tablet b11bd7c31cbb4b5cb79b4257399b46e8 (DEFAULT_TABLE table=default.Table_7e276e1364a8470a853a68d2d977a4b4 [id=44cc9872095b4aa29f052c012a4f11f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:11.587509 1024 tablet_service.cc:1552] Processing DeleteTablet for tablet 4b841723726e40bda0187fb84bf36a6a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:10 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:11.587678 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b11bd7c31cbb4b5cb79b4257399b46e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:11.587843 1024 tablet_service.cc:1505] Processing CreateTablet for tablet 7fcb1aac63c84edba0a8b8c964a8b957 (DEFAULT_TABLE table=default.Table_5ed09727711e4eb6a60deb327d188aea [id=d256ddd14762490f8fbfe401d9802d4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:11.588155 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7fcb1aac63c84edba0a8b8c964a8b957. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:11.588191 1167 tablet_bootstrap.cc:492] T 44ae2b07edca44afbe1b1fb3a5b1115d P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:11.588302 1167 ts_tablet_manager.cc:1403] T 44ae2b07edca44afbe1b1fb3a5b1115d P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.030s user 0.002s sys 0.000s
I20251029 02:52:11.589100 1167 raft_consensus.cc:359] T 44ae2b07edca44afbe1b1fb3a5b1115d P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:11.589249 1167 raft_consensus.cc:385] T 44ae2b07edca44afbe1b1fb3a5b1115d P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:11.589291 1167 raft_consensus.cc:740] T 44ae2b07edca44afbe1b1fb3a5b1115d P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:11.589445 1167 consensus_queue.cc:260] T 44ae2b07edca44afbe1b1fb3a5b1115d P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:11.590008 1167 ts_tablet_manager.cc:1434] T 44ae2b07edca44afbe1b1fb3a5b1115d P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 02:52:11.590245 1167 tablet_bootstrap.cc:492] T 217144957930495aa3e9dfcfde268e71 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:11.529090 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet 2e97d1bc06a24e8d8ba11acb6fd280d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:55 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:11.590881 1029 tablet_service.cc:1505] Processing CreateTablet for tablet 2cea2f2fe68545db804a732d0c01a876 (DEFAULT_TABLE table=default.Table_149358409d284922ad0f2c49410dca96 [id=195d147c89f240bea546a83693a6c7dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:11.591203 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2cea2f2fe68545db804a732d0c01a876. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:11.529211 1039 tablet_service.cc:1552] Processing DeleteTablet for tablet 8048506ced4540a5aee9f79bfc4599bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:06 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:11.595134 1039 tablet_service.cc:1505] Processing CreateTablet for tablet fcc2d413e3b54b2da9b5b44f8b859816 (DEFAULT_TABLE table=default.Table_9791da4fa0ac47409f6cb424408a5f3b [id=60ef33b133314f1d9007a46fbf3a5338]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:11.595510 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fcc2d413e3b54b2da9b5b44f8b859816. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:11.596825 1174 ts_tablet_manager.cc:1950] T 1bc45078a6a942c38ff3350c15885eff P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:11.597945 1167 tablet_bootstrap.cc:654] T 217144957930495aa3e9dfcfde268e71 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:11.598274 878 tablet_service.cc:1505] Processing CreateTablet for tablet 99f81f12002141e1a595a08cbc220743 (DEFAULT_TABLE table=default.Table_3db438617ecf44dcb0dbdca8f8b0b1fe [id=c623013a2a0d4c769264716bf77f9b42]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:11.598748 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99f81f12002141e1a595a08cbc220743. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:11.530062 1166 tablet_bootstrap.cc:492] T afb19ba39399410bb90b6225d7b5b5b1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:11.601858 1166 ts_tablet_manager.cc:1403] T afb19ba39399410bb90b6225d7b5b5b1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.215s user 0.003s sys 0.001s
I20251029 02:52:11.601966 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 825) took 3578 ms. Trace:
I20251029 02:52:11.602058 1174 rpcz_store.cc:276] 1029 02:52:08.023037 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:08.023100 (+ 63us) service_pool.cc:225] Handling call
1029 02:52:11.543414 (+3520314us) ts_tablet_manager.cc:1151] Deleting tablet 1bc45078a6a942c38ff3350c15885eff
1029 02:52:11.543425 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:11.553872 (+ 10447us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:11.586494 (+ 32622us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:11.601945 (+ 15451us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3520127}
I20251029 02:52:11.602377 1174 tablet_replica.cc:333] T 785733da86ff425a90ccfab2933fc89e P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:11.543205 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet 738046ab57334f2dab9f4aab979c5b42 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:04 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:11.543766 1032 tablet_service.cc:1552] Processing DeleteTablet for tablet 13ee62fc2339410b9db70dfd3d3c6fa6 with delete_type TABLET_DATA_DELETED (replaced by tablet 35d9b8dfd61e4d8db441427a49fb9dca) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:11.543895 1034 tablet_service.cc:1552] Processing DeleteTablet for tablet ba699862e6f646c6beaa39f080d0faba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:04 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:11.602594 1174 raft_consensus.cc:2243] T 785733da86ff425a90ccfab2933fc89e P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:11.602710 1034 tablet_service.cc:1552] Processing DeleteTablet for tablet 509288c033d54e4b9da3f26a30dc46a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:11 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:11.602855 1034 tablet_service.cc:1552] Processing DeleteTablet for tablet c172417d1d444c439e9cda611a2ed551 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:11 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:11.602954 1174 raft_consensus.cc:2272] T 785733da86ff425a90ccfab2933fc89e P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:11.602968 1034 tablet_service.cc:1552] Processing DeleteTablet for tablet c7a3e6d8e3fb45898d8a5e094dbf0fd7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:11 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:11.602912 1036 tablet_service.cc:1505] Processing CreateTablet for tablet 44ef8aa6597a49d6bd27ece880589522 (DEFAULT_TABLE table=default.Table_af97a1e008a94a86970089216d02b3d4 [id=cc32193d11ed47aeb9f70ce8151dff1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:11.603082 1034 tablet_service.cc:1552] Processing DeleteTablet for tablet 7dbc9c8b110a48f0a801ad468738c3a3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:11 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:11.603207 1034 tablet_service.cc:1552] Processing DeleteTablet for tablet 96f7876e5d6d4a0db236df5e7f266fd2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:11 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:11.603288 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44ef8aa6597a49d6bd27ece880589522. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:11.603315 1034 tablet_service.cc:1552] Processing DeleteTablet for tablet ecf3b900206b4c86a7adbc83a769d90e with delete_type TABLET_DATA_DELETED (replaced by tablet 2e96a398bfd74cf68cf0377a813dc1dd) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:11.603564 1174 ts_tablet_manager.cc:1916] T 785733da86ff425a90ccfab2933fc89e P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:11.603566 1034 tablet_service.cc:1505] Processing CreateTablet for tablet 89750e7fa7414914ad9f0f0d146544e1 (DEFAULT_TABLE table=default.Table_fb178c25d3c1496f8ed5db2fe26d9fb1 [id=3edaa0265abc45f18102db2e21c8c302]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:11.603891 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89750e7fa7414914ad9f0f0d146544e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:11.543949 1026 tablet_service.cc:1552] Processing DeleteTablet for tablet dabb8d760aae4aa09eb83cdb586c9788 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:07 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:11.544003 1040 tablet_service.cc:1552] Processing DeleteTablet for tablet 86fc22bb84794149b598b97df41eb881 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:58 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:11.544176 1023 tablet_service.cc:1552] Processing DeleteTablet for tablet c32c28b0e74a4af3a0e69bbf24ab1a1f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:57 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:11.544413 1038 tablet_service.cc:1552] Processing DeleteTablet for tablet 1275e192871440408a8345fef9b56260 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:56 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:11.607008 1040 tablet_service.cc:1505] Processing CreateTablet for tablet 99f81f12002141e1a595a08cbc220743 (DEFAULT_TABLE table=default.Table_3db438617ecf44dcb0dbdca8f8b0b1fe [id=c623013a2a0d4c769264716bf77f9b42]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:11.607434 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99f81f12002141e1a595a08cbc220743. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:11.607378 1038 tablet_service.cc:1505] Processing CreateTablet for tablet 9280503662894544a1f227a62a9b8a01 (DEFAULT_TABLE table=default.Table_a1ba26e950644781ba8d825e5472d758 [id=82aacc471bc241c58b676fe6a1568a78]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:11.607666 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9280503662894544a1f227a62a9b8a01. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:11.607008 1023 tablet_service.cc:1505] Processing CreateTablet for tablet 81f9503e1a344602bedffd9b451451cf (DEFAULT_TABLE table=default.Table_3fd1224741e74537ac50e1e7ee2f9719 [id=f0ea7ef1dc5d42f5ab9b1e1047457716]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:11.611349 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81f9503e1a344602bedffd9b451451cf. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:11.612282 847 consensus_peers.cc:597] T b6d4f7bf702f4834bc5f4ef2b50b92a7 P c756528642a543eeacd93cef3e335a7d -> Peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Couldn't send request to peer b495a49349324f7d81908f2da700f0d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b6d4f7bf702f4834bc5f4ef2b50b92a7. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:11.612497 1176 log.cc:1199] T d61beda662d844cdaf67fd9a2e382ea2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/d61beda662d844cdaf67fd9a2e382ea2
I20251029 02:52:11.612833 1176 ts_tablet_manager.cc:1950] T d61beda662d844cdaf67fd9a2e382ea2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:11.615314 1167 tablet_bootstrap.cc:492] T 217144957930495aa3e9dfcfde268e71 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:11.615471 1167 ts_tablet_manager.cc:1403] T 217144957930495aa3e9dfcfde268e71 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.025s user 0.001s sys 0.001s
I20251029 02:52:11.616514 1167 raft_consensus.cc:359] T 217144957930495aa3e9dfcfde268e71 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:11.616695 1167 raft_consensus.cc:385] T 217144957930495aa3e9dfcfde268e71 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:11.616747 1167 raft_consensus.cc:740] T 217144957930495aa3e9dfcfde268e71 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:11.616972 1172 tablet_bootstrap.cc:492] T 46cd75ff18b14212b556b9bcb3012dd9 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:11.617033 1167 consensus_queue.cc:260] T 217144957930495aa3e9dfcfde268e71 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:11.617631 1167 ts_tablet_manager.cc:1434] T 217144957930495aa3e9dfcfde268e71 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 02:52:11.618170 1167 tablet_bootstrap.cc:492] T d7abaf84ba5d445aa17433b3be891548 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:11.619813 1172 tablet_bootstrap.cc:654] T 46cd75ff18b14212b556b9bcb3012dd9 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:11.620116 1167 tablet_bootstrap.cc:654] T d7abaf84ba5d445aa17433b3be891548 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:11.620997 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 1bc45078a6a942c38ff3350c15885eff (table default.Table_3126aa57075d44dc8b73f08fe21aabdf [id=e8fa170a8a044f2fa1236949a91454de]) successfully deleted
I20251029 02:52:11.621534 1166 raft_consensus.cc:359] T afb19ba39399410bb90b6225d7b5b5b1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:11.621707 1166 raft_consensus.cc:385] T afb19ba39399410bb90b6225d7b5b5b1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:11.621754 1166 raft_consensus.cc:740] T afb19ba39399410bb90b6225d7b5b5b1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:11.621934 1166 consensus_queue.cc:260] T afb19ba39399410bb90b6225d7b5b5b1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:11.625614 1166 ts_tablet_manager.cc:1434] T afb19ba39399410bb90b6225d7b5b5b1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.024s user 0.002s sys 0.000s
I20251029 02:52:11.626880 1166 tablet_bootstrap.cc:492] T e65eaaae43ac435ebd665546de9d68dd P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:11.630376 1174 ts_tablet_manager.cc:1929] T 785733da86ff425a90ccfab2933fc89e P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:11.631713 1166 tablet_bootstrap.cc:654] T e65eaaae43ac435ebd665546de9d68dd P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:11.631407 1167 tablet_bootstrap.cc:492] T d7abaf84ba5d445aa17433b3be891548 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:11.637506 1167 ts_tablet_manager.cc:1403] T d7abaf84ba5d445aa17433b3be891548 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.000s
I20251029 02:52:11.638204 1167 raft_consensus.cc:359] T d7abaf84ba5d445aa17433b3be891548 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:11.638373 1167 raft_consensus.cc:385] T d7abaf84ba5d445aa17433b3be891548 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:11.638420 1167 raft_consensus.cc:740] T d7abaf84ba5d445aa17433b3be891548 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:11.638677 1167 consensus_queue.cc:260] T d7abaf84ba5d445aa17433b3be891548 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:11.639154 1167 ts_tablet_manager.cc:1434] T d7abaf84ba5d445aa17433b3be891548 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251029 02:52:11.639437 1167 tablet_bootstrap.cc:492] T adf6e736cb9c4920807ee58682e134ef P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:11.640914 1167 tablet_bootstrap.cc:654] T adf6e736cb9c4920807ee58682e134ef P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:11.641515 1174 log.cc:1199] T 785733da86ff425a90ccfab2933fc89e P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/785733da86ff425a90ccfab2933fc89e
I20251029 02:52:11.641964 1174 ts_tablet_manager.cc:1950] T 785733da86ff425a90ccfab2933fc89e P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:11.642792 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 826) took 3465 ms. Trace:
W20251029 02:52:11.642843 712 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:45003, user_credentials={real_user=slave}} blocked reactor thread for 224598us
W20251029 02:52:11.643541 712 leader_election.cc:343] T 44ae2b07edca44afbe1b1fb3a5b1115d P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:11.644014 712 leader_election.cc:304] T 44ae2b07edca44afbe1b1fb3a5b1115d P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:11.644963 712 leader_election.cc:304] T 6d830d30b26b4ba799785fec8a6f0769 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:11.644949 1174 rpcz_store.cc:276] 1029 02:52:08.177667 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:08.177745 (+ 78us) service_pool.cc:225] Handling call
1029 02:52:11.602330 (+3424585us) ts_tablet_manager.cc:1151] Deleting tablet 785733da86ff425a90ccfab2933fc89e
1029 02:52:11.602339 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:11.622767 (+ 20428us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:11.630258 (+ 7491us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:11.642764 (+ 12506us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3424355}
I20251029 02:52:11.645418 2309 raft_consensus.cc:2749] T 44ae2b07edca44afbe1b1fb3a5b1115d P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:11.645511 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 828) took 3373 ms. Trace:
I20251029 02:52:11.645586 1174 rpcz_store.cc:276] 1029 02:52:08.272145 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:08.272194 (+ 49us) service_pool.cc:225] Handling call
1029 02:52:11.645397 (+3373203us) ts_tablet_manager.cc:1151] Deleting tablet 4372eacfc07246f3be8f7c146674a736
1029 02:52:11.645407 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:11.645496 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3373019}
I20251029 02:52:11.645704 2309 raft_consensus.cc:2749] T 6d830d30b26b4ba799785fec8a6f0769 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251029 02:52:11.645999 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 827) took 3374 ms. Trace:
I20251029 02:52:11.646112 1174 rpcz_store.cc:276] 1029 02:52:08.272031 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:08.272268 (+ 237us) service_pool.cc:225] Handling call
1029 02:52:11.645913 (+3373645us) ts_tablet_manager.cc:1151] Deleting tablet 0ffd22d8610b4329a0a4f753d9e60149
1029 02:52:11.645921 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:11.645986 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3373463}
I20251029 02:52:11.650017 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 830) took 3370 ms. Trace:
I20251029 02:52:11.650485 2309 raft_consensus.cc:493] T 27b08f88b99643d28ac609e45bf8104e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:11.650808 1174 rpcz_store.cc:276] 1029 02:52:08.279367 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:08.279748 (+ 381us) service_pool.cc:225] Handling call
1029 02:52:11.649917 (+3370169us) ts_tablet_manager.cc:1151] Deleting tablet 20a762344c5f46f88481c4ac7ad6e43b
1029 02:52:11.649926 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:11.650002 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3369974}
I20251029 02:52:11.651243 2309 raft_consensus.cc:515] T 27b08f88b99643d28ac609e45bf8104e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:11.651494 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 829) took 3372 ms. Trace:
I20251029 02:52:11.651575 1174 rpcz_store.cc:276] 1029 02:52:08.279205 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:08.280029 (+ 824us) service_pool.cc:225] Handling call
1029 02:52:11.651388 (+3371359us) ts_tablet_manager.cc:1151] Deleting tablet 50ab70b30aaa4f6bbae15855431159c8
1029 02:52:11.651397 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:11.651481 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3371223}
W20251029 02:52:11.546188 989 leader_election.cc:343] T 0d84dc8dc465496f9adec798b2e03e13 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 0d84dc8dc465496f9adec798b2e03e13
I20251029 02:52:11.651870 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 831) took 3367 ms. Trace:
I20251029 02:52:11.651927 989 leader_election.cc:304] T 0d84dc8dc465496f9adec798b2e03e13 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5; no voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:11.651933 1174 rpcz_store.cc:276] 1029 02:52:08.284294 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:08.284352 (+ 58us) service_pool.cc:225] Handling call
1029 02:52:11.651780 (+3367428us) ts_tablet_manager.cc:1151] Deleting tablet 48aa20640dfe43288326ed6d6b9bc1cf
1029 02:52:11.651788 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:11.651856 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3367248}
I20251029 02:52:11.551792 1037 tablet_service.cc:1552] Processing DeleteTablet for tablet af4860c6b72f4d988b206b5ef2f22906 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:56 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:11.553018 1025 tablet_service.cc:1552] Processing DeleteTablet for tablet 9bb2166a590d4210b02dd779ef6654ad with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:55 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:11.652206 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 832) took 3362 ms. Trace:
I20251029 02:52:11.553315 1028 tablet_service.cc:1505] Processing CreateTablet for tablet 96427e60edeb45bdab9666a5055b5a99 (DEFAULT_TABLE table=default.Table_41dc74911c3747a7a2763f31d7ca40e1 [id=6ae2337c7870453fb2f1f4cbb8b5c164]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:11.652278 1174 rpcz_store.cc:276] 1029 02:52:08.289703 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:08.289775 (+ 72us) service_pool.cc:225] Handling call
1029 02:52:11.652123 (+3362348us) ts_tablet_manager.cc:1151] Deleting tablet 65e761f9f9fb490586d6415a5a8565dd
1029 02:52:11.652131 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:11.652194 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3362179}
I20251029 02:52:11.652508 1174 tablet_replica.cc:333] T d2990e9f626c40618c118266bc595684 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:11.652671 1174 raft_consensus.cc:2243] T d2990e9f626c40618c118266bc595684 P c756528642a543eeacd93cef3e335a7d [term 2 LEADER]: Raft consensus shutting down.
I20251029 02:52:11.652665 2309 leader_election.cc:290] T 27b08f88b99643d28ac609e45bf8104e P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:11.652906 1174 pending_rounds.cc:70] T d2990e9f626c40618c118266bc595684 P c756528642a543eeacd93cef3e335a7d: Trying to abort 1 pending ops.
I20251029 02:52:11.652956 1174 pending_rounds.cc:77] T d2990e9f626c40618c118266bc595684 P c756528642a543eeacd93cef3e335a7d: Aborting op as it isn't in flight: id { term: 2 index: 3 } timestamp: 7215949133062352896 op_type: NO_OP noop_request { }
I20251029 02:52:11.653050 1174 raft_consensus.cc:2889] T d2990e9f626c40618c118266bc595684 P c756528642a543eeacd93cef3e335a7d [term 2 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 02:52:11.653131 1174 raft_consensus.cc:2272] T d2990e9f626c40618c118266bc595684 P c756528642a543eeacd93cef3e335a7d [term 2 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:11.653067 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "27b08f88b99643d28ac609e45bf8104e" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:11.653156 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96427e60edeb45bdab9666a5055b5a99. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:11.653533 711 leader_election.cc:343] T 27b08f88b99643d28ac609e45bf8104e P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:11.654616 712 leader_election.cc:304] T af4860c6b72f4d988b206b5ef2f22906 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters: c756528642a543eeacd93cef3e335a7d
I20251029 02:52:11.655014 712 leader_election.cc:304] T c32c28b0e74a4af3a0e69bbf24ab1a1f P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters: c756528642a543eeacd93cef3e335a7d
I20251029 02:52:11.655278 2113 raft_consensus.cc:2804] T af4860c6b72f4d988b206b5ef2f22906 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:11.655370 2113 raft_consensus.cc:697] T af4860c6b72f4d988b206b5ef2f22906 P b495a49349324f7d81908f2da700f0d9 [term 1 LEADER]: Becoming Leader. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Running, Role: LEADER
I20251029 02:52:11.655584 2113 consensus_queue.cc:237] T af4860c6b72f4d988b206b5ef2f22906 P b495a49349324f7d81908f2da700f0d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:11.657220 2309 raft_consensus.cc:2804] T c32c28b0e74a4af3a0e69bbf24ab1a1f P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:11.657452 2309 raft_consensus.cc:493] T c32c28b0e74a4af3a0e69bbf24ab1a1f P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:11.657634 2309 raft_consensus.cc:3060] T c32c28b0e74a4af3a0e69bbf24ab1a1f P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
W20251029 02:52:11.657861 988 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20251029 02:52:11.659464 1167 tablet_bootstrap.cc:492] T adf6e736cb9c4920807ee58682e134ef P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:11.659608 1167 ts_tablet_manager.cc:1403] T adf6e736cb9c4920807ee58682e134ef P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.000s
I20251029 02:52:11.660979 1167 raft_consensus.cc:359] T adf6e736cb9c4920807ee58682e134ef P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:11.661146 1167 raft_consensus.cc:385] T adf6e736cb9c4920807ee58682e134ef P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:11.661202 1167 raft_consensus.cc:740] T adf6e736cb9c4920807ee58682e134ef P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:11.661388 1167 consensus_queue.cc:260] T adf6e736cb9c4920807ee58682e134ef P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:11.661854 1167 ts_tablet_manager.cc:1434] T adf6e736cb9c4920807ee58682e134ef P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251029 02:52:11.662124 1167 tablet_bootstrap.cc:492] T 44c0488eba0543078ff4f0b3d69366f2 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:11.663486 2309 raft_consensus.cc:515] T c32c28b0e74a4af3a0e69bbf24ab1a1f P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:11.665148 1167 tablet_bootstrap.cc:654] T 44c0488eba0543078ff4f0b3d69366f2 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:11.665494 712 leader_election.cc:304] T 8048506ced4540a5aee9f79bfc4599bf P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters: c756528642a543eeacd93cef3e335a7d
I20251029 02:52:11.666433 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c32c28b0e74a4af3a0e69bbf24ab1a1f" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
W20251029 02:52:11.666571 712 leader_election.cc:343] T e65eaaae43ac435ebd665546de9d68dd P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:11.666626 917 raft_consensus.cc:3060] T c32c28b0e74a4af3a0e69bbf24ab1a1f P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:11.666664 712 leader_election.cc:304] T e65eaaae43ac435ebd665546de9d68dd P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:11.666935 2403 raft_consensus.cc:2804] T 8048506ced4540a5aee9f79bfc4599bf P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:11.668460 2309 leader_election.cc:290] T c32c28b0e74a4af3a0e69bbf24ab1a1f P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 election: Requested vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:11.668591 2309 raft_consensus.cc:2749] T e65eaaae43ac435ebd665546de9d68dd P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:11.669369 917 raft_consensus.cc:2468] T c32c28b0e74a4af3a0e69bbf24ab1a1f P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b495a49349324f7d81908f2da700f0d9 in term 1.
I20251029 02:52:11.669867 711 leader_election.cc:304] T c32c28b0e74a4af3a0e69bbf24ab1a1f P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:11.672490 2214 raft_consensus.cc:493] T 44ae2b07edca44afbe1b1fb3a5b1115d P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:11.672664 2214 raft_consensus.cc:515] T 44ae2b07edca44afbe1b1fb3a5b1115d P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:11.673671 1052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "44ae2b07edca44afbe1b1fb3a5b1115d" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:52:11.674213 849 leader_election.cc:343] T 44ae2b07edca44afbe1b1fb3a5b1115d P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:11.674851 2214 leader_election.cc:290] T 44ae2b07edca44afbe1b1fb3a5b1115d P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:11.674924 769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "44ae2b07edca44afbe1b1fb3a5b1115d" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:11.675114 769 raft_consensus.cc:2468] T 44ae2b07edca44afbe1b1fb3a5b1115d P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:11.675581 847 leader_election.cc:304] T 44ae2b07edca44afbe1b1fb3a5b1115d P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5
I20251029 02:52:11.675307 2309 raft_consensus.cc:2804] T c32c28b0e74a4af3a0e69bbf24ab1a1f P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:11.675801 2214 raft_consensus.cc:2804] T 44ae2b07edca44afbe1b1fb3a5b1115d P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:11.675849 2309 raft_consensus.cc:697] T c32c28b0e74a4af3a0e69bbf24ab1a1f P b495a49349324f7d81908f2da700f0d9 [term 1 LEADER]: Becoming Leader. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Running, Role: LEADER
I20251029 02:52:11.675864 2214 raft_consensus.cc:493] T 44ae2b07edca44afbe1b1fb3a5b1115d P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:11.675932 2214 raft_consensus.cc:3060] T 44ae2b07edca44afbe1b1fb3a5b1115d P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:11.676081 2309 consensus_queue.cc:237] T c32c28b0e74a4af3a0e69bbf24ab1a1f P b495a49349324f7d81908f2da700f0d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:11.678926 2214 raft_consensus.cc:515] T 44ae2b07edca44afbe1b1fb3a5b1115d P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:11.679899 1052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "44ae2b07edca44afbe1b1fb3a5b1115d" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
W20251029 02:52:11.680114 512 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.60:41233, user_credentials={real_user=slave}} blocked reactor thread for 58859.5us
W20251029 02:52:11.680488 849 leader_election.cc:343] T 44ae2b07edca44afbe1b1fb3a5b1115d P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:11.680931 896 tablet_service.cc:1505] Processing CreateTablet for tablet f58bd72cee5b46f5941a3976547e2aaa (DEFAULT_TABLE table=default.Table_f08f8ab20ebf4c9a99d5b88cf17d3b7d [id=bf42ef67f58e45c58b290c7dbb1ca205]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:11.681013 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 785733da86ff425a90ccfab2933fc89e (table default.Table_bdefc7775234406788f3f7e220769b43 [id=b81476e2d6db40ef9f97211bdb79ce21]) successfully deleted
W20251029 02:52:11.681241 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 4372eacfc07246f3be8f7c146674a736 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4372eacfc07246f3be8f7c146674a736 already in progress: creating tablet
I20251029 02:52:11.681319 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f58bd72cee5b46f5941a3976547e2aaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:11.681299 2214 leader_election.cc:290] T 44ae2b07edca44afbe1b1fb3a5b1115d P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
W20251029 02:52:11.681418 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 0ffd22d8610b4329a0a4f753d9e60149 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0ffd22d8610b4329a0a4f753d9e60149 already in progress: creating tablet
W20251029 02:52:11.681608 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 20a762344c5f46f88481c4ac7ad6e43b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 20a762344c5f46f88481c4ac7ad6e43b already in progress: creating tablet
W20251029 02:52:11.681808 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 50ab70b30aaa4f6bbae15855431159c8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 50ab70b30aaa4f6bbae15855431159c8 already in progress: creating tablet
W20251029 02:52:11.682119 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 48aa20640dfe43288326ed6d6b9bc1cf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 48aa20640dfe43288326ed6d6b9bc1cf already in progress: creating tablet
W20251029 02:52:11.682319 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 65e761f9f9fb490586d6415a5a8565dd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 65e761f9f9fb490586d6415a5a8565dd already in progress: creating tablet
I20251029 02:52:11.683634 520 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 1bdb67125493430398ff5c80b73de2ac
I20251029 02:52:11.684280 2306 raft_consensus.cc:2749] T 0d84dc8dc465496f9adec798b2e03e13 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 02:52:11.684535 989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:38481, user_credentials={real_user=slave}} blocked reactor thread for 138349us
I20251029 02:52:11.685353 2179 consensus_queue.cc:1048] T 02e4588b03144ea2850e13a1f8f29f06 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251029 02:52:11.685948 989 leader_election.cc:343] T 59d03a190b784ab18a3e63a572675891 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 59d03a190b784ab18a3e63a572675891
I20251029 02:52:11.686043 989 leader_election.cc:304] T 59d03a190b784ab18a3e63a572675891 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5; no voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:11.686236 2179 raft_consensus.cc:2749] T 59d03a190b784ab18a3e63a572675891 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 02:52:11.686610 989 leader_election.cc:343] T 13ee62fc2339410b9db70dfd3d3c6fa6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 13ee62fc2339410b9db70dfd3d3c6fa6
I20251029 02:52:11.686889 894 tablet_service.cc:1505] Processing CreateTablet for tablet 09567e75ea1140e5ba506a49fbb1d77e (DEFAULT_TABLE table=default.Table_a0c3ca13b0804fa399653c68e8bf4335 [id=50a069a1fc504904a54dd6f67c5a808f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:11.687263 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09567e75ea1140e5ba506a49fbb1d77e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:11.687215 989 leader_election.cc:304] T 6d830d30b26b4ba799785fec8a6f0769 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5; no voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:11.687433 2179 raft_consensus.cc:2749] T 6d830d30b26b4ba799785fec8a6f0769 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251029 02:52:11.688432 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 886) took 3001 ms. Trace:
I20251029 02:52:11.688503 1176 rpcz_store.cc:276] 1029 02:52:08.686946 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:08.696866 (+ 9920us) service_pool.cc:225] Handling call
1029 02:52:11.399928 (+2703062us) ts_tablet_manager.cc:1151] Deleting tablet d61beda662d844cdaf67fd9a2e382ea2
1029 02:52:11.399936 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:11.522946 (+123010us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:11.525921 (+ 2975us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:11.688413 (+162492us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":33681,"tablet-delete.queue_time_us":2702853}
I20251029 02:52:11.688778 520 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet 32519fbffcef44a890161e5f3043806c (table default.Table_d64a9ac717a94c40a98f826136089824 [id=b95fe1c7d6ff4519b90f993f1fa961fd]). Expected version 2 got 0
I20251029 02:52:11.688858 1176 tablet_replica.cc:333] T f529e17e97074dca91ec2f48954cfd32 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:11.689018 1176 raft_consensus.cc:2243] T f529e17e97074dca91ec2f48954cfd32 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:11.689178 1176 raft_consensus.cc:2272] T f529e17e97074dca91ec2f48954cfd32 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:11.689667 1176 ts_tablet_manager.cc:1916] T f529e17e97074dca91ec2f48954cfd32 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:11.690799 881 tablet_service.cc:1552] Processing DeleteTablet for tablet 1bdb67125493430398ff5c80b73de2ac with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:11 UTC) from {username='slave'} at 127.0.0.1:35702
W20251029 02:52:11.691161 989 leader_election.cc:343] T 9bb2166a590d4210b02dd779ef6654ad P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 9bb2166a590d4210b02dd779ef6654ad
I20251029 02:52:11.691511 989 leader_election.cc:304] T 217144957930495aa3e9dfcfde268e71 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters: c756528642a543eeacd93cef3e335a7d
I20251029 02:52:11.692785 520 catalog_manager.cc:5649] T 946f65014d97425aab3851009b742807 P c756528642a543eeacd93cef3e335a7d reported cstate change: term changed from 0 to 1, leader changed from <none> to c756528642a543eeacd93cef3e335a7d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "c756528642a543eeacd93cef3e335a7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } health_report { overall_health: UNKNOWN } } }
I20251029 02:52:11.681178 769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "44ae2b07edca44afbe1b1fb3a5b1115d" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:11.699685 769 raft_consensus.cc:3060] T 44ae2b07edca44afbe1b1fb3a5b1115d P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:11.702495 769 raft_consensus.cc:2468] T 44ae2b07edca44afbe1b1fb3a5b1115d P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:11.703117 847 leader_election.cc:304] T 44ae2b07edca44afbe1b1fb3a5b1115d P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5
I20251029 02:52:11.703788 2205 raft_consensus.cc:2804] T 44ae2b07edca44afbe1b1fb3a5b1115d P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:11.704557 2205 raft_consensus.cc:697] T 44ae2b07edca44afbe1b1fb3a5b1115d P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:52:11.704859 2205 consensus_queue.cc:237] T 44ae2b07edca44afbe1b1fb3a5b1115d P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:11.705763 2179 raft_consensus.cc:2804] T 217144957930495aa3e9dfcfde268e71 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:11.705847 2179 raft_consensus.cc:493] T 217144957930495aa3e9dfcfde268e71 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:11.705897 2179 raft_consensus.cc:3060] T 217144957930495aa3e9dfcfde268e71 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:11.705948 2201 consensus_queue.cc:1048] T 2e97d1bc06a24e8d8ba11acb6fd280d8 P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:11.708384 2179 raft_consensus.cc:515] T 217144957930495aa3e9dfcfde268e71 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:11.709754 2179 leader_election.cc:290] T 217144957930495aa3e9dfcfde268e71 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Requested vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:11.709793 2201 consensus_queue.cc:1048] T 738046ab57334f2dab9f4aab979c5b42 P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:11.710259 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "217144957930495aa3e9dfcfde268e71" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
I20251029 02:52:11.710455 917 raft_consensus.cc:3060] T 217144957930495aa3e9dfcfde268e71 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
W20251029 02:52:11.713250 989 leader_election.cc:343] T 9bb2166a590d4210b02dd779ef6654ad P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 9bb2166a590d4210b02dd779ef6654ad
I20251029 02:52:11.713641 989 leader_election.cc:304] T 2f198c3259864f7ab3625cb7bfae20b0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters: c756528642a543eeacd93cef3e335a7d
I20251029 02:52:11.713433 917 raft_consensus.cc:2468] T 217144957930495aa3e9dfcfde268e71 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 1.
I20251029 02:52:11.714558 990 leader_election.cc:304] T 217144957930495aa3e9dfcfde268e71 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:11.715184 2179 raft_consensus.cc:2804] T 2f198c3259864f7ab3625cb7bfae20b0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:11.715286 2179 raft_consensus.cc:493] T 2f198c3259864f7ab3625cb7bfae20b0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:11.715374 2179 raft_consensus.cc:3060] T 2f198c3259864f7ab3625cb7bfae20b0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
W20251029 02:52:11.716171 712 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:45003, user_credentials={real_user=slave}} blocked reactor thread for 49602.9us
I20251029 02:52:11.716747 2403 raft_consensus.cc:697] T 8048506ced4540a5aee9f79bfc4599bf P b495a49349324f7d81908f2da700f0d9 [term 1 LEADER]: Becoming Leader. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Running, Role: LEADER
I20251029 02:52:11.717018 2403 consensus_queue.cc:237] T 8048506ced4540a5aee9f79bfc4599bf P b495a49349324f7d81908f2da700f0d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:11.681640 756 tablet_service.cc:1505] Processing CreateTablet for tablet 09567e75ea1140e5ba506a49fbb1d77e (DEFAULT_TABLE table=default.Table_a0c3ca13b0804fa399653c68e8bf4335 [id=50a069a1fc504904a54dd6f67c5a808f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:11.686388 742 tablet_service.cc:1552] Processing DeleteTablet for tablet 1bdb67125493430398ff5c80b73de2ac with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:11 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:11.717857 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09567e75ea1140e5ba506a49fbb1d77e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:11.717870 2179 raft_consensus.cc:515] T 2f198c3259864f7ab3625cb7bfae20b0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:11.718364 2179 leader_election.cc:290] T 2f198c3259864f7ab3625cb7bfae20b0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Requested vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:11.719049 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2f198c3259864f7ab3625cb7bfae20b0" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
W20251029 02:52:11.719477 990 leader_election.cc:343] T 2f198c3259864f7ab3625cb7bfae20b0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:11.719779 2179 raft_consensus.cc:2804] T 217144957930495aa3e9dfcfde268e71 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:11.719933 2179 raft_consensus.cc:697] T 217144957930495aa3e9dfcfde268e71 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Becoming Leader. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Running, Role: LEADER
I20251029 02:52:11.720626 1174 ts_tablet_manager.cc:1916] T d2990e9f626c40618c118266bc595684 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:11.720837 2179 consensus_queue.cc:237] T 217144957930495aa3e9dfcfde268e71 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
W20251029 02:52:11.716344 712 leader_election.cc:343] T da2a21c3236a4bd99872b89b12a36508 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Not found: Tablet not found: da2a21c3236a4bd99872b89b12a36508
I20251029 02:52:11.697091 740 tablet_service.cc:1505] Processing CreateTablet for tablet f58bd72cee5b46f5941a3976547e2aaa (DEFAULT_TABLE table=default.Table_f08f8ab20ebf4c9a99d5b88cf17d3b7d [id=bf42ef67f58e45c58b290c7dbb1ca205]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:11.721733 712 leader_election.cc:304] T da2a21c3236a4bd99872b89b12a36508 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
W20251029 02:52:11.721916 712 raft_consensus.cc:2686] T da2a21c3236a4bd99872b89b12a36508 P b495a49349324f7d81908f2da700f0d9: unable to run election callback: Service unavailable: Thread pool token was shut down
I20251029 02:52:11.722143 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f58bd72cee5b46f5941a3976547e2aaa. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:11.722548 712 leader_election.cc:343] T 41a6139fd9b84707a8c7eebc0381e383 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Not found: Tablet not found: 41a6139fd9b84707a8c7eebc0381e383
I20251029 02:52:11.727370 989 leader_election.cc:304] T 97b2ecae572a4b7ea1b9595074556708 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters: c756528642a543eeacd93cef3e335a7d
I20251029 02:52:11.727609 2306 raft_consensus.cc:2804] T 97b2ecae572a4b7ea1b9595074556708 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:11.727689 2306 raft_consensus.cc:493] T 97b2ecae572a4b7ea1b9595074556708 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:11.727756 2306 raft_consensus.cc:3060] T 97b2ecae572a4b7ea1b9595074556708 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:11.728572 526 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 81ee839d811a4f46b93a57ede397bd93
I20251029 02:52:11.729498 524 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 0f96af1cb0ea4b46bd846ae0739f4947
I20251029 02:52:11.730271 2306 raft_consensus.cc:515] T 97b2ecae572a4b7ea1b9595074556708 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
W20251029 02:52:11.730526 523 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 479fd93eb02540868931faee38830223 consensus state has no leader
I20251029 02:52:11.731158 2306 leader_election.cc:290] T 97b2ecae572a4b7ea1b9595074556708 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Requested vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:52:11.731285 521 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet a530fe8de8ae4be5b0fd9a5f7e94ea9e
I20251029 02:52:11.731529 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97b2ecae572a4b7ea1b9595074556708" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
I20251029 02:52:11.731748 917 raft_consensus.cc:2393] T 97b2ecae572a4b7ea1b9595074556708 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in current term 1: Already voted for candidate b495a49349324f7d81908f2da700f0d9 in this term.
I20251029 02:52:11.732558 517 master_service.cc:954] ReplaceTablet: received request to replace tablet 9bb2166a590d4210b02dd779ef6654ad from {username='slave'} at 127.0.0.1:35314
I20251029 02:52:11.732620 989 leader_election.cc:304] T 061f3d38c4604fa4b57e74ff6f6fc7da P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters: c756528642a543eeacd93cef3e335a7d
I20251029 02:52:11.732838 2306 raft_consensus.cc:2804] T 061f3d38c4604fa4b57e74ff6f6fc7da P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:11.733032 2306 raft_consensus.cc:697] T 061f3d38c4604fa4b57e74ff6f6fc7da P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Becoming Leader. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Running, Role: LEADER
I20251029 02:52:11.733482 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_38847ce61ea24d58aeeb20a5d4f5d9ba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:11.734061 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_38847ce61ea24d58aeeb20a5d4f5d9ba in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:11.733980 2306 consensus_queue.cc:237] T 061f3d38c4604fa4b57e74ff6f6fc7da P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:11.734680 522 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 426f9be3b0074f60989acc43ec1e9bd6
I20251029 02:52:11.735169 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 81ee839d811a4f46b93a57ede397bd93 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:11 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:11.735368 896 tablet_service.cc:1552] Processing DeleteTablet for tablet 0f96af1cb0ea4b46bd846ae0739f4947 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:11 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:11.735541 881 tablet_service.cc:1552] Processing DeleteTablet for tablet a530fe8de8ae4be5b0fd9a5f7e94ea9e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:11 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:11.735672 522 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_603240f584a74f1d965ee97ee1a2d2dc" } new_table_name: "default.Table_5be14f83fc964005ad4f77747b7466f7" modify_external_catalogs: true
I20251029 02:52:11.735746 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_abf33e38626c4d3e9bd944fecb1d1490"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:11.736341 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_abf33e38626c4d3e9bd944fecb1d1490 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:11.737957 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_18e8d552574143d1a42e021aca3182ed"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:11.738160 881 tablet_service.cc:1552] Processing DeleteTablet for tablet 426f9be3b0074f60989acc43ec1e9bd6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:11 UTC) from {username='slave'} at 127.0.0.1:35702
W20251029 02:52:11.738436 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_18e8d552574143d1a42e021aca3182ed in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:11.738737 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_ce61c133a61047b99c732ee57710d119"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:11.739197 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ce61c133a61047b99c732ee57710d119 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:11.739670 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_720c6ad9cd7a42e9b59130daf6492301"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:11.740119 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_720c6ad9cd7a42e9b59130daf6492301 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:11.740391 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_41d356dd41454ac680819ce36c4766a0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:11.740820 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_41d356dd41454ac680819ce36c4766a0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:52:11.741233 989 consensus_peers.cc:597] T 1275e192871440408a8345fef9b56260 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 -> Peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Couldn't send request to peer b495a49349324f7d81908f2da700f0d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1275e192871440408a8345fef9b56260. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:11.741361 989 leader_election.cc:304] T c32c28b0e74a4af3a0e69bbf24ab1a1f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters: c756528642a543eeacd93cef3e335a7d
I20251029 02:52:11.742431 2306 raft_consensus.cc:2804] T c32c28b0e74a4af3a0e69bbf24ab1a1f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:11.742501 2306 raft_consensus.cc:493] T c32c28b0e74a4af3a0e69bbf24ab1a1f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:11.742590 2306 raft_consensus.cc:3060] T c32c28b0e74a4af3a0e69bbf24ab1a1f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:11.745189 2306 raft_consensus.cc:515] T c32c28b0e74a4af3a0e69bbf24ab1a1f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:11.746171 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c32c28b0e74a4af3a0e69bbf24ab1a1f" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
I20251029 02:52:11.746435 917 raft_consensus.cc:2393] T c32c28b0e74a4af3a0e69bbf24ab1a1f P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in current term 1: Already voted for candidate b495a49349324f7d81908f2da700f0d9 in this term.
I20251029 02:52:11.747138 2306 leader_election.cc:290] T c32c28b0e74a4af3a0e69bbf24ab1a1f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Requested vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:11.747819 878 tablet_service.cc:1505] Processing CreateTablet for tablet 2965344ff7174295b7759d1b2c95ff87 (DEFAULT_TABLE table=default.Table_79052b7cce26442aa1454f10280e7e7a [id=a877951ca0224abbbc5a9112f0a598a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:11.705120 1166 tablet_bootstrap.cc:492] T e65eaaae43ac435ebd665546de9d68dd P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:11.750319 1166 ts_tablet_manager.cc:1403] T e65eaaae43ac435ebd665546de9d68dd P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.123s user 0.001s sys 0.003s
I20251029 02:52:11.750592 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2965344ff7174295b7759d1b2c95ff87. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:11.751235 1166 raft_consensus.cc:359] T e65eaaae43ac435ebd665546de9d68dd P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:11.751441 1166 raft_consensus.cc:385] T e65eaaae43ac435ebd665546de9d68dd P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:11.751492 1166 raft_consensus.cc:740] T e65eaaae43ac435ebd665546de9d68dd P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:11.751717 1166 consensus_queue.cc:260] T e65eaaae43ac435ebd665546de9d68dd P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:11.752178 2214 raft_consensus.cc:493] T af4860c6b72f4d988b206b5ef2f22906 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:11.752318 2214 raft_consensus.cc:515] T af4860c6b72f4d988b206b5ef2f22906 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:11.753131 1166 ts_tablet_manager.cc:1434] T e65eaaae43ac435ebd665546de9d68dd P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251029 02:52:11.753422 1166 tablet_bootstrap.cc:492] T d7abaf84ba5d445aa17433b3be891548 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:11.753634 2302 consensus_queue.cc:1048] T 738046ab57334f2dab9f4aab979c5b42 P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:11.755424 2214 leader_election.cc:290] T af4860c6b72f4d988b206b5ef2f22906 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:11.756319 1053 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af4860c6b72f4d988b206b5ef2f22906" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:11.756511 1053 raft_consensus.cc:2393] T af4860c6b72f4d988b206b5ef2f22906 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c756528642a543eeacd93cef3e335a7d in current term 1: Already voted for candidate b495a49349324f7d81908f2da700f0d9 in this term.
W20251029 02:52:11.756707 847 consensus_peers.cc:597] T 2e97d1bc06a24e8d8ba11acb6fd280d8 P c756528642a543eeacd93cef3e335a7d -> Peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Couldn't send request to peer b495a49349324f7d81908f2da700f0d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2e97d1bc06a24e8d8ba11acb6fd280d8. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:11.757181 1166 tablet_bootstrap.cc:654] T d7abaf84ba5d445aa17433b3be891548 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
W20251029 02:52:11.747609 989 consensus_peers.cc:597] T 9bb2166a590d4210b02dd779ef6654ad P 0cfd4bee16f84b7c8b0ff3ce14db20d5 -> Peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Couldn't send request to peer b495a49349324f7d81908f2da700f0d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9bb2166a590d4210b02dd779ef6654ad. This is attempt 1: this message will repeat every 5th retry.
W20251029 02:52:11.761122 989 leader_election.cc:343] T e7f846b294cc4eb687a26c8b5931ac19 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: e7f846b294cc4eb687a26c8b5931ac19
I20251029 02:52:11.761525 989 leader_election.cc:304] T e7f846b294cc4eb687a26c8b5931ac19 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5; no voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d
W20251029 02:52:11.765093 989 consensus_peers.cc:597] T 13ee62fc2339410b9db70dfd3d3c6fa6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 -> Peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Couldn't send request to peer b495a49349324f7d81908f2da700f0d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 13ee62fc2339410b9db70dfd3d3c6fa6. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:11.765496 989 leader_election.cc:304] T adf6e736cb9c4920807ee58682e134ef P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters: c756528642a543eeacd93cef3e335a7d
I20251029 02:52:11.765321 2179 raft_consensus.cc:2749] T e7f846b294cc4eb687a26c8b5931ac19 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:11.766360 2179 consensus_queue.cc:1048] T 95f342d1cfb84c4eb31bd3d15ea3bdc2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:11.766647 2188 raft_consensus.cc:2804] T adf6e736cb9c4920807ee58682e134ef P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:11.766727 2188 raft_consensus.cc:493] T adf6e736cb9c4920807ee58682e134ef P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:11.766788 2188 raft_consensus.cc:3060] T adf6e736cb9c4920807ee58682e134ef P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:11.704380 1176 ts_tablet_manager.cc:1929] T f529e17e97074dca91ec2f48954cfd32 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:11.767220 1176 log.cc:1199] T f529e17e97074dca91ec2f48954cfd32 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/f529e17e97074dca91ec2f48954cfd32
I20251029 02:52:11.767616 1176 ts_tablet_manager.cc:1950] T f529e17e97074dca91ec2f48954cfd32 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:11.768525 1166 tablet_bootstrap.cc:492] T d7abaf84ba5d445aa17433b3be891548 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:11.768642 1166 ts_tablet_manager.cc:1403] T d7abaf84ba5d445aa17433b3be891548 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.001s
I20251029 02:52:11.769372 2188 raft_consensus.cc:515] T adf6e736cb9c4920807ee58682e134ef P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:11.770104 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 887) took 3072 ms. Trace:
I20251029 02:52:11.770291 1176 rpcz_store.cc:276] 1029 02:52:08.697234 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:08.697274 (+ 40us) service_pool.cc:225] Handling call
1029 02:52:11.688814 (+2991540us) ts_tablet_manager.cc:1151] Deleting tablet f529e17e97074dca91ec2f48954cfd32
1029 02:52:11.688823 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:11.701151 (+ 12328us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:11.704285 (+ 3134us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:11.770083 (+ 65798us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2991415}
I20251029 02:52:11.770692 2188 leader_election.cc:290] T adf6e736cb9c4920807ee58682e134ef P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Requested vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:11.771301 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "adf6e736cb9c4920807ee58682e134ef" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
I20251029 02:52:11.771464 917 raft_consensus.cc:3060] T adf6e736cb9c4920807ee58682e134ef P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:11.774550 917 raft_consensus.cc:2468] T adf6e736cb9c4920807ee58682e134ef P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 1.
I20251029 02:52:11.774936 990 leader_election.cc:304] T adf6e736cb9c4920807ee58682e134ef P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:11.775648 2188 raft_consensus.cc:2804] T adf6e736cb9c4920807ee58682e134ef P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:11.775806 2188 raft_consensus.cc:697] T adf6e736cb9c4920807ee58682e134ef P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Becoming Leader. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Running, Role: LEADER
I20251029 02:52:11.776069 2188 consensus_queue.cc:237] T adf6e736cb9c4920807ee58682e134ef P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:11.770812 1176 tablet_replica.cc:333] T 785733da86ff425a90ccfab2933fc89e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:11.725741 742 tablet_service.cc:1505] Processing CreateTablet for tablet 2965344ff7174295b7759d1b2c95ff87 (DEFAULT_TABLE table=default.Table_79052b7cce26442aa1454f10280e7e7a [id=a877951ca0224abbbc5a9112f0a598a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:11.780757 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2965344ff7174295b7759d1b2c95ff87. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:11.780736 1176 raft_consensus.cc:2243] T 785733da86ff425a90ccfab2933fc89e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:11.781013 1176 raft_consensus.cc:2272] T 785733da86ff425a90ccfab2933fc89e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:11.781561 1176 ts_tablet_manager.cc:1916] T 785733da86ff425a90ccfab2933fc89e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:11.769357 1166 raft_consensus.cc:359] T d7abaf84ba5d445aa17433b3be891548 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:11.786666 1166 raft_consensus.cc:385] T d7abaf84ba5d445aa17433b3be891548 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:11.787609 1167 tablet_bootstrap.cc:492] T 44c0488eba0543078ff4f0b3d69366f2 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:11.787753 1167 ts_tablet_manager.cc:1403] T 44c0488eba0543078ff4f0b3d69366f2 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.126s user 0.003s sys 0.003s
I20251029 02:52:11.787807 1166 raft_consensus.cc:740] T d7abaf84ba5d445aa17433b3be891548 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:11.727246 1172 tablet_bootstrap.cc:492] T 46cd75ff18b14212b556b9bcb3012dd9 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:11.729732 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 81ee839d811a4f46b93a57ede397bd93 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:11 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:11.699301 1175 ts_tablet_manager.cc:1929] T 41a6139fd9b84707a8c7eebc0381e383 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.3
I20251029 02:52:11.744186 751 tablet_service.cc:1552] Processing DeleteTablet for tablet 426f9be3b0074f60989acc43ec1e9bd6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:11 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:11.744319 747 tablet_service.cc:1552] Processing DeleteTablet for tablet a530fe8de8ae4be5b0fd9a5f7e94ea9e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:11 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:11.744414 738 tablet_service.cc:1552] Processing DeleteTablet for tablet 0f96af1cb0ea4b46bd846ae0739f4947 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:11 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:11.753670 764 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af4860c6b72f4d988b206b5ef2f22906" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:11.788666 1167 raft_consensus.cc:359] T 44c0488eba0543078ff4f0b3d69366f2 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:11.788869 1167 raft_consensus.cc:385] T 44c0488eba0543078ff4f0b3d69366f2 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:11.788926 1167 raft_consensus.cc:740] T 44c0488eba0543078ff4f0b3d69366f2 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:11.788376 1172 ts_tablet_manager.cc:1403] T 46cd75ff18b14212b556b9bcb3012dd9 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.171s user 0.003s sys 0.001s
I20251029 02:52:11.789151 1166 consensus_queue.cc:260] T d7abaf84ba5d445aa17433b3be891548 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:11.789182 1167 consensus_queue.cc:260] T 44c0488eba0543078ff4f0b3d69366f2 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:11.789952 1166 ts_tablet_manager.cc:1434] T d7abaf84ba5d445aa17433b3be891548 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.021s user 0.001s sys 0.001s
I20251029 02:52:11.793484 849 leader_election.cc:304] T af4860c6b72f4d988b206b5ef2f22906 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9
I20251029 02:52:11.794421 2205 raft_consensus.cc:493] T d7abaf84ba5d445aa17433b3be891548 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:11.794576 2205 raft_consensus.cc:515] T d7abaf84ba5d445aa17433b3be891548 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
W20251029 02:52:11.794965 519 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d22b01dc872b4ebb8f65b586b29eea00 consensus state has no leader
I20251029 02:52:11.795101 2205 leader_election.cc:290] T d7abaf84ba5d445aa17433b3be891548 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:11.795213 2205 raft_consensus.cc:3060] T af4860c6b72f4d988b206b5ef2f22906 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:11.795675 1166 tablet_bootstrap.cc:492] T 44ae2b07edca44afbe1b1fb3a5b1115d P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:11.797996 1166 tablet_bootstrap.cc:654] T 44ae2b07edca44afbe1b1fb3a5b1115d P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:11.798357 2205 raft_consensus.cc:2749] T af4860c6b72f4d988b206b5ef2f22906 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:11.799192 1052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d7abaf84ba5d445aa17433b3be891548" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:11.799363 1052 raft_consensus.cc:2468] T d7abaf84ba5d445aa17433b3be891548 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
W20251029 02:52:11.788607 712 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:45003, user_credentials={real_user=slave}} blocked reactor thread for 66063.2us
I20251029 02:52:11.798267 764 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d7abaf84ba5d445aa17433b3be891548" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:11.799839 764 raft_consensus.cc:2468] T d7abaf84ba5d445aa17433b3be891548 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:11.799846 849 leader_election.cc:304] T d7abaf84ba5d445aa17433b3be891548 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:11.800062 2205 raft_consensus.cc:2804] T d7abaf84ba5d445aa17433b3be891548 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:11.800120 2205 raft_consensus.cc:493] T d7abaf84ba5d445aa17433b3be891548 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:11.800181 2205 raft_consensus.cc:3060] T d7abaf84ba5d445aa17433b3be891548 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:11.788897 1175 log.cc:1199] T 41a6139fd9b84707a8c7eebc0381e383 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/41a6139fd9b84707a8c7eebc0381e383
I20251029 02:52:11.800660 1175 ts_tablet_manager.cc:1950] T 41a6139fd9b84707a8c7eebc0381e383 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
W20251029 02:52:11.800880 712 leader_election.cc:343] T 67651306f3b44e7eb14dcee9a2007a81 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:11.800994 712 leader_election.cc:304] T 67651306f3b44e7eb14dcee9a2007a81 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:11.801322 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 814) took 3795 ms. Trace:
I20251029 02:52:11.801373 2309 consensus_queue.cc:1048] T 28f7ceb91a34450cad4b5b45f5285b41 P b495a49349324f7d81908f2da700f0d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:11.801738 2309 raft_consensus.cc:2749] T 67651306f3b44e7eb14dcee9a2007a81 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:11.801465 1175 rpcz_store.cc:276] 1029 02:52:08.006373 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:08.006426 (+ 53us) service_pool.cc:225] Handling call
1029 02:52:11.345394 (+3338968us) ts_tablet_manager.cc:1151] Deleting tablet 41a6139fd9b84707a8c7eebc0381e383
1029 02:52:11.345402 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:11.539653 (+194251us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:11.699195 (+159542us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:11.801307 (+102112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3338779}
I20251029 02:52:11.789947 1172 raft_consensus.cc:359] T 46cd75ff18b14212b556b9bcb3012dd9 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:11.802217 1172 raft_consensus.cc:385] T 46cd75ff18b14212b556b9bcb3012dd9 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:11.802341 1172 raft_consensus.cc:740] T 46cd75ff18b14212b556b9bcb3012dd9 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:11.802351 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 41a6139fd9b84707a8c7eebc0381e383 (table default.Table_e48aadc66a0e48658d45d0812c29cbdf [id=f45315b686594867aae0114f4ce2f26c]) successfully deleted
I20251029 02:52:11.802704 1172 consensus_queue.cc:260] T 46cd75ff18b14212b556b9bcb3012dd9 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
W20251029 02:52:11.801319 712 leader_election.cc:343] T 41a6139fd9b84707a8c7eebc0381e383 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Not found: Tablet not found: 41a6139fd9b84707a8c7eebc0381e383
I20251029 02:52:11.803337 1172 ts_tablet_manager.cc:1434] T 46cd75ff18b14212b556b9bcb3012dd9 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
W20251029 02:52:11.803604 712 leader_election.cc:343] T d041e87ceb11441983d0e38b7929c3a4 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:11.803767 2205 raft_consensus.cc:515] T d7abaf84ba5d445aa17433b3be891548 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:11.803860 712 leader_election.cc:304] T d041e87ceb11441983d0e38b7929c3a4 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:11.804216 712 leader_election.cc:304] T 71b4b91071a54d6aa66160e7955bc915 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters: c756528642a543eeacd93cef3e335a7d
I20251029 02:52:11.804558 2309 raft_consensus.cc:2749] T d041e87ceb11441983d0e38b7929c3a4 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:11.804638 1052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d7abaf84ba5d445aa17433b3be891548" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:52:11.804785 1052 raft_consensus.cc:3060] T d7abaf84ba5d445aa17433b3be891548 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:11.804955 2113 raft_consensus.cc:2764] T 71b4b91071a54d6aa66160e7955bc915 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
W20251029 02:52:11.805276 712 leader_election.cc:343] T 643e20d9974447daa78885b3401f3b77 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:11.805388 712 leader_election.cc:304] T 643e20d9974447daa78885b3401f3b77 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
W20251029 02:52:11.805682 712 leader_election.cc:343] T 59eee86f249e4d6a9d09d30276dcd659 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:11.805805 712 leader_election.cc:304] T 59eee86f249e4d6a9d09d30276dcd659 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:11.807399 1052 raft_consensus.cc:2468] T d7abaf84ba5d445aa17433b3be891548 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:11.802851 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 815) took 3528 ms. Trace:
I20251029 02:52:11.807567 1175 rpcz_store.cc:276] 1029 02:52:08.274897 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:08.275336 (+ 439us) service_pool.cc:225] Handling call
1029 02:52:11.802719 (+3527383us) ts_tablet_manager.cc:1151] Deleting tablet 0ffd22d8610b4329a0a4f753d9e60149
1029 02:52:11.802731 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:11.802831 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3527097}
I20251029 02:52:11.807978 849 leader_election.cc:304] T d7abaf84ba5d445aa17433b3be891548 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters:
W20251029 02:52:11.808101 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 0ffd22d8610b4329a0a4f753d9e60149 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0ffd22d8610b4329a0a4f753d9e60149 already in progress: creating tablet
I20251029 02:52:11.808295 2205 leader_election.cc:290] T d7abaf84ba5d445aa17433b3be891548 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:11.808419 2205 raft_consensus.cc:2804] T d7abaf84ba5d445aa17433b3be891548 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:11.808435 2113 raft_consensus.cc:2749] T 643e20d9974447daa78885b3401f3b77 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:11.808509 2205 raft_consensus.cc:697] T d7abaf84ba5d445aa17433b3be891548 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:52:11.808637 2113 raft_consensus.cc:2749] T 59eee86f249e4d6a9d09d30276dcd659 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:11.808719 2205 consensus_queue.cc:237] T d7abaf84ba5d445aa17433b3be891548 P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:11.809118 764 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d7abaf84ba5d445aa17433b3be891548" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:11.809291 764 raft_consensus.cc:3060] T d7abaf84ba5d445aa17433b3be891548 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:11.810546 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 816) took 3535 ms. Trace:
I20251029 02:52:11.810642 1175 rpcz_store.cc:276] 1029 02:52:08.275066 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:08.275647 (+ 581us) service_pool.cc:225] Handling call
1029 02:52:11.810382 (+3534735us) ts_tablet_manager.cc:1151] Deleting tablet 4372eacfc07246f3be8f7c146674a736
1029 02:52:11.810390 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:11.810496 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3534560}
W20251029 02:52:11.811051 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 4372eacfc07246f3be8f7c146674a736 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4372eacfc07246f3be8f7c146674a736 already in progress: creating tablet
I20251029 02:52:11.811333 1175 tablet_replica.cc:333] T 20a762344c5f46f88481c4ac7ad6e43b P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:11.811533 1175 raft_consensus.cc:2243] T 20a762344c5f46f88481c4ac7ad6e43b P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:11.811663 1175 raft_consensus.cc:2272] T 20a762344c5f46f88481c4ac7ad6e43b P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:11.812042 764 raft_consensus.cc:2468] T d7abaf84ba5d445aa17433b3be891548 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:11.812845 1175 ts_tablet_manager.cc:1916] T 20a762344c5f46f88481c4ac7ad6e43b P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 02:52:11.814658 712 leader_election.cc:343] T 946f65014d97425aab3851009b742807 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:11.814847 712 leader_election.cc:304] T 946f65014d97425aab3851009b742807 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:11.815141 2309 raft_consensus.cc:2749] T 946f65014d97425aab3851009b742807 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 02:52:11.815644 712 leader_election.cc:343] T f281191e7c6143f983f11d87f1c9b64e P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:11.816620 712 leader_election.cc:304] T f281191e7c6143f983f11d87f1c9b64e P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:11.817065 1167 ts_tablet_manager.cc:1434] T 44c0488eba0543078ff4f0b3d69366f2 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.029s user 0.001s sys 0.001s
W20251029 02:52:11.817041 712 leader_election.cc:343] T afb19ba39399410bb90b6225d7b5b5b1 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:11.817252 712 leader_election.cc:304] T afb19ba39399410bb90b6225d7b5b5b1 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:11.817895 2309 raft_consensus.cc:2749] T f281191e7c6143f983f11d87f1c9b64e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:11.818122 2309 raft_consensus.cc:2749] T afb19ba39399410bb90b6225d7b5b5b1 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:11.818562 712 leader_election.cc:304] T 217144957930495aa3e9dfcfde268e71 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters: c756528642a543eeacd93cef3e335a7d
W20251029 02:52:11.818979 712 leader_election.cc:343] T a9e0fcc759a64679b3f59a6c964877b6 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:11.819561 1176 ts_tablet_manager.cc:1929] T 785733da86ff425a90ccfab2933fc89e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:11.819713 712 leader_election.cc:304] T a9e0fcc759a64679b3f59a6c964877b6 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:11.819147 2403 raft_consensus.cc:2804] T 217144957930495aa3e9dfcfde268e71 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:11.820019 2403 raft_consensus.cc:493] T 217144957930495aa3e9dfcfde268e71 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:11.820159 2403 raft_consensus.cc:3060] T 217144957930495aa3e9dfcfde268e71 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:11.822835 2403 raft_consensus.cc:515] T 217144957930495aa3e9dfcfde268e71 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:11.824298 1176 log.cc:1199] T 785733da86ff425a90ccfab2933fc89e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/785733da86ff425a90ccfab2933fc89e
I20251029 02:52:11.824735 1176 ts_tablet_manager.cc:1950] T 785733da86ff425a90ccfab2933fc89e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:11.824693 881 tablet_service.cc:1505] Processing CreateTablet for tablet a52fc7119e12435faec1a1df4e5eedef (DEFAULT_TABLE table=default.Table_0f172f8e7eb649ffa6f1723cab301cc5 [id=6c216f006b8f4aadbd4633f8bdf745b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:11.825945 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 888) took 3128 ms. Trace:
I20251029 02:52:11.826085 1176 rpcz_store.cc:276] 1029 02:52:08.697484 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:08.697519 (+ 35us) service_pool.cc:225] Handling call
1029 02:52:11.770755 (+3073236us) ts_tablet_manager.cc:1151] Deleting tablet 785733da86ff425a90ccfab2933fc89e
1029 02:52:11.770764 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:11.791902 (+ 21138us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:11.819447 (+ 27545us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:11.825925 (+ 6478us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3073130}
I20251029 02:52:11.823871 2309 raft_consensus.cc:2749] T a9e0fcc759a64679b3f59a6c964877b6 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 02:52:11.824162 712 leader_election.cc:343] T 58dde9397b434b089ade9968f369006a P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:11.825743 738 tablet_service.cc:1505] Processing CreateTablet for tablet a52fc7119e12435faec1a1df4e5eedef (DEFAULT_TABLE table=default.Table_0f172f8e7eb649ffa6f1723cab301cc5 [id=6c216f006b8f4aadbd4633f8bdf745b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:11.826778 896 tablet_service.cc:1505] Processing CreateTablet for tablet 69a1106275d4458b93a29e9834c34c68 (DEFAULT_TABLE table=default.Table_11dd5d796fa041058d7cbc7ce4ef730d [id=1c7a68ea8b5746b79a9e6988b1168133]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:11.826992 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a52fc7119e12435faec1a1df4e5eedef. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:11.827454 747 tablet_service.cc:1505] Processing CreateTablet for tablet 69a1106275d4458b93a29e9834c34c68 (DEFAULT_TABLE table=default.Table_11dd5d796fa041058d7cbc7ce4ef730d [id=1c7a68ea8b5746b79a9e6988b1168133]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:11.827781 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69a1106275d4458b93a29e9834c34c68. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:11.833226 712 leader_election.cc:304] T 58dde9397b434b089ade9968f369006a P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:11.833556 2309 raft_consensus.cc:2749] T 58dde9397b434b089ade9968f369006a P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:11.834966 2309 raft_consensus.cc:493] T 6789bc16664e4b37b86702449ab27855 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:11.835170 2309 raft_consensus.cc:515] T 6789bc16664e4b37b86702449ab27855 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:11.836213 2309 leader_election.cc:290] T 6789bc16664e4b37b86702449ab27855 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:11.837741 918 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "217144957930495aa3e9dfcfde268e71" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
I20251029 02:52:11.837950 918 raft_consensus.cc:2393] T 217144957930495aa3e9dfcfde268e71 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b495a49349324f7d81908f2da700f0d9 in current term 1: Already voted for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in this term.
I20251029 02:52:11.838755 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 895) took 2945 ms. Trace:
I20251029 02:52:11.839100 2403 leader_election.cc:290] T 217144957930495aa3e9dfcfde268e71 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 election: Requested vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:11.839077 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6789bc16664e4b37b86702449ab27855" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:11.839399 1176 rpcz_store.cc:276] 1029 02:52:08.892950 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:08.893061 (+ 111us) service_pool.cc:225] Handling call
1029 02:52:11.838623 (+2945562us) ts_tablet_manager.cc:1151] Deleting tablet 372d93914abc41b38a702742b7db0bc2
1029 02:52:11.838635 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:11.838735 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2945351}
W20251029 02:52:11.839577 711 leader_election.cc:343] T 6789bc16664e4b37b86702449ab27855 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:11.839756 519 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_35f77c51187d489fad31db01129e5e98" } new_table_name: "default.Table_dbeb5b9d4261424085b86ecddb373eff" modify_external_catalogs: true
I20251029 02:52:11.840826 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 911) took 1975 ms. Trace:
I20251029 02:52:11.840971 1176 rpcz_store.cc:276] 1029 02:52:09.865487 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:09.865542 (+ 55us) service_pool.cc:225] Handling call
1029 02:52:11.840705 (+1975163us) ts_tablet_manager.cc:1151] Deleting tablet 00ebd0c68c4441f9b63c86cdab62497d
1029 02:52:11.840714 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:11.840810 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1974983}
I20251029 02:52:11.841282 1176 tablet_replica.cc:333] T dfa8b1b9a80e4e8689e0acb44e16346a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:11.840567 1166 tablet_bootstrap.cc:492] T 44ae2b07edca44afbe1b1fb3a5b1115d P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:11.841498 1166 ts_tablet_manager.cc:1403] T 44ae2b07edca44afbe1b1fb3a5b1115d P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.047s user 0.002s sys 0.002s
I20251029 02:52:11.842429 1166 raft_consensus.cc:359] T 44ae2b07edca44afbe1b1fb3a5b1115d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:11.842671 1166 raft_consensus.cc:385] T 44ae2b07edca44afbe1b1fb3a5b1115d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:11.842767 1166 raft_consensus.cc:740] T 44ae2b07edca44afbe1b1fb3a5b1115d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:11.843000 1166 consensus_queue.cc:260] T 44ae2b07edca44afbe1b1fb3a5b1115d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:11.844760 1166 ts_tablet_manager.cc:1434] T 44ae2b07edca44afbe1b1fb3a5b1115d P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20251029 02:52:11.847414 1166 tablet_bootstrap.cc:492] T 643e20d9974447daa78885b3401f3b77 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:11.850421 1176 raft_consensus.cc:2243] T dfa8b1b9a80e4e8689e0acb44e16346a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:11.850675 1176 raft_consensus.cc:2272] T dfa8b1b9a80e4e8689e0acb44e16346a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:11.853472 1166 tablet_bootstrap.cc:654] T 643e20d9974447daa78885b3401f3b77 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:11.853833 1176 ts_tablet_manager.cc:1916] T dfa8b1b9a80e4e8689e0acb44e16346a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:11.855666 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a52fc7119e12435faec1a1df4e5eedef. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:11.855660 1167 tablet_bootstrap.cc:492] T 643e20d9974447daa78885b3401f3b77 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:11.857698 1167 tablet_bootstrap.cc:654] T 643e20d9974447daa78885b3401f3b77 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:11.858723 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69a1106275d4458b93a29e9834c34c68. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:11.863986 1176 ts_tablet_manager.cc:1929] T dfa8b1b9a80e4e8689e0acb44e16346a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:11.865346 1176 log.cc:1199] T dfa8b1b9a80e4e8689e0acb44e16346a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/dfa8b1b9a80e4e8689e0acb44e16346a
I20251029 02:52:11.866362 1176 ts_tablet_manager.cc:1950] T dfa8b1b9a80e4e8689e0acb44e16346a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:11.865958 1166 tablet_bootstrap.cc:492] T 643e20d9974447daa78885b3401f3b77 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:11.866608 1166 ts_tablet_manager.cc:1403] T 643e20d9974447daa78885b3401f3b77 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.001s
I20251029 02:52:11.867439 1166 raft_consensus.cc:359] T 643e20d9974447daa78885b3401f3b77 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:11.867671 1166 raft_consensus.cc:385] T 643e20d9974447daa78885b3401f3b77 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:11.867771 1166 raft_consensus.cc:740] T 643e20d9974447daa78885b3401f3b77 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:11.868094 1166 consensus_queue.cc:260] T 643e20d9974447daa78885b3401f3b77 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:11.871454 1166 ts_tablet_manager.cc:1434] T 643e20d9974447daa78885b3401f3b77 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20251029 02:52:11.871831 1166 tablet_bootstrap.cc:492] T 731c0acfbe204e8199ad246e38ae2ccd P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:11.873503 1166 tablet_bootstrap.cc:654] T 731c0acfbe204e8199ad246e38ae2ccd P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:11.875519 1167 tablet_bootstrap.cc:492] T 643e20d9974447daa78885b3401f3b77 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:11.875042 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 912) took 2009 ms. Trace:
I20251029 02:52:11.875661 1167 ts_tablet_manager.cc:1403] T 643e20d9974447daa78885b3401f3b77 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.010s
I20251029 02:52:11.875715 1176 rpcz_store.cc:276] 1029 02:52:09.865872 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:10.041115 (+175243us) service_pool.cc:225] Handling call
1029 02:52:11.841242 (+1800127us) ts_tablet_manager.cc:1151] Deleting tablet dfa8b1b9a80e4e8689e0acb44e16346a
1029 02:52:11.841250 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:11.858689 (+ 17439us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:11.863897 (+ 5208us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:11.875020 (+ 11123us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1799874}
I20251029 02:52:11.876384 1167 raft_consensus.cc:359] T 643e20d9974447daa78885b3401f3b77 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:11.876554 1167 raft_consensus.cc:385] T 643e20d9974447daa78885b3401f3b77 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:11.876613 1167 raft_consensus.cc:740] T 643e20d9974447daa78885b3401f3b77 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:11.876606 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_eff289b2377f44aeb3411568c4854be1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:11.876842 989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:38481, user_credentials={real_user=slave}} blocked reactor thread for 111365us
I20251029 02:52:11.876801 1167 consensus_queue.cc:260] T 643e20d9974447daa78885b3401f3b77 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:11.877081 1176 tablet_replica.cc:333] T 0d84dc8dc465496f9adec798b2e03e13 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:11.877254 989 leader_election.cc:304] T 82289e0f823c4d43bfd8c4e878572b2b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters: c756528642a543eeacd93cef3e335a7d
I20251029 02:52:11.877420 1167 ts_tablet_manager.cc:1434] T 643e20d9974447daa78885b3401f3b77 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 02:52:11.877331 1176 raft_consensus.cc:2243] T 0d84dc8dc465496f9adec798b2e03e13 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:11.877671 1176 raft_consensus.cc:2272] T 0d84dc8dc465496f9adec798b2e03e13 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:11.877820 2188 raft_consensus.cc:2804] T 82289e0f823c4d43bfd8c4e878572b2b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:11.877923 2188 raft_consensus.cc:493] T 82289e0f823c4d43bfd8c4e878572b2b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:11.878029 2188 raft_consensus.cc:3060] T 82289e0f823c4d43bfd8c4e878572b2b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:11.878088 2306 consensus_queue.cc:1048] T 9269622ca92645d5aa8e73341201ccbb P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251029 02:52:11.878304 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eff289b2377f44aeb3411568c4854be1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:11.878850 1167 tablet_bootstrap.cc:492] T afb19ba39399410bb90b6225d7b5b5b1 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:11.879731 989 leader_election.cc:304] T 44c0488eba0543078ff4f0b3d69366f2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters: c756528642a543eeacd93cef3e335a7d
I20251029 02:52:11.879699 2306 consensus_queue.cc:1048] T dabb8d760aae4aa09eb83cdb586c9788 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:11.880060 2306 raft_consensus.cc:2804] T 44c0488eba0543078ff4f0b3d69366f2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:11.880168 2306 raft_consensus.cc:493] T 44c0488eba0543078ff4f0b3d69366f2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:11.880268 2306 raft_consensus.cc:3060] T 44c0488eba0543078ff4f0b3d69366f2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:11.880403 747 tablet_service.cc:1552] Processing DeleteTablet for tablet 46cd75ff18b14212b556b9bcb3012dd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:05 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:11.880628 2188 raft_consensus.cc:515] T 82289e0f823c4d43bfd8c4e878572b2b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:11.881150 2188 leader_election.cc:290] T 82289e0f823c4d43bfd8c4e878572b2b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Requested vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:11.881321 1172 tablet_bootstrap.cc:492] T ae4231daa2594258aa51419c97a2f50a P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:11.880081 989 leader_election.cc:304] T 71b4b91071a54d6aa66160e7955bc915 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters: c756528642a543eeacd93cef3e335a7d
W20251029 02:52:11.882143 989 raft_consensus.cc:2686] T 71b4b91071a54d6aa66160e7955bc915 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: unable to run election callback: Service unavailable: Thread pool token was shut down
W20251029 02:52:11.883172 712 leader_election.cc:343] T 7bb85a12f573494cbb3118d1947eda27 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:11.883339 712 leader_election.cc:304] T 7bb85a12f573494cbb3118d1947eda27 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:11.883358 2306 raft_consensus.cc:515] T 44c0488eba0543078ff4f0b3d69366f2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
W20251029 02:52:11.883991 712 leader_election.cc:343] T 378f549ff7364d018d8b9a5ef01ac878 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:11.884040 2306 leader_election.cc:290] T 44c0488eba0543078ff4f0b3d69366f2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Requested vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:11.884130 712 leader_election.cc:304] T 378f549ff7364d018d8b9a5ef01ac878 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:11.884184 2306 consensus_queue.cc:1048] T 44b518c3a0964a6fa863b2784d80c8a4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:11.884521 2403 raft_consensus.cc:2749] T 7bb85a12f573494cbb3118d1947eda27 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:11.884729 2403 raft_consensus.cc:2749] T 378f549ff7364d018d8b9a5ef01ac878 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:11.884963 2179 consensus_queue.cc:1048] T 30d03e1e42d34242bd354f9e9ec4f127 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251029 02:52:11.884737 712 leader_election.cc:343] T 20a762344c5f46f88481c4ac7ad6e43b P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:11.885762 712 leader_election.cc:304] T 20a762344c5f46f88481c4ac7ad6e43b P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
W20251029 02:52:11.885955 712 raft_consensus.cc:2686] T 20a762344c5f46f88481c4ac7ad6e43b P b495a49349324f7d81908f2da700f0d9: unable to run election callback: Service unavailable: Thread pool token was shut down
I20251029 02:52:11.886513 918 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "44c0488eba0543078ff4f0b3d69366f2" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
I20251029 02:52:11.886699 918 raft_consensus.cc:3060] T 44c0488eba0543078ff4f0b3d69366f2 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:11.887737 1176 ts_tablet_manager.cc:1916] T 0d84dc8dc465496f9adec798b2e03e13 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:11.889420 918 raft_consensus.cc:2468] T 44c0488eba0543078ff4f0b3d69366f2 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 1.
I20251029 02:52:11.886519 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82289e0f823c4d43bfd8c4e878572b2b" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
I20251029 02:52:11.889847 917 raft_consensus.cc:3060] T 82289e0f823c4d43bfd8c4e878572b2b P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:11.890264 990 leader_election.cc:304] T 44c0488eba0543078ff4f0b3d69366f2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:11.890959 2188 raft_consensus.cc:2804] T 44c0488eba0543078ff4f0b3d69366f2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:11.892156 2188 raft_consensus.cc:697] T 44c0488eba0543078ff4f0b3d69366f2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Becoming Leader. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Running, Role: LEADER
I20251029 02:52:11.892514 2188 consensus_queue.cc:237] T 44c0488eba0543078ff4f0b3d69366f2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:11.893107 917 raft_consensus.cc:2468] T 82289e0f823c4d43bfd8c4e878572b2b P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 1.
I20251029 02:52:11.893669 712 leader_election.cc:304] T c1222e30bc4141b3ae686d80053c4ef5 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters: c756528642a543eeacd93cef3e335a7d
I20251029 02:52:11.893760 1167 tablet_bootstrap.cc:654] T afb19ba39399410bb90b6225d7b5b5b1 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:11.893965 2309 raft_consensus.cc:2804] T c1222e30bc4141b3ae686d80053c4ef5 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:11.894080 2309 raft_consensus.cc:493] T c1222e30bc4141b3ae686d80053c4ef5 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:11.894184 2309 raft_consensus.cc:3060] T c1222e30bc4141b3ae686d80053c4ef5 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:11.894268 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet c9dff4dff82f49e7ad49caad5e404f09 (table default.Table_282e9d0ae9384d3284e662fa864925a9 [id=e3b3113e72b747f2a2b55f3b8fae96c3]) successfully deleted
I20251029 02:52:11.894464 990 leader_election.cc:304] T 82289e0f823c4d43bfd8c4e878572b2b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters:
W20251029 02:52:11.894640 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet d3faa2d4df584c6e8da426a3830a6454 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d3faa2d4df584c6e8da426a3830a6454 already in progress: creating tablet
I20251029 02:52:11.894723 2179 raft_consensus.cc:2804] T 82289e0f823c4d43bfd8c4e878572b2b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:11.894804 2179 raft_consensus.cc:697] T 82289e0f823c4d43bfd8c4e878572b2b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Becoming Leader. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Running, Role: LEADER
I20251029 02:52:11.894989 2179 consensus_queue.cc:237] T 82289e0f823c4d43bfd8c4e878572b2b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:11.898746 896 tablet_service.cc:1552] Processing DeleteTablet for tablet 97b2ecae572a4b7ea1b9595074556708 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:00 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:11.901902 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 71b4b91071a54d6aa66160e7955bc915 (table default.Table_457a3835f7d84fd7bf722dfc1d18d301 [id=68645701bb494d9fa65e7c1ff7c9e14c]) successfully deleted
I20251029 02:52:11.914213 520 catalog_manager.cc:5649] T d7abaf84ba5d445aa17433b3be891548 P c756528642a543eeacd93cef3e335a7d reported cstate change: term changed from 0 to 1, leader changed from <none> to c756528642a543eeacd93cef3e335a7d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "c756528642a543eeacd93cef3e335a7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } health_report { overall_health: UNKNOWN } } }
I20251029 02:52:11.914682 520 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet d7abaf84ba5d445aa17433b3be891548 (table default.Table_c1c6933af1e046aa895e0ee915178f5c [id=9e33c530a35544a4ae22e2a1238592db]). Expected version 3 got 0
I20251029 02:52:11.919544 896 tablet_service.cc:1505] Processing CreateTablet for tablet e83ef5b888e94ff4b340074c3c9bc0ac (DEFAULT_TABLE table=default.Table_2050c80c7f844f4b9fc3aeffb54f8d6e [id=24bc2bcb1e2f4091a460cf22c15f902a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:11.920217 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e83ef5b888e94ff4b340074c3c9bc0ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:11.921516 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "217144957930495aa3e9dfcfde268e71" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:11.921794 776 raft_consensus.cc:2393] T 217144957930495aa3e9dfcfde268e71 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in current term 1: Already voted for candidate b495a49349324f7d81908f2da700f0d9 in this term.
I20251029 02:52:11.922418 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2f198c3259864f7ab3625cb7bfae20b0" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:11.922638 776 raft_consensus.cc:3060] T 2f198c3259864f7ab3625cb7bfae20b0 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
W20251029 02:52:11.922647 849 consensus_peers.cc:597] T 777992acb1bd4765886b802940db74bb P c756528642a543eeacd93cef3e335a7d -> Peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Couldn't send request to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 777992acb1bd4765886b802940db74bb. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:11.925367 776 raft_consensus.cc:2468] T 2f198c3259864f7ab3625cb7bfae20b0 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 1.
I20251029 02:52:11.925963 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97b2ecae572a4b7ea1b9595074556708" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:11.926630 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c32c28b0e74a4af3a0e69bbf24ab1a1f" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:11.916198 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet d61beda662d844cdaf67fd9a2e382ea2 (table default.Table_ca4e3b5646244cafa5a0429d5a8f2ac6 [id=481e8e175f364d31bb7dbbaa2df2dad1]) successfully deleted
I20251029 02:52:11.926891 520 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet 82289e0f823c4d43bfd8c4e878572b2b (table default.Table_e7bb3e262a894c2f84b6edfc3917b30d [id=2cca3b7c50b243b5883b8ac1f353f253]). Expected version 1 got 0
I20251029 02:52:11.927112 520 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet 643e20d9974447daa78885b3401f3b77 (table default.Table_7c1cb64e05be4f6e8ac28286c982a3e7 [id=b4fc453bf54d45c4b8dddc3c49bf48f8]). Expected version 2 got 0
I20251029 02:52:11.928934 762 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "adf6e736cb9c4920807ee58682e134ef" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:11.929136 762 raft_consensus.cc:2393] T adf6e736cb9c4920807ee58682e134ef P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in current term 1: Already voted for candidate b495a49349324f7d81908f2da700f0d9 in this term.
I20251029 02:52:11.930821 762 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82289e0f823c4d43bfd8c4e878572b2b" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:11.930994 762 raft_consensus.cc:2393] T 82289e0f823c4d43bfd8c4e878572b2b P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in current term 1: Already voted for candidate b495a49349324f7d81908f2da700f0d9 in this term.
I20251029 02:52:11.937813 2443 consensus_queue.cc:1048] T 44b518c3a0964a6fa863b2784d80c8a4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:11.954025 1052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "adf6e736cb9c4920807ee58682e134ef" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:52:11.954144 1058 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40e844fd7ad74680bcbe7bef936c5b9d" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:11.954862 1058 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "82289e0f823c4d43bfd8c4e878572b2b" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:52:11.955432 1058 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "97b2ecae572a4b7ea1b9595074556708" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:52:11.955665 1058 raft_consensus.cc:2393] T 97b2ecae572a4b7ea1b9595074556708 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b495a49349324f7d81908f2da700f0d9 in current term 1: Already voted for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in this term.
I20251029 02:52:11.956123 1058 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "27b08f88b99643d28ac609e45bf8104e" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:11.956652 1058 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c32c28b0e74a4af3a0e69bbf24ab1a1f" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:52:11.956861 1058 raft_consensus.cc:2393] T c32c28b0e74a4af3a0e69bbf24ab1a1f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate b495a49349324f7d81908f2da700f0d9 in current term 1: Already voted for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in this term.
I20251029 02:52:11.959837 1053 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "217144957930495aa3e9dfcfde268e71" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:52:11.959993 1052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6789bc16664e4b37b86702449ab27855" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:11.960191 1033 tablet_service.cc:1505] Processing CreateTablet for tablet ffe664eab28c4aa88ac299c5b5e66410 (DEFAULT_TABLE table=default.Table_75f0822a7de44b8ea84235f6790a2140 [id=67ad707678dc4b98b380fdc8a9699b80]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:11.960319 1028 tablet_service.cc:1505] Processing CreateTablet for tablet 209671f4c6284613a2d77b17688c2263 (DEFAULT_TABLE table=default.Table_f0423becd1a243ff9b81ce1b985f083e [id=aedc704e0d604bf4a113882bd22a4317]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:11.960584 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ffe664eab28c4aa88ac299c5b5e66410. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:11.960790 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 209671f4c6284613a2d77b17688c2263. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:11.963472 881 tablet_service.cc:1505] Processing CreateTablet for tablet 33c16cc408084a049232e02ecf47ebea (DEFAULT_TABLE table=default.Table_85df974f5259462298e155ee87b26acb [id=5a454671d6094643a528168ed652a4b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:11.963912 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33c16cc408084a049232e02ecf47ebea. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:11.969084 1033 tablet_service.cc:1505] Processing CreateTablet for tablet 404ae4213500460386d1edd6e66f04c3 (DEFAULT_TABLE table=default.Table_5db0b0e77f584976b6253ec87f386bb8 [id=59ae93647c29498ca382f850a9d42ac3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:11.969617 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 404ae4213500460386d1edd6e66f04c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:11.969805 1028 tablet_service.cc:1505] Processing CreateTablet for tablet b11bd7c31cbb4b5cb79b4257399b46e8 (DEFAULT_TABLE table=default.Table_7e276e1364a8470a853a68d2d977a4b4 [id=44cc9872095b4aa29f052c012a4f11f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:11.927929 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet f529e17e97074dca91ec2f48954cfd32 (table default.Table_570f159e617548ec974b41a654c1d800 [id=835aa84a7f8d426d9560b819e6864715]) successfully deleted
I20251029 02:52:11.970219 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b11bd7c31cbb4b5cb79b4257399b46e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:11.971177 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 785733da86ff425a90ccfab2933fc89e (table default.Table_bdefc7775234406788f3f7e220769b43 [id=b81476e2d6db40ef9f97211bdb79ce21]) successfully deleted
W20251029 02:52:11.971426 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 372d93914abc41b38a702742b7db0bc2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 372d93914abc41b38a702742b7db0bc2 already in progress: creating tablet
W20251029 02:52:11.971616 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 00ebd0c68c4441f9b63c86cdab62497d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 00ebd0c68c4441f9b63c86cdab62497d already in progress: creating tablet
I20251029 02:52:11.971801 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet dfa8b1b9a80e4e8689e0acb44e16346a (table default.Table_3b4bdfaf4ddf43cfa2d2287594c57816 [id=222ef1b283554565b4930328c2c8975b]) successfully deleted
W20251029 02:52:11.974011 849 consensus_peers.cc:597] T 0d84dc8dc465496f9adec798b2e03e13 P c756528642a543eeacd93cef3e335a7d -> Peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Couldn't send request to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:11.974948 1038 tablet_service.cc:1505] Processing CreateTablet for tablet d557a4750fee4c2b92eaa834619a678c (DEFAULT_TABLE table=default.Table_4873cb28e4c84935ab481f06ad654716 [id=9bc3968738ca426f8ac8dec3b513b085]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:11.975281 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d557a4750fee4c2b92eaa834619a678c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:11.984500 1024 tablet_service.cc:1505] Processing CreateTablet for tablet f58bd72cee5b46f5941a3976547e2aaa (DEFAULT_TABLE table=default.Table_f08f8ab20ebf4c9a99d5b88cf17d3b7d [id=bf42ef67f58e45c58b290c7dbb1ca205]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:11.984855 520 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet adf6e736cb9c4920807ee58682e134ef (table default.Table_0a039bfb2e1f458f89ffc3ee93300b8f [id=64650da66d464074828f2dc955045f68]). Expected version 3 got 0
I20251029 02:52:11.985050 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f58bd72cee5b46f5941a3976547e2aaa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:11.985087 1034 tablet_service.cc:1552] Processing DeleteTablet for tablet 1bdb67125493430398ff5c80b73de2ac with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:11 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:11.985473 1034 tablet_service.cc:1505] Processing CreateTablet for tablet 2965344ff7174295b7759d1b2c95ff87 (DEFAULT_TABLE table=default.Table_79052b7cce26442aa1454f10280e7e7a [id=a877951ca0224abbbc5a9112f0a598a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:11.985764 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet 81ee839d811a4f46b93a57ede397bd93 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:11 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:11.985985 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet 0f96af1cb0ea4b46bd846ae0739f4947 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:11 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:11.986198 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet a530fe8de8ae4be5b0fd9a5f7e94ea9e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:11 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:11.986431 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet 426f9be3b0074f60989acc43ec1e9bd6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:11 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:11.986749 2425 consensus_queue.cc:1048] T e7f846b294cc4eb687a26c8b5931ac19 P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:11.989045 1039 tablet_service.cc:1505] Processing CreateTablet for tablet 09567e75ea1140e5ba506a49fbb1d77e (DEFAULT_TABLE table=default.Table_a0c3ca13b0804fa399653c68e8bf4335 [id=50a069a1fc504904a54dd6f67c5a808f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:11.991958 1038 tablet_service.cc:1505] Processing CreateTablet for tablet 6408901d1e5f4ac8a7bff0de708660eb (DEFAULT_TABLE table=default.Table_7fdd50180d294d44bc4c64f2cc5dc57a [id=b7cab31fff3e439d980f98446e240547]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:11.993903 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2965344ff7174295b7759d1b2c95ff87. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:11.998764 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09567e75ea1140e5ba506a49fbb1d77e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:12.001384 2453 tablet.cc:1722] T 9269622ca92645d5aa8e73341201ccbb P c756528642a543eeacd93cef3e335a7d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20251029 02:52:12.008975 2452 tablet.cc:1722] T 9269622ca92645d5aa8e73341201ccbb P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20251029 02:52:12.012548 1174 ts_tablet_manager.cc:1929] T d2990e9f626c40618c118266bc595684 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.3
I20251029 02:52:12.012779 1174 log.cc:1199] T d2990e9f626c40618c118266bc595684 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/d2990e9f626c40618c118266bc595684
I20251029 02:52:12.013157 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6408901d1e5f4ac8a7bff0de708660eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:12.013273 1174 ts_tablet_manager.cc:1950] T d2990e9f626c40618c118266bc595684 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:12.021867 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 833) took 3717 ms. Trace:
I20251029 02:52:12.029495 1024 tablet_service.cc:1505] Processing CreateTablet for tablet a52fc7119e12435faec1a1df4e5eedef (DEFAULT_TABLE table=default.Table_0f172f8e7eb649ffa6f1723cab301cc5 [id=6c216f006b8f4aadbd4633f8bdf745b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:12.029716 1042 tablet_service.cc:1505] Processing CreateTablet for tablet 33c16cc408084a049232e02ecf47ebea (DEFAULT_TABLE table=default.Table_85df974f5259462298e155ee87b26acb [id=5a454671d6094643a528168ed652a4b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:12.029896 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a52fc7119e12435faec1a1df4e5eedef. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:12.030004 1034 tablet_service.cc:1505] Processing CreateTablet for tablet 69a1106275d4458b93a29e9834c34c68 (DEFAULT_TABLE table=default.Table_11dd5d796fa041058d7cbc7ce4ef730d [id=1c7a68ea8b5746b79a9e6988b1168133]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:12.030077 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33c16cc408084a049232e02ecf47ebea. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:12.031098 522 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 57898c0257894f899613d7578088cbb3 consensus state has no leader
I20251029 02:52:12.033439 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69a1106275d4458b93a29e9834c34c68. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:12.034590 881 tablet_service.cc:1505] Processing CreateTablet for tablet 426f9be3b0074f60989acc43ec1e9bd6 (DEFAULT_TABLE table=default.Table_5e8222bfdcd948849865e2ce542ea6a5 [id=7b81bbf86d714c9dbbf8f82e99dc2a9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:12.034729 1039 tablet_service.cc:1505] Processing CreateTablet for tablet e83ef5b888e94ff4b340074c3c9bc0ac (DEFAULT_TABLE table=default.Table_2050c80c7f844f4b9fc3aeffb54f8d6e [id=24bc2bcb1e2f4091a460cf22c15f902a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:12.035109 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 426f9be3b0074f60989acc43ec1e9bd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:12.035122 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e83ef5b888e94ff4b340074c3c9bc0ac. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:12.035128 510 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 07316861d2da4f55bd41111e400677fe (table default.Table_fdb7650da3b84059a8aa5d2dc28fb8f5 [id=7a248a5faa8346e09038e9e1c892625b], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 07316861d2da4f55bd41111e400677fe consensus state has no leader
I20251029 02:52:12.042447 1167 tablet_bootstrap.cc:492] T afb19ba39399410bb90b6225d7b5b5b1 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:12.042599 1167 ts_tablet_manager.cc:1403] T afb19ba39399410bb90b6225d7b5b5b1 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.164s user 0.009s sys 0.029s
I20251029 02:52:12.042608 1166 tablet_bootstrap.cc:492] T 731c0acfbe204e8199ad246e38ae2ccd P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:12.042721 1166 ts_tablet_manager.cc:1403] T 731c0acfbe204e8199ad246e38ae2ccd P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.171s user 0.005s sys 0.003s
I20251029 02:52:12.042660 1035 tablet_service.cc:1505] Processing CreateTablet for tablet 426f9be3b0074f60989acc43ec1e9bd6 (DEFAULT_TABLE table=default.Table_5e8222bfdcd948849865e2ce542ea6a5 [id=7b81bbf86d714c9dbbf8f82e99dc2a9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:12.043308 1036 tablet_service.cc:1505] Processing CreateTablet for tablet d07bfb380e884843ae611380a1d10b36 (DEFAULT_TABLE table=default.Table_b2aa73be4c2a45c6ac352144abf10116 [id=89d9862268394db490cf7bebb3bbf454]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:12.043606 767 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "44c0488eba0543078ff4f0b3d69366f2" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:12.043606 1167 raft_consensus.cc:359] T afb19ba39399410bb90b6225d7b5b5b1 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:12.043761 1167 raft_consensus.cc:385] T afb19ba39399410bb90b6225d7b5b5b1 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:12.043788 767 raft_consensus.cc:3060] T 44c0488eba0543078ff4f0b3d69366f2 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:12.043829 1167 raft_consensus.cc:740] T afb19ba39399410bb90b6225d7b5b5b1 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:12.043917 1166 raft_consensus.cc:359] T 731c0acfbe204e8199ad246e38ae2ccd P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:12.044068 1166 raft_consensus.cc:385] T 731c0acfbe204e8199ad246e38ae2ccd P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:12.044023 1167 consensus_queue.cc:260] T afb19ba39399410bb90b6225d7b5b5b1 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:12.044111 1166 raft_consensus.cc:740] T 731c0acfbe204e8199ad246e38ae2ccd P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:12.044317 1166 consensus_queue.cc:260] T 731c0acfbe204e8199ad246e38ae2ccd P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:12.044484 1167 ts_tablet_manager.cc:1434] T afb19ba39399410bb90b6225d7b5b5b1 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251029 02:52:12.044728 1167 tablet_bootstrap.cc:492] T e65eaaae43ac435ebd665546de9d68dd P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:12.044937 1166 ts_tablet_manager.cc:1434] T 731c0acfbe204e8199ad246e38ae2ccd P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251029 02:52:12.046319 1166 tablet_bootstrap.cc:492] T a9e0fcc759a64679b3f59a6c964877b6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:12.049724 1167 tablet_bootstrap.cc:654] T e65eaaae43ac435ebd665546de9d68dd P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:12.051947 1166 tablet_bootstrap.cc:654] T a9e0fcc759a64679b3f59a6c964877b6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:12.021970 1174 rpcz_store.cc:276] 1029 02:52:08.304120 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:08.304276 (+ 156us) service_pool.cc:225] Handling call
1029 02:52:11.652469 (+3348193us) ts_tablet_manager.cc:1151] Deleting tablet d2990e9f626c40618c118266bc595684
1029 02:52:11.652476 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:11.822377 (+169901us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:12.012455 (+190078us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:12.021843 (+ 9388us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3347961}
I20251029 02:52:12.054592 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet d2990e9f626c40618c118266bc595684 (table default.Table_da047830460443a29c5269c98fc859e9 [id=f733948e92d8470da9d8050bd26a06b1]) successfully deleted
I20251029 02:52:12.055225 1174 tablet_replica.cc:333] T c764062d3ba04d4ba8b2e367ad8ed433 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:12.055382 878 tablet_service.cc:1505] Processing CreateTablet for tablet d07bfb380e884843ae611380a1d10b36 (DEFAULT_TABLE table=default.Table_b2aa73be4c2a45c6ac352144abf10116 [id=89d9862268394db490cf7bebb3bbf454]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:12.056401 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d07bfb380e884843ae611380a1d10b36. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:12.056665 747 tablet_service.cc:1505] Processing CreateTablet for tablet e83ef5b888e94ff4b340074c3c9bc0ac (DEFAULT_TABLE table=default.Table_2050c80c7f844f4b9fc3aeffb54f8d6e [id=24bc2bcb1e2f4091a460cf22c15f902a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:12.057060 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e83ef5b888e94ff4b340074c3c9bc0ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:12.060037 751 tablet_service.cc:1505] Processing CreateTablet for tablet 426f9be3b0074f60989acc43ec1e9bd6 (DEFAULT_TABLE table=default.Table_5e8222bfdcd948849865e2ce542ea6a5 [id=7b81bbf86d714c9dbbf8f82e99dc2a9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:12.060618 745 tablet_service.cc:1505] Processing CreateTablet for tablet d07bfb380e884843ae611380a1d10b36 (DEFAULT_TABLE table=default.Table_b2aa73be4c2a45c6ac352144abf10116 [id=89d9862268394db490cf7bebb3bbf454]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:12.060964 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d07bfb380e884843ae611380a1d10b36. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:12.061666 847 consensus_peers.cc:597] T e7f846b294cc4eb687a26c8b5931ac19 P c756528642a543eeacd93cef3e335a7d -> Peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Couldn't send request to peer b495a49349324f7d81908f2da700f0d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e7f846b294cc4eb687a26c8b5931ac19. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:12.064575 2188 raft_consensus.cc:493] T afb19ba39399410bb90b6225d7b5b5b1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:12.064715 2188 raft_consensus.cc:515] T afb19ba39399410bb90b6225d7b5b5b1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:12.065639 2188 leader_election.cc:290] T afb19ba39399410bb90b6225d7b5b5b1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:12.067643 1176 ts_tablet_manager.cc:1929] T 0d84dc8dc465496f9adec798b2e03e13 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:12.067761 1176 log.cc:1199] T 0d84dc8dc465496f9adec798b2e03e13 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/0d84dc8dc465496f9adec798b2e03e13
I20251029 02:52:12.067902 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "afb19ba39399410bb90b6225d7b5b5b1" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:12.068113 1176 ts_tablet_manager.cc:1950] T 0d84dc8dc465496f9adec798b2e03e13 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:12.068145 917 raft_consensus.cc:2468] T afb19ba39399410bb90b6225d7b5b5b1 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 0.
I20251029 02:52:12.068358 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 426f9be3b0074f60989acc43ec1e9bd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:12.068609 990 leader_election.cc:304] T afb19ba39399410bb90b6225d7b5b5b1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:12.068918 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d07bfb380e884843ae611380a1d10b36. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:12.068971 2179 raft_consensus.cc:2804] T afb19ba39399410bb90b6225d7b5b5b1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:12.069152 2179 raft_consensus.cc:493] T afb19ba39399410bb90b6225d7b5b5b1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:12.069270 2179 raft_consensus.cc:3060] T afb19ba39399410bb90b6225d7b5b5b1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:12.069703 1174 raft_consensus.cc:2243] T c764062d3ba04d4ba8b2e367ad8ed433 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:12.069893 1174 raft_consensus.cc:2272] T c764062d3ba04d4ba8b2e367ad8ed433 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:12.070427 1174 ts_tablet_manager.cc:1916] T c764062d3ba04d4ba8b2e367ad8ed433 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 02:52:12.071238 712 leader_election.cc:343] T d2990e9f626c40618c118266bc595684 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Not found: Tablet not found: d2990e9f626c40618c118266bc595684
I20251029 02:52:12.071365 712 leader_election.cc:304] T d2990e9f626c40618c118266bc595684 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:12.071765 712 leader_election.cc:304] T adf6e736cb9c4920807ee58682e134ef P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
W20251029 02:52:12.072099 712 leader_election.cc:343] T 40e844fd7ad74680bcbe7bef936c5b9d P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:12.072204 712 leader_election.cc:304] T 40e844fd7ad74680bcbe7bef936c5b9d P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:12.072630 2403 raft_consensus.cc:2749] T d2990e9f626c40618c118266bc595684 P b495a49349324f7d81908f2da700f0d9 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251029 02:52:12.072780 2403 raft_consensus.cc:2749] T adf6e736cb9c4920807ee58682e134ef P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251029 02:52:12.072921 2403 raft_consensus.cc:2749] T 40e844fd7ad74680bcbe7bef936c5b9d P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:12.072525 712 leader_election.cc:304] T 82289e0f823c4d43bfd8c4e878572b2b P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
W20251029 02:52:12.080974 712 leader_election.cc:343] T 27b08f88b99643d28ac609e45bf8104e P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:12.081079 712 leader_election.cc:304] T 27b08f88b99643d28ac609e45bf8104e P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:12.081528 712 leader_election.cc:304] T 217144957930495aa3e9dfcfde268e71 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
W20251029 02:52:12.081856 712 leader_election.cc:343] T 6789bc16664e4b37b86702449ab27855 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:12.081938 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 426f9be3b0074f60989acc43ec1e9bd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:12.082001 2415 raft_consensus.cc:2749] T 82289e0f823c4d43bfd8c4e878572b2b P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251029 02:52:12.082170 2415 raft_consensus.cc:2749] T 27b08f88b99643d28ac609e45bf8104e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:12.082262 2415 raft_consensus.cc:2749] T 217144957930495aa3e9dfcfde268e71 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251029 02:52:12.081938 712 leader_election.cc:304] T 6789bc16664e4b37b86702449ab27855 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:12.083539 2403 raft_consensus.cc:2749] T 6789bc16664e4b37b86702449ab27855 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:12.083860 1175 ts_tablet_manager.cc:1929] T 20a762344c5f46f88481c4ac7ad6e43b P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:12.083974 1175 log.cc:1199] T 20a762344c5f46f88481c4ac7ad6e43b P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/20a762344c5f46f88481c4ac7ad6e43b
I20251029 02:52:12.084350 1175 ts_tablet_manager.cc:1950] T 20a762344c5f46f88481c4ac7ad6e43b P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:12.085453 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 818) took 3810 ms. Trace:
I20251029 02:52:12.088323 989 leader_election.cc:304] T 2f198c3259864f7ab3625cb7bfae20b0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters: c756528642a543eeacd93cef3e335a7d
I20251029 02:52:12.085553 1175 rpcz_store.cc:276] 1029 02:52:08.275276 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:08.275846 (+ 570us) service_pool.cc:225] Handling call
1029 02:52:11.811269 (+3535423us) ts_tablet_manager.cc:1151] Deleting tablet 20a762344c5f46f88481c4ac7ad6e43b
1029 02:52:11.811278 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:12.005610 (+194332us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:12.009178 (+ 3568us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:12.085426 (+ 76248us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3535283}
I20251029 02:52:12.089311 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 20a762344c5f46f88481c4ac7ad6e43b (table default.Table_3ca4a6512b174cb897b142dfe1259549 [id=a862192912a04942a8f682fc9825b014]) successfully deleted
I20251029 02:52:12.089457 2188 raft_consensus.cc:2804] T 2f198c3259864f7ab3625cb7bfae20b0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:12.089578 2188 raft_consensus.cc:697] T 2f198c3259864f7ab3625cb7bfae20b0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Becoming Leader. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Running, Role: LEADER
I20251029 02:52:12.089787 2188 consensus_queue.cc:237] T 2f198c3259864f7ab3625cb7bfae20b0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:12.090356 2452 tablet.cc:1722] T 44b518c3a0964a6fa863b2784d80c8a4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 02:52:12.090869 2453 tablet.cc:1722] T 44b518c3a0964a6fa863b2784d80c8a4 P c756528642a543eeacd93cef3e335a7d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 02:52:12.091621 738 tablet_service.cc:1505] Processing CreateTablet for tablet 33c16cc408084a049232e02ecf47ebea (DEFAULT_TABLE table=default.Table_85df974f5259462298e155ee87b26acb [id=5a454671d6094643a528168ed652a4b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:12.092000 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33c16cc408084a049232e02ecf47ebea. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:12.092446 767 raft_consensus.cc:2468] T 44c0488eba0543078ff4f0b3d69366f2 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 1.
I20251029 02:52:12.092561 2113 raft_consensus.cc:493] T 32519fbffcef44a890161e5f3043806c P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:12.092659 2113 raft_consensus.cc:515] T 32519fbffcef44a890161e5f3043806c P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:12.093214 2113 leader_election.cc:290] T 32519fbffcef44a890161e5f3043806c P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:12.096040 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "32519fbffcef44a890161e5f3043806c" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:12.097513 1175 tablet_replica.cc:333] T 738046ab57334f2dab9f4aab979c5b42 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:12.097558 989 leader_election.cc:304] T 97b2ecae572a4b7ea1b9595074556708 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5; no voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:12.097813 1175 raft_consensus.cc:2243] T 738046ab57334f2dab9f4aab979c5b42 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:12.097901 989 leader_election.cc:304] T c32c28b0e74a4af3a0e69bbf24ab1a1f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5; no voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:12.098064 1175 raft_consensus.cc:2272] T 738046ab57334f2dab9f4aab979c5b42 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:12.098789 1175 ts_tablet_manager.cc:1916] T 738046ab57334f2dab9f4aab979c5b42 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:12.099182 2188 raft_consensus.cc:2749] T 97b2ecae572a4b7ea1b9595074556708 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251029 02:52:12.099349 2188 raft_consensus.cc:2749] T c32c28b0e74a4af3a0e69bbf24ab1a1f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20251029 02:52:12.099295 767 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "afb19ba39399410bb90b6225d7b5b5b1" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
W20251029 02:52:12.099689 711 leader_election.cc:343] T 32519fbffcef44a890161e5f3043806c P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:12.106003 767 raft_consensus.cc:2468] T afb19ba39399410bb90b6225d7b5b5b1 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 0.
I20251029 02:52:12.102278 1172 tablet_bootstrap.cc:654] T ae4231daa2594258aa51419c97a2f50a P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:12.108446 1167 tablet_bootstrap.cc:492] T e65eaaae43ac435ebd665546de9d68dd P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:12.108597 1167 ts_tablet_manager.cc:1403] T e65eaaae43ac435ebd665546de9d68dd P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.064s user 0.007s sys 0.014s
I20251029 02:52:12.109107 2452 tablet.cc:1722] T cc74e83833754f0ca19046031a24ebb9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 02:52:12.110153 1167 raft_consensus.cc:359] T e65eaaae43ac435ebd665546de9d68dd P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:12.110392 1167 raft_consensus.cc:385] T e65eaaae43ac435ebd665546de9d68dd P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:12.110482 1167 raft_consensus.cc:740] T e65eaaae43ac435ebd665546de9d68dd P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:12.110757 1167 consensus_queue.cc:260] T e65eaaae43ac435ebd665546de9d68dd P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:12.111496 2453 tablet.cc:1722] T cc74e83833754f0ca19046031a24ebb9 P c756528642a543eeacd93cef3e335a7d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 02:52:12.115856 1167 ts_tablet_manager.cc:1434] T e65eaaae43ac435ebd665546de9d68dd P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20251029 02:52:12.118558 1058 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "32519fbffcef44a890161e5f3043806c" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:52:12.119014 712 leader_election.cc:343] T 32519fbffcef44a890161e5f3043806c P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:12.119212 712 leader_election.cc:304] T 32519fbffcef44a890161e5f3043806c P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:12.119467 2113 raft_consensus.cc:2749] T 32519fbffcef44a890161e5f3043806c P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:12.124361 2302 raft_consensus.cc:493] T 643e20d9974447daa78885b3401f3b77 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:12.124555 2302 raft_consensus.cc:515] T 643e20d9974447daa78885b3401f3b77 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:12.125300 2302 leader_election.cc:290] T 643e20d9974447daa78885b3401f3b77 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:12.126081 1058 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "643e20d9974447daa78885b3401f3b77" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:12.126117 767 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "643e20d9974447daa78885b3401f3b77" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:12.126258 1058 raft_consensus.cc:2468] T 643e20d9974447daa78885b3401f3b77 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:12.126330 767 raft_consensus.cc:2468] T 643e20d9974447daa78885b3401f3b77 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:12.126642 849 leader_election.cc:304] T 643e20d9974447daa78885b3401f3b77 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:12.127069 2302 raft_consensus.cc:2804] T 643e20d9974447daa78885b3401f3b77 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:12.127139 2302 raft_consensus.cc:493] T 643e20d9974447daa78885b3401f3b77 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:12.127264 2302 raft_consensus.cc:3060] T 643e20d9974447daa78885b3401f3b77 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:12.129097 1174 ts_tablet_manager.cc:1929] T c764062d3ba04d4ba8b2e367ad8ed433 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:12.129231 1174 log.cc:1199] T c764062d3ba04d4ba8b2e367ad8ed433 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/c764062d3ba04d4ba8b2e367ad8ed433
I20251029 02:52:12.136119 1175 ts_tablet_manager.cc:1929] T 738046ab57334f2dab9f4aab979c5b42 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:12.136240 1175 log.cc:1199] T 738046ab57334f2dab9f4aab979c5b42 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/738046ab57334f2dab9f4aab979c5b42
I20251029 02:52:12.136592 1175 ts_tablet_manager.cc:1950] T 738046ab57334f2dab9f4aab979c5b42 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:12.137322 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 819) took 3861 ms. Trace:
I20251029 02:52:12.137432 1175 rpcz_store.cc:276] 1029 02:52:08.276108 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:08.276247 (+ 139us) service_pool.cc:225] Handling call
1029 02:52:12.089076 (+3812829us) ts_tablet_manager.cc:1151] Deleting tablet 738046ab57334f2dab9f4aab979c5b42
1029 02:52:12.089086 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:12.133362 (+ 44276us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:12.136054 (+ 2692us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:12.137301 (+ 1247us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3812695}
I20251029 02:52:12.137754 1175 tablet_replica.cc:333] T 061f3d38c4604fa4b57e74ff6f6fc7da P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:12.137933 1175 raft_consensus.cc:2243] T 061f3d38c4604fa4b57e74ff6f6fc7da P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:12.137996 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 738046ab57334f2dab9f4aab979c5b42 (table default.Table_31d42031714e415fab63cc5b9933fe0c [id=33b26e2658ff4270a5f1306b6cde34db]) successfully deleted
I20251029 02:52:12.138095 1175 raft_consensus.cc:2272] T 061f3d38c4604fa4b57e74ff6f6fc7da P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:12.145478 1175 ts_tablet_manager.cc:1916] T 061f3d38c4604fa4b57e74ff6f6fc7da P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 02:52:12.146813 2447 log.cc:927] Time spent T 9269622ca92645d5aa8e73341201ccbb P b495a49349324f7d81908f2da700f0d9: Append to log took a long time: real 0.056s user 0.000s sys 0.000s
I20251029 02:52:12.154181 2302 raft_consensus.cc:515] T 643e20d9974447daa78885b3401f3b77 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:12.154920 2302 leader_election.cc:290] T 643e20d9974447daa78885b3401f3b77 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:12.155078 1174 ts_tablet_manager.cc:1950] T c764062d3ba04d4ba8b2e367ad8ed433 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:12.155217 1058 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "643e20d9974447daa78885b3401f3b77" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:52:12.155452 1058 raft_consensus.cc:3060] T 643e20d9974447daa78885b3401f3b77 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:12.158720 767 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "643e20d9974447daa78885b3401f3b77" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:12.158941 767 raft_consensus.cc:3060] T 643e20d9974447daa78885b3401f3b77 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:12.161597 767 raft_consensus.cc:2468] T 643e20d9974447daa78885b3401f3b77 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:12.162092 847 leader_election.cc:304] T 643e20d9974447daa78885b3401f3b77 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:12.162679 2302 raft_consensus.cc:2804] T 643e20d9974447daa78885b3401f3b77 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:12.163029 2302 raft_consensus.cc:697] T 643e20d9974447daa78885b3401f3b77 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:52:12.163379 2302 consensus_queue.cc:237] T 643e20d9974447daa78885b3401f3b77 P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:12.162711 1058 raft_consensus.cc:2468] T 643e20d9974447daa78885b3401f3b77 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:12.179852 1175 ts_tablet_manager.cc:1929] T 061f3d38c4604fa4b57e74ff6f6fc7da P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:12.180009 1175 log.cc:1199] T 061f3d38c4604fa4b57e74ff6f6fc7da P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/061f3d38c4604fa4b57e74ff6f6fc7da
I20251029 02:52:12.180368 1175 ts_tablet_manager.cc:1950] T 061f3d38c4604fa4b57e74ff6f6fc7da P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:12.181232 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 820) took 3905 ms. Trace:
I20251029 02:52:12.181349 1175 rpcz_store.cc:276] 1029 02:52:08.276208 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:08.276396 (+ 188us) service_pool.cc:225] Handling call
1029 02:52:12.137701 (+3861305us) ts_tablet_manager.cc:1151] Deleting tablet 061f3d38c4604fa4b57e74ff6f6fc7da
1029 02:52:12.137713 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:12.172522 (+ 34809us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:12.179706 (+ 7184us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:12.181203 (+ 1497us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3861170}
I20251029 02:52:12.181902 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 061f3d38c4604fa4b57e74ff6f6fc7da (table default.Table_0f40dc35be6d46b3a991e93384fa1e1f [id=7f5f847a87ab464b932a46f3c04e43d6]) successfully deleted
I20251029 02:52:12.200757 518 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_4a6906297ac74f45b5f153339e2713c3" } modify_external_catalogs: true
I20251029 02:52:12.201028 518 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_4a6906297ac74f45b5f153339e2713c3" } modify_external_catalogs: true
I20251029 02:52:12.202909 517 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_b594044b034c485b886a94f8699bb935" } modify_external_catalogs: true
I20251029 02:52:12.203094 517 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_b594044b034c485b886a94f8699bb935" } modify_external_catalogs: true
I20251029 02:52:12.204461 523 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_3dd2917370f84ddd901c89c68f34c97d" } new_table_name: "default.Table_dd29d68d6ddd4157bc66b4ff9b54b33f" modify_external_catalogs: true
I20251029 02:52:12.205847 521 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_ffc17029e0734c73a3735b9f99cf1332" } modify_external_catalogs: true
I20251029 02:52:12.206079 521 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_ffc17029e0734c73a3735b9f99cf1332" } modify_external_catalogs: true
W20251029 02:52:12.207319 519 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3e9a076e086141ef8ed59d4c8633f961 has no consensus state
I20251029 02:52:12.207638 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_2f9f5508a47d486ca4004b3c2b50013d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:12.208209 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2f9f5508a47d486ca4004b3c2b50013d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:12.211385 524 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_aec07800b35948bba7db9168286fcf14" } modify_external_catalogs: true
I20251029 02:52:12.211567 524 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_aec07800b35948bba7db9168286fcf14" } modify_external_catalogs: true
I20251029 02:52:12.214385 1167 tablet_bootstrap.cc:492] T 2f198c3259864f7ab3625cb7bfae20b0 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:12.215411 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 834) took 3870 ms. Trace:
I20251029 02:52:12.215559 1174 rpcz_store.cc:276] 1029 02:52:08.345042 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:08.345094 (+ 52us) service_pool.cc:225] Handling call
1029 02:52:12.055171 (+3710077us) ts_tablet_manager.cc:1151] Deleting tablet c764062d3ba04d4ba8b2e367ad8ed433
1029 02:52:12.055183 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:12.075904 (+ 20721us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:12.128953 (+ 53049us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:12.215331 (+ 86378us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3709634}
I20251029 02:52:12.215920 1174 tablet_replica.cc:333] T dfa8b1b9a80e4e8689e0acb44e16346a P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:12.216111 1167 tablet_bootstrap.cc:654] T 2f198c3259864f7ab3625cb7bfae20b0 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:12.216172 1174 raft_consensus.cc:2243] T dfa8b1b9a80e4e8689e0acb44e16346a P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:12.216540 1174 raft_consensus.cc:2272] T dfa8b1b9a80e4e8689e0acb44e16346a P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:12.217121 1174 ts_tablet_manager.cc:1916] T dfa8b1b9a80e4e8689e0acb44e16346a P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:12.219019 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_47341e4770a74239b50cbc23e77a82e4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:12.219575 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_47341e4770a74239b50cbc23e77a82e4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:12.220257 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet c764062d3ba04d4ba8b2e367ad8ed433 (table default.Table_4e9d94609b4a4d1f94615dcc9830d717 [id=761ebdd0095c414488aa221cbc99d3bc]) successfully deleted
I20251029 02:52:12.233253 520 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_d844249d525b48428ebf40e1d6cb6c51" } modify_external_catalogs: true
I20251029 02:52:12.235432 526 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet 44b518c3a0964a6fa863b2784d80c8a4 (table default.Table_9120016827e946cc8a145252d2569d2b [id=e0c0cf3d456444c191a32bb180e83798]). Expected version 1 got 0
I20251029 02:52:12.235981 526 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet 9269622ca92645d5aa8e73341201ccbb (table default.Table_902ea0b243c7400c86250cfb5fae5b54 [id=d0ca7c93c8cf4dc19962a1957321c5b7]). Expected version 5 got 0
I20251029 02:52:12.236281 526 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet 95f342d1cfb84c4eb31bd3d15ea3bdc2 (table default.Table_96ea715e50e74cc48759dda025a46318 [id=c63d52811db749d6badbed72fc133943]). Expected version 1 got 0
I20251029 02:52:12.236577 526 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet cc74e83833754f0ca19046031a24ebb9 (table default.Table_6563934f48114b0987d298bfad3e9bf7 [id=57471769276f44299eae50b9d854b50c]). Expected version 2 got 0
I20251029 02:52:12.236783 526 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet c1222e30bc4141b3ae686d80053c4ef5 (table default.Table_7125184a813c44d997ab9d948a1156bd [id=31801defa6db4ca7a08fbf7f029b65c1]). Expected version 3 got 0
I20251029 02:52:12.237370 526 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet 82289e0f823c4d43bfd8c4e878572b2b (table default.Table_e7bb3e262a894c2f84b6edfc3917b30d [id=2cca3b7c50b243b5883b8ac1f353f253]). Expected version 1 got 0
I20251029 02:52:12.238724 526 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet adf6e736cb9c4920807ee58682e134ef (table default.Table_0a039bfb2e1f458f89ffc3ee93300b8f [id=64650da66d464074828f2dc955045f68]). Expected version 3 got 0
I20251029 02:52:12.240288 878 tablet_service.cc:1552] Processing DeleteTablet for tablet 44c0488eba0543078ff4f0b3d69366f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:59 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:12.240473 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 217144957930495aa3e9dfcfde268e71 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:01 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:12.245040 1167 tablet_bootstrap.cc:492] T 2f198c3259864f7ab3625cb7bfae20b0 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:12.245190 1167 ts_tablet_manager.cc:1403] T 2f198c3259864f7ab3625cb7bfae20b0 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.031s user 0.001s sys 0.007s
I20251029 02:52:12.245959 1167 raft_consensus.cc:359] T 2f198c3259864f7ab3625cb7bfae20b0 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:12.246153 1167 raft_consensus.cc:385] T 2f198c3259864f7ab3625cb7bfae20b0 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:12.246235 1167 raft_consensus.cc:740] T 2f198c3259864f7ab3625cb7bfae20b0 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:12.246490 1167 consensus_queue.cc:260] T 2f198c3259864f7ab3625cb7bfae20b0 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:12.249214 1167 ts_tablet_manager.cc:1434] T 2f198c3259864f7ab3625cb7bfae20b0 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20251029 02:52:12.250298 1167 tablet_bootstrap.cc:492] T c1222e30bc4141b3ae686d80053c4ef5 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:12.251896 1167 tablet_bootstrap.cc:654] T c1222e30bc4141b3ae686d80053c4ef5 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:12.234372 2179 raft_consensus.cc:515] T afb19ba39399410bb90b6225d7b5b5b1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:12.253644 2179 leader_election.cc:290] T afb19ba39399410bb90b6225d7b5b5b1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Requested vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:12.254139 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "afb19ba39399410bb90b6225d7b5b5b1" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
I20251029 02:52:12.254354 917 raft_consensus.cc:3060] T afb19ba39399410bb90b6225d7b5b5b1 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:12.255009 896 tablet_service.cc:1552] Processing DeleteTablet for tablet 44ae2b07edca44afbe1b1fb3a5b1115d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:00 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:12.256976 917 raft_consensus.cc:2468] T afb19ba39399410bb90b6225d7b5b5b1 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 1.
I20251029 02:52:12.257579 990 leader_election.cc:304] T afb19ba39399410bb90b6225d7b5b5b1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:12.258452 2179 raft_consensus.cc:2804] T afb19ba39399410bb90b6225d7b5b5b1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:12.258752 2179 raft_consensus.cc:697] T afb19ba39399410bb90b6225d7b5b5b1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Becoming Leader. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Running, Role: LEADER
I20251029 02:52:12.259045 2179 consensus_queue.cc:237] T afb19ba39399410bb90b6225d7b5b5b1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:12.261997 2201 consensus_queue.cc:1048] T 71b4b91071a54d6aa66160e7955bc915 P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:12.262137 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 913) took 2220 ms. Trace:
I20251029 02:52:12.262279 1176 rpcz_store.cc:276] 1029 02:52:10.041551 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:10.041598 (+ 47us) service_pool.cc:225] Handling call
1029 02:52:11.877036 (+1835438us) ts_tablet_manager.cc:1151] Deleting tablet 0d84dc8dc465496f9adec798b2e03e13
1029 02:52:11.877044 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:11.981961 (+104917us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:12.067542 (+ 85581us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:12.262094 (+194552us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1835266}
I20251029 02:52:12.266467 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 0d84dc8dc465496f9adec798b2e03e13 (table default.Table_0d5730d385074a70b295b233e6ecd79e [id=423993f75a9948aebe1dee01d3e12702]) successfully deleted
I20251029 02:52:12.267061 1176 tablet_replica.cc:333] T 59d03a190b784ab18a3e63a572675891 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:12.267467 1176 raft_consensus.cc:2243] T 59d03a190b784ab18a3e63a572675891 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:12.267856 1176 raft_consensus.cc:2272] T 59d03a190b784ab18a3e63a572675891 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:12.269719 1176 ts_tablet_manager.cc:1916] T 59d03a190b784ab18a3e63a572675891 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:12.279728 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "afb19ba39399410bb90b6225d7b5b5b1" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:12.280540 772 raft_consensus.cc:3060] T afb19ba39399410bb90b6225d7b5b5b1 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:12.285012 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 817) took 4009 ms. Trace:
I20251029 02:52:12.285378 1175 rpcz_store.cc:276] 1029 02:52:08.275173 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:08.276386 (+ 1213us) service_pool.cc:225] Handling call
1029 02:52:12.284504 (+4008118us) ts_tablet_manager.cc:1151] Deleting tablet 50ab70b30aaa4f6bbae15855431159c8
1029 02:52:12.284564 (+ 60us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:12.284937 (+ 373us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4007837}
W20251029 02:52:12.286330 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 50ab70b30aaa4f6bbae15855431159c8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 50ab70b30aaa4f6bbae15855431159c8 already in progress: creating tablet
I20251029 02:52:12.286972 1175 tablet_replica.cc:333] T 30d03e1e42d34242bd354f9e9ec4f127 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:12.289093 2443 raft_consensus.cc:493] T 44ae2b07edca44afbe1b1fb3a5b1115d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:12.289248 2443 raft_consensus.cc:515] T 44ae2b07edca44afbe1b1fb3a5b1115d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:12.289979 2443 leader_election.cc:290] T 44ae2b07edca44afbe1b1fb3a5b1115d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:12.290584 767 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "44ae2b07edca44afbe1b1fb3a5b1115d" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:12.290665 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "44ae2b07edca44afbe1b1fb3a5b1115d" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:12.290810 767 raft_consensus.cc:2393] T 44ae2b07edca44afbe1b1fb3a5b1115d P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in current term 1: Already voted for candidate c756528642a543eeacd93cef3e335a7d in this term.
I20251029 02:52:12.291199 989 leader_election.cc:304] T 44ae2b07edca44afbe1b1fb3a5b1115d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5; no voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:12.291608 2443 raft_consensus.cc:3060] T 44ae2b07edca44afbe1b1fb3a5b1115d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:12.294641 2113 raft_consensus.cc:493] T 372d93914abc41b38a702742b7db0bc2 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:12.294824 2113 raft_consensus.cc:515] T 372d93914abc41b38a702742b7db0bc2 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:12.294986 2443 raft_consensus.cc:2749] T 44ae2b07edca44afbe1b1fb3a5b1115d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:12.295567 2113 leader_election.cc:290] T 372d93914abc41b38a702742b7db0bc2 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:12.299446 1058 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "372d93914abc41b38a702742b7db0bc2" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:52:12.300315 712 leader_election.cc:343] T 372d93914abc41b38a702742b7db0bc2 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:12.302367 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "372d93914abc41b38a702742b7db0bc2" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:12.302922 711 leader_election.cc:343] T 372d93914abc41b38a702742b7db0bc2 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:12.303026 711 leader_election.cc:304] T 372d93914abc41b38a702742b7db0bc2 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:12.303449 2113 raft_consensus.cc:2749] T 372d93914abc41b38a702742b7db0bc2 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:12.305310 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet 44c0488eba0543078ff4f0b3d69366f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:59 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:12.305809 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet 0d84dc8dc465496f9adec798b2e03e13 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:56 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:12.306252 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet 2f198c3259864f7ab3625cb7bfae20b0 with delete_type TABLET_DATA_DELETED (replaced by tablet 877e278deb4a4b01ac66b06be0890baf) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:12.309024 1174 ts_tablet_manager.cc:1929] T dfa8b1b9a80e4e8689e0acb44e16346a P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:12.309424 1174 log.cc:1199] T dfa8b1b9a80e4e8689e0acb44e16346a P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/dfa8b1b9a80e4e8689e0acb44e16346a
I20251029 02:52:12.324340 2302 raft_consensus.cc:493] T ccdca31db2834de4b6ed0f2d67a10535 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0cfd4bee16f84b7c8b0ff3ce14db20d5)
I20251029 02:52:12.324513 2302 raft_consensus.cc:515] T ccdca31db2834de4b6ed0f2d67a10535 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:12.325130 2302 leader_election.cc:290] T ccdca31db2834de4b6ed0f2d67a10535 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:12.325192 517 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet fd1bf6b0f45349d0981f9ddd89269489
I20251029 02:52:12.325973 896 tablet_service.cc:1552] Processing DeleteTablet for tablet fd1bf6b0f45349d0981f9ddd89269489 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:12 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:12.326164 518 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 1150d2210a234f1aa554f486cdcabc4c
I20251029 02:52:12.326653 520 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_d844249d525b48428ebf40e1d6cb6c51" } modify_external_catalogs: true
W20251029 02:52:12.326907 849 leader_election.cc:343] T ccdca31db2834de4b6ed0f2d67a10535 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Not found: Tablet not found: ccdca31db2834de4b6ed0f2d67a10535
W20251029 02:52:12.327653 847 leader_election.cc:343] T ccdca31db2834de4b6ed0f2d67a10535 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: ccdca31db2834de4b6ed0f2d67a10535
I20251029 02:52:12.327764 847 leader_election.cc:304] T ccdca31db2834de4b6ed0f2d67a10535 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9
I20251029 02:52:12.328183 2302 raft_consensus.cc:2749] T ccdca31db2834de4b6ed0f2d67a10535 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251029 02:52:12.328449 1034 tablet_service.cc:1552] Processing DeleteTablet for tablet 1150d2210a234f1aa554f486cdcabc4c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:12 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:12.328562 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet fd1bf6b0f45349d0981f9ddd89269489 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:12 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:12.329233 896 tablet_service.cc:1552] Processing DeleteTablet for tablet 1150d2210a234f1aa554f486cdcabc4c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:12 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:12.330176 738 tablet_service.cc:1552] Processing DeleteTablet for tablet 1150d2210a234f1aa554f486cdcabc4c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:12 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:12.331343 518 master_service.cc:954] ReplaceTablet: received request to replace tablet f281191e7c6143f983f11d87f1c9b64e from {username='slave'} at 127.0.0.1:35314
I20251029 02:52:12.331164 738 tablet_service.cc:1552] Processing DeleteTablet for tablet fd1bf6b0f45349d0981f9ddd89269489 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:12 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:12.344177 1166 tablet_bootstrap.cc:492] T a9e0fcc759a64679b3f59a6c964877b6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:12.344327 1166 ts_tablet_manager.cc:1403] T a9e0fcc759a64679b3f59a6c964877b6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.299s user 0.000s sys 0.005s
I20251029 02:52:12.344992 1166 raft_consensus.cc:359] T a9e0fcc759a64679b3f59a6c964877b6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:12.345148 1166 raft_consensus.cc:385] T a9e0fcc759a64679b3f59a6c964877b6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:12.345218 1166 raft_consensus.cc:740] T a9e0fcc759a64679b3f59a6c964877b6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:12.345412 1166 consensus_queue.cc:260] T a9e0fcc759a64679b3f59a6c964877b6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:12.346184 1166 ts_tablet_manager.cc:1434] T a9e0fcc759a64679b3f59a6c964877b6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251029 02:52:12.351020 2469 tablet.cc:1722] T 02e4588b03144ea2850e13a1f8f29f06 P b495a49349324f7d81908f2da700f0d9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251029 02:52:12.355899 1174 ts_tablet_manager.cc:1950] T dfa8b1b9a80e4e8689e0acb44e16346a P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:12.356997 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 835) took 3993 ms. Trace:
I20251029 02:52:12.357131 1174 rpcz_store.cc:276] 1029 02:52:08.363416 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:08.363482 (+ 66us) service_pool.cc:225] Handling call
1029 02:52:12.215843 (+3852361us) ts_tablet_manager.cc:1151] Deleting tablet dfa8b1b9a80e4e8689e0acb44e16346a
1029 02:52:12.215853 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:12.222421 (+ 6568us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:12.308633 (+ 86212us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:12.356968 (+ 48335us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3852114}
I20251029 02:52:12.357892 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet dfa8b1b9a80e4e8689e0acb44e16346a (table default.Table_3b4bdfaf4ddf43cfa2d2287594c57816 [id=222ef1b283554565b4930328c2c8975b]) successfully deleted
W20251029 02:52:12.365007 523 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 31ca9012ddf6401ba78888a0c82b4ef2 has no consensus state
I20251029 02:52:12.365365 525 catalog_manager.cc:5649] T 643e20d9974447daa78885b3401f3b77 P c756528642a543eeacd93cef3e335a7d reported cstate change: term changed from 0 to 1, leader changed from <none> to c756528642a543eeacd93cef3e335a7d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "c756528642a543eeacd93cef3e335a7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } health_report { overall_health: UNKNOWN } } }
I20251029 02:52:12.365687 521 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 41cfa2d722034618a05de717b28a778d
I20251029 02:52:12.365837 525 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet 643e20d9974447daa78885b3401f3b77 (table default.Table_7c1cb64e05be4f6e8ac28286c982a3e7 [id=b4fc453bf54d45c4b8dddc3c49bf48f8]). Expected version 2 got 0
I20251029 02:52:12.366870 519 master_service.cc:954] ReplaceTablet: received request to replace tablet 2f198c3259864f7ab3625cb7bfae20b0 from {username='slave'} at 127.0.0.1:35314
I20251029 02:52:12.368978 738 tablet_service.cc:1552] Processing DeleteTablet for tablet 41cfa2d722034618a05de717b28a778d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:12 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:12.369477 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet 41cfa2d722034618a05de717b28a778d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:12 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:12.370664 517 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_5d956728da054438a1bd400dac5dac1e" } new_table_name: "default.Table_f2291437414b4849a6e2443780dd1c78" modify_external_catalogs: true
I20251029 02:52:12.372689 881 tablet_service.cc:1552] Processing DeleteTablet for tablet 41cfa2d722034618a05de717b28a778d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:12 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:12.373986 524 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 99d60ef5325e45059026596a756448de
I20251029 02:52:12.374742 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_2c4f1597c5d34f11a1bd7ee6550803fc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:12.374891 849 consensus_peers.cc:597] T 71b4b91071a54d6aa66160e7955bc915 P c756528642a543eeacd93cef3e335a7d -> Peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Couldn't send request to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 71b4b91071a54d6aa66160e7955bc915. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:12.374964 1166 tablet_bootstrap.cc:492] T 59eee86f249e4d6a9d09d30276dcd659 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
W20251029 02:52:12.375267 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2c4f1597c5d34f11a1bd7ee6550803fc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:12.376529 1166 tablet_bootstrap.cc:654] T 59eee86f249e4d6a9d09d30276dcd659 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:12.377032 738 tablet_service.cc:1552] Processing DeleteTablet for tablet 99d60ef5325e45059026596a756448de with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:12 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:12.377835 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet 99d60ef5325e45059026596a756448de with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:12 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:12.378608 523 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_51e08a9cce7d4d9f9b358a693206462f" } new_table_name: "default.Table_d114e6c0e5ee4562874991a545902919" modify_external_catalogs: true
I20251029 02:52:12.381507 881 tablet_service.cc:1552] Processing DeleteTablet for tablet 99d60ef5325e45059026596a756448de with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:12 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:12.381685 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_80d1a54a30fa423f809bef46b9301f29"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:12.382354 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_80d1a54a30fa423f809bef46b9301f29 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:12.387342 2462 tablet.cc:1722] T cc74e83833754f0ca19046031a24ebb9 P b495a49349324f7d81908f2da700f0d9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 02:52:12.392115 2467 tablet.cc:1722] T 44b518c3a0964a6fa863b2784d80c8a4 P b495a49349324f7d81908f2da700f0d9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 02:52:12.392827 2466 tablet.cc:1722] T cedfb53078d94c81b5e294ce7438325e P b495a49349324f7d81908f2da700f0d9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 02:52:12.395604 2113 raft_consensus.cc:493] T 46cd75ff18b14212b556b9bcb3012dd9 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:12.395728 2113 raft_consensus.cc:515] T 46cd75ff18b14212b556b9bcb3012dd9 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:12.396274 2113 leader_election.cc:290] T 46cd75ff18b14212b556b9bcb3012dd9 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:12.396960 1052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "46cd75ff18b14212b556b9bcb3012dd9" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:12.397382 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "46cd75ff18b14212b556b9bcb3012dd9" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:12.397892 711 leader_election.cc:343] T 46cd75ff18b14212b556b9bcb3012dd9 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
W20251029 02:52:12.398308 712 leader_election.cc:343] T 46cd75ff18b14212b556b9bcb3012dd9 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:12.398406 712 leader_election.cc:304] T 46cd75ff18b14212b556b9bcb3012dd9 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:12.398764 2113 raft_consensus.cc:2749] T 46cd75ff18b14212b556b9bcb3012dd9 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:12.406737 2466 tablet.cc:1722] T 9269622ca92645d5aa8e73341201ccbb P b495a49349324f7d81908f2da700f0d9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20251029 02:52:12.416411 1172 tablet_bootstrap.cc:492] T ae4231daa2594258aa51419c97a2f50a P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:12.423254 1172 ts_tablet_manager.cc:1403] T ae4231daa2594258aa51419c97a2f50a P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.542s user 0.009s sys 0.007s
I20251029 02:52:12.423420 522 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet 02e4588b03144ea2850e13a1f8f29f06 (table default.Table_0772f9db5d6142d988f6ffca2e96bdd2 [id=fd6d31d379394857a35422211af00642]). Expected version 3 got 0
I20251029 02:52:12.424172 522 master_service.cc:954] ReplaceTablet: received request to replace tablet 7dbc9c8b110a48f0a801ad468738c3a3 from {username='slave'} at 127.0.0.1:35314
I20251029 02:52:12.424679 522 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_316affb3c558411eb1aa14af326a27c2" } new_table_name: "default.Table_0161c19a25d44c07a9e403b2e619f210" modify_external_catalogs: true
I20251029 02:52:12.425302 1172 raft_consensus.cc:359] T ae4231daa2594258aa51419c97a2f50a P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:12.425460 1172 raft_consensus.cc:385] T ae4231daa2594258aa51419c97a2f50a P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:12.425508 1172 raft_consensus.cc:740] T ae4231daa2594258aa51419c97a2f50a P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:12.426389 2179 raft_consensus.cc:493] T 731c0acfbe204e8199ad246e38ae2ccd P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:12.426558 2179 raft_consensus.cc:515] T 731c0acfbe204e8199ad246e38ae2ccd P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:12.427160 2179 leader_election.cc:290] T 731c0acfbe204e8199ad246e38ae2ccd P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:12.428375 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "731c0acfbe204e8199ad246e38ae2ccd" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:12.428974 990 leader_election.cc:343] T 731c0acfbe204e8199ad246e38ae2ccd P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
W20251029 02:52:12.429070 523 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9c6f4dc4b0c74ddd80d9dc762619b1ec consensus state has no leader
I20251029 02:52:12.429383 766 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "731c0acfbe204e8199ad246e38ae2ccd" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:12.429438 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_a5ee816bf96c4eb891cb8ba5ed50f3ed"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:12.429636 766 raft_consensus.cc:2468] T 731c0acfbe204e8199ad246e38ae2ccd P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 0.
W20251029 02:52:12.429987 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a5ee816bf96c4eb891cb8ba5ed50f3ed in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:12.429996 989 leader_election.cc:304] T 731c0acfbe204e8199ad246e38ae2ccd P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters: c756528642a543eeacd93cef3e335a7d
I20251029 02:52:12.430362 2179 raft_consensus.cc:2804] T 731c0acfbe204e8199ad246e38ae2ccd P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:12.430435 2179 raft_consensus.cc:493] T 731c0acfbe204e8199ad246e38ae2ccd P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:12.430498 2179 raft_consensus.cc:3060] T 731c0acfbe204e8199ad246e38ae2ccd P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:12.431473 524 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_af03737bb25a4837aaef8c02ede1a537" } new_table_name: "default.Table_9814ab5eb726434abc36514decaf93b5" modify_external_catalogs: true
I20251029 02:52:12.432356 520 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 49aaa2d8986a44df979edc1c42612189
I20251029 02:52:12.433348 738 tablet_service.cc:1552] Processing DeleteTablet for tablet 49aaa2d8986a44df979edc1c42612189 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:12 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:12.433362 2179 raft_consensus.cc:515] T 731c0acfbe204e8199ad246e38ae2ccd P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:12.433663 2443 raft_consensus.cc:493] T e65eaaae43ac435ebd665546de9d68dd P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:12.433776 2443 raft_consensus.cc:515] T e65eaaae43ac435ebd665546de9d68dd P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:12.433897 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_0d5fcbba432c470cb80f5cdb893b6294"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:12.434334 2443 leader_election.cc:290] T e65eaaae43ac435ebd665546de9d68dd P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
W20251029 02:52:12.435081 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0d5fcbba432c470cb80f5cdb893b6294 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:12.435141 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e65eaaae43ac435ebd665546de9d68dd" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:12.435312 917 raft_consensus.cc:2468] T e65eaaae43ac435ebd665546de9d68dd P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 0.
I20251029 02:52:12.435981 2179 leader_election.cc:290] T 731c0acfbe204e8199ad246e38ae2ccd P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Requested vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:12.437160 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "731c0acfbe204e8199ad246e38ae2ccd" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
I20251029 02:52:12.436465 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet 49aaa2d8986a44df979edc1c42612189 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:12 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:12.437589 990 leader_election.cc:304] T e65eaaae43ac435ebd665546de9d68dd P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:12.437997 2443 raft_consensus.cc:2804] T e65eaaae43ac435ebd665546de9d68dd P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:12.438067 2443 raft_consensus.cc:493] T e65eaaae43ac435ebd665546de9d68dd P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:12.438133 2443 raft_consensus.cc:3060] T e65eaaae43ac435ebd665546de9d68dd P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:12.440611 2443 raft_consensus.cc:515] T e65eaaae43ac435ebd665546de9d68dd P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:12.441128 2443 leader_election.cc:290] T e65eaaae43ac435ebd665546de9d68dd P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Requested vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:52:12.441833 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e65eaaae43ac435ebd665546de9d68dd" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
I20251029 02:52:12.441988 917 raft_consensus.cc:3060] T e65eaaae43ac435ebd665546de9d68dd P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:12.444953 917 raft_consensus.cc:2468] T e65eaaae43ac435ebd665546de9d68dd P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 1.
W20251029 02:52:12.445353 990 leader_election.cc:343] T 731c0acfbe204e8199ad246e38ae2ccd P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:12.445981 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e65eaaae43ac435ebd665546de9d68dd" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:12.446092 990 leader_election.cc:304] T e65eaaae43ac435ebd665546de9d68dd P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:12.446164 776 raft_consensus.cc:3060] T e65eaaae43ac435ebd665546de9d68dd P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:12.446333 2179 raft_consensus.cc:2804] T e65eaaae43ac435ebd665546de9d68dd P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:12.446436 2179 raft_consensus.cc:697] T e65eaaae43ac435ebd665546de9d68dd P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Becoming Leader. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Running, Role: LEADER
I20251029 02:52:12.446666 2179 consensus_queue.cc:237] T e65eaaae43ac435ebd665546de9d68dd P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:12.418002 772 raft_consensus.cc:2468] T afb19ba39399410bb90b6225d7b5b5b1 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 1.
I20251029 02:52:12.418138 1175 raft_consensus.cc:2243] T 30d03e1e42d34242bd354f9e9ec4f127 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:12.448881 776 raft_consensus.cc:2468] T e65eaaae43ac435ebd665546de9d68dd P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 1.
I20251029 02:52:12.448911 881 tablet_service.cc:1552] Processing DeleteTablet for tablet 49aaa2d8986a44df979edc1c42612189 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:12 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:12.449780 1175 raft_consensus.cc:2272] T 30d03e1e42d34242bd354f9e9ec4f127 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:12.453074 763 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e65eaaae43ac435ebd665546de9d68dd" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:12.453323 763 raft_consensus.cc:2376] T e65eaaae43ac435ebd665546de9d68dd P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 1. Re-sending same reply.
I20251029 02:52:12.453872 1175 ts_tablet_manager.cc:1916] T 30d03e1e42d34242bd354f9e9ec4f127 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:12.454134 764 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "731c0acfbe204e8199ad246e38ae2ccd" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:12.454316 1174 tablet_replica.cc:333] T dabb8d760aae4aa09eb83cdb586c9788 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:12.454439 764 raft_consensus.cc:3060] T 731c0acfbe204e8199ad246e38ae2ccd P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:12.454607 1174 raft_consensus.cc:2243] T dabb8d760aae4aa09eb83cdb586c9788 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:12.454764 1174 raft_consensus.cc:2272] T dabb8d760aae4aa09eb83cdb586c9788 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:12.455260 1174 ts_tablet_manager.cc:1916] T dabb8d760aae4aa09eb83cdb586c9788 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 02:52:12.456063 849 consensus_peers.cc:597] T 59d03a190b784ab18a3e63a572675891 P c756528642a543eeacd93cef3e335a7d -> Peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Couldn't send request to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:12.457199 764 raft_consensus.cc:2468] T 731c0acfbe204e8199ad246e38ae2ccd P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 1.
W20251029 02:52:12.457473 2309 consensus_meta.cc:287] T c1222e30bc4141b3ae686d80053c4ef5 P b495a49349324f7d81908f2da700f0d9: Time spent flushing consensus metadata: real 0.563s user 0.023s sys 0.000s
I20251029 02:52:12.457564 989 leader_election.cc:304] T 731c0acfbe204e8199ad246e38ae2ccd P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters: c756528642a543eeacd93cef3e335a7d
I20251029 02:52:12.458010 2430 raft_consensus.cc:2804] T 731c0acfbe204e8199ad246e38ae2ccd P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:12.458134 2430 raft_consensus.cc:697] T 731c0acfbe204e8199ad246e38ae2ccd P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Becoming Leader. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Running, Role: LEADER
I20251029 02:52:12.458429 2430 consensus_queue.cc:237] T 731c0acfbe204e8199ad246e38ae2ccd P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:12.461510 2309 raft_consensus.cc:515] T c1222e30bc4141b3ae686d80053c4ef5 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:12.464737 2309 leader_election.cc:290] T c1222e30bc4141b3ae686d80053c4ef5 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 election: Requested vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:12.464586 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c1222e30bc4141b3ae686d80053c4ef5" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
I20251029 02:52:12.465063 1052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c1222e30bc4141b3ae686d80053c4ef5" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:52:12.465230 1052 raft_consensus.cc:3060] T c1222e30bc4141b3ae686d80053c4ef5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
W20251029 02:52:12.465423 711 leader_election.cc:343] T c1222e30bc4141b3ae686d80053c4ef5 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:12.467918 1052 raft_consensus.cc:2468] T c1222e30bc4141b3ae686d80053c4ef5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b495a49349324f7d81908f2da700f0d9 in term 1.
I20251029 02:52:12.468297 712 leader_election.cc:304] T c1222e30bc4141b3ae686d80053c4ef5 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters: c756528642a543eeacd93cef3e335a7d
I20251029 02:52:12.468713 2309 raft_consensus.cc:2804] T c1222e30bc4141b3ae686d80053c4ef5 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:12.469184 2309 raft_consensus.cc:697] T c1222e30bc4141b3ae686d80053c4ef5 P b495a49349324f7d81908f2da700f0d9 [term 1 LEADER]: Becoming Leader. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Running, Role: LEADER
I20251029 02:52:12.469480 2309 consensus_queue.cc:237] T c1222e30bc4141b3ae686d80053c4ef5 P b495a49349324f7d81908f2da700f0d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:12.486617 1172 consensus_queue.cc:260] T ae4231daa2594258aa51419c97a2f50a P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:12.487785 1172 ts_tablet_manager.cc:1434] T ae4231daa2594258aa51419c97a2f50a P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.063s user 0.001s sys 0.000s
I20251029 02:52:12.488104 1172 tablet_bootstrap.cc:492] T 6142018e44af4d32818adefa2ee43d02 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:12.490314 1172 tablet_bootstrap.cc:654] T 6142018e44af4d32818adefa2ee43d02 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:12.492463 2113 consensus_queue.cc:1048] T c32c28b0e74a4af3a0e69bbf24ab1a1f P b495a49349324f7d81908f2da700f0d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:12.493561 1175 ts_tablet_manager.cc:1929] T 30d03e1e42d34242bd354f9e9ec4f127 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:12.493664 1175 log.cc:1199] T 30d03e1e42d34242bd354f9e9ec4f127 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/30d03e1e42d34242bd354f9e9ec4f127
I20251029 02:52:12.493980 1175 ts_tablet_manager.cc:1950] T 30d03e1e42d34242bd354f9e9ec4f127 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:12.494328 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 821) took 4217 ms. Trace:
I20251029 02:52:12.500023 1166 tablet_bootstrap.cc:492] T 59eee86f249e4d6a9d09d30276dcd659 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:12.500229 1166 ts_tablet_manager.cc:1403] T 59eee86f249e4d6a9d09d30276dcd659 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.126s user 0.001s sys 0.002s
I20251029 02:52:12.501015 1166 raft_consensus.cc:359] T 59eee86f249e4d6a9d09d30276dcd659 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:12.501303 1166 raft_consensus.cc:385] T 59eee86f249e4d6a9d09d30276dcd659 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:12.501397 1166 raft_consensus.cc:740] T 59eee86f249e4d6a9d09d30276dcd659 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:12.501996 1166 consensus_queue.cc:260] T 59eee86f249e4d6a9d09d30276dcd659 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:12.504989 1167 tablet_bootstrap.cc:492] T c1222e30bc4141b3ae686d80053c4ef5 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:12.505118 1167 ts_tablet_manager.cc:1403] T c1222e30bc4141b3ae686d80053c4ef5 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.255s user 0.069s sys 0.074s
I20251029 02:52:12.505811 1167 raft_consensus.cc:359] T c1222e30bc4141b3ae686d80053c4ef5 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:12.505972 1167 raft_consensus.cc:385] T c1222e30bc4141b3ae686d80053c4ef5 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:12.506018 1167 raft_consensus.cc:740] T c1222e30bc4141b3ae686d80053c4ef5 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:12.507624 1167 consensus_queue.cc:260] T c1222e30bc4141b3ae686d80053c4ef5 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:12.509733 1167 ts_tablet_manager.cc:1434] T c1222e30bc4141b3ae686d80053c4ef5 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251029 02:52:12.511555 1167 tablet_bootstrap.cc:492] T 731c0acfbe204e8199ad246e38ae2ccd P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:12.513167 1167 tablet_bootstrap.cc:654] T 731c0acfbe204e8199ad246e38ae2ccd P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:12.522217 1176 ts_tablet_manager.cc:1929] T 59d03a190b784ab18a3e63a572675891 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:12.523049 1176 log.cc:1199] T 59d03a190b784ab18a3e63a572675891 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/59d03a190b784ab18a3e63a572675891
I20251029 02:52:12.523720 1176 ts_tablet_manager.cc:1950] T 59d03a190b784ab18a3e63a572675891 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:12.530893 2416 consensus_queue.cc:1048] T c32c28b0e74a4af3a0e69bbf24ab1a1f P b495a49349324f7d81908f2da700f0d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:12.533571 1174 ts_tablet_manager.cc:1929] T dabb8d760aae4aa09eb83cdb586c9788 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:12.533699 1174 log.cc:1199] T dabb8d760aae4aa09eb83cdb586c9788 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/dabb8d760aae4aa09eb83cdb586c9788
I20251029 02:52:12.534112 1174 ts_tablet_manager.cc:1950] T dabb8d760aae4aa09eb83cdb586c9788 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:12.494398 1175 rpcz_store.cc:276] 1029 02:52:08.276629 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:08.276669 (+ 40us) service_pool.cc:225] Handling call
1029 02:52:12.286835 (+4010166us) ts_tablet_manager.cc:1151] Deleting tablet 30d03e1e42d34242bd354f9e9ec4f127
1029 02:52:12.286848 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:12.449650 (+162802us) spinlock_profiling.cc:243] Waited 131 ms on lock 0x6160001d3854. stack: 00007f1174e9f9f6 00007f1174e9f7b0 00007f1173e7b3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f1180d6a100 00007f11820b7ae3 00007f1184c78287 00007f1184cab2d4 00007f1184c8bd70 00007f1184c8bafd 0000000000a8867c 00007f1174ee5d7f 00007f1174ee904b 00007f1174ee8ec8
1029 02:52:12.489471 (+ 39821us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:12.493480 (+ 4009us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:12.494309 (+ 829us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":300457344,"tablet-delete.queue_time_us":4009987}
I20251029 02:52:12.540334 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 30d03e1e42d34242bd354f9e9ec4f127 (table default.Table_435f2dfe42134760abc9e8880fa19f32 [id=c6e1c357fbfc49be972748da564cc9a9]) successfully deleted
I20251029 02:52:12.541601 1175 tablet_replica.cc:333] T 6d830d30b26b4ba799785fec8a6f0769 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:12.541857 1175 raft_consensus.cc:2243] T 6d830d30b26b4ba799785fec8a6f0769 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:12.542109 1175 raft_consensus.cc:2272] T 6d830d30b26b4ba799785fec8a6f0769 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:12.542323 2452 tablet.cc:1722] T 1c53fc1fff1a4bb988beb46ff91fd3be P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 02:52:12.543522 1175 ts_tablet_manager.cc:1916] T 6d830d30b26b4ba799785fec8a6f0769 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:12.547266 2443 consensus_queue.cc:1048] T adf6e736cb9c4920807ee58682e134ef P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:12.550647 1166 ts_tablet_manager.cc:1434] T 59eee86f249e4d6a9d09d30276dcd659 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.050s user 0.001s sys 0.001s
W20251029 02:52:12.552304 510 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.62:42343, user_credentials={real_user=slave}} blocked reactor thread for 44534.2us
W20251029 02:52:12.554809 847 consensus_peers.cc:597] T 738046ab57334f2dab9f4aab979c5b42 P c756528642a543eeacd93cef3e335a7d -> Peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Couldn't send request to peer b495a49349324f7d81908f2da700f0d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 738046ab57334f2dab9f4aab979c5b42. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:12.544602 2469 tablet.cc:1722] T 1c53fc1fff1a4bb988beb46ff91fd3be P b495a49349324f7d81908f2da700f0d9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 02:52:12.557940 2453 tablet.cc:1722] T 1c53fc1fff1a4bb988beb46ff91fd3be P c756528642a543eeacd93cef3e335a7d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 02:52:12.558465 1167 tablet_bootstrap.cc:492] T 731c0acfbe204e8199ad246e38ae2ccd P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:12.558835 1167 ts_tablet_manager.cc:1403] T 731c0acfbe204e8199ad246e38ae2ccd P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.048s user 0.005s sys 0.001s
I20251029 02:52:12.560586 1167 raft_consensus.cc:359] T 731c0acfbe204e8199ad246e38ae2ccd P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:12.561667 1167 raft_consensus.cc:385] T 731c0acfbe204e8199ad246e38ae2ccd P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:12.561849 1167 raft_consensus.cc:740] T 731c0acfbe204e8199ad246e38ae2ccd P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:12.562565 1172 tablet_bootstrap.cc:492] T 6142018e44af4d32818adefa2ee43d02 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:12.562945 1167 consensus_queue.cc:260] T 731c0acfbe204e8199ad246e38ae2ccd P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:12.564836 1167 ts_tablet_manager.cc:1434] T 731c0acfbe204e8199ad246e38ae2ccd P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251029 02:52:12.565237 1172 ts_tablet_manager.cc:1403] T 6142018e44af4d32818adefa2ee43d02 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.077s user 0.003s sys 0.000s
I20251029 02:52:12.566246 1172 raft_consensus.cc:359] T 6142018e44af4d32818adefa2ee43d02 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:12.566409 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 836) took 4202 ms. Trace:
I20251029 02:52:12.566494 1174 rpcz_store.cc:276] 1029 02:52:08.363917 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:08.363970 (+ 53us) service_pool.cc:225] Handling call
1029 02:52:12.454267 (+4090297us) ts_tablet_manager.cc:1151] Deleting tablet dabb8d760aae4aa09eb83cdb586c9788
1029 02:52:12.454277 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:12.500656 (+ 46379us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:12.533428 (+ 32772us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:12.566387 (+ 32959us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4090077}
I20251029 02:52:12.566969 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet dabb8d760aae4aa09eb83cdb586c9788 (table default.Table_d897a86961894f1aaad67274b687080e [id=b3e544a043444b77a597c6c65c15b795]) successfully deleted
W20251029 02:52:12.567029 849 consensus_peers.cc:597] T f529e17e97074dca91ec2f48954cfd32 P c756528642a543eeacd93cef3e335a7d -> Peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Couldn't send request to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f529e17e97074dca91ec2f48954cfd32. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:12.567621 1167 tablet_bootstrap.cc:492] T d041e87ceb11441983d0e38b7929c3a4 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:12.571094 1172 raft_consensus.cc:385] T 6142018e44af4d32818adefa2ee43d02 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:12.571331 1172 raft_consensus.cc:740] T 6142018e44af4d32818adefa2ee43d02 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:12.571501 1174 tablet_replica.cc:333] T f529e17e97074dca91ec2f48954cfd32 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:12.571741 1174 raft_consensus.cc:2243] T f529e17e97074dca91ec2f48954cfd32 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:12.572168 1174 raft_consensus.cc:2272] T f529e17e97074dca91ec2f48954cfd32 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:12.571825 1172 consensus_queue.cc:260] T 6142018e44af4d32818adefa2ee43d02 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:12.572762 1174 ts_tablet_manager.cc:1916] T f529e17e97074dca91ec2f48954cfd32 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:12.574319 1167 tablet_bootstrap.cc:654] T d041e87ceb11441983d0e38b7929c3a4 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:12.575706 1172 ts_tablet_manager.cc:1434] T 6142018e44af4d32818adefa2ee43d02 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20251029 02:52:12.576808 2443 consensus_queue.cc:1048] T adf6e736cb9c4920807ee58682e134ef P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:12.584821 526 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet d7abaf84ba5d445aa17433b3be891548 (table default.Table_c1c6933af1e046aa895e0ee915178f5c [id=9e33c530a35544a4ae22e2a1238592db]). Expected version 3 got 0
I20251029 02:52:12.585341 526 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet 643e20d9974447daa78885b3401f3b77 (table default.Table_7c1cb64e05be4f6e8ac28286c982a3e7 [id=b4fc453bf54d45c4b8dddc3c49bf48f8]). Expected version 2 got 0
I20251029 02:52:12.585564 526 catalog_manager.cc:5649] T 82289e0f823c4d43bfd8c4e878572b2b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } health_report { overall_health: UNKNOWN } } }
I20251029 02:52:12.585956 526 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet 82289e0f823c4d43bfd8c4e878572b2b (table default.Table_e7bb3e262a894c2f84b6edfc3917b30d [id=2cca3b7c50b243b5883b8ac1f353f253]). Expected version 1 got 0
I20251029 02:52:12.591732 2113 consensus_queue.cc:1048] T 97b2ecae572a4b7ea1b9595074556708 P b495a49349324f7d81908f2da700f0d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:12.591879 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_13076d1361cd4ef9ae70f064a6b8679c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:12.592510 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_13076d1361cd4ef9ae70f064a6b8679c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:12.593261 2201 raft_consensus.cc:493] T 0f80b840ed954230bfbfe7e40127cd2f P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0cfd4bee16f84b7c8b0ff3ce14db20d5)
I20251029 02:52:12.593444 2201 raft_consensus.cc:515] T 0f80b840ed954230bfbfe7e40127cd2f P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:12.593752 518 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet f281191e7c6143f983f11d87f1c9b64e
I20251029 02:52:12.594193 518 catalog_manager.cc:6719] ReplaceTablet: tablet f281191e7c6143f983f11d87f1c9b64e deleted and replaced by tablet e91d6bdca3ad40c48d93e6196644462c
I20251029 02:52:12.594192 2201 leader_election.cc:290] T 0f80b840ed954230bfbfe7e40127cd2f P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:12.594494 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_4ca293107d23430cbad5c6e53b07da9f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:12.595000 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4ca293107d23430cbad5c6e53b07da9f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:52:12.596148 517 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f3ede523203545d18b09d535fdcabb6f consensus state has no leader
I20251029 02:52:12.596485 2188 consensus_queue.cc:1048] T 061f3d38c4604fa4b57e74ff6f6fc7da P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:12.596509 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_42036229ecf348adb05a3fdaaab07e1f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:12.596995 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_42036229ecf348adb05a3fdaaab07e1f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:52:12.598138 522 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4e89d72dd274498783dc815f6f2ee031 consensus state has no leader
I20251029 02:52:12.598577 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_5e94f24436bb4574a310bd6c375107a1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:12.599052 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5e94f24436bb4574a310bd6c375107a1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:12.600973 523 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_0a146ce6c31f4dbf994a2442f2386d93" } modify_external_catalogs: true
I20251029 02:52:12.601151 523 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_0a146ce6c31f4dbf994a2442f2386d93" } modify_external_catalogs: true
W20251029 02:52:12.601696 524 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ead73d81d18a40a4b0aa9935f86421f8 consensus state has no leader
I20251029 02:52:12.602006 524 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_a1ba26e950644781ba8d825e5472d758" } modify_external_catalogs: true
I20251029 02:52:12.602164 524 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_a1ba26e950644781ba8d825e5472d758" } modify_external_catalogs: true
I20251029 02:52:12.603633 520 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_e5ebaa623385480b9cd7be9694ff024f" } new_table_name: "default.Table_fcfe86e2824842a2aa6e4188c33b4a74" modify_external_catalogs: true
I20251029 02:52:12.606276 738 tablet_service.cc:1552] Processing DeleteTablet for tablet f281191e7c6143f983f11d87f1c9b64e with delete_type TABLET_DATA_DELETED (replaced by tablet e91d6bdca3ad40c48d93e6196644462c) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:12.606389 881 tablet_service.cc:1552] Processing DeleteTablet for tablet f281191e7c6143f983f11d87f1c9b64e with delete_type TABLET_DATA_DELETED (replaced by tablet e91d6bdca3ad40c48d93e6196644462c) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:12.607854 1034 tablet_service.cc:1552] Processing DeleteTablet for tablet f281191e7c6143f983f11d87f1c9b64e with delete_type TABLET_DATA_DELETED (replaced by tablet e91d6bdca3ad40c48d93e6196644462c) from {username='slave'} at 127.0.0.1:37626
W20251029 02:52:12.610334 1968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251029 02:52:12.610903 1968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251029 02:52:12.611078 1968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20251029 02:52:12.621089 517 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_df7d932b50b64537b4dbc68a846665fc" } modify_external_catalogs: true
I20251029 02:52:12.621348 517 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_df7d932b50b64537b4dbc68a846665fc" } modify_external_catalogs: true
I20251029 02:52:12.623693 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_dcf576a6745749cc85aeddaab539d832"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:12.624336 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dcf576a6745749cc85aeddaab539d832 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:12.625958 518 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_bc803f454de540c28f741096815c1ce2" } modify_external_catalogs: true
I20251029 02:52:12.626191 518 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_bc803f454de540c28f741096815c1ce2" } modify_external_catalogs: true
W20251029 02:52:12.626371 1968 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251029 02:52:12.627592 1968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251029 02:52:12.627771 1968 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251029 02:52:12.627904 1968 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251029 02:52:12.586390 526 catalog_manager.cc:5649] T adf6e736cb9c4920807ee58682e134ef P 0cfd4bee16f84b7c8b0ff3ce14db20d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } health_report { overall_health: UNKNOWN } } }
I20251029 02:52:12.629459 526 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet adf6e736cb9c4920807ee58682e134ef (table default.Table_0a039bfb2e1f458f89ffc3ee93300b8f [id=64650da66d464074828f2dc955045f68]). Expected version 3 got 0
I20251029 02:52:12.629742 525 catalog_manager.cc:6194] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: default.Table_6563934f48114b0987d298bfad3e9bf7 [id=57471769276f44299eae50b9d854b50c] alter complete (version 2)
I20251029 02:52:12.629891 1167 tablet_bootstrap.cc:492] T d041e87ceb11441983d0e38b7929c3a4 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:12.630041 1167 ts_tablet_manager.cc:1403] T d041e87ceb11441983d0e38b7929c3a4 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.062s user 0.009s sys 0.021s
I20251029 02:52:12.630913 1167 raft_consensus.cc:359] T d041e87ceb11441983d0e38b7929c3a4 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:12.631822 1167 raft_consensus.cc:385] T d041e87ceb11441983d0e38b7929c3a4 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:12.631891 1167 raft_consensus.cc:740] T d041e87ceb11441983d0e38b7929c3a4 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:12.633119 1167 consensus_queue.cc:260] T d041e87ceb11441983d0e38b7929c3a4 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:12.633713 1167 ts_tablet_manager.cc:1434] T d041e87ceb11441983d0e38b7929c3a4 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251029 02:52:12.634150 1167 tablet_bootstrap.cc:492] T 59eee86f249e4d6a9d09d30276dcd659 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:12.635375 522 master_service.cc:954] ReplaceTablet: received request to replace tablet ca71c4638a6b419b876e22af7e6cf00a from {username='slave'} at 127.0.0.1:35314
I20251029 02:52:12.641290 1167 tablet_bootstrap.cc:654] T 59eee86f249e4d6a9d09d30276dcd659 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
W20251029 02:52:12.644739 510 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.62:42343, user_credentials={real_user=slave}} blocked reactor thread for 86248us
I20251029 02:52:12.646669 524 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 9280503662894544a1f227a62a9b8a01
I20251029 02:52:12.647729 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 914) took 2605 ms. Trace:
I20251029 02:52:12.647892 1176 rpcz_store.cc:276] 1029 02:52:10.041899 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:10.042017 (+ 118us) service_pool.cc:225] Handling call
1029 02:52:12.266924 (+2224907us) ts_tablet_manager.cc:1151] Deleting tablet 59d03a190b784ab18a3e63a572675891
1029 02:52:12.266980 (+ 56us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:12.323592 (+ 56612us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:12.522091 (+198499us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:12.647707 (+125616us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2224696}
I20251029 02:52:12.648028 738 tablet_service.cc:1552] Processing DeleteTablet for tablet 9280503662894544a1f227a62a9b8a01 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:12 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:12.648315 1176 tablet_replica.cc:333] T 49d9ce32e4f04f11be2ecefc5d523a8b P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:12.648571 1176 raft_consensus.cc:2243] T 49d9ce32e4f04f11be2ecefc5d523a8b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:12.648849 1176 raft_consensus.cc:2272] T 49d9ce32e4f04f11be2ecefc5d523a8b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:12.649462 1176 ts_tablet_manager.cc:1916] T 49d9ce32e4f04f11be2ecefc5d523a8b P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:12.650758 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 59d03a190b784ab18a3e63a572675891 (table default.Table_2ec9c2215f7147adab6bbfa2fd06d968 [id=3ede90ae95fb46b2ab910914a49a35a5]) successfully deleted
I20251029 02:52:12.651346 1166 tablet_bootstrap.cc:492] T d041e87ceb11441983d0e38b7929c3a4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:12.652161 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_56d870facffb425daf13fa7cb9937b12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:12.652740 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_56d870facffb425daf13fa7cb9937b12 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:12.653614 1166 tablet_bootstrap.cc:654] T d041e87ceb11441983d0e38b7929c3a4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:12.655258 896 tablet_service.cc:1552] Processing DeleteTablet for tablet 9280503662894544a1f227a62a9b8a01 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:12 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:12.655979 2416 consensus_queue.cc:1048] T af4860c6b72f4d988b206b5ef2f22906 P b495a49349324f7d81908f2da700f0d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:12.656386 2416 consensus_queue.cc:1048] T 97b2ecae572a4b7ea1b9595074556708 P b495a49349324f7d81908f2da700f0d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:12.658802 1054 raft_consensus.cc:1275] T 643e20d9974447daa78885b3401f3b77 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Refusing update from remote peer c756528642a543eeacd93cef3e335a7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251029 02:52:12.659005 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet 9280503662894544a1f227a62a9b8a01 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:12 UTC) from {username='slave'} at 127.0.0.1:37626
W20251029 02:52:12.659317 849 leader_election.cc:343] T 0f80b840ed954230bfbfe7e40127cd2f P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Not found: Tablet not found: 0f80b840ed954230bfbfe7e40127cd2f
I20251029 02:52:12.659668 2425 consensus_queue.cc:1048] T 946f65014d97425aab3851009b742807 P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:12.660126 2425 consensus_queue.cc:1048] T 643e20d9974447daa78885b3401f3b77 P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:12.658486 769 raft_consensus.cc:1275] T 643e20d9974447daa78885b3401f3b77 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Refusing update from remote peer c756528642a543eeacd93cef3e335a7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20251029 02:52:12.661643 847 leader_election.cc:343] T 0f80b840ed954230bfbfe7e40127cd2f P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 0f80b840ed954230bfbfe7e40127cd2f
I20251029 02:52:12.661746 847 leader_election.cc:304] T 0f80b840ed954230bfbfe7e40127cd2f P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9
I20251029 02:52:12.662478 2205 consensus_queue.cc:1048] T 946f65014d97425aab3851009b742807 P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:12.662966 2205 raft_consensus.cc:2749] T 0f80b840ed954230bfbfe7e40127cd2f P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251029 02:52:12.663149 2205 consensus_queue.cc:1048] T 643e20d9974447daa78885b3401f3b77 P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251029 02:52:12.666929 989 consensus_peers.cc:597] T 061f3d38c4604fa4b57e74ff6f6fc7da P 0cfd4bee16f84b7c8b0ff3ce14db20d5 -> Peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Couldn't send request to peer b495a49349324f7d81908f2da700f0d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 061f3d38c4604fa4b57e74ff6f6fc7da. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:12.675490 2309 consensus_queue.cc:1048] T 8048506ced4540a5aee9f79bfc4599bf P b495a49349324f7d81908f2da700f0d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:12.685242 1174 ts_tablet_manager.cc:1929] T f529e17e97074dca91ec2f48954cfd32 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:12.685387 1174 log.cc:1199] T f529e17e97074dca91ec2f48954cfd32 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/f529e17e97074dca91ec2f48954cfd32
I20251029 02:52:12.685766 1174 ts_tablet_manager.cc:1950] T f529e17e97074dca91ec2f48954cfd32 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:12.696332 1172 tablet_bootstrap.cc:492] T 72f9d4408f384e7093316d85f092e1b2 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:12.702538 2425 consensus_queue.cc:1048] T d7abaf84ba5d445aa17433b3be891548 P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:12.709296 1968 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:44691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/master-0/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:42343,127.0.104.61:34239,127.0.104.60:41233
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:42343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=46851
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 60f5e5267b92c39485a66121d3ce3cc7ef57b0e0
build type FASTDEBUG
built by None at 29 Oct 2025 02:43:25 UTC on 24a791456cd2
build id 8768
ASAN enabled
I20251029 02:52:12.711984 1176 ts_tablet_manager.cc:1929] T 49d9ce32e4f04f11be2ecefc5d523a8b P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:12.717664 1968 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251029 02:52:12.719285 2416 consensus_queue.cc:1048] T 8048506ced4540a5aee9f79bfc4599bf P b495a49349324f7d81908f2da700f0d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:12.721712 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 842) took 4120 ms. Trace:
I20251029 02:52:12.721870 1174 rpcz_store.cc:276] 1029 02:52:08.601571 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:08.601653 (+ 82us) service_pool.cc:225] Handling call
1029 02:52:12.571445 (+3969792us) ts_tablet_manager.cc:1151] Deleting tablet f529e17e97074dca91ec2f48954cfd32
1029 02:52:12.571456 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:12.667515 (+ 96059us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:12.685111 (+ 17596us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:12.721688 (+ 36577us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3969545}
I20251029 02:52:12.722572 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet f529e17e97074dca91ec2f48954cfd32 (table default.Table_570f159e617548ec974b41a654c1d800 [id=835aa84a7f8d426d9560b819e6864715]) successfully deleted
I20251029 02:52:12.723568 1176 log.cc:1199] T 49d9ce32e4f04f11be2ecefc5d523a8b P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/49d9ce32e4f04f11be2ecefc5d523a8b
I20251029 02:52:12.723840 1174 tablet_replica.cc:333] T c9dff4dff82f49e7ad49caad5e404f09 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:12.725684 1176 ts_tablet_manager.cc:1950] T 49d9ce32e4f04f11be2ecefc5d523a8b P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:12.725764 1968 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20251029 02:52:12.725999 1174 raft_consensus.cc:2243] T c9dff4dff82f49e7ad49caad5e404f09 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:12.726204 1174 raft_consensus.cc:2272] T c9dff4dff82f49e7ad49caad5e404f09 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:12.726876 1174 ts_tablet_manager.cc:1916] T c9dff4dff82f49e7ad49caad5e404f09 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 02:52:12.731459 520 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e91d6bdca3ad40c48d93e6196644462c has no consensus state
I20251029 02:52:12.731947 520 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_f17714316e664840899f4d9827cd25fb" } new_table_name: "default.Table_a5f63619b78649ea878ad36c8b5c8143" modify_external_catalogs: true
I20251029 02:52:12.735141 517 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet fe5aff0f61a44b91bfb37b42457a91f0
I20251029 02:52:12.735910 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_09fa848b582f433b9ed2b3f9ec4cf3fe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:12.736390 896 tablet_service.cc:1552] Processing DeleteTablet for tablet fe5aff0f61a44b91bfb37b42457a91f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:12 UTC) from {username='slave'} at 127.0.0.1:35702
W20251029 02:52:12.736495 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_09fa848b582f433b9ed2b3f9ec4cf3fe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:12.741202 738 tablet_service.cc:1552] Processing DeleteTablet for tablet fe5aff0f61a44b91bfb37b42457a91f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:12 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:12.743235 2425 consensus_queue.cc:1048] T 44ae2b07edca44afbe1b1fb3a5b1115d P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:12.738871 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet fe5aff0f61a44b91bfb37b42457a91f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:12 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:12.745290 1175 ts_tablet_manager.cc:1929] T 6d830d30b26b4ba799785fec8a6f0769 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:12.745424 1175 log.cc:1199] T 6d830d30b26b4ba799785fec8a6f0769 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/6d830d30b26b4ba799785fec8a6f0769
I20251029 02:52:12.745922 1175 ts_tablet_manager.cc:1950] T 6d830d30b26b4ba799785fec8a6f0769 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:12.746649 1167 tablet_bootstrap.cc:492] T 59eee86f249e4d6a9d09d30276dcd659 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:12.747272 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 822) took 4470 ms. Trace:
I20251029 02:52:12.747360 1175 rpcz_store.cc:276] 1029 02:52:08.276911 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:08.276949 (+ 38us) service_pool.cc:225] Handling call
1029 02:52:12.540707 (+4263758us) ts_tablet_manager.cc:1151] Deleting tablet 6d830d30b26b4ba799785fec8a6f0769
1029 02:52:12.541421 (+ 714us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:12.561275 (+ 19854us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:12.745147 (+183872us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:12.747249 (+ 2102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4263621}
I20251029 02:52:12.747656 2443 raft_consensus.cc:493] T 59eee86f249e4d6a9d09d30276dcd659 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:12.747823 2443 raft_consensus.cc:515] T 59eee86f249e4d6a9d09d30276dcd659 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:12.748027 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 823) took 4464 ms. Trace:
I20251029 02:52:12.748117 1175 rpcz_store.cc:276] 1029 02:52:08.284083 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:08.284157 (+ 74us) service_pool.cc:225] Handling call
1029 02:52:12.747879 (+4463722us) ts_tablet_manager.cc:1151] Deleting tablet 48aa20640dfe43288326ed6d6b9bc1cf
1029 02:52:12.747888 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:12.747991 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4463500}
I20251029 02:52:12.748466 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 824) took 4460 ms. Trace:
I20251029 02:52:12.748498 2443 leader_election.cc:290] T 59eee86f249e4d6a9d09d30276dcd659 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:12.748523 1175 rpcz_store.cc:276] 1029 02:52:08.288284 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:08.288361 (+ 77us) service_pool.cc:225] Handling call
1029 02:52:12.748354 (+4459993us) ts_tablet_manager.cc:1151] Deleting tablet 65e761f9f9fb490586d6415a5a8565dd
1029 02:52:12.748362 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:12.748451 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4459766}
I20251029 02:52:12.748723 1175 tablet_replica.cc:333] T d2990e9f626c40618c118266bc595684 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:12.748955 1175 raft_consensus.cc:2243] T d2990e9f626c40618c118266bc595684 P b495a49349324f7d81908f2da700f0d9 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:12.749006 1172 tablet_bootstrap.cc:654] T 72f9d4408f384e7093316d85f092e1b2 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:12.749137 1175 raft_consensus.cc:2272] T d2990e9f626c40618c118266bc595684 P b495a49349324f7d81908f2da700f0d9 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:12.749667 1175 ts_tablet_manager.cc:1916] T d2990e9f626c40618c118266bc595684 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:12.749774 764 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59eee86f249e4d6a9d09d30276dcd659" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:12.750664 764 raft_consensus.cc:2468] T 59eee86f249e4d6a9d09d30276dcd659 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 0.
I20251029 02:52:12.751166 989 leader_election.cc:304] T 59eee86f249e4d6a9d09d30276dcd659 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters:
I20251029 02:52:12.751729 2443 raft_consensus.cc:2804] T 59eee86f249e4d6a9d09d30276dcd659 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:12.751845 2443 raft_consensus.cc:493] T 59eee86f249e4d6a9d09d30276dcd659 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:12.751895 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 915) took 2709 ms. Trace:
I20251029 02:52:12.751945 2443 raft_consensus.cc:3060] T 59eee86f249e4d6a9d09d30276dcd659 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:12.751997 1176 rpcz_store.cc:276] 1029 02:52:10.042319 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:10.042365 (+ 46us) service_pool.cc:225] Handling call
1029 02:52:12.648259 (+2605894us) ts_tablet_manager.cc:1151] Deleting tablet 49d9ce32e4f04f11be2ecefc5d523a8b
1029 02:52:12.648268 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:12.675739 (+ 27471us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:12.711815 (+ 36076us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:12.751847 (+ 40032us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2605744}
I20251029 02:52:12.752418 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 916) took 2709 ms. Trace:
I20251029 02:52:12.752513 1176 rpcz_store.cc:276] 1029 02:52:10.042641 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:10.042683 (+ 42us) service_pool.cc:225] Handling call
1029 02:52:12.752278 (+2709595us) ts_tablet_manager.cc:1151] Deleting tablet 5815626b05204b578cc4a110f246bf10
1029 02:52:12.752287 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:12.752374 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2709415}
I20251029 02:52:12.751693 919 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59eee86f249e4d6a9d09d30276dcd659" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:12.753681 1167 ts_tablet_manager.cc:1403] T 59eee86f249e4d6a9d09d30276dcd659 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.119s user 0.029s sys 0.044s
I20251029 02:52:12.754632 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 6d830d30b26b4ba799785fec8a6f0769 (table default.Table_19165ff9f7f24a97bf835e72701455d3 [id=ae6032816e6c4b80aa294478cbd49460]) successfully deleted
I20251029 02:52:12.754657 1167 raft_consensus.cc:359] T 59eee86f249e4d6a9d09d30276dcd659 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:12.754796 1167 raft_consensus.cc:385] T 59eee86f249e4d6a9d09d30276dcd659 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:12.754841 1167 raft_consensus.cc:740] T 59eee86f249e4d6a9d09d30276dcd659 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
W20251029 02:52:12.754949 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 48aa20640dfe43288326ed6d6b9bc1cf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 48aa20640dfe43288326ed6d6b9bc1cf already in progress: creating tablet
I20251029 02:52:12.755003 1167 consensus_queue.cc:260] T 59eee86f249e4d6a9d09d30276dcd659 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
W20251029 02:52:12.755160 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 65e761f9f9fb490586d6415a5a8565dd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 65e761f9f9fb490586d6415a5a8565dd already in progress: creating tablet
I20251029 02:52:12.755487 1167 ts_tablet_manager.cc:1434] T 59eee86f249e4d6a9d09d30276dcd659 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251029 02:52:12.755772 1167 tablet_bootstrap.cc:492] T a9e0fcc759a64679b3f59a6c964877b6 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:12.756098 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 49d9ce32e4f04f11be2ecefc5d523a8b (table default.Table_af1ab75f523d41338d4471824997f935 [id=468c76c8c99a4aa59c8b1f6c8e4bfa30]) successfully deleted
W20251029 02:52:12.756382 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 5815626b05204b578cc4a110f246bf10 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5815626b05204b578cc4a110f246bf10 already in progress: creating tablet
I20251029 02:52:12.757616 1167 tablet_bootstrap.cc:654] T a9e0fcc759a64679b3f59a6c964877b6 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:12.763645 1175 ts_tablet_manager.cc:1929] T d2990e9f626c40618c118266bc595684 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251029 02:52:12.763787 1175 log.cc:1199] T d2990e9f626c40618c118266bc595684 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/d2990e9f626c40618c118266bc595684
I20251029 02:52:12.764166 1175 ts_tablet_manager.cc:1950] T d2990e9f626c40618c118266bc595684 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:12.765156 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 825) took 4462 ms. Trace:
I20251029 02:52:12.765261 1175 rpcz_store.cc:276] 1029 02:52:08.302701 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:08.302945 (+ 244us) service_pool.cc:225] Handling call
1029 02:52:12.748686 (+4445741us) ts_tablet_manager.cc:1151] Deleting tablet d2990e9f626c40618c118266bc595684
1029 02:52:12.748693 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:12.760222 (+ 11529us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:12.763510 (+ 3288us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:12.765127 (+ 1617us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4445508}
I20251029 02:52:12.765841 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet d2990e9f626c40618c118266bc595684 (table default.Table_da047830460443a29c5269c98fc859e9 [id=f733948e92d8470da9d8050bd26a06b1]) successfully deleted
I20251029 02:52:12.766923 2205 consensus_queue.cc:1048] T 44ae2b07edca44afbe1b1fb3a5b1115d P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:12.752833 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 918) took 2709 ms. Trace:
W20251029 02:52:12.755002 990 leader_election.cc:343] T 59eee86f249e4d6a9d09d30276dcd659 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:12.762982 2443 raft_consensus.cc:515] T 59eee86f249e4d6a9d09d30276dcd659 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:12.772572 2453 tablet.cc:1722] T 643e20d9974447daa78885b3401f3b77 P c756528642a543eeacd93cef3e335a7d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 02:52:12.772872 2443 leader_election.cc:290] T 59eee86f249e4d6a9d09d30276dcd659 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Requested vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:12.774070 2443 consensus_queue.cc:1048] T 82289e0f823c4d43bfd8c4e878572b2b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:12.775394 919 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59eee86f249e4d6a9d09d30276dcd659" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
I20251029 02:52:12.775600 919 raft_consensus.cc:3060] T 59eee86f249e4d6a9d09d30276dcd659 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:12.775821 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "59eee86f249e4d6a9d09d30276dcd659" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:12.776011 772 raft_consensus.cc:3060] T 59eee86f249e4d6a9d09d30276dcd659 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
W20251029 02:52:12.777456 2515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251029 02:52:12.778450 2469 tablet.cc:1722] T 643e20d9974447daa78885b3401f3b77 P b495a49349324f7d81908f2da700f0d9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 02:52:12.779295 919 raft_consensus.cc:2468] T 59eee86f249e4d6a9d09d30276dcd659 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 1.
I20251029 02:52:12.779752 990 leader_election.cc:304] T 59eee86f249e4d6a9d09d30276dcd659 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:12.780025 2443 raft_consensus.cc:2804] T 59eee86f249e4d6a9d09d30276dcd659 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:12.780179 2443 raft_consensus.cc:697] T 59eee86f249e4d6a9d09d30276dcd659 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Becoming Leader. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Running, Role: LEADER
I20251029 02:52:12.780174 2425 consensus_queue.cc:1048] T d7abaf84ba5d445aa17433b3be891548 P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:12.780436 2443 consensus_queue.cc:237] T 59eee86f249e4d6a9d09d30276dcd659 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:12.783663 772 raft_consensus.cc:2468] T 59eee86f249e4d6a9d09d30276dcd659 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 1.
I20251029 02:52:12.784148 1175 tablet_replica.cc:333] T dabb8d760aae4aa09eb83cdb586c9788 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:12.784375 1175 raft_consensus.cc:2243] T dabb8d760aae4aa09eb83cdb586c9788 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:12.784530 1175 raft_consensus.cc:2272] T dabb8d760aae4aa09eb83cdb586c9788 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:12.786587 1175 ts_tablet_manager.cc:1916] T dabb8d760aae4aa09eb83cdb586c9788 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 02:52:12.788754 989 consensus_peers.cc:597] T dabb8d760aae4aa09eb83cdb586c9788 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 -> Peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Couldn't send request to peer b495a49349324f7d81908f2da700f0d9. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20251029 02:52:12.795405 989 consensus_peers.cc:597] T 30d03e1e42d34242bd354f9e9ec4f127 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 -> Peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Couldn't send request to peer b495a49349324f7d81908f2da700f0d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 30d03e1e42d34242bd354f9e9ec4f127. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:12.803242 1175 ts_tablet_manager.cc:1929] T dabb8d760aae4aa09eb83cdb586c9788 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:12.803390 1175 log.cc:1199] T dabb8d760aae4aa09eb83cdb586c9788 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/dabb8d760aae4aa09eb83cdb586c9788
I20251029 02:52:12.803794 1175 ts_tablet_manager.cc:1950] T dabb8d760aae4aa09eb83cdb586c9788 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:12.804548 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 826) took 4444 ms. Trace:
I20251029 02:52:12.804638 1175 rpcz_store.cc:276] 1029 02:52:08.359900 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:08.359995 (+ 95us) service_pool.cc:225] Handling call
1029 02:52:12.784087 (+4424092us) ts_tablet_manager.cc:1151] Deleting tablet dabb8d760aae4aa09eb83cdb586c9788
1029 02:52:12.784097 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:12.800284 (+ 16187us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:12.803147 (+ 2863us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:12.804530 (+ 1383us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4423818}
I20251029 02:52:12.804993 1175 tablet_replica.cc:333] T 71b4b91071a54d6aa66160e7955bc915 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:12.805153 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet dabb8d760aae4aa09eb83cdb586c9788 (table default.Table_d897a86961894f1aaad67274b687080e [id=b3e544a043444b77a597c6c65c15b795]) successfully deleted
I20251029 02:52:12.813346 1176 rpcz_store.cc:276] 1029 02:52:10.043481 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:10.043525 (+ 44us) service_pool.cc:225] Handling call
1029 02:52:12.752705 (+2709180us) ts_tablet_manager.cc:1151] Deleting tablet ac162bce5b6e4080839095a045ef0cc0
1029 02:52:12.752713 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:12.752791 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2709012}
I20251029 02:52:12.814107 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 919) took 2770 ms. Trace:
I20251029 02:52:12.814304 1176 rpcz_store.cc:276] 1029 02:52:10.044027 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:10.044075 (+ 48us) service_pool.cc:225] Handling call
1029 02:52:12.813909 (+2769834us) ts_tablet_manager.cc:1151] Deleting tablet 591a6c86fc984bac9fb9eb62e35dc6c1
1029 02:52:12.813926 (+ 17us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:12.814050 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2769621}
I20251029 02:52:12.814759 1176 tablet_replica.cc:333] T 1c53fc1fff1a4bb988beb46ff91fd3be P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
W20251029 02:52:12.815266 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet ac162bce5b6e4080839095a045ef0cc0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ac162bce5b6e4080839095a045ef0cc0 already in progress: creating tablet
W20251029 02:52:12.815637 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 591a6c86fc984bac9fb9eb62e35dc6c1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 591a6c86fc984bac9fb9eb62e35dc6c1 already in progress: creating tablet
I20251029 02:52:12.826608 2430 raft_consensus.cc:493] T a9e0fcc759a64679b3f59a6c964877b6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:12.826812 2430 raft_consensus.cc:515] T a9e0fcc759a64679b3f59a6c964877b6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:12.826942 2425 raft_consensus.cc:493] T d041e87ceb11441983d0e38b7929c3a4 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:12.827078 2425 raft_consensus.cc:515] T d041e87ceb11441983d0e38b7929c3a4 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:12.827577 2430 leader_election.cc:290] T a9e0fcc759a64679b3f59a6c964877b6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:12.827656 2425 leader_election.cc:290] T d041e87ceb11441983d0e38b7929c3a4 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:12.828233 1061 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d041e87ceb11441983d0e38b7929c3a4" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:52:12.828815 849 leader_election.cc:343] T d041e87ceb11441983d0e38b7929c3a4 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:12.828809 919 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a9e0fcc759a64679b3f59a6c964877b6" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:12.830041 990 leader_election.cc:343] T a9e0fcc759a64679b3f59a6c964877b6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:12.831480 1176 raft_consensus.cc:2243] T 1c53fc1fff1a4bb988beb46ff91fd3be P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:12.831955 1176 raft_consensus.cc:2272] T 1c53fc1fff1a4bb988beb46ff91fd3be P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:12.833025 1176 ts_tablet_manager.cc:1916] T 1c53fc1fff1a4bb988beb46ff91fd3be P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:12.838357 2430 consensus_queue.cc:1048] T 82289e0f823c4d43bfd8c4e878572b2b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:12.840747 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d041e87ceb11441983d0e38b7929c3a4" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:12.840986 772 raft_consensus.cc:2468] T d041e87ceb11441983d0e38b7929c3a4 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:12.841441 847 leader_election.cc:304] T d041e87ceb11441983d0e38b7929c3a4 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5
I20251029 02:52:12.841791 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a9e0fcc759a64679b3f59a6c964877b6" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:12.842000 2205 raft_consensus.cc:2804] T d041e87ceb11441983d0e38b7929c3a4 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:12.842005 776 raft_consensus.cc:2468] T a9e0fcc759a64679b3f59a6c964877b6 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 0.
I20251029 02:52:12.842095 2205 raft_consensus.cc:493] T d041e87ceb11441983d0e38b7929c3a4 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:12.842170 2205 raft_consensus.cc:3060] T d041e87ceb11441983d0e38b7929c3a4 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:12.843024 989 leader_election.cc:304] T a9e0fcc759a64679b3f59a6c964877b6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters: c756528642a543eeacd93cef3e335a7d
I20251029 02:52:12.843466 2179 raft_consensus.cc:2804] T a9e0fcc759a64679b3f59a6c964877b6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:12.843590 2179 raft_consensus.cc:493] T a9e0fcc759a64679b3f59a6c964877b6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:12.843691 2179 raft_consensus.cc:3060] T a9e0fcc759a64679b3f59a6c964877b6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:12.849000 2205 raft_consensus.cc:515] T d041e87ceb11441983d0e38b7929c3a4 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:12.849977 1055 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d041e87ceb11441983d0e38b7929c3a4" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
W20251029 02:52:12.850608 849 leader_election.cc:343] T d041e87ceb11441983d0e38b7929c3a4 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:12.850976 2179 raft_consensus.cc:515] T a9e0fcc759a64679b3f59a6c964877b6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:12.851532 2179 leader_election.cc:290] T a9e0fcc759a64679b3f59a6c964877b6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Requested vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:12.852118 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a9e0fcc759a64679b3f59a6c964877b6" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
W20251029 02:52:12.852710 990 leader_election.cc:343] T a9e0fcc759a64679b3f59a6c964877b6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:12.856262 2205 leader_election.cc:290] T d041e87ceb11441983d0e38b7929c3a4 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:12.856213 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d041e87ceb11441983d0e38b7929c3a4" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:12.856431 772 raft_consensus.cc:3060] T d041e87ceb11441983d0e38b7929c3a4 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:12.856999 1167 tablet_bootstrap.cc:492] T a9e0fcc759a64679b3f59a6c964877b6 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:12.857158 1167 ts_tablet_manager.cc:1403] T a9e0fcc759a64679b3f59a6c964877b6 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.101s user 0.003s sys 0.000s
I20251029 02:52:12.858152 1167 raft_consensus.cc:359] T a9e0fcc759a64679b3f59a6c964877b6 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:12.858351 1167 raft_consensus.cc:385] T a9e0fcc759a64679b3f59a6c964877b6 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:12.858281 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a9e0fcc759a64679b3f59a6c964877b6" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:12.858398 1167 raft_consensus.cc:740] T a9e0fcc759a64679b3f59a6c964877b6 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:12.858467 776 raft_consensus.cc:3060] T a9e0fcc759a64679b3f59a6c964877b6 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:12.858619 1167 consensus_queue.cc:260] T a9e0fcc759a64679b3f59a6c964877b6 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:12.859397 1167 ts_tablet_manager.cc:1434] T a9e0fcc759a64679b3f59a6c964877b6 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251029 02:52:12.859659 1167 tablet_bootstrap.cc:492] T 67651306f3b44e7eb14dcee9a2007a81 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:12.864718 776 raft_consensus.cc:2468] T a9e0fcc759a64679b3f59a6c964877b6 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 1.
I20251029 02:52:12.865305 989 leader_election.cc:304] T a9e0fcc759a64679b3f59a6c964877b6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters: c756528642a543eeacd93cef3e335a7d
I20251029 02:52:12.865993 2430 raft_consensus.cc:2804] T a9e0fcc759a64679b3f59a6c964877b6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:12.866159 2430 raft_consensus.cc:697] T a9e0fcc759a64679b3f59a6c964877b6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Becoming Leader. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Running, Role: LEADER
I20251029 02:52:12.866412 2430 consensus_queue.cc:237] T a9e0fcc759a64679b3f59a6c964877b6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:12.874259 2416 raft_consensus.cc:493] T ae4231daa2594258aa51419c97a2f50a P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:12.877013 1174 ts_tablet_manager.cc:1929] T c9dff4dff82f49e7ad49caad5e404f09 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:12.877156 1174 log.cc:1199] T c9dff4dff82f49e7ad49caad5e404f09 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/c9dff4dff82f49e7ad49caad5e404f09
I20251029 02:52:12.877525 1174 ts_tablet_manager.cc:1950] T c9dff4dff82f49e7ad49caad5e404f09 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:12.874399 2416 raft_consensus.cc:515] T ae4231daa2594258aa51419c97a2f50a P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:12.880070 2416 leader_election.cc:290] T ae4231daa2594258aa51419c97a2f50a P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:52:12.881121 1055 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae4231daa2594258aa51419c97a2f50a" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:52:12.881657 712 leader_election.cc:343] T ae4231daa2594258aa51419c97a2f50a P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:12.885438 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae4231daa2594258aa51419c97a2f50a" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:12.885898 711 leader_election.cc:343] T ae4231daa2594258aa51419c97a2f50a P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:12.885995 711 leader_election.cc:304] T ae4231daa2594258aa51419c97a2f50a P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:12.886411 2416 raft_consensus.cc:2749] T ae4231daa2594258aa51419c97a2f50a P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:12.894443 2205 raft_consensus.cc:493] T 2f198c3259864f7ab3625cb7bfae20b0 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:12.894640 2205 raft_consensus.cc:515] T 2f198c3259864f7ab3625cb7bfae20b0 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:12.895777 1055 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2f198c3259864f7ab3625cb7bfae20b0" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:12.896735 2205 leader_election.cc:290] T 2f198c3259864f7ab3625cb7bfae20b0 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
W20251029 02:52:12.899627 2513 log.cc:927] Time spent T 643e20d9974447daa78885b3401f3b77 P b495a49349324f7d81908f2da700f0d9: Append to log took a long time: real 0.102s user 0.000s sys 0.000s
I20251029 02:52:12.901855 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 846) took 4299 ms. Trace:
I20251029 02:52:12.901957 1174 rpcz_store.cc:276] 1029 02:52:08.601880 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:08.601938 (+ 58us) service_pool.cc:225] Handling call
1029 02:52:12.723783 (+4121845us) ts_tablet_manager.cc:1151] Deleting tablet c9dff4dff82f49e7ad49caad5e404f09
1029 02:52:12.723796 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:12.848385 (+124589us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:12.876875 (+ 28490us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:12.901836 (+ 24961us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4121671}
I20251029 02:52:12.902402 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet c9dff4dff82f49e7ad49caad5e404f09 (table default.Table_282e9d0ae9384d3284e662fa864925a9 [id=e3b3113e72b747f2a2b55f3b8fae96c3]) successfully deleted
I20251029 02:52:12.903079 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 845) took 4301 ms. Trace:
I20251029 02:52:12.903286 1174 rpcz_store.cc:276] 1029 02:52:08.601815 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:08.601945 (+ 130us) service_pool.cc:225] Handling call
1029 02:52:12.902963 (+4301018us) ts_tablet_manager.cc:1151] Deleting tablet d3faa2d4df584c6e8da426a3830a6454
1029 02:52:12.902973 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:12.903062 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4300783}
W20251029 02:52:12.904006 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet d3faa2d4df584c6e8da426a3830a6454 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d3faa2d4df584c6e8da426a3830a6454 already in progress: creating tablet
I20251029 02:52:12.904409 1174 tablet_replica.cc:333] T 71b4b91071a54d6aa66160e7955bc915 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:12.904692 1174 raft_consensus.cc:2243] T 71b4b91071a54d6aa66160e7955bc915 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:12.905074 1174 raft_consensus.cc:2272] T 71b4b91071a54d6aa66160e7955bc915 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:12.906054 1174 ts_tablet_manager.cc:1916] T 71b4b91071a54d6aa66160e7955bc915 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:12.914817 1167 tablet_bootstrap.cc:654] T 67651306f3b44e7eb14dcee9a2007a81 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:12.926210 772 raft_consensus.cc:2468] T d041e87ceb11441983d0e38b7929c3a4 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:12.928870 2179 consensus_queue.cc:1048] T 44c0488eba0543078ff4f0b3d69366f2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:12.934024 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2f198c3259864f7ab3625cb7bfae20b0" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:12.934310 772 raft_consensus.cc:2393] T 2f198c3259864f7ab3625cb7bfae20b0 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c756528642a543eeacd93cef3e335a7d in current term 1: Already voted for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in this term.
I20251029 02:52:12.934616 2179 consensus_queue.cc:1048] T 217144957930495aa3e9dfcfde268e71 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:12.935417 847 leader_election.cc:304] T d041e87ceb11441983d0e38b7929c3a4 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5
I20251029 02:52:12.936113 2425 raft_consensus.cc:2804] T d041e87ceb11441983d0e38b7929c3a4 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:12.936259 2425 raft_consensus.cc:697] T d041e87ceb11441983d0e38b7929c3a4 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:52:12.937129 2425 consensus_queue.cc:237] T d041e87ceb11441983d0e38b7929c3a4 P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:12.937712 2416 consensus_queue.cc:1048] T af4860c6b72f4d988b206b5ef2f22906 P b495a49349324f7d81908f2da700f0d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:12.938855 1172 tablet_bootstrap.cc:492] T 72f9d4408f384e7093316d85f092e1b2 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:12.939007 1172 ts_tablet_manager.cc:1403] T 72f9d4408f384e7093316d85f092e1b2 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.243s user 0.007s sys 0.005s
I20251029 02:52:12.940783 1175 raft_consensus.cc:2243] T 71b4b91071a54d6aa66160e7955bc915 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:12.940984 1175 raft_consensus.cc:2272] T 71b4b91071a54d6aa66160e7955bc915 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:12.941645 1176 ts_tablet_manager.cc:1929] T 1c53fc1fff1a4bb988beb46ff91fd3be P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251029 02:52:12.941838 1176 log.cc:1199] T 1c53fc1fff1a4bb988beb46ff91fd3be P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/1c53fc1fff1a4bb988beb46ff91fd3be
I20251029 02:52:12.942234 1175 ts_tablet_manager.cc:1916] T 71b4b91071a54d6aa66160e7955bc915 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:12.942301 1176 ts_tablet_manager.cc:1950] T 1c53fc1fff1a4bb988beb46ff91fd3be P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:12.942698 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 920) took 2898 ms. Trace:
I20251029 02:52:12.942799 1176 rpcz_store.cc:276] 1029 02:52:10.044404 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:10.044448 (+ 44us) service_pool.cc:225] Handling call
1029 02:52:12.814674 (+2770226us) ts_tablet_manager.cc:1151] Deleting tablet 1c53fc1fff1a4bb988beb46ff91fd3be
1029 02:52:12.814686 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:12.865953 (+ 51267us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:12.941480 (+ 75527us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:12.942661 (+ 1181us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2770068}
I20251029 02:52:12.943339 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 1c53fc1fff1a4bb988beb46ff91fd3be (table default.Table_dd4934aabcd442039db111954d819f6d [id=9abdb27ccc0a48e991b7a4619808fdf8]) successfully deleted
I20251029 02:52:12.945428 847 leader_election.cc:304] T 2f198c3259864f7ab3625cb7bfae20b0 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9
I20251029 02:52:12.947535 1176 tablet_replica.cc:333] T c9e923724ca9402682f13b252dbc37aa P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:12.948239 1176 raft_consensus.cc:2243] T c9e923724ca9402682f13b252dbc37aa P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:12.948598 1167 tablet_bootstrap.cc:492] T 67651306f3b44e7eb14dcee9a2007a81 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:12.948854 1167 ts_tablet_manager.cc:1403] T 67651306f3b44e7eb14dcee9a2007a81 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.089s user 0.004s sys 0.001s
I20251029 02:52:12.949848 1167 raft_consensus.cc:359] T 67651306f3b44e7eb14dcee9a2007a81 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:12.950085 1167 raft_consensus.cc:385] T 67651306f3b44e7eb14dcee9a2007a81 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:12.950178 1167 raft_consensus.cc:740] T 67651306f3b44e7eb14dcee9a2007a81 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:12.950867 1167 consensus_queue.cc:260] T 67651306f3b44e7eb14dcee9a2007a81 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:12.951802 2302 raft_consensus.cc:3060] T 2f198c3259864f7ab3625cb7bfae20b0 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:12.952488 1167 ts_tablet_manager.cc:1434] T 67651306f3b44e7eb14dcee9a2007a81 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251029 02:52:12.953274 1167 tablet_bootstrap.cc:492] T f281191e7c6143f983f11d87f1c9b64e P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:12.954734 2302 raft_consensus.cc:2749] T 2f198c3259864f7ab3625cb7bfae20b0 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:12.961179 2179 consensus_queue.cc:1048] T 44c0488eba0543078ff4f0b3d69366f2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:12.965699 1176 raft_consensus.cc:2272] T c9e923724ca9402682f13b252dbc37aa P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:12.966491 1176 ts_tablet_manager.cc:1916] T c9e923724ca9402682f13b252dbc37aa P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:12.971275 2452 tablet.cc:1722] T 643e20d9974447daa78885b3401f3b77 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 02:52:12.977552 1175 ts_tablet_manager.cc:1929] T 71b4b91071a54d6aa66160e7955bc915 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:12.977701 1175 log.cc:1199] T 71b4b91071a54d6aa66160e7955bc915 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/71b4b91071a54d6aa66160e7955bc915
I20251029 02:52:12.979003 1175 ts_tablet_manager.cc:1950] T 71b4b91071a54d6aa66160e7955bc915 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:12.979907 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 831) took 4449 ms. Trace:
I20251029 02:52:12.980005 1175 rpcz_store.cc:276] 1029 02:52:08.530902 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:08.530966 (+ 64us) service_pool.cc:225] Handling call
1029 02:52:12.804946 (+4273980us) ts_tablet_manager.cc:1151] Deleting tablet 71b4b91071a54d6aa66160e7955bc915
1029 02:52:12.804955 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:12.946276 (+141321us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:12.977421 (+ 31145us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:12.979883 (+ 2462us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":135516,"tablet-delete.queue_time_us":4273814}
I20251029 02:52:12.980417 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 832) took 4448 ms. Trace:
I20251029 02:52:12.980491 1175 rpcz_store.cc:276] 1029 02:52:08.531821 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:08.531874 (+ 53us) service_pool.cc:225] Handling call
1029 02:52:12.980294 (+4448420us) ts_tablet_manager.cc:1151] Deleting tablet d3faa2d4df584c6e8da426a3830a6454
1029 02:52:12.980304 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:12.980401 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4448269}
I20251029 02:52:12.980901 1175 tablet_replica.cc:333] T c9dff4dff82f49e7ad49caad5e404f09 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:12.981127 1175 raft_consensus.cc:2243] T c9dff4dff82f49e7ad49caad5e404f09 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:12.981312 1175 raft_consensus.cc:2272] T c9dff4dff82f49e7ad49caad5e404f09 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:12.981916 1175 ts_tablet_manager.cc:1916] T c9dff4dff82f49e7ad49caad5e404f09 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:12.983644 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 71b4b91071a54d6aa66160e7955bc915 (table default.Table_457a3835f7d84fd7bf722dfc1d18d301 [id=68645701bb494d9fa65e7c1ff7c9e14c]) successfully deleted
W20251029 02:52:12.984021 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet d3faa2d4df584c6e8da426a3830a6454 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d3faa2d4df584c6e8da426a3830a6454 already in progress: creating tablet
I20251029 02:52:12.990157 2430 consensus_queue.cc:1048] T 2f198c3259864f7ab3625cb7bfae20b0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:12.992286 2430 consensus_queue.cc:1048] T 217144957930495aa3e9dfcfde268e71 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:12.995534 1172 raft_consensus.cc:359] T 72f9d4408f384e7093316d85f092e1b2 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:12.995750 1172 raft_consensus.cc:385] T 72f9d4408f384e7093316d85f092e1b2 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:12.995810 1172 raft_consensus.cc:740] T 72f9d4408f384e7093316d85f092e1b2 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:12.996030 1172 consensus_queue.cc:260] T 72f9d4408f384e7093316d85f092e1b2 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:12.996577 1172 ts_tablet_manager.cc:1434] T 72f9d4408f384e7093316d85f092e1b2 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.057s user 0.001s sys 0.000s
I20251029 02:52:12.996923 1172 tablet_bootstrap.cc:492] T eca016aed05e4ce09e1a2bf96e2286f5 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:12.997572 521 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_0427cb4a53de4e62a4b89829586a2a53" } modify_external_catalogs: true
I20251029 02:52:12.997843 521 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_0427cb4a53de4e62a4b89829586a2a53" } modify_external_catalogs: true
I20251029 02:52:12.999732 1167 tablet_bootstrap.cc:654] T f281191e7c6143f983f11d87f1c9b64e P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:13.001822 519 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet 95f342d1cfb84c4eb31bd3d15ea3bdc2 (table default.Table_96ea715e50e74cc48759dda025a46318 [id=c63d52811db749d6badbed72fc133943]). Expected version 1 got 0
I20251029 02:52:13.002373 522 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet ca71c4638a6b419b876e22af7e6cf00a
I20251029 02:52:13.003007 522 catalog_manager.cc:6719] ReplaceTablet: tablet ca71c4638a6b419b876e22af7e6cf00a deleted and replaced by tablet c977cf36300648d6a6a2236d358fc02d
I20251029 02:52:13.003366 522 master_service.cc:954] ReplaceTablet: received request to replace tablet f2d47238761e4ed49c88abdbb677b18a from {username='slave'} at 127.0.0.1:35314
I20251029 02:52:13.006675 896 tablet_service.cc:1552] Processing DeleteTablet for tablet ca71c4638a6b419b876e22af7e6cf00a with delete_type TABLET_DATA_DELETED (replaced by tablet c977cf36300648d6a6a2236d358fc02d) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:13.007006 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_8dbb47d3c55c40a1b33c0567d0eff749"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:13.007157 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet ca71c4638a6b419b876e22af7e6cf00a with delete_type TABLET_DATA_DELETED (replaced by tablet c977cf36300648d6a6a2236d358fc02d) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:13.007440 525 catalog_manager.cc:6194] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: default.Table_902ea0b243c7400c86250cfb5fae5b54 [id=d0ca7c93c8cf4dc19962a1957321c5b7] alter complete (version 5)
I20251029 02:52:13.007617 1176 ts_tablet_manager.cc:1929] T c9e923724ca9402682f13b252dbc37aa P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20251029 02:52:13.007668 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8dbb47d3c55c40a1b33c0567d0eff749 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:13.007776 1176 log.cc:1199] T c9e923724ca9402682f13b252dbc37aa P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/c9e923724ca9402682f13b252dbc37aa
I20251029 02:52:13.008114 1174 ts_tablet_manager.cc:1929] T 71b4b91071a54d6aa66160e7955bc915 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:13.008244 1174 log.cc:1199] T 71b4b91071a54d6aa66160e7955bc915 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/71b4b91071a54d6aa66160e7955bc915
I20251029 02:52:13.008316 738 tablet_service.cc:1552] Processing DeleteTablet for tablet ca71c4638a6b419b876e22af7e6cf00a with delete_type TABLET_DATA_DELETED (replaced by tablet c977cf36300648d6a6a2236d358fc02d) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:13.008580 1174 ts_tablet_manager.cc:1950] T 71b4b91071a54d6aa66160e7955bc915 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:13.009416 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 844) took 4407 ms. Trace:
I20251029 02:52:13.010084 522 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_0936756f462041d6a93c1401ceb56158" } new_table_name: "default.Table_b52bfade79b84f65973bee918424d4a0" modify_external_catalogs: true
W20251029 02:52:13.016302 520 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cdb558f14ab34d5ea4b1e9280db3c358 consensus state has no leader
I20251029 02:52:13.009526 1174 rpcz_store.cc:276] 1029 02:52:08.601745 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:08.602207 (+ 462us) service_pool.cc:225] Handling call
1029 02:52:12.904354 (+4302147us) ts_tablet_manager.cc:1151] Deleting tablet 71b4b91071a54d6aa66160e7955bc915
1029 02:52:12.904367 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:12.910004 (+ 5637us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:13.004079 (+ 94075us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:13.009393 (+ 5314us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4302019}
I20251029 02:52:13.016731 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_957f906aceb6449fb91b40e913a605c0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:13.017024 1174 tablet_replica.cc:333] T c9bd3f2deb5d4af09b7918269a98509f P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:13.017211 1174 raft_consensus.cc:2243] T c9bd3f2deb5d4af09b7918269a98509f P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus shutting down.
W20251029 02:52:13.017323 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_957f906aceb6449fb91b40e913a605c0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:13.017357 1174 raft_consensus.cc:2272] T c9bd3f2deb5d4af09b7918269a98509f P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:13.017892 1174 ts_tablet_manager.cc:1916] T c9bd3f2deb5d4af09b7918269a98509f P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:13.019774 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 71b4b91071a54d6aa66160e7955bc915 (table default.Table_457a3835f7d84fd7bf722dfc1d18d301 [id=68645701bb494d9fa65e7c1ff7c9e14c]) successfully deleted
I20251029 02:52:13.020747 1175 ts_tablet_manager.cc:1929] T c9dff4dff82f49e7ad49caad5e404f09 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:13.020865 1175 log.cc:1199] T c9dff4dff82f49e7ad49caad5e404f09 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/c9dff4dff82f49e7ad49caad5e404f09
I20251029 02:52:13.029079 1175 ts_tablet_manager.cc:1950] T c9dff4dff82f49e7ad49caad5e404f09 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:13.030000 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 833) took 4495 ms. Trace:
I20251029 02:52:13.030118 1175 rpcz_store.cc:276] 1029 02:52:08.534574 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:08.534621 (+ 47us) service_pool.cc:225] Handling call
1029 02:52:12.980689 (+4446068us) ts_tablet_manager.cc:1151] Deleting tablet c9dff4dff82f49e7ad49caad5e404f09
1029 02:52:12.980697 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:12.987397 (+ 6700us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:13.020629 (+ 33232us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:13.029971 (+ 9342us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4445910}
I20251029 02:52:13.030663 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet c9dff4dff82f49e7ad49caad5e404f09 (table default.Table_282e9d0ae9384d3284e662fa864925a9 [id=e3b3113e72b747f2a2b55f3b8fae96c3]) successfully deleted
I20251029 02:52:13.031379 1175 tablet_replica.cc:333] T 372d93914abc41b38a702742b7db0bc2 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:13.031674 1175 raft_consensus.cc:2243] T 372d93914abc41b38a702742b7db0bc2 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:13.031912 1175 raft_consensus.cc:2272] T 372d93914abc41b38a702742b7db0bc2 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:13.034890 1176 ts_tablet_manager.cc:1950] T c9e923724ca9402682f13b252dbc37aa P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:13.035563 1166 tablet_bootstrap.cc:492] T d041e87ceb11441983d0e38b7929c3a4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:13.035812 1166 ts_tablet_manager.cc:1403] T d041e87ceb11441983d0e38b7929c3a4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.385s user 0.040s sys 0.081s
I20251029 02:52:13.035876 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 934) took 2757 ms. Trace:
I20251029 02:52:13.035989 1176 rpcz_store.cc:276] 1029 02:52:10.278485 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:10.324150 (+ 45665us) service_pool.cc:225] Handling call
1029 02:52:12.947456 (+2623306us) ts_tablet_manager.cc:1151] Deleting tablet c9e923724ca9402682f13b252dbc37aa
1029 02:52:12.947466 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:12.970500 (+ 23034us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:13.007484 (+ 36984us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:13.035825 (+ 28341us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2623091}
I20251029 02:52:13.036535 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet c9e923724ca9402682f13b252dbc37aa (table default.Table_443f33587b8f428484b99e07355d69ca [id=03a25420ce6a46fc93a9fb753246bc7b]) successfully deleted
I20251029 02:52:13.037009 1166 raft_consensus.cc:359] T d041e87ceb11441983d0e38b7929c3a4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:13.037335 1166 raft_consensus.cc:385] T d041e87ceb11441983d0e38b7929c3a4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:13.037468 1166 raft_consensus.cc:740] T d041e87ceb11441983d0e38b7929c3a4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:13.037817 1166 consensus_queue.cc:260] T d041e87ceb11441983d0e38b7929c3a4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
W20251029 02:52:13.038704 510 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet cdb558f14ab34d5ea4b1e9280db3c358 (table default.Table_a5f63619b78649ea878ad36c8b5c8143 [id=aea283176c6f49c1b94bc313494d486d], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet cdb558f14ab34d5ea4b1e9280db3c358 consensus state has no leader
I20251029 02:52:13.041484 2453 tablet.cc:1722] T 1bdb67125493430398ff5c80b73de2ac P c756528642a543eeacd93cef3e335a7d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251029 02:52:13.049863 1175 ts_tablet_manager.cc:1916] T 372d93914abc41b38a702742b7db0bc2 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:13.051847 2452 tablet.cc:1722] T 1bdb67125493430398ff5c80b73de2ac P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251029 02:52:13.055300 2188 consensus_queue.cc:1048] T 2f198c3259864f7ab3625cb7bfae20b0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:13.065165 1166 ts_tablet_manager.cc:1434] T d041e87ceb11441983d0e38b7929c3a4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.029s user 0.001s sys 0.001s
W20251029 02:52:13.065428 2517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251029 02:52:13.065533 1166 tablet_bootstrap.cc:492] T 67651306f3b44e7eb14dcee9a2007a81 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:13.066045 1968 server_base.cc:1047] running on GCE node
W20251029 02:52:13.066558 2543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251029 02:52:13.067783 1968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251029 02:52:13.072602 1166 tablet_bootstrap.cc:654] T 67651306f3b44e7eb14dcee9a2007a81 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
W20251029 02:52:13.080514 990 consensus_peers.cc:597] T dabb8d760aae4aa09eb83cdb586c9788 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 -> Peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Couldn't send request to peer c756528642a543eeacd93cef3e335a7d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dabb8d760aae4aa09eb83cdb586c9788. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:13.084846 1167 tablet_bootstrap.cc:492] T f281191e7c6143f983f11d87f1c9b64e P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:13.084993 1167 ts_tablet_manager.cc:1403] T f281191e7c6143f983f11d87f1c9b64e P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.132s user 0.004s sys 0.001s
I20251029 02:52:13.085714 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_6aacefe556ee42399e96836738dab1f2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:13.086383 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6aacefe556ee42399e96836738dab1f2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:13.086452 526 catalog_manager.cc:6194] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: default.Table_9120016827e946cc8a145252d2569d2b [id=e0c0cf3d456444c191a32bb180e83798] alter complete (version 1)
W20251029 02:52:13.087265 525 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.2:37633 (request call id 138) took 785 ms (client timeout 1000 ms). Trace:
I20251029 02:52:13.088100 1968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
W20251029 02:52:13.087385 525 rpcz_store.cc:269] 1029 02:52:12.302162 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:12.317783 (+ 15621us) service_pool.cc:225] Handling call
1029 02:52:12.362951 (+ 45168us) spinlock_profiling.cc:243] Waited 44 ms on lock 0x616000041254. stack: 00007f4b1e4899f6 00007f4b1e4897b0 00007f4b1d4653ad 0000000000983f04 0000000000983ecd 00007f4b2d7906bd 00007f4b2d7907b3 00007f4b2a357441 00007f4b320119dd 00007f4b32198aa8 00007f4b28cabca6 00007f4b28cab9d5 00007f4b2542827e 00007f4b2542715e 00007f4b2542d395 00007f4b2542dcbf
1029 02:52:13.087210 (+724259us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1029 02:52:12.493364 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1029 02:52:12.494576 (+ 1212us) write_op.cc:432] Acquiring schema lock in shared mode
1029 02:52:12.494592 (+ 16us) write_op.cc:435] Acquired schema lock
1029 02:52:12.494597 (+ 5us) tablet.cc:660] Decoding operations
1029 02:52:12.494733 (+ 136us) write_op.cc:620] Acquiring the partition lock for write op
1029 02:52:12.494752 (+ 19us) write_op.cc:641] Partition lock acquired for write op
1029 02:52:12.494759 (+ 7us) tablet.cc:683] Acquiring locks for 1 operations
1029 02:52:12.494846 (+ 87us) tablet.cc:699] Row locks acquired
1029 02:52:12.494850 (+ 4us) write_op.cc:260] PREPARE: finished
1029 02:52:12.494904 (+ 54us) write_op.cc:270] Start()
1029 02:52:12.494962 (+ 58us) write_op.cc:276] Timestamp: P: 1761706332494849 usec, L: 0
1029 02:52:12.494968 (+ 6us) op_driver.cc:348] REPLICATION: starting
1029 02:52:12.495178 (+ 210us) log.cc:844] Serialized 527 byte log entry
1029 02:52:12.578900 (+ 83722us) op_driver.cc:464] REPLICATION: finished
1029 02:52:12.579128 (+ 228us) write_op.cc:301] APPLY: starting
1029 02:52:12.579145 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
1029 02:52:12.579181 (+ 36us) tablet.cc:1368] finished BulkCheckPresence
1029 02:52:12.579204 (+ 23us) tablet.cc:1370] starting ApplyRowOperation cycle
1029 02:52:12.579252 (+ 48us) tablet.cc:1381] finished ApplyRowOperation cycle
1029 02:52:12.579282 (+ 30us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1029 02:52:12.579288 (+ 6us) write_op.cc:312] APPLY: finished
1029 02:52:12.579371 (+ 83us) log.cc:844] Serialized 23 byte log entry
1029 02:52:12.579421 (+ 50us) write_op.cc:489] Releasing partition, row and schema locks
1029 02:52:12.579437 (+ 16us) write_op.cc:454] Released schema lock
1029 02:52:12.579443 (+ 6us) write_op.cc:341] FINISH: Updating metrics
1029 02:52:12.579517 (+ 74us) write_op.cc:489] Releasing partition, row and schema locks
1029 02:52:12.579521 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
1029 02:52:12.586902 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1029 02:52:12.587067 (+ 165us) write_op.cc:432] Acquiring schema lock in shared mode
1029 02:52:12.587073 (+ 6us) write_op.cc:435] Acquired schema lock
1029 02:52:12.587075 (+ 2us) tablet.cc:660] Decoding operations
1029 02:52:12.587182 (+ 107us) write_op.cc:620] Acquiring the partition lock for write op
1029 02:52:12.587193 (+ 11us) write_op.cc:641] Partition lock acquired for write op
1029 02:52:12.587197 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
1029 02:52:12.587264 (+ 67us) tablet.cc:699] Row locks acquired
1029 02:52:12.587267 (+ 3us) write_op.cc:260] PREPARE: finished
1029 02:52:12.587304 (+ 37us) write_op.cc:270] Start()
1029 02:52:12.587345 (+ 41us) write_op.cc:276] Timestamp: P: 1761706332587240 usec, L: 0
1029 02:52:12.587348 (+ 3us) op_driver.cc:348] REPLICATION: starting
1029 02:52:12.587795 (+ 447us) log.cc:844] Serialized 356 byte log entry
1029 02:52:12.613606 (+ 25811us) op_driver.cc:464] REPLICATION: finished
1029 02:52:12.613711 (+ 105us) write_op.cc:301] APPLY: starting
1029 02:52:12.613725 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
1029 02:52:12.613762 (+ 37us) tablet.cc:1368] finished BulkCheckPresence
1029 02:52:12.613789 (+ 27us) tablet.cc:1370] starting ApplyRowOperation cycle
1029 02:52:12.613843 (+ 54us) tablet.cc:1381] finished ApplyRowOperation cycle
1029 02:52:12.613878 (+ 35us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1029 02:52:12.613885 (+ 7us) write_op.cc:312] APPLY: finished
1029 02:52:12.613958 (+ 73us) log.cc:844] Serialized 23 byte log entry
1029 02:52:12.614017 (+ 59us) write_op.cc:489] Releasing partition, row and schema locks
1029 02:52:12.614034 (+ 17us) write_op.cc:454] Released schema lock
1029 02:52:12.614040 (+ 6us) write_op.cc:341] FINISH: Updating metrics
1029 02:52:12.614117 (+ 77us) write_op.cc:489] Releasing partition, row and schema locks
1029 02:52:12.614121 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
1029 02:52:12.735042 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1029 02:52:12.735132 (+ 90us) write_op.cc:432] Acquiring schema lock in shared mode
1029 02:52:12.735137 (+ 5us) write_op.cc:435] Acquired schema lock
1029 02:52:12.735139 (+ 2us) tablet.cc:660] Decoding operations
1029 02:52:12.735214 (+ 75us) write_op.cc:620] Acquiring the partition lock for write op
1029 02:52:12.735222 (+ 8us) write_op.cc:641] Partition lock acquired for write op
1029 02:52:12.735225 (+ 3us) tablet.cc:683] Acquiring locks for 1 operations
1029 02:52:12.735294 (+ 69us) tablet.cc:699] Row locks acquired
1029 02:52:12.735296 (+ 2us) write_op.cc:260] PREPARE: finished
1029 02:52:12.735321 (+ 25us) write_op.cc:270] Start()
1029 02:52:12.735856 (+ 535us) write_op.cc:276] Timestamp: P: 1761706332735300 usec, L: 0
1029 02:52:12.735861 (+ 5us) op_driver.cc:348] REPLICATION: starting
1029 02:52:12.736013 (+ 152us) log.cc:844] Serialized 356 byte log entry
1029 02:52:12.999242 (+263229us) op_driver.cc:464] REPLICATION: finished
1029 02:52:13.000168 (+ 926us) write_op.cc:301] APPLY: starting
1029 02:52:13.000184 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
1029 02:52:13.000228 (+ 44us) tablet.cc:1368] finished BulkCheckPresence
1029 02:52:13.000261 (+ 33us) tablet.cc:1370] starting ApplyRowOperation cycle
1029 02:52:13.000315 (+ 54us) tablet.cc:1381] finished ApplyRowOperation cycle
1029 02:52:13.000354 (+ 39us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1029 02:52:13.000366 (+ 12us) write_op.cc:312] APPLY: finished
1029 02:52:13.000431 (+ 65us) log.cc:844] Serialized 23 byte log entry
1029 02:52:13.000470 (+ 39us) write_op.cc:489] Releasing partition, row and schema locks
1029 02:52:13.000484 (+ 14us) write_op.cc:454] Released schema lock
1029 02:52:13.000490 (+ 6us) write_op.cc:341] FINISH: Updating metrics
1029 02:52:13.000556 (+ 66us) write_op.cc:489] Releasing partition, row and schema locks
1029 02:52:13.000562 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":9,"spinlock_wait_cycles":101237504,"child_traces":[["op",{"apply.queue_time_us":171,"apply.run_cpu_time_us":381,"apply.run_wall_time_us":380,"num_ops":1,"prepare.queue_time_us":112236,"prepare.run_cpu_time_us":846,"prepare.run_wall_time_us":1926,"raft.queue_time_us":297,"raft.run_cpu_time_us":197,"raft.run_wall_time_us":198,"replication_time_us":83885}],["op",{"apply.queue_time_us":57,"apply.run_cpu_time_us":397,"apply.run_wall_time_us":396,"num_ops":1,"prepare.queue_time_us":664,"prepare.run_cpu_time_us":756,"prepare.run_wall_time_us":987,"replication_time_us":26223}],["op",{"apply.queue_time_us":869,"apply.run_cpu_time_us":386,"apply.run_wall_time_us":386,"num_ops":1,"prepare.queue_time_us":104339,"prepare.run_cpu_time_us":559,"prepare.run_wall_time_us":1379,"raft.queue_time_us":198,"raft.run_cpu_time_us":3,"raft.run_wall_time_us":4,"replication_time_us":263356}]]}
I20251029 02:52:13.090178 738 tablet_service.cc:1552] Processing DeleteTablet for tablet 71b4b91071a54d6aa66160e7955bc915 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:08 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:13.090382 751 tablet_service.cc:1552] Processing DeleteTablet for tablet 30d03e1e42d34242bd354f9e9ec4f127 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:58 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:13.086385 1167 raft_consensus.cc:359] T f281191e7c6143f983f11d87f1c9b64e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:13.090617 1167 raft_consensus.cc:385] T f281191e7c6143f983f11d87f1c9b64e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:13.090685 1167 raft_consensus.cc:740] T f281191e7c6143f983f11d87f1c9b64e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:13.090886 751 tablet_service.cc:1552] Processing DeleteTablet for tablet dabb8d760aae4aa09eb83cdb586c9788 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:07 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:13.090890 1167 consensus_queue.cc:260] T f281191e7c6143f983f11d87f1c9b64e P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:13.091364 1167 ts_tablet_manager.cc:1434] T f281191e7c6143f983f11d87f1c9b64e P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20251029 02:52:13.091516 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet 44ae2b07edca44afbe1b1fb3a5b1115d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:00 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:13.091614 1167 tablet_bootstrap.cc:492] T 378f549ff7364d018d8b9a5ef01ac878 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:13.092097 1034 tablet_service.cc:1552] Processing DeleteTablet for tablet e65eaaae43ac435ebd665546de9d68dd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:01 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:13.092312 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet afb19ba39399410bb90b6225d7b5b5b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:57 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:13.093199 1167 tablet_bootstrap.cc:654] T 378f549ff7364d018d8b9a5ef01ac878 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:13.095528 1968 hybrid_clock.cc:648] HybridClock initialized: now 1761706333093654 us; error 1859 us; skew 500 ppm
I20251029 02:52:13.096694 1968 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251029 02:52:13.102771 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 955) took 2487 ms. Trace:
I20251029 02:52:13.102888 1176 rpcz_store.cc:276] 1029 02:52:10.615020 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:10.615078 (+ 58us) service_pool.cc:225] Handling call
1029 02:52:13.102612 (+2487534us) ts_tablet_manager.cc:1151] Deleting tablet 23856678b7aa4f9ebf65fb4564a4bf91
1029 02:52:13.102625 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:13.102725 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2487348}
W20251029 02:52:13.103325 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 23856678b7aa4f9ebf65fb4564a4bf91 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 23856678b7aa4f9ebf65fb4564a4bf91 already in progress: creating tablet
I20251029 02:52:13.103410 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 956) took 2488 ms. Trace:
I20251029 02:52:13.103497 1176 rpcz_store.cc:276] 1029 02:52:10.615132 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:10.620330 (+ 5198us) service_pool.cc:225] Handling call
1029 02:52:13.103279 (+2482949us) ts_tablet_manager.cc:1151] Deleting tablet 43a76b984da3414da701221f87349337
1029 02:52:13.103287 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:13.103367 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2482769}
I20251029 02:52:13.103911 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 964) took 2465 ms. Trace:
W20251029 02:52:13.103911 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 43a76b984da3414da701221f87349337 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 43a76b984da3414da701221f87349337 already in progress: creating tablet
I20251029 02:52:13.104003 1176 rpcz_store.cc:276] 1029 02:52:10.638445 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:10.638508 (+ 63us) service_pool.cc:225] Handling call
1029 02:52:13.103805 (+2465297us) ts_tablet_manager.cc:1151] Deleting tablet 4660fd0534df49e0af43c36fb1122d84
1029 02:52:13.103813 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:13.103876 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2465018}
W20251029 02:52:13.104487 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 4660fd0534df49e0af43c36fb1122d84 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4660fd0534df49e0af43c36fb1122d84 already in progress: creating tablet
I20251029 02:52:13.104532 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 967) took 2368 ms. Trace:
I20251029 02:52:13.104602 1176 rpcz_store.cc:276] 1029 02:52:10.736298 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:10.744904 (+ 8606us) service_pool.cc:225] Handling call
1029 02:52:13.104426 (+2359522us) ts_tablet_manager.cc:1151] Deleting tablet eeb75fc46e2a41f78252868ad5256966
1029 02:52:13.104434 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:13.104501 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2359287}
I20251029 02:52:13.105005 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 976) took 2293 ms. Trace:
I20251029 02:52:13.105103 1176 rpcz_store.cc:276] 1029 02:52:10.811071 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:10.812404 (+ 1333us) service_pool.cc:225] Handling call
1029 02:52:13.104890 (+2292486us) ts_tablet_manager.cc:1151] Deleting tablet 877e278deb4a4b01ac66b06be0890baf
1029 02:52:13.104899 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:13.104966 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2154285}
I20251029 02:52:13.105526 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 983) took 2291 ms. Trace:
W20251029 02:52:13.105639 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet eeb75fc46e2a41f78252868ad5256966 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eeb75fc46e2a41f78252868ad5256966 already in progress: creating tablet
I20251029 02:52:13.105616 1176 rpcz_store.cc:276] 1029 02:52:10.813749 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:10.813826 (+ 77us) service_pool.cc:225] Handling call
1029 02:52:13.105401 (+2291575us) ts_tablet_manager.cc:1151] Deleting tablet 0700b93cc398403d8b3602e84182a3ce
1029 02:52:13.105408 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:13.105471 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2148795}
W20251029 02:52:13.105873 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 877e278deb4a4b01ac66b06be0890baf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 877e278deb4a4b01ac66b06be0890baf already in progress: creating tablet
I20251029 02:52:13.106019 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 984) took 2292 ms. Trace:
W20251029 02:52:13.106133 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 0700b93cc398403d8b3602e84182a3ce because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0700b93cc398403d8b3602e84182a3ce already in progress: creating tablet
I20251029 02:52:13.106117 1176 rpcz_store.cc:276] 1029 02:52:10.813854 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:10.814001 (+ 147us) service_pool.cc:225] Handling call
1029 02:52:13.105912 (+2291911us) ts_tablet_manager.cc:1151] Deleting tablet bfb9115f324f44c98f876a2d56bb23a1
1029 02:52:13.105920 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:13.105988 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2148855}
I20251029 02:52:13.106526 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 985) took 2292 ms. Trace:
I20251029 02:52:13.106532 896 tablet_service.cc:1552] Processing DeleteTablet for tablet afb19ba39399410bb90b6225d7b5b5b1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:57 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:13.106619 1176 rpcz_store.cc:276] 1029 02:52:10.813933 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:10.813994 (+ 61us) service_pool.cc:225] Handling call
1029 02:52:13.106401 (+2292407us) ts_tablet_manager.cc:1151] Deleting tablet ef2e9dd8f90a47389d9062d1f99a1da1
1029 02:52:13.106409 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:13.106474 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2149034}
I20251029 02:52:13.107044 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 987) took 2287 ms. Trace:
I20251029 02:52:13.107117 896 tablet_service.cc:1552] Processing DeleteTablet for tablet e65eaaae43ac435ebd665546de9d68dd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:01 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:13.107137 1176 rpcz_store.cc:276] 1029 02:52:10.819298 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:10.819358 (+ 60us) service_pool.cc:225] Handling call
1029 02:52:13.106942 (+2287584us) ts_tablet_manager.cc:1151] Deleting tablet 34d141bee474452693dba1b9d7f2b023
1029 02:52:13.106949 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:13.107008 (+ 59us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2136343}
I20251029 02:52:13.107307 881 tablet_service.cc:1552] Processing DeleteTablet for tablet 2f198c3259864f7ab3625cb7bfae20b0 with delete_type TABLET_DATA_DELETED (replaced by tablet 877e278deb4a4b01ac66b06be0890baf) from {username='slave'} at 127.0.0.1:35702
W20251029 02:52:13.108448 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet bfb9115f324f44c98f876a2d56bb23a1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bfb9115f324f44c98f876a2d56bb23a1 already in progress: creating tablet
W20251029 02:52:13.108668 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet ef2e9dd8f90a47389d9062d1f99a1da1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ef2e9dd8f90a47389d9062d1f99a1da1 already in progress: creating tablet
W20251029 02:52:13.108932 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 34d141bee474452693dba1b9d7f2b023 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 34d141bee474452693dba1b9d7f2b023 already in progress: creating tablet
I20251029 02:52:13.124440 1174 ts_tablet_manager.cc:1929] T c9bd3f2deb5d4af09b7918269a98509f P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:13.124570 1174 log.cc:1199] T c9bd3f2deb5d4af09b7918269a98509f P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/c9bd3f2deb5d4af09b7918269a98509f
I20251029 02:52:13.124846 1174 ts_tablet_manager.cc:1950] T c9bd3f2deb5d4af09b7918269a98509f P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:13.127071 1166 tablet_bootstrap.cc:492] T 67651306f3b44e7eb14dcee9a2007a81 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:13.127246 1166 ts_tablet_manager.cc:1403] T 67651306f3b44e7eb14dcee9a2007a81 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.062s user 0.002s sys 0.002s
I20251029 02:52:13.128072 1166 raft_consensus.cc:359] T 67651306f3b44e7eb14dcee9a2007a81 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:13.128309 1166 raft_consensus.cc:385] T 67651306f3b44e7eb14dcee9a2007a81 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:13.128396 1166 raft_consensus.cc:740] T 67651306f3b44e7eb14dcee9a2007a81 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:13.128731 1166 consensus_queue.cc:260] T 67651306f3b44e7eb14dcee9a2007a81 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:13.129418 1166 ts_tablet_manager.cc:1434] T 67651306f3b44e7eb14dcee9a2007a81 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 02:52:13.129715 1166 tablet_bootstrap.cc:492] T f281191e7c6143f983f11d87f1c9b64e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:13.133093 1175 ts_tablet_manager.cc:1929] T 372d93914abc41b38a702742b7db0bc2 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:13.133236 1175 log.cc:1199] T 372d93914abc41b38a702742b7db0bc2 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/372d93914abc41b38a702742b7db0bc2
I20251029 02:52:13.133563 1166 tablet_bootstrap.cc:654] T f281191e7c6143f983f11d87f1c9b64e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:13.134688 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 992) took 2315 ms. Trace:
I20251029 02:52:13.134795 1176 rpcz_store.cc:276] 1029 02:52:10.819671 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:10.821164 (+ 1493us) service_pool.cc:225] Handling call
1029 02:52:13.131381 (+2310217us) ts_tablet_manager.cc:1151] Deleting tablet 5a1aa131cf4e4edca5bb2515edd7556a
1029 02:52:13.134543 (+ 3162us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:13.134649 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2160677}
W20251029 02:52:13.135273 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 5a1aa131cf4e4edca5bb2515edd7556a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5a1aa131cf4e4edca5bb2515edd7556a already in progress: creating tablet
I20251029 02:52:13.135627 1175 ts_tablet_manager.cc:1950] T 372d93914abc41b38a702742b7db0bc2 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:13.136336 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 844) took 4376 ms. Trace:
I20251029 02:52:13.136430 1175 rpcz_store.cc:276] 1029 02:52:08.760317 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:08.760373 (+ 56us) service_pool.cc:225] Handling call
1029 02:52:13.031316 (+4270943us) ts_tablet_manager.cc:1151] Deleting tablet 372d93914abc41b38a702742b7db0bc2
1029 02:52:13.031328 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:13.085029 (+ 53701us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:13.132936 (+ 47907us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:13.136316 (+ 3380us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15957,"tablet-delete.queue_time_us":4270752}
I20251029 02:52:13.136785 1175 tablet_replica.cc:333] T 28f7ceb91a34450cad4b5b45f5285b41 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:13.137004 1175 raft_consensus.cc:2243] T 28f7ceb91a34450cad4b5b45f5285b41 P b495a49349324f7d81908f2da700f0d9 [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:13.137310 1175 raft_consensus.cc:2272] T 28f7ceb91a34450cad4b5b45f5285b41 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:13.137810 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 372d93914abc41b38a702742b7db0bc2 (table default.Table_4e9d94609b4a4d1f94615dcc9830d717 [id=761ebdd0095c414488aa221cbc99d3bc]) successfully deleted
I20251029 02:52:13.138460 1175 ts_tablet_manager.cc:1916] T 28f7ceb91a34450cad4b5b45f5285b41 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:13.141884 1172 tablet_bootstrap.cc:654] T eca016aed05e4ce09e1a2bf96e2286f5 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:13.142135 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 986) took 2327 ms. Trace:
I20251029 02:52:13.142262 1176 rpcz_store.cc:276] 1029 02:52:10.814494 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:10.821613 (+ 7119us) service_pool.cc:225] Handling call
1029 02:52:13.135561 (+2313948us) ts_tablet_manager.cc:1151] Deleting tablet 862d268a526545ba8f25bb91c2992321
1029 02:52:13.141972 (+ 6411us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:13.142075 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2164759}
W20251029 02:52:13.142727 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 862d268a526545ba8f25bb91c2992321 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 862d268a526545ba8f25bb91c2992321 already in progress: creating tablet
I20251029 02:52:13.143765 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 993) took 2323 ms. Trace:
I20251029 02:52:13.143863 1176 rpcz_store.cc:276] 1029 02:52:10.819734 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:10.935534 (+115800us) service_pool.cc:225] Handling call
1029 02:52:13.143103 (+2207569us) ts_tablet_manager.cc:1151] Deleting tablet 8378d67f1da743cca91803e37ea54281
1029 02:52:13.143113 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:13.143195 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2133588}
W20251029 02:52:13.144358 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 8378d67f1da743cca91803e37ea54281 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8378d67f1da743cca91803e37ea54281 already in progress: creating tablet
I20251029 02:52:13.145411 1167 tablet_bootstrap.cc:492] T 378f549ff7364d018d8b9a5ef01ac878 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:13.145567 1167 ts_tablet_manager.cc:1403] T 378f549ff7364d018d8b9a5ef01ac878 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.054s user 0.002s sys 0.001s
I20251029 02:52:13.146822 2526 raft_consensus.cc:493] T 72f9d4408f384e7093316d85f092e1b2 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:13.146982 2526 raft_consensus.cc:515] T 72f9d4408f384e7093316d85f092e1b2 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:13.158269 2469 tablet.cc:1722] T 1bdb67125493430398ff5c80b73de2ac P b495a49349324f7d81908f2da700f0d9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251029 02:52:13.158713 1167 raft_consensus.cc:359] T 378f549ff7364d018d8b9a5ef01ac878 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:13.158932 1167 raft_consensus.cc:385] T 378f549ff7364d018d8b9a5ef01ac878 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:13.158996 1167 raft_consensus.cc:740] T 378f549ff7364d018d8b9a5ef01ac878 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:13.159279 1167 consensus_queue.cc:260] T 378f549ff7364d018d8b9a5ef01ac878 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:13.160017 1167 ts_tablet_manager.cc:1434] T 378f549ff7364d018d8b9a5ef01ac878 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.014s user 0.001s sys 0.001s
I20251029 02:52:13.160342 1167 tablet_bootstrap.cc:492] T 7bb85a12f573494cbb3118d1947eda27 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:13.162420 1167 tablet_bootstrap.cc:654] T 7bb85a12f573494cbb3118d1947eda27 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:13.163580 525 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_aff419c3c59d4480b8c26d6bbb53f05b" } new_table_name: "default.Table_b74b95d8844643009cb9312d31770ded" modify_external_catalogs: true
I20251029 02:52:13.165081 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_39b69ff4c04748ffaf18565c7f306693"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:13.165645 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_39b69ff4c04748ffaf18565c7f306693 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:13.167730 526 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_c0fb3766a9c04aa9b2e3caca2e7b9b3a" } new_table_name: "default.Table_3f37e9c1a92248ad88895ef2da37188c" modify_external_catalogs: true
I20251029 02:52:13.168125 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 843) took 4566 ms. Trace:
I20251029 02:52:13.168412 1174 rpcz_store.cc:276] 1029 02:52:08.601671 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:08.606559 (+ 4888us) service_pool.cc:225] Handling call
1029 02:52:13.016970 (+4410411us) ts_tablet_manager.cc:1151] Deleting tablet c9bd3f2deb5d4af09b7918269a98509f
1029 02:52:13.016981 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:13.100546 (+ 83565us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:13.124321 (+ 23775us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:13.168101 (+ 43780us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4410189}
I20251029 02:52:13.168907 1174 tablet_replica.cc:333] T 777992acb1bd4765886b802940db74bb P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:13.169087 1174 raft_consensus.cc:2243] T 777992acb1bd4765886b802940db74bb P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:13.169440 1174 raft_consensus.cc:2272] T 777992acb1bd4765886b802940db74bb P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:13.170060 1174 ts_tablet_manager.cc:1916] T 777992acb1bd4765886b802940db74bb P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 02:52:13.173592 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.3:45003, user_credentials={real_user=slave}} blocked reactor thread for 55109.8us
I20251029 02:52:13.175889 2526 leader_election.cc:290] T 72f9d4408f384e7093316d85f092e1b2 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:52:13.176472 2205 raft_consensus.cc:493] T 731c0acfbe204e8199ad246e38ae2ccd P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:13.176860 2205 raft_consensus.cc:515] T 731c0acfbe204e8199ad246e38ae2ccd P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:13.178113 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1003) took 1669 ms. Trace:
I20251029 02:52:13.178275 1176 rpcz_store.cc:276] 1029 02:52:11.508668 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:11.508722 (+ 54us) service_pool.cc:225] Handling call
1029 02:52:13.177920 (+1669198us) ts_tablet_manager.cc:1151] Deleting tablet 12432c84ca104b0e99926a247318906a
1029 02:52:13.177932 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:13.178037 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1668962}
I20251029 02:52:13.178455 2205 leader_election.cc:290] T 731c0acfbe204e8199ad246e38ae2ccd P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:13.178817 1055 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "731c0acfbe204e8199ad246e38ae2ccd" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:13.179723 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "72f9d4408f384e7093316d85f092e1b2" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:13.180383 711 leader_election.cc:343] T 72f9d4408f384e7093316d85f092e1b2 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:13.182173 766 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "731c0acfbe204e8199ad246e38ae2ccd" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:13.182389 766 raft_consensus.cc:2393] T 731c0acfbe204e8199ad246e38ae2ccd P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c756528642a543eeacd93cef3e335a7d in current term 1: Already voted for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in this term.
I20251029 02:52:13.181516 1055 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "72f9d4408f384e7093316d85f092e1b2" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:13.182852 847 leader_election.cc:304] T 731c0acfbe204e8199ad246e38ae2ccd P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9
I20251029 02:52:13.184707 2205 raft_consensus.cc:3060] T 731c0acfbe204e8199ad246e38ae2ccd P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
W20251029 02:52:13.186901 712 leader_election.cc:343] T 72f9d4408f384e7093316d85f092e1b2 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:13.187011 712 leader_election.cc:304] T 72f9d4408f384e7093316d85f092e1b2 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:13.187335 2526 raft_consensus.cc:2749] T 72f9d4408f384e7093316d85f092e1b2 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:13.187695 1166 tablet_bootstrap.cc:492] T f281191e7c6143f983f11d87f1c9b64e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:13.188163 1166 ts_tablet_manager.cc:1403] T f281191e7c6143f983f11d87f1c9b64e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.059s user 0.003s sys 0.007s
I20251029 02:52:13.190037 1166 raft_consensus.cc:359] T f281191e7c6143f983f11d87f1c9b64e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:13.193326 1175 ts_tablet_manager.cc:1929] T 28f7ceb91a34450cad4b5b45f5285b41 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:13.193430 1175 log.cc:1199] T 28f7ceb91a34450cad4b5b45f5285b41 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/28f7ceb91a34450cad4b5b45f5285b41
I20251029 02:52:13.193758 1175 ts_tablet_manager.cc:1950] T 28f7ceb91a34450cad4b5b45f5285b41 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:13.200928 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet c9bd3f2deb5d4af09b7918269a98509f (table default.Table_47a1a59db0444fa49f7274509385ff0c [id=283c09ba60114ba9b165e2ff144a6c35]) successfully deleted
W20251029 02:52:13.201668 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 12432c84ca104b0e99926a247318906a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 12432c84ca104b0e99926a247318906a already in progress: creating tablet
I20251029 02:52:13.208827 1172 tablet_bootstrap.cc:492] T eca016aed05e4ce09e1a2bf96e2286f5 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:13.208967 1172 ts_tablet_manager.cc:1403] T eca016aed05e4ce09e1a2bf96e2286f5 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.212s user 0.003s sys 0.006s
I20251029 02:52:13.209645 1172 raft_consensus.cc:359] T eca016aed05e4ce09e1a2bf96e2286f5 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:13.209800 1172 raft_consensus.cc:385] T eca016aed05e4ce09e1a2bf96e2286f5 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:13.209858 1172 raft_consensus.cc:740] T eca016aed05e4ce09e1a2bf96e2286f5 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:13.210050 1172 consensus_queue.cc:260] T eca016aed05e4ce09e1a2bf96e2286f5 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:13.210470 1172 ts_tablet_manager.cc:1434] T eca016aed05e4ce09e1a2bf96e2286f5 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20251029 02:52:13.210888 1172 tablet_bootstrap.cc:492] T d29cc2a0c8b84e3ebadc7f571d666847 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:13.212885 1172 tablet_bootstrap.cc:654] T d29cc2a0c8b84e3ebadc7f571d666847 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
W20251029 02:52:13.217617 522 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 76a7e8517db140e8bdd3b67ce423ec90 consensus state has no leader
I20251029 02:52:13.217952 2205 raft_consensus.cc:2749] T 731c0acfbe204e8199ad246e38ae2ccd P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:13.194442 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 858) took 4338 ms. Trace:
I20251029 02:52:13.218402 1175 rpcz_store.cc:276] 1029 02:52:08.855788 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:08.855958 (+ 170us) service_pool.cc:225] Handling call
1029 02:52:13.136731 (+4280773us) ts_tablet_manager.cc:1151] Deleting tablet 28f7ceb91a34450cad4b5b45f5285b41
1029 02:52:13.136743 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:13.182131 (+ 45388us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:13.193223 (+ 11092us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:13.194424 (+ 1201us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4279421}
I20251029 02:52:13.218874 1175 tablet_replica.cc:333] T 8048506ced4540a5aee9f79bfc4599bf P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:13.219331 1175 raft_consensus.cc:2243] T 8048506ced4540a5aee9f79bfc4599bf P b495a49349324f7d81908f2da700f0d9 [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:13.219626 1175 raft_consensus.cc:2272] T 8048506ced4540a5aee9f79bfc4599bf P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251029 02:52:13.219697 525 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c92a3c77be8943fb929d321559e3a46c consensus state has no leader
W20251029 02:52:13.219735 849 consensus_peers.cc:597] T 49d9ce32e4f04f11be2ecefc5d523a8b P c756528642a543eeacd93cef3e335a7d -> Peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Couldn't send request to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 49d9ce32e4f04f11be2ecefc5d523a8b. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:13.220438 1175 ts_tablet_manager.cc:1916] T 8048506ced4540a5aee9f79bfc4599bf P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:13.204253 2430 consensus_queue.cc:1048] T afb19ba39399410bb90b6225d7b5b5b1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:13.224355 1166 raft_consensus.cc:385] T f281191e7c6143f983f11d87f1c9b64e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:13.224843 1166 raft_consensus.cc:740] T f281191e7c6143f983f11d87f1c9b64e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:13.225394 1166 consensus_queue.cc:260] T f281191e7c6143f983f11d87f1c9b64e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:13.188196 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1004) took 1679 ms. Trace:
I20251029 02:52:13.226130 1176 rpcz_store.cc:276] 1029 02:52:11.509084 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:11.509123 (+ 39us) service_pool.cc:225] Handling call
1029 02:52:13.188029 (+1678906us) ts_tablet_manager.cc:1151] Deleting tablet e74eda48dca647efacabe525ef316944
1029 02:52:13.188040 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:13.188148 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1669452}
I20251029 02:52:13.228950 2443 consensus_queue.cc:1048] T afb19ba39399410bb90b6225d7b5b5b1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251029 02:52:13.229053 1175 ts_tablet_manager.cc:1929] T 8048506ced4540a5aee9f79bfc4599bf P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:13.229168 1175 log.cc:1199] T 8048506ced4540a5aee9f79bfc4599bf P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/8048506ced4540a5aee9f79bfc4599bf
I20251029 02:52:13.229600 1175 ts_tablet_manager.cc:1950] T 8048506ced4540a5aee9f79bfc4599bf P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:13.229598 1166 ts_tablet_manager.cc:1434] T f281191e7c6143f983f11d87f1c9b64e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.041s user 0.001s sys 0.001s
I20251029 02:52:13.230610 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 28f7ceb91a34450cad4b5b45f5285b41 (table default.Table_5580c367c15f4f3a87626302cfde455f [id=89406c2781fd4674b634e3b1dbd06dc7]) successfully deleted
I20251029 02:52:13.230751 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 861) took 4250 ms. Trace:
I20251029 02:52:13.230876 1175 rpcz_store.cc:276] 1029 02:52:08.979604 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:08.980035 (+ 431us) service_pool.cc:225] Handling call
1029 02:52:13.218817 (+4238782us) ts_tablet_manager.cc:1151] Deleting tablet 8048506ced4540a5aee9f79bfc4599bf
1029 02:52:13.218826 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:13.224807 (+ 5981us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:13.228967 (+ 4160us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:13.230409 (+ 1442us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4238420}
W20251029 02:52:13.231355 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet e74eda48dca647efacabe525ef316944 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e74eda48dca647efacabe525ef316944 already in progress: creating tablet
I20251029 02:52:13.231432 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 862) took 4138 ms. Trace:
I20251029 02:52:13.231658 1175 rpcz_store.cc:276] 1029 02:52:09.092907 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:09.092969 (+ 62us) service_pool.cc:225] Handling call
1029 02:52:13.231276 (+4138307us) ts_tablet_manager.cc:1151] Deleting tablet 00ebd0c68c4441f9b63c86cdab62497d
1029 02:52:13.231285 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:13.231405 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4138091}
I20251029 02:52:13.232333 1175 tablet_replica.cc:333] T af4860c6b72f4d988b206b5ef2f22906 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:13.232585 1175 raft_consensus.cc:2243] T af4860c6b72f4d988b206b5ef2f22906 P b495a49349324f7d81908f2da700f0d9 [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:13.233567 1166 tablet_bootstrap.cc:492] T 378f549ff7364d018d8b9a5ef01ac878 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:13.234597 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 8048506ced4540a5aee9f79bfc4599bf (table default.Table_f2a72498cf4540949baff93eba1f5ef0 [id=a047685192f34bf1a1f3d1361cdaf272]) successfully deleted
W20251029 02:52:13.234879 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 00ebd0c68c4441f9b63c86cdab62497d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 00ebd0c68c4441f9b63c86cdab62497d already in progress: creating tablet
I20251029 02:52:13.233948 1176 tablet_replica.cc:333] T 217144957930495aa3e9dfcfde268e71 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:13.232921 1175 raft_consensus.cc:2272] T af4860c6b72f4d988b206b5ef2f22906 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:13.237293 1176 raft_consensus.cc:2243] T 217144957930495aa3e9dfcfde268e71 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:13.238121 1175 ts_tablet_manager.cc:1916] T af4860c6b72f4d988b206b5ef2f22906 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:13.238572 1176 raft_consensus.cc:2272] T 217144957930495aa3e9dfcfde268e71 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:13.243630 1176 ts_tablet_manager.cc:1916] T 217144957930495aa3e9dfcfde268e71 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:13.244029 2443 consensus_queue.cc:1048] T e65eaaae43ac435ebd665546de9d68dd P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:13.247541 1167 tablet_bootstrap.cc:492] T 7bb85a12f573494cbb3118d1947eda27 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:13.247666 1167 ts_tablet_manager.cc:1403] T 7bb85a12f573494cbb3118d1947eda27 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.087s user 0.003s sys 0.003s
I20251029 02:52:13.248379 1167 raft_consensus.cc:359] T 7bb85a12f573494cbb3118d1947eda27 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:13.248525 1167 raft_consensus.cc:385] T 7bb85a12f573494cbb3118d1947eda27 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:13.248574 1167 raft_consensus.cc:740] T 7bb85a12f573494cbb3118d1947eda27 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:13.248843 2201 raft_consensus.cc:493] T caef1c93e33a4390bcada59a41f43465 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:13.249275 2201 raft_consensus.cc:515] T caef1c93e33a4390bcada59a41f43465 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:13.249610 1172 tablet_bootstrap.cc:492] T d29cc2a0c8b84e3ebadc7f571d666847 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:13.249763 1172 ts_tablet_manager.cc:1403] T d29cc2a0c8b84e3ebadc7f571d666847 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.039s user 0.001s sys 0.005s
W20251029 02:52:13.252496 847 leader_election.cc:343] T caef1c93e33a4390bcada59a41f43465 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: caef1c93e33a4390bcada59a41f43465
I20251029 02:52:13.252748 1172 raft_consensus.cc:359] T d29cc2a0c8b84e3ebadc7f571d666847 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:13.252952 1172 raft_consensus.cc:385] T d29cc2a0c8b84e3ebadc7f571d666847 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:13.253046 1172 raft_consensus.cc:740] T d29cc2a0c8b84e3ebadc7f571d666847 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:13.253301 1167 consensus_queue.cc:260] T 7bb85a12f573494cbb3118d1947eda27 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:13.253932 1172 consensus_queue.cc:260] T d29cc2a0c8b84e3ebadc7f571d666847 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:13.254568 1172 ts_tablet_manager.cc:1434] T d29cc2a0c8b84e3ebadc7f571d666847 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20251029 02:52:13.254824 1172 tablet_bootstrap.cc:492] T 6f8cf09f23984b879b60ab6b27c7d52e P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:13.253435 2201 leader_election.cc:290] T caef1c93e33a4390bcada59a41f43465 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:13.256850 1172 tablet_bootstrap.cc:654] T 6f8cf09f23984b879b60ab6b27c7d52e P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:13.260322 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_02d8535577114a268daf473744621d3c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:13.260890 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_02d8535577114a268daf473744621d3c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:13.262079 1174 ts_tablet_manager.cc:1929] T 777992acb1bd4765886b802940db74bb P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:13.262195 1174 log.cc:1199] T 777992acb1bd4765886b802940db74bb P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/777992acb1bd4765886b802940db74bb
I20251029 02:52:13.262089 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_d5b1b53f211a44bf944f53dfde769c5d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:13.262646 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d5b1b53f211a44bf944f53dfde769c5d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:13.264277 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_f861c96de6f04741b28da03e79652f8b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:13.264652 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f861c96de6f04741b28da03e79652f8b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:52:13.269887 526 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c3ce63a996da494ba0775ba533c1a007 consensus state has no leader
I20251029 02:52:13.274885 1968 webserver.cc:492] Webserver started at http://127.0.104.62:46851/ using document root <none> and password file <none>
I20251029 02:52:13.275038 519 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_dd29d68d6ddd4157bc66b4ff9b54b33f" } modify_external_catalogs: true
I20251029 02:52:13.275223 519 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_dd29d68d6ddd4157bc66b4ff9b54b33f" } modify_external_catalogs: true
I20251029 02:52:13.276211 1968 fs_manager.cc:362] Metadata directory not provided
I20251029 02:52:13.276355 1968 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
W20251029 02:52:13.254401 849 leader_election.cc:343] T caef1c93e33a4390bcada59a41f43465 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Not found: Tablet not found: caef1c93e33a4390bcada59a41f43465
I20251029 02:52:13.280804 849 leader_election.cc:304] T caef1c93e33a4390bcada59a41f43465 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9
I20251029 02:52:13.281222 2426 raft_consensus.cc:2749] T caef1c93e33a4390bcada59a41f43465 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:13.275085 2201 raft_consensus.cc:493] T 67651306f3b44e7eb14dcee9a2007a81 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:13.253808 1167 ts_tablet_manager.cc:1434] T 7bb85a12f573494cbb3118d1947eda27 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251029 02:52:13.281514 2201 raft_consensus.cc:515] T 67651306f3b44e7eb14dcee9a2007a81 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:13.281705 1167 tablet_bootstrap.cc:492] T 58dde9397b434b089ade9968f369006a P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:13.282049 2201 leader_election.cc:290] T 67651306f3b44e7eb14dcee9a2007a81 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:13.282750 766 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "67651306f3b44e7eb14dcee9a2007a81" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:13.283078 766 raft_consensus.cc:2468] T 67651306f3b44e7eb14dcee9a2007a81 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:13.283640 847 leader_election.cc:304] T 67651306f3b44e7eb14dcee9a2007a81 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:13.283619 1055 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "67651306f3b44e7eb14dcee9a2007a81" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:13.283834 1055 raft_consensus.cc:2468] T 67651306f3b44e7eb14dcee9a2007a81 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:13.284312 2201 raft_consensus.cc:2804] T 67651306f3b44e7eb14dcee9a2007a81 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:13.284415 2201 raft_consensus.cc:493] T 67651306f3b44e7eb14dcee9a2007a81 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:13.284489 2201 raft_consensus.cc:3060] T 67651306f3b44e7eb14dcee9a2007a81 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:13.287354 2426 raft_consensus.cc:493] T c1222e30bc4141b3ae686d80053c4ef5 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:13.287489 2426 raft_consensus.cc:515] T c1222e30bc4141b3ae686d80053c4ef5 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:13.288247 2426 leader_election.cc:290] T c1222e30bc4141b3ae686d80053c4ef5 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:13.288872 766 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c1222e30bc4141b3ae686d80053c4ef5" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:13.290591 1035 tablet_service.cc:1505] Processing CreateTablet for tablet cdcc527f611a4f288655de30b5714697 (DEFAULT_TABLE table=default.Table_41d356dd41454ac680819ce36c4766a0 [id=9ea51e6784bb4b78b4b0963a9fea74c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:13.290759 751 tablet_service.cc:1505] Processing CreateTablet for tablet cdcc527f611a4f288655de30b5714697 (DEFAULT_TABLE table=default.Table_41d356dd41454ac680819ce36c4766a0 [id=9ea51e6784bb4b78b4b0963a9fea74c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:13.291059 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cdcc527f611a4f288655de30b5714697. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:13.291247 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cdcc527f611a4f288655de30b5714697. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:13.292961 522 catalog_manager.cc:5649] T c1222e30bc4141b3ae686d80053c4ef5 P b495a49349324f7d81908f2da700f0d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to b495a49349324f7d81908f2da700f0d9 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "b495a49349324f7d81908f2da700f0d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } health_report { overall_health: HEALTHY } } }
I20251029 02:52:13.293393 522 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet c1222e30bc4141b3ae686d80053c4ef5 (table default.Table_7125184a813c44d997ab9d948a1156bd [id=31801defa6db4ca7a08fbf7f029b65c1]). Expected version 3 got 0
I20251029 02:52:13.293798 522 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet d7abaf84ba5d445aa17433b3be891548 (table default.Table_c1c6933af1e046aa895e0ee915178f5c [id=9e33c530a35544a4ae22e2a1238592db]). Expected version 3 got 0
I20251029 02:52:13.294034 522 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet 82289e0f823c4d43bfd8c4e878572b2b (table default.Table_e7bb3e262a894c2f84b6edfc3917b30d [id=2cca3b7c50b243b5883b8ac1f353f253]). Expected version 1 got 0
I20251029 02:52:13.295022 525 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_52cba56ce9fb44bd84c35ef2f9219e56" } modify_external_catalogs: true
I20251029 02:52:13.295022 1055 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c1222e30bc4141b3ae686d80053c4ef5" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:13.295270 1055 raft_consensus.cc:2393] T c1222e30bc4141b3ae686d80053c4ef5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c756528642a543eeacd93cef3e335a7d in current term 1: Already voted for candidate b495a49349324f7d81908f2da700f0d9 in this term.
I20251029 02:52:13.295212 525 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_52cba56ce9fb44bd84c35ef2f9219e56" } modify_external_catalogs: true
I20251029 02:52:13.295892 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_08c8b82cbf9a424c8514bea1a6324fb1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:13.295985 849 leader_election.cc:304] T c1222e30bc4141b3ae686d80053c4ef5 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9
I20251029 02:52:13.295571 1968 fs_manager.cc:714] Time spent opening directory manager: real 0.016s user 0.004s sys 0.006s
W20251029 02:52:13.296466 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_08c8b82cbf9a424c8514bea1a6324fb1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:13.296643 1034 tablet_service.cc:1505] Processing CreateTablet for tablet ebdec273d5074107b2ebc4d4d6f4e9c1 (DEFAULT_TABLE table=default.Table_ce61c133a61047b99c732ee57710d119 [id=a34f22065980475384b1531ae2cd76ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:13.296849 517 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_715191bdf4ad4b2fbf349552331f0f1e" } new_table_name: "default.Table_c67b547f18944a3490f903739cdb7436" modify_external_catalogs: true
I20251029 02:52:13.297051 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ebdec273d5074107b2ebc4d4d6f4e9c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:13.299556 2426 raft_consensus.cc:3060] T c1222e30bc4141b3ae686d80053c4ef5 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:13.299682 738 tablet_service.cc:1505] Processing CreateTablet for tablet ebdec273d5074107b2ebc4d4d6f4e9c1 (DEFAULT_TABLE table=default.Table_ce61c133a61047b99c732ee57710d119 [id=a34f22065980475384b1531ae2cd76ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:13.300081 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ebdec273d5074107b2ebc4d4d6f4e9c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:13.303061 894 tablet_service.cc:1505] Processing CreateTablet for tablet ebdec273d5074107b2ebc4d4d6f4e9c1 (DEFAULT_TABLE table=default.Table_ce61c133a61047b99c732ee57710d119 [id=a34f22065980475384b1531ae2cd76ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:13.303148 900 tablet_service.cc:1505] Processing CreateTablet for tablet f3dfd1d767294ea08ef5e957b349ee12 (DEFAULT_TABLE table=default.Table_697672643d0d4986badc08e39aa0bb14 [id=4e8fb698bb064ad29ede263026d29776]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:13.303421 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ebdec273d5074107b2ebc4d4d6f4e9c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:13.303635 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3dfd1d767294ea08ef5e957b349ee12. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:13.303795 879 tablet_service.cc:1505] Processing CreateTablet for tablet 3e9a076e086141ef8ed59d4c8633f961 (DEFAULT_TABLE table=default.Table_dbeb5b9d4261424085b86ecddb373eff [id=f403173676454b2d9ca997f2fb46598d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:13.303920 878 tablet_service.cc:1505] Processing CreateTablet for tablet 53b2c53eaef64cfd96a4c849f9eec686 (DEFAULT_TABLE table=default.Table_38847ce61ea24d58aeeb20a5d4f5d9ba [id=e9993caf01d44e5f868506775e43a5ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:13.304083 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e9a076e086141ef8ed59d4c8633f961. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:13.304219 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53b2c53eaef64cfd96a4c849f9eec686. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:13.304188 1036 tablet_service.cc:1505] Processing CreateTablet for tablet 53b2c53eaef64cfd96a4c849f9eec686 (DEFAULT_TABLE table=default.Table_38847ce61ea24d58aeeb20a5d4f5d9ba [id=e9993caf01d44e5f868506775e43a5ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:13.304463 895 tablet_service.cc:1505] Processing CreateTablet for tablet 8471db993eab4fcb9e6f317d48ad8f80 (DEFAULT_TABLE table=default.Table_ad45e164be6a481c871f036851e127f2 [id=592b3a043426469193575e824213d36e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:13.304525 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53b2c53eaef64cfd96a4c849f9eec686. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:13.304745 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8471db993eab4fcb9e6f317d48ad8f80. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:13.303893 896 tablet_service.cc:1505] Processing CreateTablet for tablet cdcc527f611a4f288655de30b5714697 (DEFAULT_TABLE table=default.Table_41d356dd41454ac680819ce36c4766a0 [id=9ea51e6784bb4b78b4b0963a9fea74c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:13.306855 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cdcc527f611a4f288655de30b5714697. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:13.306782 1039 tablet_service.cc:1505] Processing CreateTablet for tablet f3dfd1d767294ea08ef5e957b349ee12 (DEFAULT_TABLE table=default.Table_697672643d0d4986badc08e39aa0bb14 [id=4e8fb698bb064ad29ede263026d29776]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:13.307111 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3dfd1d767294ea08ef5e957b349ee12. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:13.310766 1024 tablet_service.cc:1505] Processing CreateTablet for tablet 3e9a076e086141ef8ed59d4c8633f961 (DEFAULT_TABLE table=default.Table_dbeb5b9d4261424085b86ecddb373eff [id=f403173676454b2d9ca997f2fb46598d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:13.311157 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e9a076e086141ef8ed59d4c8633f961. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:13.314788 1042 tablet_service.cc:1505] Processing CreateTablet for tablet 8471db993eab4fcb9e6f317d48ad8f80 (DEFAULT_TABLE table=default.Table_ad45e164be6a481c871f036851e127f2 [id=592b3a043426469193575e824213d36e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:13.315147 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8471db993eab4fcb9e6f317d48ad8f80. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:13.315847 2426 raft_consensus.cc:2749] T c1222e30bc4141b3ae686d80053c4ef5 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:13.316552 1167 tablet_bootstrap.cc:654] T 58dde9397b434b089ade9968f369006a P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:13.321051 2201 raft_consensus.cc:515] T 67651306f3b44e7eb14dcee9a2007a81 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:13.321288 745 tablet_service.cc:1505] Processing CreateTablet for tablet f3dfd1d767294ea08ef5e957b349ee12 (DEFAULT_TABLE table=default.Table_697672643d0d4986badc08e39aa0bb14 [id=4e8fb698bb064ad29ede263026d29776]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:13.321707 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3dfd1d767294ea08ef5e957b349ee12. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:13.322242 766 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "67651306f3b44e7eb14dcee9a2007a81" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:13.322431 766 raft_consensus.cc:3060] T 67651306f3b44e7eb14dcee9a2007a81 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:13.323720 751 tablet_service.cc:1505] Processing CreateTablet for tablet 53b2c53eaef64cfd96a4c849f9eec686 (DEFAULT_TABLE table=default.Table_38847ce61ea24d58aeeb20a5d4f5d9ba [id=e9993caf01d44e5f868506775e43a5ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:13.324097 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53b2c53eaef64cfd96a4c849f9eec686. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:13.325223 766 raft_consensus.cc:2468] T 67651306f3b44e7eb14dcee9a2007a81 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:13.325662 847 leader_election.cc:304] T 67651306f3b44e7eb14dcee9a2007a81 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:13.325800 747 tablet_service.cc:1505] Processing CreateTablet for tablet 3e9a076e086141ef8ed59d4c8633f961 (DEFAULT_TABLE table=default.Table_dbeb5b9d4261424085b86ecddb373eff [id=f403173676454b2d9ca997f2fb46598d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:13.325927 2425 raft_consensus.cc:2804] T 67651306f3b44e7eb14dcee9a2007a81 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:13.326035 2425 raft_consensus.cc:697] T 67651306f3b44e7eb14dcee9a2007a81 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:52:13.326292 756 tablet_service.cc:1505] Processing CreateTablet for tablet 8471db993eab4fcb9e6f317d48ad8f80 (DEFAULT_TABLE table=default.Table_ad45e164be6a481c871f036851e127f2 [id=592b3a043426469193575e824213d36e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:13.326282 2425 consensus_queue.cc:237] T 67651306f3b44e7eb14dcee9a2007a81 P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:13.327864 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8471db993eab4fcb9e6f317d48ad8f80. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:13.328742 1055 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "67651306f3b44e7eb14dcee9a2007a81" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:52:13.328951 1055 raft_consensus.cc:3060] T 67651306f3b44e7eb14dcee9a2007a81 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:13.330116 1172 tablet_bootstrap.cc:492] T 6f8cf09f23984b879b60ab6b27c7d52e P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:13.330269 1172 ts_tablet_manager.cc:1403] T 6f8cf09f23984b879b60ab6b27c7d52e P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.075s user 0.000s sys 0.003s
I20251029 02:52:13.330724 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e9a076e086141ef8ed59d4c8633f961. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:13.331048 1172 raft_consensus.cc:359] T 6f8cf09f23984b879b60ab6b27c7d52e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:13.331249 1172 raft_consensus.cc:385] T 6f8cf09f23984b879b60ab6b27c7d52e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:13.331318 1172 raft_consensus.cc:740] T 6f8cf09f23984b879b60ab6b27c7d52e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:13.331550 1172 consensus_queue.cc:260] T 6f8cf09f23984b879b60ab6b27c7d52e P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:13.331827 1055 raft_consensus.cc:2468] T 67651306f3b44e7eb14dcee9a2007a81 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:13.334015 896 tablet_service.cc:1505] Processing CreateTablet for tablet 36b639d133404370bdb911277973ceea (DEFAULT_TABLE table=default.Table_4bae416d7e1144d1bb614a134d8cccba [id=1aa4f4dc72e34e628adc1b3d8c2aadf0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:13.334290 881 tablet_service.cc:1505] Processing CreateTablet for tablet 3e48adc758b24fa0b3163410f7af7505 (DEFAULT_TABLE table=default.Table_1b4647439c30401a9285f7281255f7dd [id=f97f52c89ec9414f846c3347527cda65]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:13.334372 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36b639d133404370bdb911277973ceea. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:13.334416 900 tablet_service.cc:1505] Processing CreateTablet for tablet b8120ed944d24152950ea1c43834076e (DEFAULT_TABLE table=default.Table_f565624917764d89a8a93c2e498f1105 [id=ecaf632aec734e7f9771df9c17abc441]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:13.334615 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e48adc758b24fa0b3163410f7af7505. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:13.334760 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8120ed944d24152950ea1c43834076e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:13.337754 2201 leader_election.cc:290] T 67651306f3b44e7eb14dcee9a2007a81 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:13.340131 895 tablet_service.cc:1505] Processing CreateTablet for tablet 15bdb124f57d46b5b73fd628b7b7083a (DEFAULT_TABLE table=default.Table_1a2a463f983a4f408912990b262bc2a6 [id=476acaa58c49476895cac34f1e1fb417]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:13.340469 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15bdb124f57d46b5b73fd628b7b7083a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:13.341037 894 tablet_service.cc:1505] Processing CreateTablet for tablet 07005fd4923f46a6b62e405649172944 (DEFAULT_TABLE table=default.Table_ad87203491b8455d98e85df6212fe26f [id=fc3536c37d4a41aaa2fe76c12f701a86]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:13.341377 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07005fd4923f46a6b62e405649172944. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:13.342182 739 tablet_service.cc:1505] Processing CreateTablet for tablet 2c424a54c9714fe490677e64e3dd2931 (DEFAULT_TABLE table=default.Table_1963d5301e1a4a8ea388fbedfa53c6df [id=d3be1cfa7f004b3da084a30c82a57d64]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:13.342398 742 tablet_service.cc:1505] Processing CreateTablet for tablet 07005fd4923f46a6b62e405649172944 (DEFAULT_TABLE table=default.Table_ad87203491b8455d98e85df6212fe26f [id=fc3536c37d4a41aaa2fe76c12f701a86]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:13.342599 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c424a54c9714fe490677e64e3dd2931. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:13.342901 740 tablet_service.cc:1505] Processing CreateTablet for tablet 36b639d133404370bdb911277973ceea (DEFAULT_TABLE table=default.Table_4bae416d7e1144d1bb614a134d8cccba [id=1aa4f4dc72e34e628adc1b3d8c2aadf0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:13.343020 2179 consensus_queue.cc:1048] T e65eaaae43ac435ebd665546de9d68dd P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:13.343228 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36b639d133404370bdb911277973ceea. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:13.343662 1028 tablet_service.cc:1505] Processing CreateTablet for tablet 2c424a54c9714fe490677e64e3dd2931 (DEFAULT_TABLE table=default.Table_1963d5301e1a4a8ea388fbedfa53c6df [id=d3be1cfa7f004b3da084a30c82a57d64]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:13.344087 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c424a54c9714fe490677e64e3dd2931. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:13.345996 749 tablet_service.cc:1505] Processing CreateTablet for tablet 3e48adc758b24fa0b3163410f7af7505 (DEFAULT_TABLE table=default.Table_1b4647439c30401a9285f7281255f7dd [id=f97f52c89ec9414f846c3347527cda65]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:13.346343 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e48adc758b24fa0b3163410f7af7505. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:13.346774 743 tablet_service.cc:1505] Processing CreateTablet for tablet 15bdb124f57d46b5b73fd628b7b7083a (DEFAULT_TABLE table=default.Table_1a2a463f983a4f408912990b262bc2a6 [id=476acaa58c49476895cac34f1e1fb417]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:13.346767 1029 tablet_service.cc:1505] Processing CreateTablet for tablet 15bdb124f57d46b5b73fd628b7b7083a (DEFAULT_TABLE table=default.Table_1a2a463f983a4f408912990b262bc2a6 [id=476acaa58c49476895cac34f1e1fb417]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:13.347119 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15bdb124f57d46b5b73fd628b7b7083a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:13.347221 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15bdb124f57d46b5b73fd628b7b7083a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:13.350800 1040 tablet_service.cc:1505] Processing CreateTablet for tablet b8120ed944d24152950ea1c43834076e (DEFAULT_TABLE table=default.Table_f565624917764d89a8a93c2e498f1105 [id=ecaf632aec734e7f9771df9c17abc441]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:13.350836 744 tablet_service.cc:1505] Processing CreateTablet for tablet b8120ed944d24152950ea1c43834076e (DEFAULT_TABLE table=default.Table_f565624917764d89a8a93c2e498f1105 [id=ecaf632aec734e7f9771df9c17abc441]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:13.351158 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8120ed944d24152950ea1c43834076e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:13.351269 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8120ed944d24152950ea1c43834076e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:13.262513 1174 ts_tablet_manager.cc:1950] T 777992acb1bd4765886b802940db74bb P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:13.354545 1031 tablet_service.cc:1505] Processing CreateTablet for tablet 3e48adc758b24fa0b3163410f7af7505 (DEFAULT_TABLE table=default.Table_1b4647439c30401a9285f7281255f7dd [id=f97f52c89ec9414f846c3347527cda65]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:13.354903 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e48adc758b24fa0b3163410f7af7505. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:13.354804 1033 tablet_service.cc:1505] Processing CreateTablet for tablet 07005fd4923f46a6b62e405649172944 (DEFAULT_TABLE table=default.Table_ad87203491b8455d98e85df6212fe26f [id=fc3536c37d4a41aaa2fe76c12f701a86]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:13.355178 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07005fd4923f46a6b62e405649172944. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:13.355401 1030 tablet_service.cc:1505] Processing CreateTablet for tablet 76063e06f32c442296a1cb17051d2b9b (DEFAULT_TABLE table=default.Table_ec03963009c14e09b1fecad8b28ec9bb [id=12b5033da17f4c58b6cc6df36677b016]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:13.355561 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 852) took 4510 ms. Trace:
I20251029 02:52:13.355710 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76063e06f32c442296a1cb17051d2b9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:13.355641 1174 rpcz_store.cc:276] 1029 02:52:08.845374 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:08.846054 (+ 680us) service_pool.cc:225] Handling call
1029 02:52:13.168848 (+4322794us) ts_tablet_manager.cc:1151] Deleting tablet 777992acb1bd4765886b802940db74bb
1029 02:52:13.168862 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:13.247987 (+ 79125us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:13.261970 (+ 13983us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:13.355536 (+ 93566us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4322590}
I20251029 02:52:13.358646 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 777992acb1bd4765886b802940db74bb (table default.Table_d9fe74a49f4843b089aae7fca9e4a826 [id=152882c742b2459ba863fd41083284c1]) successfully deleted
I20251029 02:52:13.358760 1038 tablet_service.cc:1505] Processing CreateTablet for tablet 36b639d133404370bdb911277973ceea (DEFAULT_TABLE table=default.Table_4bae416d7e1144d1bb614a134d8cccba [id=1aa4f4dc72e34e628adc1b3d8c2aadf0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:13.359081 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36b639d133404370bdb911277973ceea. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:13.360322 1175 ts_tablet_manager.cc:1929] T af4860c6b72f4d988b206b5ef2f22906 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:13.360421 1175 log.cc:1199] T af4860c6b72f4d988b206b5ef2f22906 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/af4860c6b72f4d988b206b5ef2f22906
I20251029 02:52:13.360755 1175 ts_tablet_manager.cc:1950] T af4860c6b72f4d988b206b5ef2f22906 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:13.362876 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07005fd4923f46a6b62e405649172944. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:13.363529 878 tablet_service.cc:1505] Processing CreateTablet for tablet 2c424a54c9714fe490677e64e3dd2931 (DEFAULT_TABLE table=default.Table_1963d5301e1a4a8ea388fbedfa53c6df [id=d3be1cfa7f004b3da084a30c82a57d64]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:13.363912 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c424a54c9714fe490677e64e3dd2931. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:13.364792 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 857) took 4452 ms. Trace:
I20251029 02:52:13.364871 1174 rpcz_store.cc:276] 1029 02:52:08.912829 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:08.913054 (+ 225us) service_pool.cc:225] Handling call
1029 02:52:13.364668 (+4451614us) ts_tablet_manager.cc:1151] Deleting tablet 372d93914abc41b38a702742b7db0bc2
1029 02:52:13.364678 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:13.364773 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4451397}
W20251029 02:52:13.366820 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 372d93914abc41b38a702742b7db0bc2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 372d93914abc41b38a702742b7db0bc2 already in progress: creating tablet
I20251029 02:52:13.378728 1174 tablet_replica.cc:333] T ccdca31db2834de4b6ed0f2d67a10535 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:13.378950 1174 raft_consensus.cc:2243] T ccdca31db2834de4b6ed0f2d67a10535 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:13.379137 1174 raft_consensus.cc:2272] T ccdca31db2834de4b6ed0f2d67a10535 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:13.379688 1174 ts_tablet_manager.cc:1916] T ccdca31db2834de4b6ed0f2d67a10535 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:13.386610 2523 consensus_queue.cc:1048] T c1222e30bc4141b3ae686d80053c4ef5 P b495a49349324f7d81908f2da700f0d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:13.386715 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 863) took 4244 ms. Trace:
I20251029 02:52:13.386811 1175 rpcz_store.cc:276] 1029 02:52:09.142177 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:09.142254 (+ 77us) service_pool.cc:225] Handling call
1029 02:52:13.232041 (+4089787us) ts_tablet_manager.cc:1151] Deleting tablet af4860c6b72f4d988b206b5ef2f22906
1029 02:52:13.232260 (+ 219us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:13.242113 (+ 9853us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:13.360211 (+118098us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:13.386695 (+ 26484us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4089580}
I20251029 02:52:13.387374 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet af4860c6b72f4d988b206b5ef2f22906 (table default.Table_f5db3e3913ca4ea98b02981cd04a5965 [id=7eb08c679e0e441bb1ad4be13b2494ef]) successfully deleted
I20251029 02:52:13.395438 2443 raft_consensus.cc:493] T d041e87ceb11441983d0e38b7929c3a4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:13.395628 2443 raft_consensus.cc:515] T d041e87ceb11441983d0e38b7929c3a4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:13.396152 2443 leader_election.cc:290] T d041e87ceb11441983d0e38b7929c3a4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:13.397351 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d041e87ceb11441983d0e38b7929c3a4" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:13.398152 762 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d041e87ceb11441983d0e38b7929c3a4" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:13.398350 762 raft_consensus.cc:2393] T d041e87ceb11441983d0e38b7929c3a4 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in current term 1: Already voted for candidate c756528642a543eeacd93cef3e335a7d in this term.
I20251029 02:52:13.398734 989 leader_election.cc:304] T d041e87ceb11441983d0e38b7929c3a4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5; no voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:13.399108 2443 raft_consensus.cc:3060] T d041e87ceb11441983d0e38b7929c3a4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:13.401623 2443 raft_consensus.cc:2749] T d041e87ceb11441983d0e38b7929c3a4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:13.402449 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 864) took 4200 ms. Trace:
I20251029 02:52:13.402583 1175 rpcz_store.cc:276] 1029 02:52:09.202171 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:09.202269 (+ 98us) service_pool.cc:225] Handling call
1029 02:52:13.387220 (+4184951us) ts_tablet_manager.cc:1151] Deleting tablet 5815626b05204b578cc4a110f246bf10
1029 02:52:13.387228 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:13.402427 (+ 15199us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4184764}
W20251029 02:52:13.404876 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 5815626b05204b578cc4a110f246bf10 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5815626b05204b578cc4a110f246bf10 already in progress: creating tablet
I20251029 02:52:13.405540 1175 tablet_replica.cc:333] T c32c28b0e74a4af3a0e69bbf24ab1a1f P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:13.405720 1175 raft_consensus.cc:2243] T c32c28b0e74a4af3a0e69bbf24ab1a1f P b495a49349324f7d81908f2da700f0d9 [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:13.406011 1175 raft_consensus.cc:2272] T c32c28b0e74a4af3a0e69bbf24ab1a1f P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:13.411406 1175 ts_tablet_manager.cc:1916] T c32c28b0e74a4af3a0e69bbf24ab1a1f P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:13.411697 518 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 15cadd681ceb4daba178a9cc2b7aec08
I20251029 02:52:13.412242 2523 raft_consensus.cc:493] T 6142018e44af4d32818adefa2ee43d02 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:13.412376 518 master_service.cc:954] ReplaceTablet: received request to replace tablet 82289e0f823c4d43bfd8c4e878572b2b from {username='slave'} at 127.0.0.1:35314
I20251029 02:52:13.412370 2523 raft_consensus.cc:515] T 6142018e44af4d32818adefa2ee43d02 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:13.412374 521 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 21c73cb2f1084827b581fb09cf7239a1
I20251029 02:52:13.412809 525 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet eb3c20822d684bea89eb358251e21e33
I20251029 02:52:13.413004 2523 leader_election.cc:290] T 6142018e44af4d32818adefa2ee43d02 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:52:13.413393 523 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 416e7c48daa849cd9635656529488922
I20251029 02:52:13.414259 519 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 31ca9012ddf6401ba78888a0c82b4ef2
W20251029 02:52:13.412609 517 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5b44578ba7cc480199492195aca27112 consensus state has no leader
I20251029 02:52:13.416002 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6142018e44af4d32818adefa2ee43d02" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:13.417447 711 leader_election.cc:343] T 6142018e44af4d32818adefa2ee43d02 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:13.418658 1055 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6142018e44af4d32818adefa2ee43d02" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:13.418687 2416 consensus_queue.cc:1048] T c1222e30bc4141b3ae686d80053c4ef5 P b495a49349324f7d81908f2da700f0d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:13.419142 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 15cadd681ceb4daba178a9cc2b7aec08 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:12 UTC) from {username='slave'} at 127.0.0.1:35702
W20251029 02:52:13.419469 712 leader_election.cc:343] T 6142018e44af4d32818adefa2ee43d02 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:13.419569 712 leader_election.cc:304] T 6142018e44af4d32818adefa2ee43d02 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:13.419920 2416 raft_consensus.cc:2749] T 6142018e44af4d32818adefa2ee43d02 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:13.420966 896 tablet_service.cc:1552] Processing DeleteTablet for tablet 31ca9012ddf6401ba78888a0c82b4ef2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:13 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:13.421104 1042 tablet_service.cc:1552] Processing DeleteTablet for tablet 31ca9012ddf6401ba78888a0c82b4ef2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:13 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:13.421229 878 tablet_service.cc:1552] Processing DeleteTablet for tablet 21c73cb2f1084827b581fb09cf7239a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:12 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:13.421597 751 tablet_service.cc:1552] Processing DeleteTablet for tablet 21c73cb2f1084827b581fb09cf7239a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:12 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:13.421761 743 tablet_service.cc:1552] Processing DeleteTablet for tablet 416e7c48daa849cd9635656529488922 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:12 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:13.421973 745 tablet_service.cc:1552] Processing DeleteTablet for tablet eb3c20822d684bea89eb358251e21e33 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:13 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:13.422089 1030 tablet_service.cc:1552] Processing DeleteTablet for tablet 416e7c48daa849cd9635656529488922 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:12 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:13.422612 744 tablet_service.cc:1552] Processing DeleteTablet for tablet 15cadd681ceb4daba178a9cc2b7aec08 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:12 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:13.425433 756 tablet_service.cc:1552] Processing DeleteTablet for tablet 31ca9012ddf6401ba78888a0c82b4ef2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:13 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:13.425722 1040 tablet_service.cc:1552] Processing DeleteTablet for tablet eb3c20822d684bea89eb358251e21e33 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:13 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:13.427486 895 tablet_service.cc:1552] Processing DeleteTablet for tablet 416e7c48daa849cd9635656529488922 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:12 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:13.428089 894 tablet_service.cc:1552] Processing DeleteTablet for tablet eb3c20822d684bea89eb358251e21e33 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:13 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:13.431069 1172 ts_tablet_manager.cc:1434] T 6f8cf09f23984b879b60ab6b27c7d52e P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.101s user 0.002s sys 0.000s
I20251029 02:52:13.431984 1172 tablet_bootstrap.cc:492] T 6dff036b32904ddb9598be4b79504bb6 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:13.426239 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet 21c73cb2f1084827b581fb09cf7239a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:12 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:13.428098 1031 tablet_service.cc:1552] Processing DeleteTablet for tablet 15cadd681ceb4daba178a9cc2b7aec08 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:12 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:13.433513 1172 tablet_bootstrap.cc:654] T 6dff036b32904ddb9598be4b79504bb6 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:13.436278 1167 tablet_bootstrap.cc:492] T 58dde9397b434b089ade9968f369006a P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:13.436427 1167 ts_tablet_manager.cc:1403] T 58dde9397b434b089ade9968f369006a P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.155s user 0.000s sys 0.006s
I20251029 02:52:13.438777 747 tablet_service.cc:1505] Processing CreateTablet for tablet 76063e06f32c442296a1cb17051d2b9b (DEFAULT_TABLE table=default.Table_ec03963009c14e09b1fecad8b28ec9bb [id=12b5033da17f4c58b6cc6df36677b016]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:13.439311 522 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet adf6e736cb9c4920807ee58682e134ef (table default.Table_0a039bfb2e1f458f89ffc3ee93300b8f [id=64650da66d464074828f2dc955045f68]). Expected version 3 got 0
I20251029 02:52:13.441156 521 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_e3cc9c53ee9e4fafb7f9684f6c518714" } new_table_name: "default.Table_c158037e7a6640a1855dfbee88745e54" modify_external_catalogs: true
I20251029 02:52:13.441290 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76063e06f32c442296a1cb17051d2b9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:13.442448 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_290bd51633d34507910b6de728d4d129"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:13.443146 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_290bd51633d34507910b6de728d4d129 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:13.444345 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_742111c9d1cc4930b57efd12f4190058"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:13.444677 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_15d2f76cb37642f5a8f3611fbcdd5259"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:13.444875 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_742111c9d1cc4930b57efd12f4190058 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:52:13.445133 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_15d2f76cb37642f5a8f3611fbcdd5259 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:13.446606 517 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_6ed43c1a2617493882d7cb6c568d21b7" } new_table_name: "default.Table_1e69e259edba466f99ed7a41dcf6367f" modify_external_catalogs: true
I20251029 02:52:13.447232 881 tablet_service.cc:1505] Processing CreateTablet for tablet 76063e06f32c442296a1cb17051d2b9b (DEFAULT_TABLE table=default.Table_ec03963009c14e09b1fecad8b28ec9bb [id=12b5033da17f4c58b6cc6df36677b016]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:13.447619 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76063e06f32c442296a1cb17051d2b9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:13.447991 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_28d4296f305b4867bf3ef9aac130e5ad"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:13.448403 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_28d4296f305b4867bf3ef9aac130e5ad in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:13.450970 2453 tablet.cc:1722] T d5741dde463a43d6a776ab92dcfb04bf P c756528642a543eeacd93cef3e335a7d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 02:52:13.455186 1031 tablet_service.cc:1505] Processing CreateTablet for tablet 31ca9012ddf6401ba78888a0c82b4ef2 (DEFAULT_TABLE table=default.Table_dd29d68d6ddd4157bc66b4ff9b54b33f [id=0004aaa3c92c4a18a50fc5c7249a0ff5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:13.455687 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31ca9012ddf6401ba78888a0c82b4ef2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:13.456009 2452 tablet.cc:1722] T d5741dde463a43d6a776ab92dcfb04bf P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 02:52:13.457324 2575 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251029 02:52:13.457439 744 tablet_service.cc:1505] Processing CreateTablet for tablet e63eda985ee84c64b38a7e924add1182 (DEFAULT_TABLE table=default.Table_27d80833eb694669a044be7eb64e4aa4 [id=7b9a3b0cb6bd413fba5d459a5bd80c63]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:13.457774 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e63eda985ee84c64b38a7e924add1182. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:13.458950 756 tablet_service.cc:1505] Processing CreateTablet for tablet 31ca9012ddf6401ba78888a0c82b4ef2 (DEFAULT_TABLE table=default.Table_dd29d68d6ddd4157bc66b4ff9b54b33f [id=0004aaa3c92c4a18a50fc5c7249a0ff5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:13.459276 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31ca9012ddf6401ba78888a0c82b4ef2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:13.460455 894 tablet_service.cc:1505] Processing CreateTablet for tablet 31ca9012ddf6401ba78888a0c82b4ef2 (DEFAULT_TABLE table=default.Table_dd29d68d6ddd4157bc66b4ff9b54b33f [id=0004aaa3c92c4a18a50fc5c7249a0ff5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:13.460826 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31ca9012ddf6401ba78888a0c82b4ef2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:13.464681 1036 tablet_service.cc:1505] Processing CreateTablet for tablet e63eda985ee84c64b38a7e924add1182 (DEFAULT_TABLE table=default.Table_27d80833eb694669a044be7eb64e4aa4 [id=7b9a3b0cb6bd413fba5d459a5bd80c63]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:13.465124 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e63eda985ee84c64b38a7e924add1182. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:13.472599 1167 raft_consensus.cc:359] T 58dde9397b434b089ade9968f369006a P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
W20251029 02:52:13.472705 521 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 25129594fe3b40f28d87c0c1b7c025af consensus state has no leader
I20251029 02:52:13.472787 1167 raft_consensus.cc:385] T 58dde9397b434b089ade9968f369006a P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:13.472927 1167 raft_consensus.cc:740] T 58dde9397b434b089ade9968f369006a P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:13.473127 1167 consensus_queue.cc:260] T 58dde9397b434b089ade9968f369006a P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:13.476981 1174 ts_tablet_manager.cc:1929] T ccdca31db2834de4b6ed0f2d67a10535 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:13.477100 1167 ts_tablet_manager.cc:1434] T 58dde9397b434b089ade9968f369006a P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.041s user 0.000s sys 0.002s
I20251029 02:52:13.477421 1167 tablet_bootstrap.cc:492] T 27b08f88b99643d28ac609e45bf8104e P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:13.478361 2179 consensus_queue.cc:1048] T 731c0acfbe204e8199ad246e38ae2ccd P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:13.483999 1175 ts_tablet_manager.cc:1929] T c32c28b0e74a4af3a0e69bbf24ab1a1f P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:13.484154 1175 log.cc:1199] T c32c28b0e74a4af3a0e69bbf24ab1a1f P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/c32c28b0e74a4af3a0e69bbf24ab1a1f
I20251029 02:52:13.484457 1040 tablet_service.cc:1505] Processing CreateTablet for tablet 3fc2ba404f6b4df6a6eea4facaf358dc (DEFAULT_TABLE table=default.Table_492f50b2191a46ee8e6bb05e69377006 [id=cdf9ac3648ff4741af78f94b930e67a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:13.484546 1175 ts_tablet_manager.cc:1950] T c32c28b0e74a4af3a0e69bbf24ab1a1f P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:13.485215 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fc2ba404f6b4df6a6eea4facaf358dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:13.477099 1174 log.cc:1199] T ccdca31db2834de4b6ed0f2d67a10535 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/ccdca31db2834de4b6ed0f2d67a10535
I20251029 02:52:13.485704 1174 ts_tablet_manager.cc:1950] T ccdca31db2834de4b6ed0f2d67a10535 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:13.486454 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 873) took 4472 ms. Trace:
I20251029 02:52:13.486557 1174 rpcz_store.cc:276] 1029 02:52:09.014355 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:09.014861 (+ 506us) service_pool.cc:225] Handling call
1029 02:52:13.378653 (+4363792us) ts_tablet_manager.cc:1151] Deleting tablet ccdca31db2834de4b6ed0f2d67a10535
1029 02:52:13.378665 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:13.409481 (+ 30816us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:13.476889 (+ 67408us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:13.486435 (+ 9546us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4363626}
I20251029 02:52:13.487109 1029 tablet_service.cc:1505] Processing CreateTablet for tablet aef3021f7d684df197eb1cfd5c6b8772 (DEFAULT_TABLE table=default.Table_3ddda212ceb444adae605530b0985d6a [id=74c2d784492043779f50c41eb961880a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:13.487262 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 874) took 4367 ms. Trace:
I20251029 02:52:13.487339 1174 rpcz_store.cc:276] 1029 02:52:09.119342 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:09.119399 (+ 57us) service_pool.cc:225] Handling call
1029 02:52:13.487154 (+4367755us) ts_tablet_manager.cc:1151] Deleting tablet 00ebd0c68c4441f9b63c86cdab62497d
1029 02:52:13.487163 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:13.487248 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4367465}
I20251029 02:52:13.487633 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aef3021f7d684df197eb1cfd5c6b8772. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:13.487689 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet ccdca31db2834de4b6ed0f2d67a10535 (table default.Table_75341a6f40d84ad3938442923cf3131e [id=72e9fc37abc14519b6b87ebfc83c292c]) successfully deleted
W20251029 02:52:13.487919 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 00ebd0c68c4441f9b63c86cdab62497d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 00ebd0c68c4441f9b63c86cdab62497d already in progress: creating tablet
I20251029 02:52:13.487979 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 875) took 4236 ms. Trace:
I20251029 02:52:13.488050 1174 rpcz_store.cc:276] 1029 02:52:09.251050 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:09.258595 (+ 7545us) service_pool.cc:225] Handling call
1029 02:52:13.487888 (+4229293us) ts_tablet_manager.cc:1151] Deleting tablet 5815626b05204b578cc4a110f246bf10
1029 02:52:13.487897 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:13.487967 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4229084}
W20251029 02:52:13.488382 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 5815626b05204b578cc4a110f246bf10 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5815626b05204b578cc4a110f246bf10 already in progress: creating tablet
I20251029 02:52:13.488631 1174 tablet_replica.cc:333] T 9bb2166a590d4210b02dd779ef6654ad P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:13.488862 1174 raft_consensus.cc:2243] T 9bb2166a590d4210b02dd779ef6654ad P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:13.489017 1174 raft_consensus.cc:2272] T 9bb2166a590d4210b02dd779ef6654ad P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:13.489519 1174 ts_tablet_manager.cc:1916] T 9bb2166a590d4210b02dd779ef6654ad P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:13.490031 744 tablet_service.cc:1505] Processing CreateTablet for tablet aef3021f7d684df197eb1cfd5c6b8772 (DEFAULT_TABLE table=default.Table_3ddda212ceb444adae605530b0985d6a [id=74c2d784492043779f50c41eb961880a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:13.490615 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aef3021f7d684df197eb1cfd5c6b8772. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:13.491778 900 tablet_service.cc:1505] Processing CreateTablet for tablet e63eda985ee84c64b38a7e924add1182 (DEFAULT_TABLE table=default.Table_27d80833eb694669a044be7eb64e4aa4 [id=7b9a3b0cb6bd413fba5d459a5bd80c63]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:13.492118 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e63eda985ee84c64b38a7e924add1182. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:13.492266 1167 tablet_bootstrap.cc:654] T 27b08f88b99643d28ac609e45bf8104e P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:13.494462 739 tablet_service.cc:1505] Processing CreateTablet for tablet 3fc2ba404f6b4df6a6eea4facaf358dc (DEFAULT_TABLE table=default.Table_492f50b2191a46ee8e6bb05e69377006 [id=cdf9ac3648ff4741af78f94b930e67a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:13.495282 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_7ca8e3e58d62408c8010ea1aeb204a94"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:13.495764 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7ca8e3e58d62408c8010ea1aeb204a94 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:13.496379 895 tablet_service.cc:1505] Processing CreateTablet for tablet aef3021f7d684df197eb1cfd5c6b8772 (DEFAULT_TABLE table=default.Table_3ddda212ceb444adae605530b0985d6a [id=74c2d784492043779f50c41eb961880a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:13.496686 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aef3021f7d684df197eb1cfd5c6b8772. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:13.497399 745 tablet_service.cc:1505] Processing CreateTablet for tablet dc65e73a3424432dbab26d58996ed6dc (DEFAULT_TABLE table=default.Table_720c6ad9cd7a42e9b59130daf6492301 [id=d014ec8634be4270a249436672533402]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:13.498132 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc65e73a3424432dbab26d58996ed6dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:13.498719 1167 tablet_bootstrap.cc:492] T 27b08f88b99643d28ac609e45bf8104e P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:13.498837 1167 ts_tablet_manager.cc:1403] T 27b08f88b99643d28ac609e45bf8104e P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.000s
I20251029 02:52:13.498947 1030 tablet_service.cc:1505] Processing CreateTablet for tablet dc65e73a3424432dbab26d58996ed6dc (DEFAULT_TABLE table=default.Table_720c6ad9cd7a42e9b59130daf6492301 [id=d014ec8634be4270a249436672533402]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:13.499251 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc65e73a3424432dbab26d58996ed6dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:13.499756 878 tablet_service.cc:1505] Processing CreateTablet for tablet dc65e73a3424432dbab26d58996ed6dc (DEFAULT_TABLE table=default.Table_720c6ad9cd7a42e9b59130daf6492301 [id=d014ec8634be4270a249436672533402]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:13.500058 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc65e73a3424432dbab26d58996ed6dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:13.501035 1167 raft_consensus.cc:359] T 27b08f88b99643d28ac609e45bf8104e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:13.501171 1167 raft_consensus.cc:385] T 27b08f88b99643d28ac609e45bf8104e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:13.501216 1167 raft_consensus.cc:740] T 27b08f88b99643d28ac609e45bf8104e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:13.502931 1042 tablet_service.cc:1505] Processing CreateTablet for tablet f1e975b267464e64a5bbe24ef4e308a3 (DEFAULT_TABLE table=default.Table_abf33e38626c4d3e9bd944fecb1d1490 [id=e93a75bd377f4e6f9eec37f7d49d7373]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:13.503279 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1e975b267464e64a5bbe24ef4e308a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:13.503629 743 tablet_service.cc:1505] Processing CreateTablet for tablet f1e975b267464e64a5bbe24ef4e308a3 (DEFAULT_TABLE table=default.Table_abf33e38626c4d3e9bd944fecb1d1490 [id=e93a75bd377f4e6f9eec37f7d49d7373]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:13.503965 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1e975b267464e64a5bbe24ef4e308a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:13.507807 894 tablet_service.cc:1505] Processing CreateTablet for tablet 3fc2ba404f6b4df6a6eea4facaf358dc (DEFAULT_TABLE table=default.Table_492f50b2191a46ee8e6bb05e69377006 [id=cdf9ac3648ff4741af78f94b930e67a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:13.508169 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fc2ba404f6b4df6a6eea4facaf358dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:13.511916 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fc2ba404f6b4df6a6eea4facaf358dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:13.513157 878 tablet_service.cc:1505] Processing CreateTablet for tablet f1e975b267464e64a5bbe24ef4e308a3 (DEFAULT_TABLE table=default.Table_abf33e38626c4d3e9bd944fecb1d1490 [id=e93a75bd377f4e6f9eec37f7d49d7373]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:13.513556 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1e975b267464e64a5bbe24ef4e308a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:13.513509 1167 consensus_queue.cc:260] T 27b08f88b99643d28ac609e45bf8104e P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:13.513789 1039 tablet_service.cc:1505] Processing CreateTablet for tablet 52889d0b28ed47aca11c3432159c807f (DEFAULT_TABLE table=default.Table_18e8d552574143d1a42e021aca3182ed [id=8bf6740582704314bb27497c1d298b62]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:13.514200 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52889d0b28ed47aca11c3432159c807f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:13.514475 1172 tablet_bootstrap.cc:492] T 6dff036b32904ddb9598be4b79504bb6 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:13.514612 1172 ts_tablet_manager.cc:1403] T 6dff036b32904ddb9598be4b79504bb6 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.083s user 0.001s sys 0.002s
I20251029 02:52:13.515278 745 tablet_service.cc:1505] Processing CreateTablet for tablet 52889d0b28ed47aca11c3432159c807f (DEFAULT_TABLE table=default.Table_18e8d552574143d1a42e021aca3182ed [id=8bf6740582704314bb27497c1d298b62]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:13.515344 1172 raft_consensus.cc:359] T 6dff036b32904ddb9598be4b79504bb6 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:13.516563 1172 raft_consensus.cc:385] T 6dff036b32904ddb9598be4b79504bb6 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:13.516633 1172 raft_consensus.cc:740] T 6dff036b32904ddb9598be4b79504bb6 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:13.516812 1172 consensus_queue.cc:260] T 6dff036b32904ddb9598be4b79504bb6 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:13.516906 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52889d0b28ed47aca11c3432159c807f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:13.517314 1172 ts_tablet_manager.cc:1434] T 6dff036b32904ddb9598be4b79504bb6 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20251029 02:52:13.517555 1172 tablet_bootstrap.cc:492] T d31e548feae24b739087c1e211d514f9 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:13.518891 1172 tablet_bootstrap.cc:654] T d31e548feae24b739087c1e211d514f9 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:13.522914 1167 ts_tablet_manager.cc:1434] T 27b08f88b99643d28ac609e45bf8104e P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.024s user 0.000s sys 0.002s
I20251029 02:52:13.526026 895 tablet_service.cc:1505] Processing CreateTablet for tablet 52889d0b28ed47aca11c3432159c807f (DEFAULT_TABLE table=default.Table_18e8d552574143d1a42e021aca3182ed [id=8bf6740582704314bb27497c1d298b62]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:13.526438 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52889d0b28ed47aca11c3432159c807f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:13.526664 743 tablet_service.cc:1505] Processing CreateTablet for tablet 2e96a398bfd74cf68cf0377a813dc1dd (DEFAULT_TABLE table=default.Table_9cd59d4e874c4239b50b475aa8def07c [id=1d130644f20d4ea5bcbe83e01d970ce7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:13.527006 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e96a398bfd74cf68cf0377a813dc1dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:13.527329 896 tablet_service.cc:1505] Processing CreateTablet for tablet 2e96a398bfd74cf68cf0377a813dc1dd (DEFAULT_TABLE table=default.Table_9cd59d4e874c4239b50b475aa8def07c [id=1d130644f20d4ea5bcbe83e01d970ce7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:13.527619 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e96a398bfd74cf68cf0377a813dc1dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:13.536623 1167 tablet_bootstrap.cc:492] T ae4231daa2594258aa51419c97a2f50a P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:13.538256 1167 tablet_bootstrap.cc:654] T ae4231daa2594258aa51419c97a2f50a P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:13.538743 519 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_10d0ed02cd464576877b041b08d9af61" } new_table_name: "default.Table_eddccb434b6d4c048d894eefcee2065b" modify_external_catalogs: true
I20251029 02:52:13.539628 1024 tablet_service.cc:1505] Processing CreateTablet for tablet 2e96a398bfd74cf68cf0377a813dc1dd (DEFAULT_TABLE table=default.Table_9cd59d4e874c4239b50b475aa8def07c [id=1d130644f20d4ea5bcbe83e01d970ce7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:13.539999 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e96a398bfd74cf68cf0377a813dc1dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:13.540678 744 tablet_service.cc:1505] Processing CreateTablet for tablet f3794a0d2f4f421f8b2d0c5649bcb2ec (DEFAULT_TABLE table=default.Table_eb6ed8b2aee44b00a5ac4f8d2a357f28 [id=3a9c3a7a33d94cfbb0f9ab02886e16b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:13.541450 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3794a0d2f4f421f8b2d0c5649bcb2ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:13.547369 896 tablet_service.cc:1505] Processing CreateTablet for tablet f3794a0d2f4f421f8b2d0c5649bcb2ec (DEFAULT_TABLE table=default.Table_eb6ed8b2aee44b00a5ac4f8d2a357f28 [id=3a9c3a7a33d94cfbb0f9ab02886e16b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:13.547538 1968 fs_manager.cc:730] Time spent opening block manager: real 0.249s user 0.003s sys 0.002s
I20251029 02:52:13.547941 1166 tablet_bootstrap.cc:654] T 378f549ff7364d018d8b9a5ef01ac878 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:13.548091 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3794a0d2f4f421f8b2d0c5649bcb2ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:13.552887 520 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_9dce7f70d51d4a3c8d7f98a2435cbce3" } new_table_name: "default.Table_d0d8770fdb3d4d34933b7b4264831ad9" modify_external_catalogs: true
I20251029 02:52:13.557865 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 865) took 4330 ms. Trace:
I20251029 02:52:13.557972 1175 rpcz_store.cc:276] 1029 02:52:09.227266 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:09.227642 (+ 376us) service_pool.cc:225] Handling call
1029 02:52:13.405492 (+4177850us) ts_tablet_manager.cc:1151] Deleting tablet c32c28b0e74a4af3a0e69bbf24ab1a1f
1029 02:52:13.405501 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:13.438376 (+ 32875us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:13.483876 (+ 45500us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:13.557839 (+ 73963us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4177671}
I20251029 02:52:13.558429 1175 tablet_replica.cc:333] T 44ae2b07edca44afbe1b1fb3a5b1115d P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:13.558561 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet c32c28b0e74a4af3a0e69bbf24ab1a1f (table default.Table_3d94f8b10e884dc0b338dcfbb9c82309 [id=3db1589f811041eaa1cd7412a26391bc]) successfully deleted
I20251029 02:52:13.558624 1175 raft_consensus.cc:2243] T 44ae2b07edca44afbe1b1fb3a5b1115d P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:13.558785 1175 raft_consensus.cc:2272] T 44ae2b07edca44afbe1b1fb3a5b1115d P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:13.560910 2469 tablet.cc:1722] T d5741dde463a43d6a776ab92dcfb04bf P b495a49349324f7d81908f2da700f0d9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 02:52:13.567675 1175 ts_tablet_manager.cc:1916] T 44ae2b07edca44afbe1b1fb3a5b1115d P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:13.573972 1166 tablet_bootstrap.cc:492] T 378f549ff7364d018d8b9a5ef01ac878 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:13.574385 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_dd83c199ef234bc2914714ced85fd52f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:13.574657 1172 tablet_bootstrap.cc:492] T d31e548feae24b739087c1e211d514f9 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:13.574775 1172 ts_tablet_manager.cc:1403] T d31e548feae24b739087c1e211d514f9 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.057s user 0.002s sys 0.000s
W20251029 02:52:13.574965 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dd83c199ef234bc2914714ced85fd52f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:13.575443 1172 raft_consensus.cc:359] T d31e548feae24b739087c1e211d514f9 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:13.575599 1172 raft_consensus.cc:385] T d31e548feae24b739087c1e211d514f9 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:13.575647 1172 raft_consensus.cc:740] T d31e548feae24b739087c1e211d514f9 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:13.575830 1172 consensus_queue.cc:260] T d31e548feae24b739087c1e211d514f9 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:13.576298 1172 ts_tablet_manager.cc:1434] T d31e548feae24b739087c1e211d514f9 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251029 02:52:13.576542 1172 tablet_bootstrap.cc:492] T ad214c870ee649279ed7321c5b69cbee P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:13.577962 1172 tablet_bootstrap.cc:654] T ad214c870ee649279ed7321c5b69cbee P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
W20251029 02:52:13.579260 517 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 748777c6cd6e4f49ab49912b719e48bd consensus state has no leader
I20251029 02:52:13.579553 517 master_service.cc:954] ReplaceTablet: received request to replace tablet dab6ed2266ed42508738399d388c4da3 from {username='slave'} at 127.0.0.1:35314
I20251029 02:52:13.588233 2443 consensus_queue.cc:1048] T 59eee86f249e4d6a9d09d30276dcd659 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:13.598418 2430 consensus_queue.cc:1048] T 731c0acfbe204e8199ad246e38ae2ccd P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:13.600606 1174 ts_tablet_manager.cc:1929] T 9bb2166a590d4210b02dd779ef6654ad P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:13.600752 1174 log.cc:1199] T 9bb2166a590d4210b02dd779ef6654ad P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/9bb2166a590d4210b02dd779ef6654ad
I20251029 02:52:13.601091 1174 ts_tablet_manager.cc:1950] T 9bb2166a590d4210b02dd779ef6654ad P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:13.603099 1172 tablet_bootstrap.cc:492] T ad214c870ee649279ed7321c5b69cbee P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:13.603247 1172 ts_tablet_manager.cc:1403] T ad214c870ee649279ed7321c5b69cbee P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.027s user 0.001s sys 0.002s
I20251029 02:52:13.604502 1172 raft_consensus.cc:359] T ad214c870ee649279ed7321c5b69cbee P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:13.604691 1172 raft_consensus.cc:385] T ad214c870ee649279ed7321c5b69cbee P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:13.604748 1172 raft_consensus.cc:740] T ad214c870ee649279ed7321c5b69cbee P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:13.604996 1172 consensus_queue.cc:260] T ad214c870ee649279ed7321c5b69cbee P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:13.605537 1172 ts_tablet_manager.cc:1434] T ad214c870ee649279ed7321c5b69cbee P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 02:52:13.605831 1172 tablet_bootstrap.cc:492] T 23b9b2ea814645eba63e5f6b12cfb966 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:13.609056 1175 ts_tablet_manager.cc:1929] T 44ae2b07edca44afbe1b1fb3a5b1115d P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:13.609169 1175 log.cc:1199] T 44ae2b07edca44afbe1b1fb3a5b1115d P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/44ae2b07edca44afbe1b1fb3a5b1115d
I20251029 02:52:13.609517 1175 ts_tablet_manager.cc:1950] T 44ae2b07edca44afbe1b1fb3a5b1115d P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:13.610267 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 866) took 4382 ms. Trace:
I20251029 02:52:13.611580 1167 tablet_bootstrap.cc:492] T ae4231daa2594258aa51419c97a2f50a P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:13.611758 1167 ts_tablet_manager.cc:1403] T ae4231daa2594258aa51419c97a2f50a P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.089s user 0.001s sys 0.005s
I20251029 02:52:13.612663 1167 raft_consensus.cc:359] T ae4231daa2594258aa51419c97a2f50a P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:13.612895 1167 raft_consensus.cc:385] T ae4231daa2594258aa51419c97a2f50a P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:13.612772 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_d90269480e524bccbef447534992b673"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:13.612957 1167 raft_consensus.cc:740] T ae4231daa2594258aa51419c97a2f50a P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:13.613236 1167 consensus_queue.cc:260] T ae4231daa2594258aa51419c97a2f50a P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
W20251029 02:52:13.613502 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d90269480e524bccbef447534992b673 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:13.613926 1167 ts_tablet_manager.cc:1434] T ae4231daa2594258aa51419c97a2f50a P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20251029 02:52:13.614548 1167 tablet_bootstrap.cc:492] T 6789bc16664e4b37b86702449ab27855 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:13.622746 1172 tablet_bootstrap.cc:654] T 23b9b2ea814645eba63e5f6b12cfb966 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:13.623654 518 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 82289e0f823c4d43bfd8c4e878572b2b
I20251029 02:52:13.624154 1167 tablet_bootstrap.cc:654] T 6789bc16664e4b37b86702449ab27855 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:13.624176 518 catalog_manager.cc:6719] ReplaceTablet: tablet 82289e0f823c4d43bfd8c4e878572b2b deleted and replaced by tablet 3d06c88ce8394f37ac8af308b7be042b
I20251029 02:52:13.625139 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 876) took 4366 ms. Trace:
I20251029 02:52:13.625286 526 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet c1222e30bc4141b3ae686d80053c4ef5 (table default.Table_7125184a813c44d997ab9d948a1156bd [id=31801defa6db4ca7a08fbf7f029b65c1]). Expected version 3 got 0
I20251029 02:52:13.625228 1174 rpcz_store.cc:276] 1029 02:52:09.259185 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:09.259262 (+ 77us) service_pool.cc:225] Handling call
1029 02:52:13.488591 (+4229329us) ts_tablet_manager.cc:1151] Deleting tablet 9bb2166a590d4210b02dd779ef6654ad
1029 02:52:13.488600 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:13.563859 (+ 75259us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:13.600505 (+ 36646us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:13.625119 (+ 24614us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4229174}
I20251029 02:52:13.625540 1174 tablet_replica.cc:333] T 1275e192871440408a8345fef9b56260 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:13.625582 526 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet a9e0fcc759a64679b3f59a6c964877b6 (table default.Table_9dce7f70d51d4a3c8d7f98a2435cbce3 [id=657afc87656d4250b4ca87b473fbb752]). Expected version 3 got 0
I20251029 02:52:13.625730 1174 raft_consensus.cc:2243] T 1275e192871440408a8345fef9b56260 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:13.625799 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 9bb2166a590d4210b02dd779ef6654ad (table default.Table_50252b3b3cc4403da5e259f9f91be915 [id=ff64a323de354792997beb99bae4a52c]) successfully deleted
I20251029 02:52:13.625886 1174 raft_consensus.cc:2272] T 1275e192871440408a8345fef9b56260 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:13.625888 526 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet adf6e736cb9c4920807ee58682e134ef (table default.Table_0a039bfb2e1f458f89ffc3ee93300b8f [id=64650da66d464074828f2dc955045f68]). Expected version 3 got 0
I20251029 02:52:13.626001 1166 ts_tablet_manager.cc:1403] T 378f549ff7364d018d8b9a5ef01ac878 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.393s user 0.002s sys 0.002s
I20251029 02:52:13.626084 526 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet d7abaf84ba5d445aa17433b3be891548 (table default.Table_c1c6933af1e046aa895e0ee915178f5c [id=9e33c530a35544a4ae22e2a1238592db]). Expected version 3 got 0
I20251029 02:52:13.626375 1174 ts_tablet_manager.cc:1916] T 1275e192871440408a8345fef9b56260 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:13.626495 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 82289e0f823c4d43bfd8c4e878572b2b with delete_type TABLET_DATA_DELETED (replaced by tablet 3d06c88ce8394f37ac8af308b7be042b) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:13.626780 1166 raft_consensus.cc:359] T 378f549ff7364d018d8b9a5ef01ac878 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:13.626961 1166 raft_consensus.cc:385] T 378f549ff7364d018d8b9a5ef01ac878 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:13.627023 1166 raft_consensus.cc:740] T 378f549ff7364d018d8b9a5ef01ac878 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:13.627254 1166 consensus_queue.cc:260] T 378f549ff7364d018d8b9a5ef01ac878 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:13.627810 1166 ts_tablet_manager.cc:1434] T 378f549ff7364d018d8b9a5ef01ac878 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251029 02:52:13.628978 524 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet 95f342d1cfb84c4eb31bd3d15ea3bdc2 (table default.Table_96ea715e50e74cc48759dda025a46318 [id=c63d52811db749d6badbed72fc133943]). Expected version 1 got 0
I20251029 02:52:13.629441 524 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet adf6e736cb9c4920807ee58682e134ef (table default.Table_0a039bfb2e1f458f89ffc3ee93300b8f [id=64650da66d464074828f2dc955045f68]). Expected version 3 got 0
I20251029 02:52:13.629624 524 catalog_manager.cc:5649] T a9e0fcc759a64679b3f59a6c964877b6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } health_report { overall_health: UNKNOWN } } }
I20251029 02:52:13.629941 524 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet a9e0fcc759a64679b3f59a6c964877b6 (table default.Table_9dce7f70d51d4a3c8d7f98a2435cbce3 [id=657afc87656d4250b4ca87b473fbb752]). Expected version 3 got 0
I20251029 02:52:13.630177 524 catalog_manager.cc:5649] T 731c0acfbe204e8199ad246e38ae2ccd P 0cfd4bee16f84b7c8b0ff3ce14db20d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } health_report { overall_health: UNKNOWN } } }
I20251029 02:52:13.630921 1038 tablet_service.cc:1552] Processing DeleteTablet for tablet 82289e0f823c4d43bfd8c4e878572b2b with delete_type TABLET_DATA_DELETED (replaced by tablet 3d06c88ce8394f37ac8af308b7be042b) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:13.631453 524 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet d7abaf84ba5d445aa17433b3be891548 (table default.Table_c1c6933af1e046aa895e0ee915178f5c [id=9e33c530a35544a4ae22e2a1238592db]). Expected version 3 got 0
I20251029 02:52:13.632305 744 tablet_service.cc:1552] Processing DeleteTablet for tablet 82289e0f823c4d43bfd8c4e878572b2b with delete_type TABLET_DATA_DELETED (replaced by tablet 3d06c88ce8394f37ac8af308b7be042b) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:13.610349 1175 rpcz_store.cc:276] 1029 02:52:09.227388 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:09.227860 (+ 472us) service_pool.cc:225] Handling call
1029 02:52:13.558381 (+4330521us) ts_tablet_manager.cc:1151] Deleting tablet 44ae2b07edca44afbe1b1fb3a5b1115d
1029 02:52:13.558391 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:13.589695 (+ 31304us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:13.608955 (+ 19260us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:13.610247 (+ 1292us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4330397}
I20251029 02:52:13.637588 1024 tablet_service.cc:1505] Processing CreateTablet for tablet f3794a0d2f4f421f8b2d0c5649bcb2ec (DEFAULT_TABLE table=default.Table_eb6ed8b2aee44b00a5ac4f8d2a357f28 [id=3a9c3a7a33d94cfbb0f9ab02886e16b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:13.637969 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3794a0d2f4f421f8b2d0c5649bcb2ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:13.637933 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 44ae2b07edca44afbe1b1fb3a5b1115d (table default.Table_c2719a3251c84739a6681f80f1d8f94d [id=a746921b2aba4cd6af22a293991968e8]) successfully deleted
I20251029 02:52:13.638003 1175 tablet_replica.cc:333] T 97b2ecae572a4b7ea1b9595074556708 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:13.638239 1175 raft_consensus.cc:2243] T 97b2ecae572a4b7ea1b9595074556708 P b495a49349324f7d81908f2da700f0d9 [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:13.638654 1175 raft_consensus.cc:2272] T 97b2ecae572a4b7ea1b9595074556708 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:13.638809 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_f88524c2f85e4721931d212083db66ba"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:13.639333 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f88524c2f85e4721931d212083db66ba in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:13.639629 1175 ts_tablet_manager.cc:1916] T 97b2ecae572a4b7ea1b9595074556708 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:13.644665 1172 tablet_bootstrap.cc:492] T 23b9b2ea814645eba63e5f6b12cfb966 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:13.644798 1172 ts_tablet_manager.cc:1403] T 23b9b2ea814645eba63e5f6b12cfb966 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.039s user 0.003s sys 0.000s
I20251029 02:52:13.645499 1172 raft_consensus.cc:359] T 23b9b2ea814645eba63e5f6b12cfb966 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:13.645655 1172 raft_consensus.cc:385] T 23b9b2ea814645eba63e5f6b12cfb966 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:13.645709 1172 raft_consensus.cc:740] T 23b9b2ea814645eba63e5f6b12cfb966 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:13.645931 1172 consensus_queue.cc:260] T 23b9b2ea814645eba63e5f6b12cfb966 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:13.646826 1172 ts_tablet_manager.cc:1434] T 23b9b2ea814645eba63e5f6b12cfb966 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20251029 02:52:13.589037 1968 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/master-0/wal
uuid: "e9672114a6a742e2864c16ff60ba246e"
format_stamp: "Formatted at 2025-10-29 02:51:46 on dist-test-slave-qwrb"
I20251029 02:52:13.647416 1968 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251029 02:52:13.648166 522 catalog_manager.cc:6194] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: default.Table_7c1cb64e05be4f6e8ac28286c982a3e7 [id=b4fc453bf54d45c4b8dddc3c49bf48f8] alter complete (version 2)
I20251029 02:52:13.650830 756 tablet_service.cc:1552] Processing DeleteTablet for tablet 72f9d4408f384e7093316d85f092e1b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:05 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:13.651067 751 tablet_service.cc:1552] Processing DeleteTablet for tablet 8048506ced4540a5aee9f79bfc4599bf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:06 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:13.651166 744 tablet_service.cc:1552] Processing DeleteTablet for tablet 44ae2b07edca44afbe1b1fb3a5b1115d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:00 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:13.651373 756 tablet_service.cc:1552] Processing DeleteTablet for tablet 6142018e44af4d32818adefa2ee43d02 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:01 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:13.652215 743 tablet_service.cc:1552] Processing DeleteTablet for tablet af4860c6b72f4d988b206b5ef2f22906 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:56 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:13.654426 2205 raft_consensus.cc:493] T 27b08f88b99643d28ac609e45bf8104e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:13.654578 2205 raft_consensus.cc:515] T 27b08f88b99643d28ac609e45bf8104e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:13.655465 1058 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "27b08f88b99643d28ac609e45bf8104e" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:52:13.656018 849 leader_election.cc:343] T 27b08f88b99643d28ac609e45bf8104e P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:13.656530 2205 leader_election.cc:290] T 27b08f88b99643d28ac609e45bf8104e P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:13.656823 747 tablet_service.cc:1552] Processing DeleteTablet for tablet 28f7ceb91a34450cad4b5b45f5285b41 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:59 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:13.658687 1167 tablet_bootstrap.cc:492] T 6789bc16664e4b37b86702449ab27855 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:13.658812 1167 ts_tablet_manager.cc:1403] T 6789bc16664e4b37b86702449ab27855 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.044s user 0.004s sys 0.000s
I20251029 02:52:13.659138 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_6fa2b2dfa5c240ba95baf05031690542"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:13.659426 2416 raft_consensus.cc:493] T d29cc2a0c8b84e3ebadc7f571d666847 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:13.659467 1167 raft_consensus.cc:359] T 6789bc16664e4b37b86702449ab27855 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:13.659605 1167 raft_consensus.cc:385] T 6789bc16664e4b37b86702449ab27855 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:13.659538 2416 raft_consensus.cc:515] T d29cc2a0c8b84e3ebadc7f571d666847 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:13.659646 1167 raft_consensus.cc:740] T 6789bc16664e4b37b86702449ab27855 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
W20251029 02:52:13.659701 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6fa2b2dfa5c240ba95baf05031690542 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:13.659845 1167 consensus_queue.cc:260] T 6789bc16664e4b37b86702449ab27855 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:13.660102 2416 leader_election.cc:290] T d29cc2a0c8b84e3ebadc7f571d666847 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:52:13.660552 1058 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d29cc2a0c8b84e3ebadc7f571d666847" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:13.660843 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d29cc2a0c8b84e3ebadc7f571d666847" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:13.661360 711 leader_election.cc:343] T d29cc2a0c8b84e3ebadc7f571d666847 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
W20251029 02:52:13.661507 712 leader_election.cc:343] T d29cc2a0c8b84e3ebadc7f571d666847 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:13.661592 712 leader_election.cc:304] T d29cc2a0c8b84e3ebadc7f571d666847 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:13.661799 2416 raft_consensus.cc:2749] T d29cc2a0c8b84e3ebadc7f571d666847 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:13.665341 1167 ts_tablet_manager.cc:1434] T 6789bc16664e4b37b86702449ab27855 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20251029 02:52:13.665985 521 master_service.cc:954] ReplaceTablet: received request to replace tablet 1c53fc1fff1a4bb988beb46ff91fd3be from {username='slave'} at 127.0.0.1:35314
I20251029 02:52:13.665531 1166 tablet_bootstrap.cc:492] T 58dde9397b434b089ade9968f369006a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:13.666313 1167 tablet_bootstrap.cc:492] T 372d93914abc41b38a702742b7db0bc2 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:13.667654 1167 tablet_bootstrap.cc:654] T 372d93914abc41b38a702742b7db0bc2 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:13.667887 1166 tablet_bootstrap.cc:654] T 58dde9397b434b089ade9968f369006a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:13.670974 1176 ts_tablet_manager.cc:1929] T 217144957930495aa3e9dfcfde268e71 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:13.671142 1176 log.cc:1199] T 217144957930495aa3e9dfcfde268e71 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/217144957930495aa3e9dfcfde268e71
I20251029 02:52:13.671514 1176 ts_tablet_manager.cc:1950] T 217144957930495aa3e9dfcfde268e71 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:13.672902 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "27b08f88b99643d28ac609e45bf8104e" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:13.673080 773 raft_consensus.cc:2468] T 27b08f88b99643d28ac609e45bf8104e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:13.676055 847 leader_election.cc:304] T 27b08f88b99643d28ac609e45bf8104e P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5
I20251029 02:52:13.676759 2205 raft_consensus.cc:2804] T 27b08f88b99643d28ac609e45bf8104e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:13.676683 2179 consensus_queue.cc:1048] T 59eee86f249e4d6a9d09d30276dcd659 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:13.676838 2205 raft_consensus.cc:493] T 27b08f88b99643d28ac609e45bf8104e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:13.676896 2205 raft_consensus.cc:3060] T 27b08f88b99643d28ac609e45bf8104e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:13.679294 1172 tablet_bootstrap.cc:492] T 70b008011d5547bbaf3f45685394139f P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:13.680124 2205 raft_consensus.cc:515] T 27b08f88b99643d28ac609e45bf8104e P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:13.681226 1172 tablet_bootstrap.cc:654] T 70b008011d5547bbaf3f45685394139f P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:13.681540 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "27b08f88b99643d28ac609e45bf8104e" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:13.681649 739 tablet_service.cc:1552] Processing DeleteTablet for tablet c32c28b0e74a4af3a0e69bbf24ab1a1f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:57 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:13.681692 773 raft_consensus.cc:3060] T 27b08f88b99643d28ac609e45bf8104e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:13.683530 1175 ts_tablet_manager.cc:1929] T 97b2ecae572a4b7ea1b9595074556708 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:13.683642 1175 log.cc:1199] T 97b2ecae572a4b7ea1b9595074556708 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/97b2ecae572a4b7ea1b9595074556708
I20251029 02:52:13.684062 1175 ts_tablet_manager.cc:1950] T 97b2ecae572a4b7ea1b9595074556708 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:13.684331 773 raft_consensus.cc:2468] T 27b08f88b99643d28ac609e45bf8104e P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:13.684899 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 867) took 4457 ms. Trace:
I20251029 02:52:13.684935 847 leader_election.cc:304] T 27b08f88b99643d28ac609e45bf8104e P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:13.684983 1175 rpcz_store.cc:276] 1029 02:52:09.227500 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:09.228014 (+ 514us) service_pool.cc:225] Handling call
1029 02:52:13.637941 (+4409927us) ts_tablet_manager.cc:1151] Deleting tablet 97b2ecae572a4b7ea1b9595074556708
1029 02:52:13.637952 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:13.659904 (+ 21952us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:13.683408 (+ 23504us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:13.684882 (+ 1474us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4409811}
I20251029 02:52:13.685216 2426 raft_consensus.cc:2804] T 27b08f88b99643d28ac609e45bf8104e P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:13.685333 2426 raft_consensus.cc:697] T 27b08f88b99643d28ac609e45bf8104e P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:52:13.685361 1174 ts_tablet_manager.cc:1929] T 1275e192871440408a8345fef9b56260 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:13.685405 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 97b2ecae572a4b7ea1b9595074556708 (table default.Table_3b4bdfaf4ddf43cfa2d2287594c57816 [id=222ef1b283554565b4930328c2c8975b]) successfully deleted
I20251029 02:52:13.685456 1174 log.cc:1199] T 1275e192871440408a8345fef9b56260 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/1275e192871440408a8345fef9b56260
I20251029 02:52:13.685547 2426 consensus_queue.cc:237] T 27b08f88b99643d28ac609e45bf8104e P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:13.685748 1174 ts_tablet_manager.cc:1950] T 1275e192871440408a8345fef9b56260 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:13.686417 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 877) took 4426 ms. Trace:
I20251029 02:52:13.686486 1174 rpcz_store.cc:276] 1029 02:52:09.259569 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:09.260079 (+ 510us) service_pool.cc:225] Handling call
1029 02:52:13.625493 (+4365414us) ts_tablet_manager.cc:1151] Deleting tablet 1275e192871440408a8345fef9b56260
1029 02:52:13.625502 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:13.630280 (+ 4778us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:13.677534 (+ 47254us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:13.686400 (+ 8866us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4365275}
I20251029 02:52:13.686789 1174 tablet_replica.cc:333] T b6d4f7bf702f4834bc5f4ef2b50b92a7 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:13.686898 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 1275e192871440408a8345fef9b56260 (table default.Table_5265805d6c3746209f9466df88b164b9 [id=5c4f8f38dfc748dda3abf03f7882cf35]) successfully deleted
I20251029 02:52:13.686972 1174 raft_consensus.cc:2243] T b6d4f7bf702f4834bc5f4ef2b50b92a7 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:13.687309 1174 raft_consensus.cc:2272] T b6d4f7bf702f4834bc5f4ef2b50b92a7 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:13.691053 2205 leader_election.cc:290] T 27b08f88b99643d28ac609e45bf8104e P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:13.694548 2452 tablet.cc:1722] T 95f342d1cfb84c4eb31bd3d15ea3bdc2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 02:52:13.696154 1174 ts_tablet_manager.cc:1916] T b6d4f7bf702f4834bc5f4ef2b50b92a7 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:13.698048 2453 tablet.cc:1722] T 95f342d1cfb84c4eb31bd3d15ea3bdc2 P c756528642a543eeacd93cef3e335a7d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 02:52:13.698453 2205 raft_consensus.cc:493] T f281191e7c6143f983f11d87f1c9b64e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:13.698658 2205 raft_consensus.cc:515] T f281191e7c6143f983f11d87f1c9b64e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:13.700016 2205 leader_election.cc:290] T f281191e7c6143f983f11d87f1c9b64e P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:13.700327 766 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f281191e7c6143f983f11d87f1c9b64e" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
W20251029 02:52:13.700858 519 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3a2b95997454452bb985c51fc02d77e2 consensus state has no leader
I20251029 02:52:13.700909 766 raft_consensus.cc:2468] T f281191e7c6143f983f11d87f1c9b64e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:13.701519 847 leader_election.cc:304] T f281191e7c6143f983f11d87f1c9b64e P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:13.702090 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_a2f9a06d661940bbbc587bf4efb7ec28"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:13.702672 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a2f9a06d661940bbbc587bf4efb7ec28 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:13.706653 2302 raft_consensus.cc:2804] T f281191e7c6143f983f11d87f1c9b64e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:13.706730 2302 raft_consensus.cc:493] T f281191e7c6143f983f11d87f1c9b64e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:13.706784 2302 raft_consensus.cc:3060] T f281191e7c6143f983f11d87f1c9b64e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:13.709319 2302 raft_consensus.cc:515] T f281191e7c6143f983f11d87f1c9b64e P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:13.709880 2302 leader_election.cc:290] T f281191e7c6143f983f11d87f1c9b64e P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:13.711272 1167 tablet_bootstrap.cc:492] T 372d93914abc41b38a702742b7db0bc2 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:13.712991 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f281191e7c6143f983f11d87f1c9b64e" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:13.713172 773 raft_consensus.cc:3060] T f281191e7c6143f983f11d87f1c9b64e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
W20251029 02:52:13.714686 520 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a9e0fcc759a64679b3f59a6c964877b6 consensus state has no leader
I20251029 02:52:13.716849 773 raft_consensus.cc:2468] T f281191e7c6143f983f11d87f1c9b64e P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:13.717301 847 leader_election.cc:304] T f281191e7c6143f983f11d87f1c9b64e P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:13.717646 2302 raft_consensus.cc:2804] T f281191e7c6143f983f11d87f1c9b64e P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:13.717926 520 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_dbeb5b9d4261424085b86ecddb373eff" } modify_external_catalogs: true
I20251029 02:52:13.717979 2302 raft_consensus.cc:697] T f281191e7c6143f983f11d87f1c9b64e P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:52:13.718072 520 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_dbeb5b9d4261424085b86ecddb373eff" } modify_external_catalogs: true
I20251029 02:52:13.718222 2302 consensus_queue.cc:237] T f281191e7c6143f983f11d87f1c9b64e P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:13.719689 894 tablet_service.cc:1552] Processing DeleteTablet for tablet d041e87ceb11441983d0e38b7929c3a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:01 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:13.720372 900 tablet_service.cc:1552] Processing DeleteTablet for tablet 67651306f3b44e7eb14dcee9a2007a81 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:03 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:13.720558 895 tablet_service.cc:1552] Processing DeleteTablet for tablet 59eee86f249e4d6a9d09d30276dcd659 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:56 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:13.721598 881 tablet_service.cc:1552] Processing DeleteTablet for tablet 777992acb1bd4765886b802940db74bb with delete_type TABLET_DATA_DELETED (replaced by tablet e65eaaae43ac435ebd665546de9d68dd) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:13.722057 526 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_39a7937d49ce4ee290f72ed5295a2a3e" } modify_external_catalogs: true
I20251029 02:52:13.722301 526 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_39a7937d49ce4ee290f72ed5295a2a3e" } modify_external_catalogs: true
I20251029 02:52:13.722503 1167 ts_tablet_manager.cc:1403] T 372d93914abc41b38a702742b7db0bc2 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.046s user 0.002s sys 0.000s
I20251029 02:52:13.723429 1167 raft_consensus.cc:359] T 372d93914abc41b38a702742b7db0bc2 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:13.723603 1167 raft_consensus.cc:385] T 372d93914abc41b38a702742b7db0bc2 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:13.723667 1167 raft_consensus.cc:740] T 372d93914abc41b38a702742b7db0bc2 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:13.724576 1167 consensus_queue.cc:260] T 372d93914abc41b38a702742b7db0bc2 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:13.726603 523 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_aedb5a569ca74ef2a9092a847ad5a3f6" } modify_external_catalogs: true
I20251029 02:52:13.726742 523 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_aedb5a569ca74ef2a9092a847ad5a3f6" } modify_external_catalogs: true
I20251029 02:52:13.726790 1167 ts_tablet_manager.cc:1434] T 372d93914abc41b38a702742b7db0bc2 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20251029 02:52:13.727555 1167 tablet_bootstrap.cc:492] T 72f9d4408f384e7093316d85f092e1b2 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:13.731124 1172 tablet_bootstrap.cc:492] T 70b008011d5547bbaf3f45685394139f P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:13.731269 1172 ts_tablet_manager.cc:1403] T 70b008011d5547bbaf3f45685394139f P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.052s user 0.003s sys 0.003s
I20251029 02:52:13.731976 1172 raft_consensus.cc:359] T 70b008011d5547bbaf3f45685394139f P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:13.732134 1172 raft_consensus.cc:385] T 70b008011d5547bbaf3f45685394139f P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:13.732189 1172 raft_consensus.cc:740] T 70b008011d5547bbaf3f45685394139f P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:13.732474 1172 consensus_queue.cc:260] T 70b008011d5547bbaf3f45685394139f P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:13.733120 517 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet dab6ed2266ed42508738399d388c4da3
I20251029 02:52:13.733547 1172 ts_tablet_manager.cc:1434] T 70b008011d5547bbaf3f45685394139f P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 02:52:13.733590 517 catalog_manager.cc:6719] ReplaceTablet: tablet dab6ed2266ed42508738399d388c4da3 deleted and replaced by tablet 06b1df46901343b6b28be4c17dd61bed
I20251029 02:52:13.734776 1031 tablet_service.cc:1552] Processing DeleteTablet for tablet dab6ed2266ed42508738399d388c4da3 with delete_type TABLET_DATA_DELETED (replaced by tablet 06b1df46901343b6b28be4c17dd61bed) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:13.735621 881 tablet_service.cc:1552] Processing DeleteTablet for tablet dab6ed2266ed42508738399d388c4da3 with delete_type TABLET_DATA_DELETED (replaced by tablet 06b1df46901343b6b28be4c17dd61bed) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:13.736536 2469 tablet.cc:1722] T 95f342d1cfb84c4eb31bd3d15ea3bdc2 P b495a49349324f7d81908f2da700f0d9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 02:52:13.738306 1166 tablet_bootstrap.cc:492] T 58dde9397b434b089ade9968f369006a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:13.739183 1167 tablet_bootstrap.cc:654] T 72f9d4408f384e7093316d85f092e1b2 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:13.740502 739 tablet_service.cc:1552] Processing DeleteTablet for tablet dab6ed2266ed42508738399d388c4da3 with delete_type TABLET_DATA_DELETED (replaced by tablet 06b1df46901343b6b28be4c17dd61bed) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:13.746230 525 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_149358409d284922ad0f2c49410dca96" } modify_external_catalogs: true
I20251029 02:52:13.746395 525 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_149358409d284922ad0f2c49410dca96" } modify_external_catalogs: true
I20251029 02:52:13.747869 517 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_dca7cb07591d4d238e29e54969bca2b8" } modify_external_catalogs: true
I20251029 02:52:13.748044 517 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_dca7cb07591d4d238e29e54969bca2b8" } modify_external_catalogs: true
I20251029 02:52:13.748260 2205 raft_consensus.cc:493] T a9e0fcc759a64679b3f59a6c964877b6 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:13.748378 2205 raft_consensus.cc:515] T a9e0fcc759a64679b3f59a6c964877b6 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:13.749615 2205 leader_election.cc:290] T a9e0fcc759a64679b3f59a6c964877b6 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:13.749749 2205 raft_consensus.cc:493] T d61beda662d844cdaf67fd9a2e382ea2 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:13.749866 2205 raft_consensus.cc:515] T d61beda662d844cdaf67fd9a2e382ea2 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:13.750012 762 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a9e0fcc759a64679b3f59a6c964877b6" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
W20251029 02:52:13.751048 847 leader_election.cc:343] T d61beda662d844cdaf67fd9a2e382ea2 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: d61beda662d844cdaf67fd9a2e382ea2
W20251029 02:52:13.751164 849 leader_election.cc:343] T d61beda662d844cdaf67fd9a2e382ea2 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Not found: Tablet not found: d61beda662d844cdaf67fd9a2e382ea2
I20251029 02:52:13.751228 847 leader_election.cc:304] T d61beda662d844cdaf67fd9a2e382ea2 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9
I20251029 02:52:13.751354 2205 leader_election.cc:290] T d61beda662d844cdaf67fd9a2e382ea2 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:13.751461 2302 raft_consensus.cc:2749] T d61beda662d844cdaf67fd9a2e382ea2 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251029 02:52:13.752993 1174 ts_tablet_manager.cc:1929] T b6d4f7bf702f4834bc5f4ef2b50b92a7 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:13.750262 762 raft_consensus.cc:2393] T a9e0fcc759a64679b3f59a6c964877b6 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c756528642a543eeacd93cef3e335a7d in current term 1: Already voted for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in this term.
I20251029 02:52:13.753098 1174 log.cc:1199] T b6d4f7bf702f4834bc5f4ef2b50b92a7 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/b6d4f7bf702f4834bc5f4ef2b50b92a7
I20251029 02:52:13.753458 1174 ts_tablet_manager.cc:1950] T b6d4f7bf702f4834bc5f4ef2b50b92a7 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:13.754168 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 878) took 4493 ms. Trace:
I20251029 02:52:13.754248 1174 rpcz_store.cc:276] 1029 02:52:09.260375 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:09.260548 (+ 173us) service_pool.cc:225] Handling call
1029 02:52:13.686747 (+4426199us) ts_tablet_manager.cc:1151] Deleting tablet b6d4f7bf702f4834bc5f4ef2b50b92a7
1029 02:52:13.686757 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:13.734523 (+ 47766us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:13.752896 (+ 18373us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:13.754151 (+ 1255us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4426057}
I20251029 02:52:13.754675 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet b6d4f7bf702f4834bc5f4ef2b50b92a7 (table default.Table_c70eb9f284cd49109fd09e430a1cea6c [id=be817eb6ec33406eb316d36575c134fb]) successfully deleted
I20251029 02:52:13.756192 1172 tablet_bootstrap.cc:492] T 589a521f5cc24e6289c37b9cf256bce5 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:13.757660 1172 tablet_bootstrap.cc:654] T 589a521f5cc24e6289c37b9cf256bce5 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:13.766168 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 879) took 4452 ms. Trace:
I20251029 02:52:13.766284 1174 rpcz_store.cc:276] 1029 02:52:09.313899 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:09.314550 (+ 651us) service_pool.cc:225] Handling call
1029 02:52:13.766016 (+4451466us) ts_tablet_manager.cc:1151] Deleting tablet ac162bce5b6e4080839095a045ef0cc0
1029 02:52:13.766028 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:13.766143 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4451261}
I20251029 02:52:13.773490 1167 tablet_bootstrap.cc:492] T 72f9d4408f384e7093316d85f092e1b2 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:13.773825 1167 ts_tablet_manager.cc:1403] T 72f9d4408f384e7093316d85f092e1b2 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.047s user 0.003s sys 0.000s
I20251029 02:52:13.774580 1167 raft_consensus.cc:359] T 72f9d4408f384e7093316d85f092e1b2 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
W20251029 02:52:13.778182 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet ac162bce5b6e4080839095a045ef0cc0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ac162bce5b6e4080839095a045ef0cc0 already in progress: creating tablet
I20251029 02:52:13.778384 1167 raft_consensus.cc:385] T 72f9d4408f384e7093316d85f092e1b2 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:13.778560 1167 raft_consensus.cc:740] T 72f9d4408f384e7093316d85f092e1b2 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:13.779105 1167 consensus_queue.cc:260] T 72f9d4408f384e7093316d85f092e1b2 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:13.779485 2302 raft_consensus.cc:493] T 378f549ff7364d018d8b9a5ef01ac878 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:13.779585 2302 raft_consensus.cc:515] T 378f549ff7364d018d8b9a5ef01ac878 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:13.780563 2302 leader_election.cc:290] T 378f549ff7364d018d8b9a5ef01ac878 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:13.783023 916 raft_consensus.cc:3060] T a9e0fcc759a64679b3f59a6c964877b6 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:13.784235 2416 raft_consensus.cc:493] T 6f8cf09f23984b879b60ab6b27c7d52e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:13.784374 2416 raft_consensus.cc:515] T 6f8cf09f23984b879b60ab6b27c7d52e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:13.784981 2416 leader_election.cc:290] T 6f8cf09f23984b879b60ab6b27c7d52e P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:52:13.785759 762 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "378f549ff7364d018d8b9a5ef01ac878" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:13.785929 762 raft_consensus.cc:2468] T 378f549ff7364d018d8b9a5ef01ac878 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:13.785946 916 raft_consensus.cc:1275] T a9e0fcc759a64679b3f59a6c964877b6 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Refusing update from remote peer 0cfd4bee16f84b7c8b0ff3ce14db20d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251029 02:52:13.786432 847 leader_election.cc:304] T 378f549ff7364d018d8b9a5ef01ac878 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:13.786731 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f8cf09f23984b879b60ab6b27c7d52e" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:13.786796 2205 raft_consensus.cc:2804] T 378f549ff7364d018d8b9a5ef01ac878 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:13.786866 2205 raft_consensus.cc:493] T 378f549ff7364d018d8b9a5ef01ac878 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:13.786922 2205 raft_consensus.cc:3060] T 378f549ff7364d018d8b9a5ef01ac878 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
W20251029 02:52:13.787609 711 leader_election.cc:343] T 6f8cf09f23984b879b60ab6b27c7d52e P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:13.787820 1167 ts_tablet_manager.cc:1434] T 72f9d4408f384e7093316d85f092e1b2 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.014s user 0.001s sys 0.000s
I20251029 02:52:13.788321 1167 tablet_bootstrap.cc:492] T 6142018e44af4d32818adefa2ee43d02 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:13.789484 518 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_28e9d77a943b43588e674c3d4825c74f" } new_table_name: "default.Table_e2177489f8c84dfa9aa73bd309e20652" modify_external_catalogs: true
I20251029 02:52:13.789515 2205 raft_consensus.cc:515] T 378f549ff7364d018d8b9a5ef01ac878 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:13.790105 1167 tablet_bootstrap.cc:654] T 6142018e44af4d32818adefa2ee43d02 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:13.790861 762 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "378f549ff7364d018d8b9a5ef01ac878" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:13.791023 762 raft_consensus.cc:3060] T 378f549ff7364d018d8b9a5ef01ac878 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:13.791004 522 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_3ccbb335620141bca4af176e8a8b8c8f" } new_table_name: "default.Table_5a4b5d7211d4473f9bc57d3198c1e87d" modify_external_catalogs: true
I20251029 02:52:13.792261 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_3db2442366f54d5e9b10cff1fddc2091"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:13.792771 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3db2442366f54d5e9b10cff1fddc2091 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:13.738499 1058 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "27b08f88b99643d28ac609e45bf8104e" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:52:13.738651 1055 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f281191e7c6143f983f11d87f1c9b64e" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:13.793619 1055 raft_consensus.cc:2468] T f281191e7c6143f983f11d87f1c9b64e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:13.775542 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 880) took 4360 ms. Trace:
I20251029 02:52:13.738790 1061 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f281191e7c6143f983f11d87f1c9b64e" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:52:13.793733 1174 rpcz_store.cc:276] 1029 02:52:09.414790 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:09.414848 (+ 58us) service_pool.cc:225] Handling call
1029 02:52:13.775159 (+4360311us) ts_tablet_manager.cc:1151] Deleting tablet 591a6c86fc984bac9fb9eb62e35dc6c1
1029 02:52:13.775169 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:13.775523 (+ 354us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4360109}
I20251029 02:52:13.793877 1061 raft_consensus.cc:3060] T f281191e7c6143f983f11d87f1c9b64e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
W20251029 02:52:13.794180 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 591a6c86fc984bac9fb9eb62e35dc6c1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 591a6c86fc984bac9fb9eb62e35dc6c1 already in progress: creating tablet
I20251029 02:52:13.794203 762 raft_consensus.cc:2468] T 378f549ff7364d018d8b9a5ef01ac878 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:13.794489 1174 tablet_replica.cc:333] T 1c53fc1fff1a4bb988beb46ff91fd3be P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:13.794898 766 raft_consensus.cc:1275] T a9e0fcc759a64679b3f59a6c964877b6 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Refusing update from remote peer 0cfd4bee16f84b7c8b0ff3ce14db20d5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251029 02:52:13.794981 1174 raft_consensus.cc:2243] T 1c53fc1fff1a4bb988beb46ff91fd3be P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:13.795158 1174 raft_consensus.cc:2272] T 1c53fc1fff1a4bb988beb46ff91fd3be P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:13.795209 847 leader_election.cc:304] T 378f549ff7364d018d8b9a5ef01ac878 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:13.795662 1174 ts_tablet_manager.cc:1916] T 1c53fc1fff1a4bb988beb46ff91fd3be P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:13.795675 2302 raft_consensus.cc:2804] T 378f549ff7364d018d8b9a5ef01ac878 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:13.795923 2302 raft_consensus.cc:697] T 378f549ff7364d018d8b9a5ef01ac878 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:52:13.796186 2302 consensus_queue.cc:237] T 378f549ff7364d018d8b9a5ef01ac878 P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:13.796634 1061 raft_consensus.cc:2468] T f281191e7c6143f983f11d87f1c9b64e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
W20251029 02:52:13.797091 849 leader_election.cc:343] T 27b08f88b99643d28ac609e45bf8104e P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:13.797438 526 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet b4621bf9e6324eaeaa64a6ca51cd5ca9
I20251029 02:52:13.798115 739 tablet_service.cc:1552] Processing DeleteTablet for tablet b4621bf9e6324eaeaa64a6ca51cd5ca9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:13 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:13.798830 881 tablet_service.cc:1552] Processing DeleteTablet for tablet b4621bf9e6324eaeaa64a6ca51cd5ca9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:13 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:13.800648 1040 tablet_service.cc:1552] Processing DeleteTablet for tablet b4621bf9e6324eaeaa64a6ca51cd5ca9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:13 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:13.742805 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1009) took 2215 ms. Trace:
I20251029 02:52:13.749330 1053 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a9e0fcc759a64679b3f59a6c964877b6" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:13.801405 849 leader_election.cc:304] T a9e0fcc759a64679b3f59a6c964877b6 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9
I20251029 02:52:13.779361 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet d041e87ceb11441983d0e38b7929c3a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:01 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:13.801636 2425 raft_consensus.cc:2749] T a9e0fcc759a64679b3f59a6c964877b6 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:13.780835 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet 59eee86f249e4d6a9d09d30276dcd659 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:56 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:13.786211 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "378f549ff7364d018d8b9a5ef01ac878" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:13.801846 1060 raft_consensus.cc:2468] T 378f549ff7364d018d8b9a5ef01ac878 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:13.788050 2188 consensus_queue.cc:1048] T a9e0fcc759a64679b3f59a6c964877b6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:13.790390 1054 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "378f549ff7364d018d8b9a5ef01ac878" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:52:13.792840 1052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f8cf09f23984b879b60ab6b27c7d52e" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:13.802431 1054 raft_consensus.cc:3060] T 378f549ff7364d018d8b9a5ef01ac878 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:13.804021 2179 consensus_queue.cc:1048] T a9e0fcc759a64679b3f59a6c964877b6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:13.804566 1176 rpcz_store.cc:276] 1029 02:52:11.527842 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:11.527885 (+ 43us) service_pool.cc:225] Handling call
1029 02:52:13.229842 (+1701957us) ts_tablet_manager.cc:1151] Deleting tablet 217144957930495aa3e9dfcfde268e71
1029 02:52:13.233882 (+ 4040us) spinlock_profiling.cc:243] Waited 3.82 ms on lock 0x61800000d518. stack: 00007f34717829f6 00007f34717827b0 00007f347075e3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f347176b2fd 00007f347176c36e 00007f34853a7c9d 00007f3485326203 00007f3481559646 00007f348155a9c9 00007f348158e2d4 00007f348156ed70 00007f348156eafd 0000000000a8867c
1029 02:52:13.233891 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:13.557244 (+323353us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:13.670848 (+113604us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:13.742787 (+ 71939us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8781952,"tablet-delete.queue_time_us":1659220}
I20251029 02:52:13.805338 1176 tablet_replica.cc:333] T 97b2ecae572a4b7ea1b9595074556708 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:13.805562 1176 raft_consensus.cc:2243] T 97b2ecae572a4b7ea1b9595074556708 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:13.805768 1176 raft_consensus.cc:2272] T 97b2ecae572a4b7ea1b9595074556708 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:13.806344 1176 ts_tablet_manager.cc:1916] T 97b2ecae572a4b7ea1b9595074556708 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 02:52:13.807132 712 leader_election.cc:343] T 6f8cf09f23984b879b60ab6b27c7d52e P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:13.807250 712 leader_election.cc:304] T 6f8cf09f23984b879b60ab6b27c7d52e P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:13.807619 2416 raft_consensus.cc:2749] T 6f8cf09f23984b879b60ab6b27c7d52e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:13.809532 2205 leader_election.cc:290] T 378f549ff7364d018d8b9a5ef01ac878 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:13.813436 523 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet eaf4018c0d624b1cb8c6528eea7b689e
I20251029 02:52:13.815289 1167 tablet_bootstrap.cc:492] T 6142018e44af4d32818adefa2ee43d02 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:13.815450 1167 ts_tablet_manager.cc:1403] T 6142018e44af4d32818adefa2ee43d02 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.000s
I20251029 02:52:13.815485 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 217144957930495aa3e9dfcfde268e71 (table default.Table_15cde6ff49784e5082de7519ce61c81d [id=3241aaa6ea9148b7a101833462ab3eeb]) successfully deleted
I20251029 02:52:13.815610 739 tablet_service.cc:1552] Processing DeleteTablet for tablet eaf4018c0d624b1cb8c6528eea7b689e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:13 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:13.816002 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_4aceae27cbe54148b726e134b4d22799"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:13.816181 1167 raft_consensus.cc:359] T 6142018e44af4d32818adefa2ee43d02 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:13.816330 1167 raft_consensus.cc:385] T 6142018e44af4d32818adefa2ee43d02 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:13.816375 1167 raft_consensus.cc:740] T 6142018e44af4d32818adefa2ee43d02 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
W20251029 02:52:13.816584 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4aceae27cbe54148b726e134b4d22799 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:13.700340 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 868) took 4392 ms. Trace:
I20251029 02:52:13.818300 1175 rpcz_store.cc:276] 1029 02:52:09.307928 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:09.308016 (+ 88us) service_pool.cc:225] Handling call
1029 02:52:13.700182 (+4392166us) ts_tablet_manager.cc:1151] Deleting tablet ac162bce5b6e4080839095a045ef0cc0
1029 02:52:13.700193 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:13.700320 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4391983}
W20251029 02:52:13.818781 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet ac162bce5b6e4080839095a045ef0cc0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ac162bce5b6e4080839095a045ef0cc0 already in progress: creating tablet
I20251029 02:52:13.819088 1175 tablet_replica.cc:333] T 217144957930495aa3e9dfcfde268e71 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:13.819288 1175 raft_consensus.cc:2243] T 217144957930495aa3e9dfcfde268e71 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:13.819461 1175 raft_consensus.cc:2272] T 217144957930495aa3e9dfcfde268e71 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:13.821069 1174 ts_tablet_manager.cc:1929] T 1c53fc1fff1a4bb988beb46ff91fd3be P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251029 02:52:13.821170 1174 log.cc:1199] T 1c53fc1fff1a4bb988beb46ff91fd3be P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/1c53fc1fff1a4bb988beb46ff91fd3be
I20251029 02:52:13.821486 1174 ts_tablet_manager.cc:1950] T 1c53fc1fff1a4bb988beb46ff91fd3be P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:13.822175 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 881) took 4227 ms. Trace:
I20251029 02:52:13.822254 1174 rpcz_store.cc:276] 1029 02:52:09.595087 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:09.595149 (+ 62us) service_pool.cc:225] Handling call
1029 02:52:13.794435 (+4199286us) ts_tablet_manager.cc:1151] Deleting tablet 1c53fc1fff1a4bb988beb46ff91fd3be
1029 02:52:13.794447 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:13.817826 (+ 23379us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:13.820981 (+ 3155us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:13.822155 (+ 1174us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4199099}
I20251029 02:52:13.822662 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 1c53fc1fff1a4bb988beb46ff91fd3be (table default.Table_dd4934aabcd442039db111954d819f6d [id=9abdb27ccc0a48e991b7a4619808fdf8]) successfully deleted
I20251029 02:52:13.823500 1175 ts_tablet_manager.cc:1916] T 217144957930495aa3e9dfcfde268e71 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:13.825955 517 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet e40d4910365f45e9a45d9b3bc5037c95
I20251029 02:52:13.828202 881 tablet_service.cc:1552] Processing DeleteTablet for tablet e40d4910365f45e9a45d9b3bc5037c95 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:13 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:13.829152 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_6aa358a4fc5c4bce9803079e4b61fdff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:13.829778 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6aa358a4fc5c4bce9803079e4b61fdff in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:13.805576 1054 raft_consensus.cc:2468] T 378f549ff7364d018d8b9a5ef01ac878 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:13.830750 739 tablet_service.cc:1552] Processing DeleteTablet for tablet e40d4910365f45e9a45d9b3bc5037c95 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:13 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:13.814728 1166 ts_tablet_manager.cc:1403] T 58dde9397b434b089ade9968f369006a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.149s user 0.003s sys 0.001s
I20251029 02:52:13.815987 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet eaf4018c0d624b1cb8c6528eea7b689e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:13 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:13.836246 1166 raft_consensus.cc:359] T 58dde9397b434b089ade9968f369006a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:13.836686 1166 raft_consensus.cc:385] T 58dde9397b434b089ade9968f369006a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:13.836738 1166 raft_consensus.cc:740] T 58dde9397b434b089ade9968f369006a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:13.836967 1166 consensus_queue.cc:260] T 58dde9397b434b089ade9968f369006a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:13.837440 1166 ts_tablet_manager.cc:1434] T 58dde9397b434b089ade9968f369006a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 02:52:13.838672 1166 tablet_bootstrap.cc:492] T 40e844fd7ad74680bcbe7bef936c5b9d P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:13.838367 1167 consensus_queue.cc:260] T 6142018e44af4d32818adefa2ee43d02 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
W20251029 02:52:13.839407 522 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c108828dc9ca4c329dcb8a0240bf675f consensus state has no leader
W20251029 02:52:13.839449 518 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2c28131d740c411dbb07898d222630ff consensus state has no leader
I20251029 02:52:13.839733 1167 ts_tablet_manager.cc:1434] T 6142018e44af4d32818adefa2ee43d02 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.024s user 0.001s sys 0.001s
I20251029 02:52:13.841537 518 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_47341e4770a74239b50cbc23e77a82e4" } new_table_name: "default.Table_8aa16ebd94174be79b8fd9d2206599a1" modify_external_catalogs: true
I20251029 02:52:13.841840 1167 tablet_bootstrap.cc:492] T cd597c319af645909b8f3d1212fcb702 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:13.843307 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_7e47b9d463974ab595d765fc15c82ca0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:13.843766 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7e47b9d463974ab595d765fc15c82ca0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:13.851094 1174 tablet_replica.cc:333] T caef1c93e33a4390bcada59a41f43465 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:13.851749 1174 raft_consensus.cc:2243] T caef1c93e33a4390bcada59a41f43465 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:13.852234 1174 raft_consensus.cc:2272] T caef1c93e33a4390bcada59a41f43465 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:13.853024 1174 ts_tablet_manager.cc:1916] T caef1c93e33a4390bcada59a41f43465 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:13.855583 1024 tablet_service.cc:1552] Processing DeleteTablet for tablet e40d4910365f45e9a45d9b3bc5037c95 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:13 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:13.857750 1166 tablet_bootstrap.cc:654] T 40e844fd7ad74680bcbe7bef936c5b9d P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:13.858122 1172 tablet_bootstrap.cc:492] T 589a521f5cc24e6289c37b9cf256bce5 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:13.858310 1172 ts_tablet_manager.cc:1403] T 589a521f5cc24e6289c37b9cf256bce5 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.103s user 0.005s sys 0.012s
I20251029 02:52:13.859129 1172 raft_consensus.cc:359] T 589a521f5cc24e6289c37b9cf256bce5 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:13.859371 1172 raft_consensus.cc:385] T 589a521f5cc24e6289c37b9cf256bce5 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:13.859467 1172 raft_consensus.cc:740] T 589a521f5cc24e6289c37b9cf256bce5 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:13.859732 1172 consensus_queue.cc:260] T 589a521f5cc24e6289c37b9cf256bce5 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:13.860738 1172 ts_tablet_manager.cc:1434] T 589a521f5cc24e6289c37b9cf256bce5 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251029 02:52:13.861383 1172 tablet_bootstrap.cc:492] T 748777c6cd6e4f49ab49912b719e48bd P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:13.862313 2452 tablet.cc:1722] T a9e0fcc759a64679b3f59a6c964877b6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20251029 02:52:13.862977 1172 tablet_bootstrap.cc:654] T 748777c6cd6e4f49ab49912b719e48bd P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:13.864754 2453 tablet.cc:1722] T a9e0fcc759a64679b3f59a6c964877b6 P c756528642a543eeacd93cef3e335a7d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20251029 02:52:13.870260 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_4ef22a62446e4805a5a9f18357f4142d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:13.870820 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4ef22a62446e4805a5a9f18357f4142d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:13.872354 1167 tablet_bootstrap.cc:654] T cd597c319af645909b8f3d1212fcb702 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:13.874382 2205 raft_consensus.cc:493] T ae4231daa2594258aa51419c97a2f50a P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:13.874588 2205 raft_consensus.cc:515] T ae4231daa2594258aa51419c97a2f50a P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:13.875057 517 master_service.cc:954] ReplaceTablet: received request to replace tablet 786d9148fb1d417a8c2c8b8c53665899 from {username='slave'} at 127.0.0.1:35314
I20251029 02:52:13.875425 1174 ts_tablet_manager.cc:1929] T caef1c93e33a4390bcada59a41f43465 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:13.876026 762 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae4231daa2594258aa51419c97a2f50a" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:13.876144 1174 log.cc:1199] T caef1c93e33a4390bcada59a41f43465 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/caef1c93e33a4390bcada59a41f43465
I20251029 02:52:13.876211 762 raft_consensus.cc:2468] T ae4231daa2594258aa51419c97a2f50a P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:13.876397 1174 ts_tablet_manager.cc:1950] T caef1c93e33a4390bcada59a41f43465 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:13.876550 847 leader_election.cc:304] T ae4231daa2594258aa51419c97a2f50a P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:13.876822 2302 raft_consensus.cc:2804] T ae4231daa2594258aa51419c97a2f50a P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:13.876889 2302 raft_consensus.cc:493] T ae4231daa2594258aa51419c97a2f50a P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:13.876955 2302 raft_consensus.cc:3060] T ae4231daa2594258aa51419c97a2f50a P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:13.877105 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 882) took 4109 ms. Trace:
I20251029 02:52:13.877180 1174 rpcz_store.cc:276] 1029 02:52:09.767949 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:09.768283 (+ 334us) service_pool.cc:225] Handling call
1029 02:52:13.851036 (+4082753us) ts_tablet_manager.cc:1151] Deleting tablet caef1c93e33a4390bcada59a41f43465
1029 02:52:13.851049 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:13.868155 (+ 17106us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:13.875353 (+ 7198us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:13.877088 (+ 1735us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4082567}
I20251029 02:52:13.877449 1174 tablet_replica.cc:333] T ba699862e6f646c6beaa39f080d0faba P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:13.877560 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet caef1c93e33a4390bcada59a41f43465 (table default.Table_b9f8911906c3473da4de53896081cb2c [id=8cccaacc256b40deabe999de1d75032e]) successfully deleted
I20251029 02:52:13.879225 1174 raft_consensus.cc:2243] T ba699862e6f646c6beaa39f080d0faba P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:13.879390 2302 raft_consensus.cc:515] T ae4231daa2594258aa51419c97a2f50a P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:13.879515 1174 raft_consensus.cc:2272] T ba699862e6f646c6beaa39f080d0faba P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:13.880020 1174 ts_tablet_manager.cc:1916] T ba699862e6f646c6beaa39f080d0faba P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:13.880218 1052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae4231daa2594258aa51419c97a2f50a" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
W20251029 02:52:13.880458 518 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 44a0c058700040f8b5c30a621103a524 has no consensus state
W20251029 02:52:13.880663 849 leader_election.cc:343] T ae4231daa2594258aa51419c97a2f50a P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:13.881299 762 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae4231daa2594258aa51419c97a2f50a" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:13.881523 762 raft_consensus.cc:3060] T ae4231daa2594258aa51419c97a2f50a P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:13.884595 762 raft_consensus.cc:2468] T ae4231daa2594258aa51419c97a2f50a P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:13.884801 521 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet 6dff036b32904ddb9598be4b79504bb6 (table default.Table_0ae102f09fdc41bcbead0148653df9e2 [id=65ec61bf400a4628b98e64e7d43c025c]). Expected version 2 got 0
I20251029 02:52:13.885056 521 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet c1222e30bc4141b3ae686d80053c4ef5 (table default.Table_7125184a813c44d997ab9d948a1156bd [id=31801defa6db4ca7a08fbf7f029b65c1]). Expected version 3 got 0
I20251029 02:52:13.885049 847 leader_election.cc:304] T ae4231daa2594258aa51419c97a2f50a P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5
I20251029 02:52:13.885283 521 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet 6f8cf09f23984b879b60ab6b27c7d52e (table default.Table_96e017f445bc4f1fbebddf65c722b8de [id=98f28a15eced4860ac13a60abdc8e0e2]). Expected version 3 got 0
I20251029 02:52:13.885329 2425 raft_consensus.cc:2804] T ae4231daa2594258aa51419c97a2f50a P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:13.885434 2425 raft_consensus.cc:697] T ae4231daa2594258aa51419c97a2f50a P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:52:13.885597 521 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet d7abaf84ba5d445aa17433b3be891548 (table default.Table_c1c6933af1e046aa895e0ee915178f5c [id=9e33c530a35544a4ae22e2a1238592db]). Expected version 3 got 0
I20251029 02:52:13.885663 2425 consensus_queue.cc:237] T ae4231daa2594258aa51419c97a2f50a P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:13.885763 521 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet adf6e736cb9c4920807ee58682e134ef (table default.Table_0a039bfb2e1f458f89ffc3ee93300b8f [id=64650da66d464074828f2dc955045f68]). Expected version 3 got 0
I20251029 02:52:13.887889 2205 leader_election.cc:290] T ae4231daa2594258aa51419c97a2f50a P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:13.888077 526 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet adf6e736cb9c4920807ee58682e134ef (table default.Table_0a039bfb2e1f458f89ffc3ee93300b8f [id=64650da66d464074828f2dc955045f68]). Expected version 3 got 0
I20251029 02:52:13.888295 526 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet c1222e30bc4141b3ae686d80053c4ef5 (table default.Table_7125184a813c44d997ab9d948a1156bd [id=31801defa6db4ca7a08fbf7f029b65c1]). Expected version 3 got 0
I20251029 02:52:13.888402 526 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet 378f549ff7364d018d8b9a5ef01ac878 (table default.Table_8c5d559a2d7f427094b719d1c0b6cff4 [id=0e6e6f5dc35241c5a04c9d6cf4c39b26]). Expected version 1 got 0
I20251029 02:52:13.890153 2302 leader_election.cc:290] T ae4231daa2594258aa51419c97a2f50a P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:13.896737 524 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet 378f549ff7364d018d8b9a5ef01ac878 (table default.Table_8c5d559a2d7f427094b719d1c0b6cff4 [id=0e6e6f5dc35241c5a04c9d6cf4c39b26]). Expected version 1 got 0
I20251029 02:52:13.896996 524 catalog_manager.cc:5649] T 27b08f88b99643d28ac609e45bf8104e P c756528642a543eeacd93cef3e335a7d reported cstate change: term changed from 0 to 1, leader changed from <none> to c756528642a543eeacd93cef3e335a7d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "c756528642a543eeacd93cef3e335a7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } health_report { overall_health: UNKNOWN } } }
I20251029 02:52:13.897373 524 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet 27b08f88b99643d28ac609e45bf8104e (table default.Table_a9e50760813844af85a85c5f8d22f8e0 [id=74a2a41c908146ec8e16c6a3227b94f2]). Expected version 1 got 0
I20251029 02:52:13.897809 524 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet d7abaf84ba5d445aa17433b3be891548 (table default.Table_c1c6933af1e046aa895e0ee915178f5c [id=9e33c530a35544a4ae22e2a1238592db]). Expected version 3 got 0
I20251029 02:52:13.898031 524 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet 6789bc16664e4b37b86702449ab27855 (table default.Table_9a7d7f4f9073437097aca9f7d6dc665e [id=ab8aeaf90f6c4bc2ad5a98780945c40a]). Expected version 2 got 0
I20251029 02:52:13.898175 524 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet c1222e30bc4141b3ae686d80053c4ef5 (table default.Table_7125184a813c44d997ab9d948a1156bd [id=31801defa6db4ca7a08fbf7f029b65c1]). Expected version 3 got 0
I20251029 02:52:13.899694 521 catalog_manager.cc:6194] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: default.Table_966779a0d37a4dc4ab3f89d2bf7c6b06 [id=08e3b2ebcf37415d9a2886ad5bbda068] alter complete (version 1)
I20251029 02:52:13.899945 520 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 3e9a076e086141ef8ed59d4c8633f961
I20251029 02:52:13.901172 739 tablet_service.cc:1552] Processing DeleteTablet for tablet 3e9a076e086141ef8ed59d4c8633f961 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:13 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:13.901633 525 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 2cea2f2fe68545db804a732d0c01a876
I20251029 02:52:13.901640 739 tablet_service.cc:1552] Processing DeleteTablet for tablet d31e548feae24b739087c1e211d514f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:00 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:13.902428 739 tablet_service.cc:1552] Processing DeleteTablet for tablet 2cea2f2fe68545db804a732d0c01a876 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:13 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:13.903048 739 tablet_service.cc:1552] Processing DeleteTablet for tablet 23b9b2ea814645eba63e5f6b12cfb966 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:57 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:13.903215 525 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_bb4222ae83ec442f8f62fbb51e9ab8eb" } new_table_name: "default.Table_76410db5d9f74f4d9c4e6adb8f3a7f40" modify_external_catalogs: true
I20251029 02:52:13.905021 881 tablet_service.cc:1552] Processing DeleteTablet for tablet 3e9a076e086141ef8ed59d4c8633f961 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:13 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:13.905613 739 tablet_service.cc:1552] Processing DeleteTablet for tablet d29cc2a0c8b84e3ebadc7f571d666847 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:00 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:13.905668 895 tablet_service.cc:1552] Processing DeleteTablet for tablet 2cea2f2fe68545db804a732d0c01a876 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:13 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:13.906312 739 tablet_service.cc:1552] Processing DeleteTablet for tablet eca016aed05e4ce09e1a2bf96e2286f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:58 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:13.906715 1175 ts_tablet_manager.cc:1929] T 217144957930495aa3e9dfcfde268e71 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:13.909796 526 catalog_manager.cc:6194] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: default.Table_96ea715e50e74cc48759dda025a46318 [id=c63d52811db749d6badbed72fc133943] alter complete (version 1)
I20251029 02:52:13.910161 896 tablet_service.cc:1552] Processing DeleteTablet for tablet eaf4018c0d624b1cb8c6528eea7b689e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:13 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:13.910426 517 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 786d9148fb1d417a8c2c8b8c53665899
I20251029 02:52:13.910914 517 catalog_manager.cc:6719] ReplaceTablet: tablet 786d9148fb1d417a8c2c8b8c53665899 deleted and replaced by tablet 15d41a61ba5943fcbc085943031907c8
I20251029 02:52:13.913414 896 tablet_service.cc:1552] Processing DeleteTablet for tablet 786d9148fb1d417a8c2c8b8c53665899 with delete_type TABLET_DATA_DELETED (replaced by tablet 15d41a61ba5943fcbc085943031907c8) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:13.914311 1054 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae4231daa2594258aa51419c97a2f50a" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:52:13.914880 849 leader_election.cc:343] T ae4231daa2594258aa51419c97a2f50a P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:13.916620 1176 ts_tablet_manager.cc:1929] T 97b2ecae572a4b7ea1b9595074556708 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:13.916785 1176 log.cc:1199] T 97b2ecae572a4b7ea1b9595074556708 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/97b2ecae572a4b7ea1b9595074556708
I20251029 02:52:13.916994 526 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_b52bfade79b84f65973bee918424d4a0" } new_table_name: "default.Table_c41d6ebff8244ed28dabdfe88c53bcd9" modify_external_catalogs: true
I20251029 02:52:13.917273 1176 ts_tablet_manager.cc:1950] T 97b2ecae572a4b7ea1b9595074556708 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:13.919341 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet 3e9a076e086141ef8ed59d4c8633f961 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:13 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:13.919878 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet 2cea2f2fe68545db804a732d0c01a876 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:13 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:13.922938 521 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet a9e0fcc759a64679b3f59a6c964877b6 (table default.Table_d0d8770fdb3d4d34933b7b4264831ad9 [id=657afc87656d4250b4ca87b473fbb752]). Expected version 4 got 0
I20251029 02:52:13.927404 896 tablet_service.cc:1552] Processing DeleteTablet for tablet 58dde9397b434b089ade9968f369006a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:59 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:13.927541 895 tablet_service.cc:1552] Processing DeleteTablet for tablet 7bb85a12f573494cbb3118d1947eda27 with delete_type TABLET_DATA_DELETED (replaced by tablet a9f78f61ee2841dea71bbbfceacc3ea6) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:13.929412 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_863430b577d14e0696d54412522d528f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:13.929975 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_863430b577d14e0696d54412522d528f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:52:13.930630 990 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:41701, user_credentials={real_user=slave}} blocked reactor thread for 46767.1us
W20251029 02:52:13.932617 526 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 76a7e8517db140e8bdd3b67ce423ec90 consensus state has no leader
I20251029 02:52:13.941659 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1010) took 2413 ms. Trace:
I20251029 02:52:13.941872 1176 rpcz_store.cc:276] 1029 02:52:11.528073 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:11.528116 (+ 43us) service_pool.cc:225] Handling call
1029 02:52:13.805289 (+2277173us) ts_tablet_manager.cc:1151] Deleting tablet 97b2ecae572a4b7ea1b9595074556708
1029 02:52:13.805299 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:13.834435 (+ 29136us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:13.916515 (+ 82080us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:13.941640 (+ 25125us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2230665}
I20251029 02:52:13.942338 1176 tablet_replica.cc:333] T 28f7ceb91a34450cad4b5b45f5285b41 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:13.942881 1176 raft_consensus.cc:2243] T 28f7ceb91a34450cad4b5b45f5285b41 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:13.942823 526 catalog_manager.cc:5649] T 378f549ff7364d018d8b9a5ef01ac878 P c756528642a543eeacd93cef3e335a7d reported cstate change: term changed from 0 to 1, leader changed from <none> to c756528642a543eeacd93cef3e335a7d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "c756528642a543eeacd93cef3e335a7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } health_report { overall_health: UNKNOWN } } }
I20251029 02:52:13.943212 1176 raft_consensus.cc:2272] T 28f7ceb91a34450cad4b5b45f5285b41 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:13.943215 526 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet 378f549ff7364d018d8b9a5ef01ac878 (table default.Table_8c5d559a2d7f427094b719d1c0b6cff4 [id=0e6e6f5dc35241c5a04c9d6cf4c39b26]). Expected version 1 got 0
I20251029 02:52:13.943478 526 catalog_manager.cc:5649] T ae4231daa2594258aa51419c97a2f50a P c756528642a543eeacd93cef3e335a7d reported cstate change: term changed from 0 to 1, leader changed from <none> to c756528642a543eeacd93cef3e335a7d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "c756528642a543eeacd93cef3e335a7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } health_report { overall_health: UNKNOWN } } }
I20251029 02:52:13.943951 1176 ts_tablet_manager.cc:1916] T 28f7ceb91a34450cad4b5b45f5285b41 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:13.944132 895 tablet_service.cc:1505] Processing CreateTablet for tablet f7d3441024c54b1984f9c2ccc28aaeb1 (DEFAULT_TABLE table=default.Table_02d8535577114a268daf473744621d3c [id=af32e21ffd164db9a3c2dd9f618ec038]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:13.944525 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7d3441024c54b1984f9c2ccc28aaeb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:13.944892 1175 log.cc:1199] T 217144957930495aa3e9dfcfde268e71 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/217144957930495aa3e9dfcfde268e71
I20251029 02:52:13.944875 2302 raft_consensus.cc:493] T 7a1866afc7564217ad777e7ce53786c5 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:13.906971 739 tablet_service.cc:1552] Processing DeleteTablet for tablet ad214c870ee649279ed7321c5b69cbee with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:05 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:13.944998 2302 raft_consensus.cc:515] T 7a1866afc7564217ad777e7ce53786c5 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:13.945287 1175 ts_tablet_manager.cc:1950] T 217144957930495aa3e9dfcfde268e71 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:13.945402 744 tablet_service.cc:1505] Processing CreateTablet for tablet 90c2ed692a954a25bdf79b9884f6c5ce (DEFAULT_TABLE table=default.Table_d5b1b53f211a44bf944f53dfde769c5d [id=23c4701379f748a6be7c96fe8ecc9e20]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:13.945724 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90c2ed692a954a25bdf79b9884f6c5ce. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:13.946223 847 leader_election.cc:343] T 7a1866afc7564217ad777e7ce53786c5 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 7a1866afc7564217ad777e7ce53786c5
I20251029 02:52:13.946700 2302 leader_election.cc:290] T 7a1866afc7564217ad777e7ce53786c5 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:13.947027 1024 tablet_service.cc:1505] Processing CreateTablet for tablet f7d3441024c54b1984f9c2ccc28aaeb1 (DEFAULT_TABLE table=default.Table_02d8535577114a268daf473744621d3c [id=af32e21ffd164db9a3c2dd9f618ec038]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:13.947350 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7d3441024c54b1984f9c2ccc28aaeb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:13.947588 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 869) took 4610 ms. Trace:
I20251029 02:52:13.947669 1175 rpcz_store.cc:276] 1029 02:52:09.337599 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:09.337655 (+ 56us) service_pool.cc:225] Handling call
1029 02:52:13.819022 (+4481367us) ts_tablet_manager.cc:1151] Deleting tablet 217144957930495aa3e9dfcfde268e71
1029 02:52:13.819033 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:13.858944 (+ 39911us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:13.906611 (+ 47667us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:13.947569 (+ 40958us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4481172}
I20251029 02:52:13.947968 1175 tablet_replica.cc:333] T e65eaaae43ac435ebd665546de9d68dd P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:13.948148 1175 raft_consensus.cc:2243] T e65eaaae43ac435ebd665546de9d68dd P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:13.948292 1175 raft_consensus.cc:2272] T e65eaaae43ac435ebd665546de9d68dd P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251029 02:52:13.948971 849 leader_election.cc:343] T 7a1866afc7564217ad777e7ce53786c5 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Not found: Tablet not found: 7a1866afc7564217ad777e7ce53786c5
I20251029 02:52:13.949075 849 leader_election.cc:304] T 7a1866afc7564217ad777e7ce53786c5 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9
I20251029 02:52:13.949409 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_ce5d7c1b786d4a58aa81cf3ac8e03aa0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:13.949728 2302 raft_consensus.cc:2749] T 7a1866afc7564217ad777e7ce53786c5 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 02:52:13.949944 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ce5d7c1b786d4a58aa81cf3ac8e03aa0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:13.909525 745 tablet_service.cc:1505] Processing CreateTablet for tablet f7d3441024c54b1984f9c2ccc28aaeb1 (DEFAULT_TABLE table=default.Table_02d8535577114a268daf473744621d3c [id=af32e21ffd164db9a3c2dd9f618ec038]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:13.950767 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7d3441024c54b1984f9c2ccc28aaeb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:13.952140 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet 786d9148fb1d417a8c2c8b8c53665899 with delete_type TABLET_DATA_DELETED (replaced by tablet 15d41a61ba5943fcbc085943031907c8) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:13.952561 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet 67651306f3b44e7eb14dcee9a2007a81 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:03 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:13.958992 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 217144957930495aa3e9dfcfde268e71 (table default.Table_15cde6ff49784e5082de7519ce61c81d [id=3241aaa6ea9148b7a101833462ab3eeb]) successfully deleted
I20251029 02:52:13.959705 1036 tablet_service.cc:1505] Processing CreateTablet for tablet 90c2ed692a954a25bdf79b9884f6c5ce (DEFAULT_TABLE table=default.Table_d5b1b53f211a44bf944f53dfde769c5d [id=23c4701379f748a6be7c96fe8ecc9e20]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:13.965235 2302 raft_consensus.cc:493] T 7bb85a12f573494cbb3118d1947eda27 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:13.965411 2302 raft_consensus.cc:515] T 7bb85a12f573494cbb3118d1947eda27 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:13.966547 1054 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7bb85a12f573494cbb3118d1947eda27" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:13.913313 747 tablet_service.cc:1552] Processing DeleteTablet for tablet 786d9148fb1d417a8c2c8b8c53665899 with delete_type TABLET_DATA_DELETED (replaced by tablet 15d41a61ba5943fcbc085943031907c8) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:13.925285 743 tablet_service.cc:1552] Processing DeleteTablet for tablet 589a521f5cc24e6289c37b9cf256bce5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:57 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:13.926364 756 tablet_service.cc:1552] Processing DeleteTablet for tablet 70b008011d5547bbaf3f45685394139f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:03 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:13.934712 2416 raft_consensus.cc:493] T eca016aed05e4ce09e1a2bf96e2286f5 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251029 02:52:13.967193 849 leader_election.cc:343] T 7bb85a12f573494cbb3118d1947eda27 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:13.967255 2416 raft_consensus.cc:515] T eca016aed05e4ce09e1a2bf96e2286f5 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:13.967866 2416 leader_election.cc:290] T eca016aed05e4ce09e1a2bf96e2286f5 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:52:13.967986 2416 raft_consensus.cc:493] T 70b008011d5547bbaf3f45685394139f P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:13.968062 762 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7bb85a12f573494cbb3118d1947eda27" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:13.968405 762 raft_consensus.cc:2468] T 7bb85a12f573494cbb3118d1947eda27 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:13.968077 2416 raft_consensus.cc:515] T 70b008011d5547bbaf3f45685394139f P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:13.969264 2416 leader_election.cc:290] T 70b008011d5547bbaf3f45685394139f P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:52:13.969868 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "70b008011d5547bbaf3f45685394139f" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:13.970407 711 leader_election.cc:343] T 70b008011d5547bbaf3f45685394139f P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:13.971041 847 leader_election.cc:304] T 7bb85a12f573494cbb3118d1947eda27 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5
I20251029 02:52:13.971177 1174 ts_tablet_manager.cc:1929] T ba699862e6f646c6beaa39f080d0faba P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:13.971275 1174 log.cc:1199] T ba699862e6f646c6beaa39f080d0faba P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/ba699862e6f646c6beaa39f080d0faba
I20251029 02:52:13.971457 2425 raft_consensus.cc:493] T 372d93914abc41b38a702742b7db0bc2 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:13.971576 1174 ts_tablet_manager.cc:1950] T ba699862e6f646c6beaa39f080d0faba P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:13.971678 2425 raft_consensus.cc:515] T 372d93914abc41b38a702742b7db0bc2 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:13.972275 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 883) took 4204 ms. Trace:
I20251029 02:52:13.972359 1174 rpcz_store.cc:276] 1029 02:52:09.768056 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:09.768501 (+ 445us) service_pool.cc:225] Handling call
1029 02:52:13.877401 (+4108900us) ts_tablet_manager.cc:1151] Deleting tablet ba699862e6f646c6beaa39f080d0faba
1029 02:52:13.877412 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:13.958033 (+ 80621us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:13.971068 (+ 13035us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:13.972256 (+ 1188us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4108780}
I20251029 02:52:13.972693 1174 tablet_replica.cc:333] T 5db519edc75d4bd0985b32322825ffae P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:13.972899 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet ba699862e6f646c6beaa39f080d0faba (table default.Table_84e9fe6549cc42ed94c0f4c0b489f726 [id=feb1f56ae44249c6adce65637017bb43]) successfully deleted
I20251029 02:52:13.972960 1054 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "372d93914abc41b38a702742b7db0bc2" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:52:13.973570 849 leader_election.cc:343] T 372d93914abc41b38a702742b7db0bc2 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:13.973762 2425 leader_election.cc:290] T 372d93914abc41b38a702742b7db0bc2 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:13.974045 2425 raft_consensus.cc:2804] T 7bb85a12f573494cbb3118d1947eda27 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:13.974761 2425 raft_consensus.cc:493] T 7bb85a12f573494cbb3118d1947eda27 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:13.975033 2425 raft_consensus.cc:3060] T 7bb85a12f573494cbb3118d1947eda27 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:13.975281 1174 raft_consensus.cc:2243] T 5db519edc75d4bd0985b32322825ffae P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus shutting down.
W20251029 02:52:13.974941 847 leader_election.cc:343] T 372d93914abc41b38a702742b7db0bc2 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 372d93914abc41b38a702742b7db0bc2
I20251029 02:52:13.975427 847 leader_election.cc:304] T 372d93914abc41b38a702742b7db0bc2 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9
I20251029 02:52:13.975648 2426 raft_consensus.cc:2749] T 372d93914abc41b38a702742b7db0bc2 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:13.977067 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_44608f8fced543919b531792404e12a6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:13.977139 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_0cd699a8b604487f89d72e6d948f2bc2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:13.977649 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_44608f8fced543919b531792404e12a6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:52:13.977705 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0cd699a8b604487f89d72e6d948f2bc2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:13.979071 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_94100a9aca744e9192f154cb3e467ee1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:13.979526 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_94100a9aca744e9192f154cb3e467ee1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:52:13.981806 990 consensus_peers.cc:597] T 1275e192871440408a8345fef9b56260 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 -> Peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Couldn't send request to peer c756528642a543eeacd93cef3e335a7d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1275e192871440408a8345fef9b56260. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:13.983599 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 97b2ecae572a4b7ea1b9595074556708 (table default.Table_3b4bdfaf4ddf43cfa2d2287594c57816 [id=222ef1b283554565b4930328c2c8975b]) successfully deleted
I20251029 02:52:13.984033 2302 leader_election.cc:290] T 7bb85a12f573494cbb3118d1947eda27 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:13.985208 2425 raft_consensus.cc:515] T 7bb85a12f573494cbb3118d1947eda27 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:13.985642 2416 raft_consensus.cc:493] T d31e548feae24b739087c1e211d514f9 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:13.985796 2416 raft_consensus.cc:515] T d31e548feae24b739087c1e211d514f9 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:13.986387 2416 leader_election.cc:290] T d31e548feae24b739087c1e211d514f9 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:52:13.987166 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d31e548feae24b739087c1e211d514f9" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:13.987668 711 leader_election.cc:343] T d31e548feae24b739087c1e211d514f9 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:13.988597 2425 leader_election.cc:290] T 7bb85a12f573494cbb3118d1947eda27 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:13.990131 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90c2ed692a954a25bdf79b9884f6c5ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:13.992321 1054 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7bb85a12f573494cbb3118d1947eda27" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
W20251029 02:52:13.992883 849 leader_election.cc:343] T 7bb85a12f573494cbb3118d1947eda27 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:13.996522 762 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7bb85a12f573494cbb3118d1947eda27" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:13.996754 762 raft_consensus.cc:3060] T 7bb85a12f573494cbb3118d1947eda27 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:13.998128 526 catalog_manager.cc:6194] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: default.Table_d0d8770fdb3d4d34933b7b4264831ad9 [id=657afc87656d4250b4ca87b473fbb752] alter complete (version 4)
I20251029 02:52:14.001308 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "eca016aed05e4ce09e1a2bf96e2286f5" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:14.001363 896 tablet_service.cc:1552] Processing DeleteTablet for tablet 72f9d4408f384e7093316d85f092e1b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:05 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:14.001446 881 tablet_service.cc:1552] Processing DeleteTablet for tablet 372d93914abc41b38a702742b7db0bc2 with delete_type TABLET_DATA_DELETED (replaced by tablet dab6ed2266ed42508738399d388c4da3) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:14.001451 895 tablet_service.cc:1552] Processing DeleteTablet for tablet 6142018e44af4d32818adefa2ee43d02 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:01 UTC) from {username='slave'} at 127.0.0.1:35702
W20251029 02:52:14.002068 711 leader_election.cc:343] T eca016aed05e4ce09e1a2bf96e2286f5 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:14.002746 762 raft_consensus.cc:2468] T 7bb85a12f573494cbb3118d1947eda27 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:14.004011 524 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_41dc74911c3747a7a2763f31d7ca40e1" } modify_external_catalogs: true
I20251029 02:52:14.004295 524 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_41dc74911c3747a7a2763f31d7ca40e1" } modify_external_catalogs: true
I20251029 02:52:14.005180 847 leader_election.cc:304] T 7bb85a12f573494cbb3118d1947eda27 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5
I20251029 02:52:14.005941 2425 raft_consensus.cc:2804] T 7bb85a12f573494cbb3118d1947eda27 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:14.006156 2425 raft_consensus.cc:697] T 7bb85a12f573494cbb3118d1947eda27 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:52:14.008172 1167 tablet_bootstrap.cc:492] T cd597c319af645909b8f3d1212fcb702 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:14.008301 1167 ts_tablet_manager.cc:1403] T cd597c319af645909b8f3d1212fcb702 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.168s user 0.001s sys 0.001s
I20251029 02:52:14.012137 2453 tablet.cc:1722] T d7abaf84ba5d445aa17433b3be891548 P c756528642a543eeacd93cef3e335a7d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251029 02:52:14.012966 1167 raft_consensus.cc:359] T cd597c319af645909b8f3d1212fcb702 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:14.013142 1167 raft_consensus.cc:385] T cd597c319af645909b8f3d1212fcb702 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:14.013197 1167 raft_consensus.cc:740] T cd597c319af645909b8f3d1212fcb702 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:14.013361 1167 consensus_queue.cc:260] T cd597c319af645909b8f3d1212fcb702 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:14.013774 1167 ts_tablet_manager.cc:1434] T cd597c319af645909b8f3d1212fcb702 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251029 02:52:14.014060 1167 tablet_bootstrap.cc:492] T 6dff036b32904ddb9598be4b79504bb6 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:14.015935 1167 tablet_bootstrap.cc:654] T 6dff036b32904ddb9598be4b79504bb6 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:14.020409 525 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_d90269480e524bccbef447534992b673" } modify_external_catalogs: true
I20251029 02:52:14.020726 525 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_d90269480e524bccbef447534992b673" } modify_external_catalogs: true
W20251029 02:52:14.023196 847 consensus_peers.cc:597] T 44ae2b07edca44afbe1b1fb3a5b1115d P c756528642a543eeacd93cef3e335a7d -> Peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Couldn't send request to peer b495a49349324f7d81908f2da700f0d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 44ae2b07edca44afbe1b1fb3a5b1115d. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:14.023233 2469 tablet.cc:1722] T d7abaf84ba5d445aa17433b3be891548 P b495a49349324f7d81908f2da700f0d9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251029 02:52:14.007067 2425 consensus_queue.cc:237] T 7bb85a12f573494cbb3118d1947eda27 P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:13.975416 1174 raft_consensus.cc:2272] T 5db519edc75d4bd0985b32322825ffae P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:14.027597 1174 ts_tablet_manager.cc:1916] T 5db519edc75d4bd0985b32322825ffae P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:14.031433 525 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet baf0e49f57514e1cb977c860d244eeb1
I20251029 02:52:14.034745 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet 58dde9397b434b089ade9968f369006a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:59 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:14.038623 526 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet c1222e30bc4141b3ae686d80053c4ef5 (table default.Table_7125184a813c44d997ab9d948a1156bd [id=31801defa6db4ca7a08fbf7f029b65c1]). Expected version 3 got 0
I20251029 02:52:14.039390 521 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_07d2f4a3c8074ece8b4982e3819caabc" } modify_external_catalogs: true
I20251029 02:52:14.040330 521 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_07d2f4a3c8074ece8b4982e3819caabc" } modify_external_catalogs: true
I20251029 02:52:14.042307 523 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_f6c47489fbba4d65bbab23c587dddb63" } new_table_name: "default.Table_39155702001247058bc1f99a94b3b672" modify_external_catalogs: true
I20251029 02:52:14.043872 524 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 96427e60edeb45bdab9666a5055b5a99
W20251029 02:52:14.044255 513 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 6dc30c614be64b8ba629d9884d450290 (table default.Table_95133cb1a69941d0b8a6321091752822 [id=69f28220165541a388a16c079fc88733], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 6dc30c614be64b8ba629d9884d450290 consensus state has no leader
I20251029 02:52:13.996521 1175 ts_tablet_manager.cc:1916] T e65eaaae43ac435ebd665546de9d68dd P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:14.046125 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_81de1a2ccffd4a439e22e14cbb0c7cc3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:14.046598 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_81de1a2ccffd4a439e22e14cbb0c7cc3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:14.047520 895 tablet_service.cc:1552] Processing DeleteTablet for tablet 96427e60edeb45bdab9666a5055b5a99 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:13 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:14.050890 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 96427e60edeb45bdab9666a5055b5a99 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:13 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:14.052203 895 tablet_service.cc:1505] Processing CreateTablet for tablet 90c2ed692a954a25bdf79b9884f6c5ce (DEFAULT_TABLE table=default.Table_d5b1b53f211a44bf944f53dfde769c5d [id=23c4701379f748a6be7c96fe8ecc9e20]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:14.052608 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 90c2ed692a954a25bdf79b9884f6c5ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:14.056182 881 tablet_service.cc:1505] Processing CreateTablet for tablet 271447d01969452c99aad86a2e39744b (DEFAULT_TABLE table=default.Table_39b69ff4c04748ffaf18565c7f306693 [id=ba2a225baf8b437583df49e8d6c496c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:14.057104 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 271447d01969452c99aad86a2e39744b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:14.058284 525 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_b8fe14e0ff054e67b0594e8b97483c71" } new_table_name: "default.Table_24deffef51354bcda12021bdf6bab67b" modify_external_catalogs: true
I20251029 02:52:14.058768 745 tablet_service.cc:1505] Processing CreateTablet for tablet 271447d01969452c99aad86a2e39744b (DEFAULT_TABLE table=default.Table_39b69ff4c04748ffaf18565c7f306693 [id=ba2a225baf8b437583df49e8d6c496c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:14.059145 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 271447d01969452c99aad86a2e39744b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:14.059442 526 catalog_manager.cc:6194] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: default.Table_c1c6933af1e046aa895e0ee915178f5c [id=9e33c530a35544a4ae22e2a1238592db] alter complete (version 3)
I20251029 02:52:14.068722 1172 tablet_bootstrap.cc:492] T 748777c6cd6e4f49ab49912b719e48bd P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:14.068928 1172 ts_tablet_manager.cc:1403] T 748777c6cd6e4f49ab49912b719e48bd P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.208s user 0.008s sys 0.022s
I20251029 02:52:14.069826 1172 raft_consensus.cc:359] T 748777c6cd6e4f49ab49912b719e48bd P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:14.070060 1172 raft_consensus.cc:385] T 748777c6cd6e4f49ab49912b719e48bd P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:14.070120 1172 raft_consensus.cc:740] T 748777c6cd6e4f49ab49912b719e48bd P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:14.070339 1172 consensus_queue.cc:260] T 748777c6cd6e4f49ab49912b719e48bd P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:14.073415 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet 96427e60edeb45bdab9666a5055b5a99 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:13 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:14.074252 1167 tablet_bootstrap.cc:492] T 6dff036b32904ddb9598be4b79504bb6 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:14.074406 1167 ts_tablet_manager.cc:1403] T 6dff036b32904ddb9598be4b79504bb6 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.060s user 0.003s sys 0.000s
I20251029 02:52:14.076467 745 tablet_service.cc:1505] Processing CreateTablet for tablet f84a6e6b464d432d9dfff61e1e07b281 (DEFAULT_TABLE table=default.Table_290bd51633d34507910b6de728d4d129 [id=77127b4a1d3246eeb08f73304677740d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:14.078682 881 tablet_service.cc:1505] Processing CreateTablet for tablet f84a6e6b464d432d9dfff61e1e07b281 (DEFAULT_TABLE table=default.Table_290bd51633d34507910b6de728d4d129 [id=77127b4a1d3246eeb08f73304677740d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:14.079146 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f84a6e6b464d432d9dfff61e1e07b281. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:14.079741 1176 ts_tablet_manager.cc:1929] T 28f7ceb91a34450cad4b5b45f5285b41 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:14.079850 1176 log.cc:1199] T 28f7ceb91a34450cad4b5b45f5285b41 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/28f7ceb91a34450cad4b5b45f5285b41
I20251029 02:52:14.080193 1176 ts_tablet_manager.cc:1950] T 28f7ceb91a34450cad4b5b45f5285b41 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:14.083510 492 master_runner.cc:186] Error getting master registration for 127.0.104.62:42343: Network error: Client connection negotiation failed: client connection to 127.0.104.62:42343: connect: Connection refused (error 111),
I20251029 02:52:14.083657 492 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251029 02:52:14.083776 1167 raft_consensus.cc:359] T 6dff036b32904ddb9598be4b79504bb6 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:14.083951 1167 raft_consensus.cc:385] T 6dff036b32904ddb9598be4b79504bb6 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:14.084009 1167 raft_consensus.cc:740] T 6dff036b32904ddb9598be4b79504bb6 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:14.084506 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_69cd5bad2bb64a15908025872e382616"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:14.085062 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_69cd5bad2bb64a15908025872e382616 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:14.085248 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_3eb06cb0b47545f2854d25ae8849681f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:14.085690 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3eb06cb0b47545f2854d25ae8849681f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:52:14.086544 523 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 195ddd7cae484f488d9d6277db9fdd15 consensus state has no leader
I20251029 02:52:14.086879 521 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 7751599f5dfc43baa20dc3487c6eddd9
I20251029 02:52:14.087019 2205 consensus_queue.cc:1048] T d041e87ceb11441983d0e38b7929c3a4 P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:14.088938 519 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet 748777c6cd6e4f49ab49912b719e48bd (table default.Table_1e69e259edba466f99ed7a41dcf6367f [id=a2f3b36dc78a4e94bd054a6a5712b384]). Expected version 1 got 0
I20251029 02:52:14.085059 522 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_4873cb28e4c84935ab481f06ad654716" } new_table_name: "default.Table_4b3a575375ab4bb19ef4d9f7d9ab74cb" modify_external_catalogs: true
W20251029 02:52:14.091094 2610 log.cc:927] Time spent T aacb6f4b9284457bb6607e3a85d5c809 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Append to log took a long time: real 0.126s user 0.000s sys 0.000s
I20251029 02:52:14.091722 1029 tablet_service.cc:1505] Processing CreateTablet for tablet 271447d01969452c99aad86a2e39744b (DEFAULT_TABLE table=default.Table_39b69ff4c04748ffaf18565c7f306693 [id=ba2a225baf8b437583df49e8d6c496c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:14.093178 896 tablet_service.cc:1505] Processing CreateTablet for tablet 2c2a3a7e103f456f94fe4c42bbc88c4a (DEFAULT_TABLE table=default.Table_6aacefe556ee42399e96836738dab1f2 [id=608835d4103148828838f08fd0143227]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:14.093263 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_fe999fb018f84d38b900fd1852effeaf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:14.093598 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c2a3a7e103f456f94fe4c42bbc88c4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:14.094954 900 tablet_service.cc:1505] Processing CreateTablet for tablet b36325568c89477d9eead4dc1e45cbce (DEFAULT_TABLE table=default.Table_957f906aceb6449fb91b40e913a605c0 [id=c79c1b0d4825456babd2ca2cd8272cd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:14.095279 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b36325568c89477d9eead4dc1e45cbce. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:14.095537 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fe999fb018f84d38b900fd1852effeaf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:14.070891 1172 ts_tablet_manager.cc:1434] T 748777c6cd6e4f49ab49912b719e48bd P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 02:52:14.097942 1172 tablet_bootstrap.cc:492] T cd597c319af645909b8f3d1212fcb702 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:14.098821 756 tablet_service.cc:1505] Processing CreateTablet for tablet 2c2a3a7e103f456f94fe4c42bbc88c4a (DEFAULT_TABLE table=default.Table_6aacefe556ee42399e96836738dab1f2 [id=608835d4103148828838f08fd0143227]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:14.099195 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c2a3a7e103f456f94fe4c42bbc88c4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:14.099776 894 tablet_service.cc:1505] Processing CreateTablet for tablet b54dc289b6ab4872a4c272098e1adb17 (DEFAULT_TABLE table=default.Table_09fa848b582f433b9ed2b3f9ec4cf3fe [id=9799f0adb93747929af88a16b0be8bc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:14.100119 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b54dc289b6ab4872a4c272098e1adb17. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:14.100430 510 consensus_peers.cc:597] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0 -> Peer e9672114a6a742e2864c16ff60ba246e (127.0.104.62:42343): Couldn't send request to peer e9672114a6a742e2864c16ff60ba246e. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.62:42343: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20251029 02:52:14.101306 1172 tablet_bootstrap.cc:654] T cd597c319af645909b8f3d1212fcb702 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:14.101627 1167 consensus_queue.cc:260] T 6dff036b32904ddb9598be4b79504bb6 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:14.103874 747 tablet_service.cc:1505] Processing CreateTablet for tablet b54dc289b6ab4872a4c272098e1adb17 (DEFAULT_TABLE table=default.Table_09fa848b582f433b9ed2b3f9ec4cf3fe [id=9799f0adb93747929af88a16b0be8bc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:14.104063 739 tablet_service.cc:1505] Processing CreateTablet for tablet 7e340c0adca84ee0bddeb4ba68ad05df (DEFAULT_TABLE table=default.Table_56d870facffb425daf13fa7cb9937b12 [id=59823dfcf94949ce82a0610f75538769]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:14.104272 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b54dc289b6ab4872a4c272098e1adb17. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:14.103879 743 tablet_service.cc:1505] Processing CreateTablet for tablet b36325568c89477d9eead4dc1e45cbce (DEFAULT_TABLE table=default.Table_957f906aceb6449fb91b40e913a605c0 [id=c79c1b0d4825456babd2ca2cd8272cd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:14.104633 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b36325568c89477d9eead4dc1e45cbce. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:14.110770 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e340c0adca84ee0bddeb4ba68ad05df. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:14.111979 1167 ts_tablet_manager.cc:1434] T 6dff036b32904ddb9598be4b79504bb6 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.037s user 0.003s sys 0.000s
I20251029 02:52:14.112320 1167 tablet_bootstrap.cc:492] T 20a762344c5f46f88481c4ac7ad6e43b P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:14.113225 1172 tablet_bootstrap.cc:492] T cd597c319af645909b8f3d1212fcb702 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:14.113372 1172 ts_tablet_manager.cc:1403] T cd597c319af645909b8f3d1212fcb702 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.001s
I20251029 02:52:14.114053 1167 tablet_bootstrap.cc:654] T 20a762344c5f46f88481c4ac7ad6e43b P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:14.114117 1172 raft_consensus.cc:359] T cd597c319af645909b8f3d1212fcb702 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:14.114286 1172 raft_consensus.cc:385] T cd597c319af645909b8f3d1212fcb702 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:14.114351 1172 raft_consensus.cc:740] T cd597c319af645909b8f3d1212fcb702 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:14.114588 1172 consensus_queue.cc:260] T cd597c319af645909b8f3d1212fcb702 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:14.115496 2453 tablet.cc:1722] T aacb6f4b9284457bb6607e3a85d5c809 P c756528642a543eeacd93cef3e335a7d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20251029 02:52:14.117082 517 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_054598fd8c4b4a87b9b5944a738dec88" } modify_external_catalogs: true
I20251029 02:52:14.117290 517 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_054598fd8c4b4a87b9b5944a738dec88" } modify_external_catalogs: true
I20251029 02:52:14.117611 1172 ts_tablet_manager.cc:1434] T cd597c319af645909b8f3d1212fcb702 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251029 02:52:14.118706 523 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_4780e4a5339245d1acdf62a82234a1d2" } modify_external_catalogs: true
I20251029 02:52:14.118875 523 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_4780e4a5339245d1acdf62a82234a1d2" } modify_external_catalogs: true
I20251029 02:52:14.119663 1172 tablet_bootstrap.cc:492] T 84993b2f63e7421290a161dde4bb0607 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:14.120261 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_5461f53d600440a8bf5bb171829677e1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:14.120795 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5461f53d600440a8bf5bb171829677e1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:14.127228 895 tablet_service.cc:1552] Processing DeleteTablet for tablet 7751599f5dfc43baa20dc3487c6eddd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:14 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:14.127418 900 tablet_service.cc:1505] Processing CreateTablet for tablet 7e340c0adca84ee0bddeb4ba68ad05df (DEFAULT_TABLE table=default.Table_56d870facffb425daf13fa7cb9937b12 [id=59823dfcf94949ce82a0610f75538769]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:14.127837 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e340c0adca84ee0bddeb4ba68ad05df. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:14.128428 526 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet 6dff036b32904ddb9598be4b79504bb6 (table default.Table_0ae102f09fdc41bcbead0148653df9e2 [id=65ec61bf400a4628b98e64e7d43c025c]). Expected version 2 got 0
I20251029 02:52:14.130474 1172 tablet_bootstrap.cc:654] T 84993b2f63e7421290a161dde4bb0607 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
W20251029 02:52:14.130769 525 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d8c66a892c90475c834f08c0734f41db consensus state has no leader
I20251029 02:52:14.136132 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f84a6e6b464d432d9dfff61e1e07b281. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:14.136634 521 master_service.cc:954] ReplaceTablet: received request to replace tablet 738046ab57334f2dab9f4aab979c5b42 from {username='slave'} at 127.0.0.1:35314
I20251029 02:52:14.136945 744 tablet_service.cc:1552] Processing DeleteTablet for tablet 7751599f5dfc43baa20dc3487c6eddd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:14 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:14.144601 525 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_9120016827e946cc8a145252d2569d2b" } new_table_name: "default.Table_9e67f2d45e8f44c6bff1b72166c092a2" modify_external_catalogs: true
I20251029 02:52:14.091928 2452 tablet.cc:1722] T aacb6f4b9284457bb6607e3a85d5c809 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20251029 02:52:14.149737 2469 tablet.cc:1722] T a9e0fcc759a64679b3f59a6c964877b6 P b495a49349324f7d81908f2da700f0d9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20251029 02:52:14.149875 526 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_d64a9ac717a94c40a98f826136089824" } new_table_name: "default.Table_a78f5837df7b44e791631e421aa7de46" modify_external_catalogs: true
I20251029 02:52:14.152947 520 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet 6dff036b32904ddb9598be4b79504bb6 (table default.Table_0ae102f09fdc41bcbead0148653df9e2 [id=65ec61bf400a4628b98e64e7d43c025c]). Expected version 2 got 0
I20251029 02:52:14.154850 1024 tablet_service.cc:1505] Processing CreateTablet for tablet f84a6e6b464d432d9dfff61e1e07b281 (DEFAULT_TABLE table=default.Table_290bd51633d34507910b6de728d4d129 [id=77127b4a1d3246eeb08f73304677740d]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 02:52:14.160056 522 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d557a4750fee4c2b92eaa834619a678c consensus state has no leader
I20251029 02:52:14.160056 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1025) took 2615 ms. Trace:
I20251029 02:52:14.160151 1176 rpcz_store.cc:276] 1029 02:52:11.544582 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:11.574657 (+ 30075us) service_pool.cc:225] Handling call
1029 02:52:13.942289 (+2367632us) ts_tablet_manager.cc:1151] Deleting tablet 28f7ceb91a34450cad4b5b45f5285b41
1029 02:52:13.942299 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:14.020450 (+ 78151us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:14.079638 (+ 59188us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:14.160034 (+ 80396us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2367457}
I20251029 02:52:14.160487 1176 tablet_replica.cc:333] T 6d830d30b26b4ba799785fec8a6f0769 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:14.160682 1176 raft_consensus.cc:2243] T 6d830d30b26b4ba799785fec8a6f0769 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:14.160820 1176 raft_consensus.cc:2272] T 6d830d30b26b4ba799785fec8a6f0769 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:14.161355 1176 ts_tablet_manager.cc:1916] T 6d830d30b26b4ba799785fec8a6f0769 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:14.162761 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 271447d01969452c99aad86a2e39744b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:14.164839 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f84a6e6b464d432d9dfff61e1e07b281. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:14.176766 517 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 595b4073cc0446cea041838097af7471
I20251029 02:52:14.177870 743 tablet_service.cc:1552] Processing DeleteTablet for tablet 595b4073cc0446cea041838097af7471 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:14 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:14.181697 523 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet e2c8983cded84a1c8b4205c48a4eac33
I20251029 02:52:14.182744 743 tablet_service.cc:1552] Processing DeleteTablet for tablet e2c8983cded84a1c8b4205c48a4eac33 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:14 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:14.183653 896 tablet_service.cc:1552] Processing DeleteTablet for tablet e2c8983cded84a1c8b4205c48a4eac33 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:14 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:14.186187 1040 tablet_service.cc:1552] Processing DeleteTablet for tablet 7751599f5dfc43baa20dc3487c6eddd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:14 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:14.186899 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_93495adef2ff4b039f9aebdb6c748b49"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:14.187511 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_93495adef2ff4b039f9aebdb6c748b49 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:14.191795 881 tablet_service.cc:1552] Processing DeleteTablet for tablet 595b4073cc0446cea041838097af7471 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:14 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:14.193658 2302 consensus_queue.cc:1048] T d041e87ceb11441983d0e38b7929c3a4 P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:14.193914 524 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet 748777c6cd6e4f49ab49912b719e48bd (table default.Table_1e69e259edba466f99ed7a41dcf6367f [id=a2f3b36dc78a4e94bd054a6a5712b384]). Expected version 1 got 0
I20251029 02:52:14.195464 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_a0cab891f6e34cd4bce7592b55b68979"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:14.195722 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_16b74e4bf0c240649a053af2967ae6bb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:14.196190 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a0cab891f6e34cd4bce7592b55b68979 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:52:14.196275 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_16b74e4bf0c240649a053af2967ae6bb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:14.198125 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_f352e3374c0142cb90edcd42caf06ef4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:14.198647 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f352e3374c0142cb90edcd42caf06ef4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:14.200207 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_97f3a6c87c274315b8c9651e5e1c1714"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:14.200680 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_97f3a6c87c274315b8c9651e5e1c1714 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:14.194689 2302 raft_consensus.cc:493] T 58dde9397b434b089ade9968f369006a P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:14.201877 2302 raft_consensus.cc:515] T 58dde9397b434b089ade9968f369006a P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:14.203454 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "58dde9397b434b089ade9968f369006a" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:14.203698 773 raft_consensus.cc:2468] T 58dde9397b434b089ade9968f369006a P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:14.204866 847 leader_election.cc:304] T 58dde9397b434b089ade9968f369006a P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:14.205250 2205 raft_consensus.cc:2804] T 58dde9397b434b089ade9968f369006a P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:14.205350 2205 raft_consensus.cc:493] T 58dde9397b434b089ade9968f369006a P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:14.205430 2205 raft_consensus.cc:3060] T 58dde9397b434b089ade9968f369006a P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:14.205619 520 catalog_manager.cc:6194] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: default.Table_76410db5d9f74f4d9c4e6adb8f3a7f40 [id=9f95160639ad40e995a6c89e297a0fad] alter complete (version 4)
I20251029 02:52:14.206136 1040 tablet_service.cc:1505] Processing CreateTablet for tablet b54dc289b6ab4872a4c272098e1adb17 (DEFAULT_TABLE table=default.Table_09fa848b582f433b9ed2b3f9ec4cf3fe [id=9799f0adb93747929af88a16b0be8bc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:14.206647 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b54dc289b6ab4872a4c272098e1adb17. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:14.208947 2205 raft_consensus.cc:515] T 58dde9397b434b089ade9968f369006a P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:14.209009 1036 tablet_service.cc:1505] Processing CreateTablet for tablet 2c2a3a7e103f456f94fe4c42bbc88c4a (DEFAULT_TABLE table=default.Table_6aacefe556ee42399e96836738dab1f2 [id=608835d4103148828838f08fd0143227]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:14.210482 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "58dde9397b434b089ade9968f369006a" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:14.210669 773 raft_consensus.cc:3060] T 58dde9397b434b089ade9968f369006a P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:14.212550 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c2a3a7e103f456f94fe4c42bbc88c4a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:14.213327 773 raft_consensus.cc:2468] T 58dde9397b434b089ade9968f369006a P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:14.213538 743 tablet_service.cc:1505] Processing CreateTablet for tablet 42c4e397583842d7b1a22640e63d4106 (DEFAULT_TABLE table=default.Table_5e94f24436bb4574a310bd6c375107a1 [id=bc6f099eedfa4ca59263c5ace494ede4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:14.213840 847 leader_election.cc:304] T 58dde9397b434b089ade9968f369006a P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:14.213920 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42c4e397583842d7b1a22640e63d4106. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:14.214112 2426 raft_consensus.cc:2804] T 58dde9397b434b089ade9968f369006a P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:14.214217 2426 raft_consensus.cc:697] T 58dde9397b434b089ade9968f369006a P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:52:14.214428 2426 consensus_queue.cc:237] T 58dde9397b434b089ade9968f369006a P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:14.217226 2205 leader_election.cc:290] T 58dde9397b434b089ade9968f369006a P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:14.221276 2302 leader_election.cc:290] T 58dde9397b434b089ade9968f369006a P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
W20251029 02:52:14.222142 526 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 32519fbffcef44a890161e5f3043806c consensus state has no leader
I20251029 02:52:14.224853 1166 tablet_bootstrap.cc:492] T 40e844fd7ad74680bcbe7bef936c5b9d P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:14.224995 1166 ts_tablet_manager.cc:1403] T 40e844fd7ad74680bcbe7bef936c5b9d P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.387s user 0.004s sys 0.009s
I20251029 02:52:14.225417 1031 tablet_service.cc:1505] Processing CreateTablet for tablet 7e340c0adca84ee0bddeb4ba68ad05df (DEFAULT_TABLE table=default.Table_56d870facffb425daf13fa7cb9937b12 [id=59823dfcf94949ce82a0610f75538769]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:14.225777 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e340c0adca84ee0bddeb4ba68ad05df. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:14.225735 1038 tablet_service.cc:1505] Processing CreateTablet for tablet 42c4e397583842d7b1a22640e63d4106 (DEFAULT_TABLE table=default.Table_5e94f24436bb4574a310bd6c375107a1 [id=bc6f099eedfa4ca59263c5ace494ede4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:14.226106 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42c4e397583842d7b1a22640e63d4106. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:14.226184 1042 tablet_service.cc:1552] Processing DeleteTablet for tablet 595b4073cc0446cea041838097af7471 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:14 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:14.226378 1030 tablet_service.cc:1552] Processing DeleteTablet for tablet e2c8983cded84a1c8b4205c48a4eac33 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:14 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:14.236048 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 28f7ceb91a34450cad4b5b45f5285b41 (table default.Table_5580c367c15f4f3a87626302cfde455f [id=89406c2781fd4674b634e3b1dbd06dc7]) successfully deleted
W20251029 02:52:14.237623 990 consensus_peers.cc:597] T 9bb2166a590d4210b02dd779ef6654ad P 0cfd4bee16f84b7c8b0ff3ce14db20d5 -> Peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Couldn't send request to peer c756528642a543eeacd93cef3e335a7d. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9bb2166a590d4210b02dd779ef6654ad. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:14.244109 2430 raft_consensus.cc:493] T 58dde9397b434b089ade9968f369006a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:14.244257 2430 raft_consensus.cc:515] T 58dde9397b434b089ade9968f369006a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:14.244148 743 tablet_service.cc:1505] Processing CreateTablet for tablet fb23c471d1d348a1b94b2ffb69b9a115 (DEFAULT_TABLE table=default.Table_42036229ecf348adb05a3fdaaab07e1f [id=497f4e20bfc946af90522b4acc0d182b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:14.244853 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb23c471d1d348a1b94b2ffb69b9a115. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:14.245913 526 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_a2eda0beb1fb4fd3aa2a64eded1e17ad" } modify_external_catalogs: true
I20251029 02:52:14.246004 2611 raft_consensus.cc:493] T 23b9b2ea814645eba63e5f6b12cfb966 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:14.246124 526 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_a2eda0beb1fb4fd3aa2a64eded1e17ad" } modify_external_catalogs: true
I20251029 02:52:14.246122 2611 raft_consensus.cc:515] T 23b9b2ea814645eba63e5f6b12cfb966 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:14.246593 1029 tablet_service.cc:1505] Processing CreateTablet for tablet b36325568c89477d9eead4dc1e45cbce (DEFAULT_TABLE table=default.Table_957f906aceb6449fb91b40e913a605c0 [id=c79c1b0d4825456babd2ca2cd8272cd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:14.246795 2611 leader_election.cc:290] T 23b9b2ea814645eba63e5f6b12cfb966 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:52:14.247484 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "23b9b2ea814645eba63e5f6b12cfb966" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:14.247952 711 leader_election.cc:343] T 23b9b2ea814645eba63e5f6b12cfb966 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:14.249342 881 tablet_service.cc:1505] Processing CreateTablet for tablet 42c4e397583842d7b1a22640e63d4106 (DEFAULT_TABLE table=default.Table_5e94f24436bb4574a310bd6c375107a1 [id=bc6f099eedfa4ca59263c5ace494ede4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:14.250341 1176 ts_tablet_manager.cc:1929] T 6d830d30b26b4ba799785fec8a6f0769 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:14.250464 1176 log.cc:1199] T 6d830d30b26b4ba799785fec8a6f0769 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/6d830d30b26b4ba799785fec8a6f0769
I20251029 02:52:14.251729 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42c4e397583842d7b1a22640e63d4106. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:14.253335 1176 ts_tablet_manager.cc:1950] T 6d830d30b26b4ba799785fec8a6f0769 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:14.257597 1175 ts_tablet_manager.cc:1929] T e65eaaae43ac435ebd665546de9d68dd P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:14.257704 1175 log.cc:1199] T e65eaaae43ac435ebd665546de9d68dd P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/e65eaaae43ac435ebd665546de9d68dd
I20251029 02:52:14.258033 1175 ts_tablet_manager.cc:1950] T e65eaaae43ac435ebd665546de9d68dd P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:14.258085 1167 tablet_bootstrap.cc:492] T 20a762344c5f46f88481c4ac7ad6e43b P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:14.258234 1167 ts_tablet_manager.cc:1403] T 20a762344c5f46f88481c4ac7ad6e43b P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.146s user 0.003s sys 0.000s
I20251029 02:52:14.259563 1968 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251029 02:52:14.260622 1968 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251029 02:52:14.260941 1968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251029 02:52:14.264838 522 master_service.cc:954] ReplaceTablet: received request to replace tablet c0936e54f1d246388a2d82dbc52aac42 from {username='slave'} at 127.0.0.1:35314
I20251029 02:52:14.265566 1172 tablet_bootstrap.cc:492] T 84993b2f63e7421290a161dde4bb0607 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:14.265712 1172 ts_tablet_manager.cc:1403] T 84993b2f63e7421290a161dde4bb0607 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.148s user 0.008s sys 0.001s
I20251029 02:52:14.266696 1172 raft_consensus.cc:359] T 84993b2f63e7421290a161dde4bb0607 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:14.266919 1172 raft_consensus.cc:385] T 84993b2f63e7421290a161dde4bb0607 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:14.266970 1172 raft_consensus.cc:740] T 84993b2f63e7421290a161dde4bb0607 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:14.267799 1167 raft_consensus.cc:359] T 20a762344c5f46f88481c4ac7ad6e43b P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:14.268020 1167 raft_consensus.cc:385] T 20a762344c5f46f88481c4ac7ad6e43b P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:14.268069 1167 raft_consensus.cc:740] T 20a762344c5f46f88481c4ac7ad6e43b P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:14.272599 896 tablet_service.cc:1505] Processing CreateTablet for tablet fb23c471d1d348a1b94b2ffb69b9a115 (DEFAULT_TABLE table=default.Table_42036229ecf348adb05a3fdaaab07e1f [id=497f4e20bfc946af90522b4acc0d182b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:14.273084 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb23c471d1d348a1b94b2ffb69b9a115. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:14.274964 1167 consensus_queue.cc:260] T 20a762344c5f46f88481c4ac7ad6e43b P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:14.277815 744 tablet_service.cc:1505] Processing CreateTablet for tablet 21605b1dabf440cf87e8962749590a4f (DEFAULT_TABLE table=default.Table_0d5fcbba432c470cb80f5cdb893b6294 [id=2eae726c7cbc4029b6b4c1343cb4e36c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:14.278218 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21605b1dabf440cf87e8962749590a4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:14.279757 1167 ts_tablet_manager.cc:1434] T 20a762344c5f46f88481c4ac7ad6e43b P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.021s user 0.002s sys 0.000s
I20251029 02:52:14.280040 1167 tablet_bootstrap.cc:492] T 40e844fd7ad74680bcbe7bef936c5b9d P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:14.284574 2425 consensus_queue.cc:1048] T 67651306f3b44e7eb14dcee9a2007a81 P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:14.286716 756 tablet_service.cc:1505] Processing CreateTablet for tablet 48e74fe2c98f4df5ac686a4405c4521b (DEFAULT_TABLE table=default.Table_a5ee816bf96c4eb891cb8ba5ed50f3ed [id=ec71481cc6464181926e080b7ba40da9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:14.287247 895 tablet_service.cc:1505] Processing CreateTablet for tablet 48e74fe2c98f4df5ac686a4405c4521b (DEFAULT_TABLE table=default.Table_a5ee816bf96c4eb891cb8ba5ed50f3ed [id=ec71481cc6464181926e080b7ba40da9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:14.293596 900 tablet_service.cc:1505] Processing CreateTablet for tablet 21605b1dabf440cf87e8962749590a4f (DEFAULT_TABLE table=default.Table_0d5fcbba432c470cb80f5cdb893b6294 [id=2eae726c7cbc4029b6b4c1343cb4e36c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:14.294313 517 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_f6666e856c244ff0b9b28f1ffd730bf9" } modify_external_catalogs: true
I20251029 02:52:14.294560 517 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_f6666e856c244ff0b9b28f1ffd730bf9" } modify_external_catalogs: true
I20251029 02:52:14.296144 523 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_6a3e403847304172bc58735bfe02dbd0" } new_table_name: "default.Table_94e148312a6f4d779555f99520b66dca" modify_external_catalogs: true
I20251029 02:52:14.296186 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_69fcb07379dc4054b90c1c6d3e9c7985"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:14.296728 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_69fcb07379dc4054b90c1c6d3e9c7985 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:14.297057 739 tablet_service.cc:1505] Processing CreateTablet for tablet 5b5f9dfe571c44af9c22e8de843c4207 (DEFAULT_TABLE table=default.Table_2c4f1597c5d34f11a1bd7ee6550803fc [id=ace38e96a5cb4d5bb821ea7d2cfb0a3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:14.297361 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48e74fe2c98f4df5ac686a4405c4521b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:14.297585 2425 raft_consensus.cc:493] T 6d830d30b26b4ba799785fec8a6f0769 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:14.297796 2425 raft_consensus.cc:515] T 6d830d30b26b4ba799785fec8a6f0769 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:14.299790 2425 leader_election.cc:290] T 6d830d30b26b4ba799785fec8a6f0769 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:14.300055 526 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet f93b0c39d7174dabbe018b1546dbc198
I20251029 02:52:14.300251 522 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet c0936e54f1d246388a2d82dbc52aac42
I20251029 02:52:14.300822 522 catalog_manager.cc:6719] ReplaceTablet: tablet c0936e54f1d246388a2d82dbc52aac42 deleted and replaced by tablet 0573fe69797f48539df5caf4b3777b4d
I20251029 02:52:14.301534 896 tablet_service.cc:1552] Processing DeleteTablet for tablet c0936e54f1d246388a2d82dbc52aac42 with delete_type TABLET_DATA_DELETED (replaced by tablet 0573fe69797f48539df5caf4b3777b4d) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:14.301923 878 tablet_service.cc:1505] Processing CreateTablet for tablet 5b5f9dfe571c44af9c22e8de843c4207 (DEFAULT_TABLE table=default.Table_2c4f1597c5d34f11a1bd7ee6550803fc [id=ace38e96a5cb4d5bb821ea7d2cfb0a3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:14.301955 751 tablet_service.cc:1552] Processing DeleteTablet for tablet c0936e54f1d246388a2d82dbc52aac42 with delete_type TABLET_DATA_DELETED (replaced by tablet 0573fe69797f48539df5caf4b3777b4d) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:14.302249 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b5f9dfe571c44af9c22e8de843c4207. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:14.304225 520 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_ca4bf5512ff048ddb5feb708b19e3f05" } modify_external_catalogs: true
I20251029 02:52:14.304416 520 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_ca4bf5512ff048ddb5feb708b19e3f05" } modify_external_catalogs: true
I20251029 02:52:14.305645 747 tablet_service.cc:1552] Processing DeleteTablet for tablet f93b0c39d7174dabbe018b1546dbc198 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:14 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:14.305969 894 tablet_service.cc:1552] Processing DeleteTablet for tablet f93b0c39d7174dabbe018b1546dbc198 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:14 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:14.307324 1054 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "eca016aed05e4ce09e1a2bf96e2286f5" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:14.309092 1166 raft_consensus.cc:359] T 40e844fd7ad74680bcbe7bef936c5b9d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:14.309249 1166 raft_consensus.cc:385] T 40e844fd7ad74680bcbe7bef936c5b9d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:14.309314 1166 raft_consensus.cc:740] T 40e844fd7ad74680bcbe7bef936c5b9d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:14.309688 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b36325568c89477d9eead4dc1e45cbce. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:14.309473 1166 consensus_queue.cc:260] T 40e844fd7ad74680bcbe7bef936c5b9d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:14.310384 1166 ts_tablet_manager.cc:1434] T 40e844fd7ad74680bcbe7bef936c5b9d P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.085s user 0.001s sys 0.001s
I20251029 02:52:14.310639 1166 tablet_bootstrap.cc:492] T 7bb85a12f573494cbb3118d1947eda27 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:14.310957 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1026) took 2766 ms. Trace:
I20251029 02:52:14.311023 1176 rpcz_store.cc:276] 1029 02:52:11.544658 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:11.574849 (+ 30191us) service_pool.cc:225] Handling call
1029 02:52:14.160440 (+2585591us) ts_tablet_manager.cc:1151] Deleting tablet 6d830d30b26b4ba799785fec8a6f0769
1029 02:52:14.160449 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:14.175182 (+ 14733us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:14.250216 (+ 75034us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:14.310941 (+ 60725us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2585465}
I20251029 02:52:14.311712 1176 tablet_replica.cc:333] T b6d4f7bf702f4834bc5f4ef2b50b92a7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:14.312258 1176 raft_consensus.cc:2243] T b6d4f7bf702f4834bc5f4ef2b50b92a7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:14.312639 1176 raft_consensus.cc:2272] T b6d4f7bf702f4834bc5f4ef2b50b92a7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:14.313388 1176 ts_tablet_manager.cc:1916] T b6d4f7bf702f4834bc5f4ef2b50b92a7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:14.314666 1061 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "70b008011d5547bbaf3f45685394139f" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:14.315191 1054 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "58dde9397b434b089ade9968f369006a" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:14.315338 1054 raft_consensus.cc:2468] T 58dde9397b434b089ade9968f369006a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:14.309823 2430 leader_election.cc:290] T 58dde9397b434b089ade9968f369006a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:14.317441 1174 ts_tablet_manager.cc:1929] T 5db519edc75d4bd0985b32322825ffae P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:14.318110 1174 log.cc:1199] T 5db519edc75d4bd0985b32322825ffae P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/5db519edc75d4bd0985b32322825ffae
I20251029 02:52:14.318565 1174 ts_tablet_manager.cc:1950] T 5db519edc75d4bd0985b32322825ffae P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:14.319121 517 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 3c3ef62323d643dda28f0ca269274600
I20251029 02:52:14.320240 745 tablet_service.cc:1552] Processing DeleteTablet for tablet 3c3ef62323d643dda28f0ca269274600 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:14 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:14.315232 1053 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "58dde9397b434b089ade9968f369006a" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:52:14.320442 1053 raft_consensus.cc:3060] T 58dde9397b434b089ade9968f369006a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:14.320963 1172 consensus_queue.cc:260] T 84993b2f63e7421290a161dde4bb0607 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:14.321218 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 3c3ef62323d643dda28f0ca269274600 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:14 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:14.322878 1172 ts_tablet_manager.cc:1434] T 84993b2f63e7421290a161dde4bb0607 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.057s user 0.000s sys 0.003s
I20251029 02:52:14.324103 1167 tablet_bootstrap.cc:654] T 40e844fd7ad74680bcbe7bef936c5b9d P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:14.327842 1061 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d31e548feae24b739087c1e211d514f9" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:52:14.328177 523 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 60a964a5181a45a186921c31cbdd3db7 consensus state has no leader
I20251029 02:52:14.328541 1176 ts_tablet_manager.cc:1929] T b6d4f7bf702f4834bc5f4ef2b50b92a7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:14.328652 1176 log.cc:1199] T b6d4f7bf702f4834bc5f4ef2b50b92a7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/b6d4f7bf702f4834bc5f4ef2b50b92a7
W20251029 02:52:14.330157 849 leader_election.cc:343] T 6d830d30b26b4ba799785fec8a6f0769 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Not found: Tablet not found: 6d830d30b26b4ba799785fec8a6f0769
I20251029 02:52:14.330385 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "58dde9397b434b089ade9968f369006a" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:14.332597 518 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_fd38cb7a06ca47d9b65efd17ae9c5d28" } new_table_name: "default.Table_0057d5992fe1462eb1839b7acf7486de" modify_external_catalogs: true
I20251029 02:52:14.333503 1061 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "23b9b2ea814645eba63e5f6b12cfb966" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:52:14.334283 712 leader_election.cc:343] T eca016aed05e4ce09e1a2bf96e2286f5 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:14.334406 712 leader_election.cc:304] T eca016aed05e4ce09e1a2bf96e2286f5 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
W20251029 02:52:14.334825 712 leader_election.cc:343] T 70b008011d5547bbaf3f45685394139f P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:14.334939 712 leader_election.cc:304] T 70b008011d5547bbaf3f45685394139f P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
W20251029 02:52:14.335269 712 leader_election.cc:343] T d31e548feae24b739087c1e211d514f9 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:14.335376 712 leader_election.cc:304] T d31e548feae24b739087c1e211d514f9 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:14.335426 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 884) took 4567 ms. Trace:
I20251029 02:52:14.335508 1174 rpcz_store.cc:276] 1029 02:52:09.768150 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:09.768651 (+ 501us) service_pool.cc:225] Handling call
1029 02:52:13.972643 (+4203992us) ts_tablet_manager.cc:1151] Deleting tablet 5db519edc75d4bd0985b32322825ffae
1029 02:52:13.972654 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:14.187590 (+214936us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:14.317337 (+129747us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:14.335404 (+ 18067us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4203843}
I20251029 02:52:14.335784 2611 raft_consensus.cc:2749] T eca016aed05e4ce09e1a2bf96e2286f5 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:14.335949 2611 raft_consensus.cc:2749] T 70b008011d5547bbaf3f45685394139f P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:14.336081 2611 raft_consensus.cc:2749] T d31e548feae24b739087c1e211d514f9 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:14.336715 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 870) took 4991 ms. Trace:
I20251029 02:52:14.336753 878 tablet_service.cc:1552] Processing DeleteTablet for tablet 20a762344c5f46f88481c4ac7ad6e43b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:08 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:14.336807 1175 rpcz_store.cc:276] 1029 02:52:09.345370 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:09.345441 (+ 71us) service_pool.cc:225] Handling call
1029 02:52:13.947927 (+4602486us) ts_tablet_manager.cc:1151] Deleting tablet e65eaaae43ac435ebd665546de9d68dd
1029 02:52:13.947936 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:14.179578 (+231642us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:14.257497 (+ 77919us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:14.336696 (+ 79199us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4602264}
I20251029 02:52:14.337194 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet e65eaaae43ac435ebd665546de9d68dd (table default.Table_d9fe74a49f4843b089aae7fca9e4a826 [id=152882c742b2459ba863fd41083284c1]) successfully deleted
I20251029 02:52:14.337751 520 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet b81f26fb2739487dbc9af7b6e9848a4a
I20251029 02:52:14.338631 745 tablet_service.cc:1552] Processing DeleteTablet for tablet b81f26fb2739487dbc9af7b6e9848a4a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:14 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:14.338739 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21605b1dabf440cf87e8962749590a4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:14.339852 1167 tablet_bootstrap.cc:492] T 40e844fd7ad74680bcbe7bef936c5b9d P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:14.340003 1167 ts_tablet_manager.cc:1403] T 40e844fd7ad74680bcbe7bef936c5b9d P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.060s user 0.004s sys 0.000s
I20251029 02:52:14.339969 745 tablet_service.cc:1505] Processing CreateTablet for tablet d9ee15c3f45a490bb7a1925e3542b0bb (DEFAULT_TABLE table=default.Table_eff289b2377f44aeb3411568c4854be1 [id=c3d49bcba6da4fbf939a0baa1137e5b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:14.340764 1167 raft_consensus.cc:359] T 40e844fd7ad74680bcbe7bef936c5b9d P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:14.340940 1167 raft_consensus.cc:385] T 40e844fd7ad74680bcbe7bef936c5b9d P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:14.340993 1167 raft_consensus.cc:740] T 40e844fd7ad74680bcbe7bef936c5b9d P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:14.341416 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 6d830d30b26b4ba799785fec8a6f0769 (table default.Table_19165ff9f7f24a97bf835e72701455d3 [id=ae6032816e6c4b80aa294478cbd49460]) successfully deleted
I20251029 02:52:14.343129 878 tablet_service.cc:1552] Processing DeleteTablet for tablet b81f26fb2739487dbc9af7b6e9848a4a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:14 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:14.343070 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 5db519edc75d4bd0985b32322825ffae (table default.Table_b8ddec1d965444948171ef0cca0ff011 [id=60a528f028d24bbabd6f5a927fb7853b]) successfully deleted
I20251029 02:52:14.339090 1174 tablet_replica.cc:333] T c9e923724ca9402682f13b252dbc37aa P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:14.343386 1174 raft_consensus.cc:2243] T c9e923724ca9402682f13b252dbc37aa P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:14.343549 1174 raft_consensus.cc:2272] T c9e923724ca9402682f13b252dbc37aa P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:14.344028 1174 ts_tablet_manager.cc:1916] T c9e923724ca9402682f13b252dbc37aa P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:14.345742 1167 consensus_queue.cc:260] T 40e844fd7ad74680bcbe7bef936c5b9d P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:14.346351 1167 ts_tablet_manager.cc:1434] T 40e844fd7ad74680bcbe7bef936c5b9d P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251029 02:52:14.350237 1167 tablet_bootstrap.cc:492] T 6f8cf09f23984b879b60ab6b27c7d52e P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:14.353137 1053 raft_consensus.cc:2468] T 58dde9397b434b089ade9968f369006a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
W20251029 02:52:14.353389 518 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 04528b4b700744aab24f6b1d5cb2787a consensus state has no leader
I20251029 02:52:14.354271 2452 tablet.cc:1722] T d7abaf84ba5d445aa17433b3be891548 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251029 02:52:14.356642 881 tablet_service.cc:1505] Processing CreateTablet for tablet d9ee15c3f45a490bb7a1925e3542b0bb (DEFAULT_TABLE table=default.Table_eff289b2377f44aeb3411568c4854be1 [id=c3d49bcba6da4fbf939a0baa1137e5b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:14.357530 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9ee15c3f45a490bb7a1925e3542b0bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:14.357992 878 tablet_service.cc:1505] Processing CreateTablet for tablet 1194e03172954aadb819cc9a6a5050fe (DEFAULT_TABLE table=default.Table_13076d1361cd4ef9ae70f064a6b8679c [id=8c1e0a13dd3440598a3e1845bdc6bb8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:14.358443 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1194e03172954aadb819cc9a6a5050fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:14.358826 743 tablet_service.cc:1505] Processing CreateTablet for tablet 1194e03172954aadb819cc9a6a5050fe (DEFAULT_TABLE table=default.Table_13076d1361cd4ef9ae70f064a6b8679c [id=8c1e0a13dd3440598a3e1845bdc6bb8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:14.359747 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48e74fe2c98f4df5ac686a4405c4521b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:14.360939 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_5c2a494d41d0490e9c6eb102db44abbe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:14.361469 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5c2a494d41d0490e9c6eb102db44abbe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:14.361771 2205 consensus_queue.cc:1048] T 67651306f3b44e7eb14dcee9a2007a81 P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251029 02:52:14.362488 847 leader_election.cc:343] T 6d830d30b26b4ba799785fec8a6f0769 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 6d830d30b26b4ba799785fec8a6f0769
I20251029 02:52:14.362638 1175 tablet_replica.cc:333] T 2f198c3259864f7ab3625cb7bfae20b0 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:14.362625 847 leader_election.cc:304] T 6d830d30b26b4ba799785fec8a6f0769 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9
I20251029 02:52:14.363061 2201 raft_consensus.cc:2749] T 6d830d30b26b4ba799785fec8a6f0769 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251029 02:52:14.363086 1175 raft_consensus.cc:2243] T 2f198c3259864f7ab3625cb7bfae20b0 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:14.363278 1175 raft_consensus.cc:2272] T 2f198c3259864f7ab3625cb7bfae20b0 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:14.363853 1175 ts_tablet_manager.cc:1916] T 2f198c3259864f7ab3625cb7bfae20b0 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:14.365702 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9ee15c3f45a490bb7a1925e3542b0bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:14.366618 1172 tablet_bootstrap.cc:492] T c081d2eacfac4ca8a0048e2a1c4f8f83 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:14.368335 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b5f9dfe571c44af9c22e8de843c4207. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:14.368366 1172 tablet_bootstrap.cc:654] T c081d2eacfac4ca8a0048e2a1c4f8f83 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:14.371557 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_0b7ca974a3574a30afa0150eea5fe8bb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:14.371884 744 tablet_service.cc:1505] Processing CreateTablet for tablet c977cf36300648d6a6a2236d358fc02d (DEFAULT_TABLE table=default.Table_31f9807cb2934a47a4fb2f67f9507053 [id=c3b6a55d9eaa4248aadb99c5ae1a9be5]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 02:52:14.372206 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0b7ca974a3574a30afa0150eea5fe8bb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:14.372257 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c977cf36300648d6a6a2236d358fc02d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:14.373905 517 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_5461f53d600440a8bf5bb171829677e1" } new_table_name: "default.Table_75bd13ed4bc340139c8219e1de2c4402" modify_external_catalogs: true
I20251029 02:52:14.377350 1031 tablet_service.cc:1505] Processing CreateTablet for tablet fb23c471d1d348a1b94b2ffb69b9a115 (DEFAULT_TABLE table=default.Table_42036229ecf348adb05a3fdaaab07e1f [id=497f4e20bfc946af90522b4acc0d182b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:14.377784 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb23c471d1d348a1b94b2ffb69b9a115. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:14.378274 747 tablet_service.cc:1505] Processing CreateTablet for tablet 3cb36eaa1ffe4db2928e87764eb65a35 (DEFAULT_TABLE table=default.Table_08c8b82cbf9a424c8514bea1a6324fb1 [id=de904db373d742ea9fcdd7aac63357bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:14.378415 751 tablet_service.cc:1505] Processing CreateTablet for tablet 8c028683f8db4c02a10a37780e20f6dc (DEFAULT_TABLE table=default.Table_2f9f5508a47d486ca4004b3c2b50013d [id=379b8725da8749ab8f218856073e0ce5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:14.378688 738 tablet_service.cc:1505] Processing CreateTablet for tablet 6959a3652c4d4c2d846cd68c3c03dd4b (DEFAULT_TABLE table=default.Table_8dbb47d3c55c40a1b33c0567d0eff749 [id=2093f9393d8a474dbe9a51ae8e336b3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:14.378804 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c028683f8db4c02a10a37780e20f6dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:14.378948 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3cb36eaa1ffe4db2928e87764eb65a35. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:14.382162 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6959a3652c4d4c2d846cd68c3c03dd4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:14.382741 1175 ts_tablet_manager.cc:1929] T 2f198c3259864f7ab3625cb7bfae20b0 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:14.383983 1175 log.cc:1199] T 2f198c3259864f7ab3625cb7bfae20b0 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/2f198c3259864f7ab3625cb7bfae20b0
I20251029 02:52:14.384721 1175 ts_tablet_manager.cc:1950] T 2f198c3259864f7ab3625cb7bfae20b0 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:14.388545 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 871) took 5012 ms. Trace:
I20251029 02:52:14.391237 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_98b6a6ee7ae34368bef7d883e216978d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:14.391906 1167 tablet_bootstrap.cc:654] T 6f8cf09f23984b879b60ab6b27c7d52e P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
W20251029 02:52:14.391904 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_98b6a6ee7ae34368bef7d883e216978d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:14.394112 1175 rpcz_store.cc:276] 1029 02:52:09.375664 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:09.375717 (+ 53us) service_pool.cc:225] Handling call
1029 02:52:14.340300 (+4964583us) ts_tablet_manager.cc:1151] Deleting tablet 2f198c3259864f7ab3625cb7bfae20b0
1029 02:52:14.362583 (+ 22283us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:14.370030 (+ 7447us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:14.382656 (+ 12626us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:14.388522 (+ 5866us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4964419}
I20251029 02:52:14.395685 1175 tablet_replica.cc:333] T 44c0488eba0543078ff4f0b3d69366f2 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:14.395954 1175 raft_consensus.cc:2243] T 44c0488eba0543078ff4f0b3d69366f2 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:14.396186 1175 raft_consensus.cc:2272] T 44c0488eba0543078ff4f0b3d69366f2 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:14.396801 1175 ts_tablet_manager.cc:1916] T 44c0488eba0543078ff4f0b3d69366f2 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:14.399638 519 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_a0cab891f6e34cd4bce7592b55b68979" } modify_external_catalogs: true
I20251029 02:52:14.399869 519 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_a0cab891f6e34cd4bce7592b55b68979" } modify_external_catalogs: true
I20251029 02:52:14.401749 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_cba115e38a014588b60259fe951f80c9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:14.403287 895 tablet_service.cc:1505] Processing CreateTablet for tablet 3cb36eaa1ffe4db2928e87764eb65a35 (DEFAULT_TABLE table=default.Table_08c8b82cbf9a424c8514bea1a6324fb1 [id=de904db373d742ea9fcdd7aac63357bc]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 02:52:14.402441 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cba115e38a014588b60259fe951f80c9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:14.403717 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3cb36eaa1ffe4db2928e87764eb65a35. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:14.404301 522 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_022e72d689654b18adde8de9ec2c065a" } modify_external_catalogs: true
I20251029 02:52:14.401790 524 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_9c6ff90fbbd9431c9ae4dbf96e02b307" } new_table_name: "default.Table_21ec6b8b68ee4e62ab9808a41c4551c4" modify_external_catalogs: true
I20251029 02:52:14.404457 522 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_022e72d689654b18adde8de9ec2c065a" } modify_external_catalogs: true
I20251029 02:52:14.406795 896 tablet_service.cc:1505] Processing CreateTablet for tablet 8c028683f8db4c02a10a37780e20f6dc (DEFAULT_TABLE table=default.Table_2f9f5508a47d486ca4004b3c2b50013d [id=379b8725da8749ab8f218856073e0ce5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:14.407159 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c028683f8db4c02a10a37780e20f6dc. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:14.410322 712 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:45003, user_credentials={real_user=slave}} blocked reactor thread for 75046.2us
I20251029 02:52:14.410414 879 tablet_service.cc:1505] Processing CreateTablet for tablet 6959a3652c4d4c2d846cd68c3c03dd4b (DEFAULT_TABLE table=default.Table_8dbb47d3c55c40a1b33c0567d0eff749 [id=2093f9393d8a474dbe9a51ae8e336b3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:14.403359 894 tablet_service.cc:1505] Processing CreateTablet for tablet c977cf36300648d6a6a2236d358fc02d (DEFAULT_TABLE table=default.Table_31f9807cb2934a47a4fb2f67f9507053 [id=c3b6a55d9eaa4248aadb99c5ae1a9be5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:14.412719 739 tablet_service.cc:1505] Processing CreateTablet for tablet 2fccba4e76b44949a37f04466ef00dfc (DEFAULT_TABLE table=default.Table_4ca293107d23430cbad5c6e53b07da9f [id=a4b75e76165e4a1a8d0c0ca8c97e856d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:14.413198 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2fccba4e76b44949a37f04466ef00dfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:14.419162 1172 tablet_bootstrap.cc:492] T c081d2eacfac4ca8a0048e2a1c4f8f83 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:14.419377 1172 ts_tablet_manager.cc:1403] T c081d2eacfac4ca8a0048e2a1c4f8f83 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.060s user 0.006s sys 0.003s
I20251029 02:52:14.420163 1172 raft_consensus.cc:359] T c081d2eacfac4ca8a0048e2a1c4f8f83 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:14.420395 1172 raft_consensus.cc:385] T c081d2eacfac4ca8a0048e2a1c4f8f83 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:14.420482 1172 raft_consensus.cc:740] T c081d2eacfac4ca8a0048e2a1c4f8f83 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:14.420727 1172 consensus_queue.cc:260] T c081d2eacfac4ca8a0048e2a1c4f8f83 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:14.421357 1172 ts_tablet_manager.cc:1434] T c081d2eacfac4ca8a0048e2a1c4f8f83 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 02:52:14.421651 1172 tablet_bootstrap.cc:492] T fcb3db8399aa4cf188b07dda220f7f5b P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:14.421979 878 tablet_service.cc:1505] Processing CreateTablet for tablet 2fccba4e76b44949a37f04466ef00dfc (DEFAULT_TABLE table=default.Table_4ca293107d23430cbad5c6e53b07da9f [id=a4b75e76165e4a1a8d0c0ca8c97e856d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:14.422844 900 tablet_service.cc:1505] Processing CreateTablet for tablet 9dd2cb0e1747472084e49a2d83766761 (DEFAULT_TABLE table=default.Table_80d1a54a30fa423f809bef46b9301f29 [id=dbb0431e6f784b9992013b534e99a221]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:14.423257 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9dd2cb0e1747472084e49a2d83766761. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:14.423693 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c977cf36300648d6a6a2236d358fc02d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:14.424141 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6959a3652c4d4c2d846cd68c3c03dd4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:14.424016 518 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_80d1a54a30fa423f809bef46b9301f29" } new_table_name: "default.Table_f6e0414ea7e04ea2973933ee1f4c4fc5" modify_external_catalogs: true
W20251029 02:52:14.426476 517 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4c418961f37044c887d278f4e0fe2248 has no consensus state
I20251029 02:52:14.427196 519 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet c587ed98423e450a90543f456de0755c
I20251029 02:52:14.431126 747 tablet_service.cc:1552] Processing DeleteTablet for tablet c081d2eacfac4ca8a0048e2a1c4f8f83 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:04 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:14.431841 1174 ts_tablet_manager.cc:1929] T c9e923724ca9402682f13b252dbc37aa P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251029 02:52:14.436241 1174 log.cc:1199] T c9e923724ca9402682f13b252dbc37aa P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/c9e923724ca9402682f13b252dbc37aa
W20251029 02:52:14.437374 524 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0de19a308a534cdaa38f46ab43789322 consensus state has no leader
I20251029 02:52:14.438279 1174 ts_tablet_manager.cc:1950] T c9e923724ca9402682f13b252dbc37aa P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:14.438836 522 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 0e9a9dd2f6e4424cb1500682d5a49cc4
W20251029 02:52:14.439416 518 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9dd2cb0e1747472084e49a2d83766761 consensus state has no leader
I20251029 02:52:14.439723 751 tablet_service.cc:1552] Processing DeleteTablet for tablet 0e9a9dd2f6e4424cb1500682d5a49cc4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:14 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:14.439881 747 tablet_service.cc:1505] Processing CreateTablet for tablet e91d6bdca3ad40c48d93e6196644462c (DEFAULT_TABLE table=default.Table_fcfe86e2824842a2aa6e4188c33b4a74 [id=cc3e1f3858f04176acb8c39c65396a5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:14.440275 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e91d6bdca3ad40c48d93e6196644462c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:14.441167 885 tablet_service.cc:1552] Processing DeleteTablet for tablet 0e9a9dd2f6e4424cb1500682d5a49cc4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:14 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:14.442149 894 tablet_service.cc:1505] Processing CreateTablet for tablet e91d6bdca3ad40c48d93e6196644462c (DEFAULT_TABLE table=default.Table_fcfe86e2824842a2aa6e4188c33b4a74 [id=cc3e1f3858f04176acb8c39c65396a5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:14.442564 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e91d6bdca3ad40c48d93e6196644462c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:14.446791 896 tablet_service.cc:1505] Processing CreateTablet for tablet 98509b563a7e4fb6be3cb8a939398de6 (DEFAULT_TABLE table=default.Table_dcf576a6745749cc85aeddaab539d832 [id=013f84786dc8453ea00661a2503cbc99]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:14.447312 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98509b563a7e4fb6be3cb8a939398de6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:14.447719 756 tablet_service.cc:1505] Processing CreateTablet for tablet 98509b563a7e4fb6be3cb8a939398de6 (DEFAULT_TABLE table=default.Table_dcf576a6745749cc85aeddaab539d832 [id=013f84786dc8453ea00661a2503cbc99]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:14.447839 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 897) took 4552 ms. Trace:
I20251029 02:52:14.447921 1174 rpcz_store.cc:276] 1029 02:52:09.895825 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:09.900536 (+ 4711us) service_pool.cc:225] Handling call
1029 02:52:14.339039 (+4438503us) ts_tablet_manager.cc:1151] Deleting tablet c9e923724ca9402682f13b252dbc37aa
1029 02:52:14.339049 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:14.347376 (+ 8327us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:14.431782 (+ 84406us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:14.447818 (+ 16036us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4438286}
I20251029 02:52:14.448097 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98509b563a7e4fb6be3cb8a939398de6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:14.450644 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2fccba4e76b44949a37f04466ef00dfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:14.451460 747 tablet_service.cc:1505] Processing CreateTablet for tablet 2a92dd79bc6e42a2a5d85bc523fd5d23 (DEFAULT_TABLE table=default.Table_f861c96de6f04741b28da03e79652f8b [id=260e71eaf2894512bba512e34b3f8431]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:14.451754 1176 ts_tablet_manager.cc:1950] T b6d4f7bf702f4834bc5f4ef2b50b92a7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:14.451962 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a92dd79bc6e42a2a5d85bc523fd5d23. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:14.452055 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1027) took 2907 ms. Trace:
I20251029 02:52:14.452132 1176 rpcz_store.cc:276] 1029 02:52:11.544725 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:11.574989 (+ 30264us) service_pool.cc:225] Handling call
1029 02:52:14.311668 (+2736679us) ts_tablet_manager.cc:1151] Deleting tablet b6d4f7bf702f4834bc5f4ef2b50b92a7
1029 02:52:14.311678 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:14.321009 (+ 9331us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:14.328426 (+ 7417us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:14.452041 (+123615us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2736563}
I20251029 02:52:14.452417 1176 tablet_replica.cc:333] T e7f846b294cc4eb687a26c8b5931ac19 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:14.452600 1176 raft_consensus.cc:2243] T e7f846b294cc4eb687a26c8b5931ac19 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:14.452752 1176 raft_consensus.cc:2272] T e7f846b294cc4eb687a26c8b5931ac19 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:14.453291 1176 ts_tablet_manager.cc:1916] T e7f846b294cc4eb687a26c8b5931ac19 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:14.449478 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 917) took 4423 ms. Trace:
I20251029 02:52:14.454918 1174 rpcz_store.cc:276] 1029 02:52:10.025605 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:10.032294 (+ 6689us) service_pool.cc:225] Handling call
1029 02:52:14.449358 (+4417064us) ts_tablet_manager.cc:1151] Deleting tablet 23856678b7aa4f9ebf65fb4564a4bf91
1029 02:52:14.449369 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:14.449461 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4416804}
I20251029 02:52:14.455457 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 919) took 4420 ms. Trace:
I20251029 02:52:14.455551 1174 rpcz_store.cc:276] 1029 02:52:10.034618 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:10.038132 (+ 3514us) service_pool.cc:225] Handling call
1029 02:52:14.455328 (+4417196us) ts_tablet_manager.cc:1151] Deleting tablet 43a76b984da3414da701221f87349337
1029 02:52:14.455339 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:14.455440 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4416999}
I20251029 02:52:14.455703 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_30bf322e8193402e80523ebfba5c8e43"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:14.456332 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_30bf322e8193402e80523ebfba5c8e43 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:14.457427 751 tablet_service.cc:1505] Processing CreateTablet for tablet 44a0c058700040f8b5c30a621103a524 (DEFAULT_TABLE table=default.Table_8aa16ebd94174be79b8fd9d2206599a1 [id=300df93a696b407cbf826cfb9ca66919]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:14.457816 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44a0c058700040f8b5c30a621103a524. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:14.457895 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_442850f577aa437a820314bbf3df9ac4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:14.458375 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_442850f577aa437a820314bbf3df9ac4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:14.458279 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_4098c28776dd4f41864c61d2299b90cd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:14.458832 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4098c28776dd4f41864c61d2299b90cd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:14.459619 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_590e7ccd21564447951d30fbf609598e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:14.460404 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_590e7ccd21564447951d30fbf609598e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:14.462215 744 tablet_service.cc:1505] Processing CreateTablet for tablet 9dd2cb0e1747472084e49a2d83766761 (DEFAULT_TABLE table=default.Table_80d1a54a30fa423f809bef46b9301f29 [id=dbb0431e6f784b9992013b534e99a221]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:14.462919 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9dd2cb0e1747472084e49a2d83766761. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:14.463233 1167 tablet_bootstrap.cc:492] T 6f8cf09f23984b879b60ab6b27c7d52e P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:14.463368 1167 ts_tablet_manager.cc:1403] T 6f8cf09f23984b879b60ab6b27c7d52e P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.113s user 0.004s sys 0.000s
I20251029 02:52:14.464159 1167 raft_consensus.cc:359] T 6f8cf09f23984b879b60ab6b27c7d52e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:14.464323 1167 raft_consensus.cc:385] T 6f8cf09f23984b879b60ab6b27c7d52e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:14.464370 1167 raft_consensus.cc:740] T 6f8cf09f23984b879b60ab6b27c7d52e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:14.464588 1167 consensus_queue.cc:260] T 6f8cf09f23984b879b60ab6b27c7d52e P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:14.465129 1036 tablet_service.cc:1505] Processing CreateTablet for tablet 21605b1dabf440cf87e8962749590a4f (DEFAULT_TABLE table=default.Table_0d5fcbba432c470cb80f5cdb893b6294 [id=2eae726c7cbc4029b6b4c1343cb4e36c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:14.465257 1167 ts_tablet_manager.cc:1434] T 6f8cf09f23984b879b60ab6b27c7d52e P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 02:52:14.461855 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 926) took 4310 ms. Trace:
I20251029 02:52:14.465509 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21605b1dabf440cf87e8962749590a4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:14.465502 1174 rpcz_store.cc:276] 1029 02:52:10.151226 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:10.154313 (+ 3087us) service_pool.cc:225] Handling call
1029 02:52:14.461710 (+4307397us) ts_tablet_manager.cc:1151] Deleting tablet 4660fd0534df49e0af43c36fb1122d84
1029 02:52:14.461723 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:14.461834 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4307104}
I20251029 02:52:14.467986 738 tablet_service.cc:1552] Processing DeleteTablet for tablet 84993b2f63e7421290a161dde4bb0607 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:57 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:14.468068 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet c9e923724ca9402682f13b252dbc37aa (table default.Table_443f33587b8f428484b99e07355d69ca [id=03a25420ce6a46fc93a9fb753246bc7b]) successfully deleted
W20251029 02:52:14.468326 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 23856678b7aa4f9ebf65fb4564a4bf91 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 23856678b7aa4f9ebf65fb4564a4bf91 already in progress: creating tablet
W20251029 02:52:14.468520 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 43a76b984da3414da701221f87349337 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 43a76b984da3414da701221f87349337 already in progress: creating tablet
W20251029 02:52:14.468887 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 4660fd0534df49e0af43c36fb1122d84 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4660fd0534df49e0af43c36fb1122d84 already in progress: creating tablet
I20251029 02:52:14.479010 1172 tablet_bootstrap.cc:654] T fcb3db8399aa4cf188b07dda220f7f5b P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:14.481735 2302 raft_consensus.cc:493] T 40e844fd7ad74680bcbe7bef936c5b9d P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:14.481892 2302 raft_consensus.cc:515] T 40e844fd7ad74680bcbe7bef936c5b9d P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:14.482568 2302 leader_election.cc:290] T 40e844fd7ad74680bcbe7bef936c5b9d P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:14.483318 762 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40e844fd7ad74680bcbe7bef936c5b9d" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:14.483495 762 raft_consensus.cc:2468] T 40e844fd7ad74680bcbe7bef936c5b9d P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:14.484481 847 leader_election.cc:304] T 40e844fd7ad74680bcbe7bef936c5b9d P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:14.485597 2302 raft_consensus.cc:2804] T 40e844fd7ad74680bcbe7bef936c5b9d P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:14.485690 2302 raft_consensus.cc:493] T 40e844fd7ad74680bcbe7bef936c5b9d P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:14.485765 2302 raft_consensus.cc:3060] T 40e844fd7ad74680bcbe7bef936c5b9d P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:14.488323 2611 raft_consensus.cc:493] T 6dff036b32904ddb9598be4b79504bb6 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:14.488446 2611 raft_consensus.cc:515] T 6dff036b32904ddb9598be4b79504bb6 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:14.488868 2302 raft_consensus.cc:515] T 40e844fd7ad74680bcbe7bef936c5b9d P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:14.489092 2611 leader_election.cc:290] T 6dff036b32904ddb9598be4b79504bb6 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:52:14.484359 1061 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40e844fd7ad74680bcbe7bef936c5b9d" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:14.490145 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dff036b32904ddb9598be4b79504bb6" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:14.490302 917 raft_consensus.cc:2468] T 6dff036b32904ddb9598be4b79504bb6 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b495a49349324f7d81908f2da700f0d9 in term 0.
I20251029 02:52:14.490310 1061 raft_consensus.cc:2468] T 40e844fd7ad74680bcbe7bef936c5b9d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:14.490648 1038 tablet_service.cc:1552] Processing DeleteTablet for tablet c0936e54f1d246388a2d82dbc52aac42 with delete_type TABLET_DATA_DELETED (replaced by tablet 0573fe69797f48539df5caf4b3777b4d) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:14.491007 711 leader_election.cc:304] T 6dff036b32904ddb9598be4b79504bb6 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:14.491257 2611 raft_consensus.cc:2804] T 6dff036b32904ddb9598be4b79504bb6 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:14.491331 2611 raft_consensus.cc:493] T 6dff036b32904ddb9598be4b79504bb6 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:14.491389 2611 raft_consensus.cc:3060] T 6dff036b32904ddb9598be4b79504bb6 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:14.491652 2302 leader_election.cc:290] T 40e844fd7ad74680bcbe7bef936c5b9d P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:14.493652 762 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40e844fd7ad74680bcbe7bef936c5b9d" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:14.493830 762 raft_consensus.cc:3060] T 40e844fd7ad74680bcbe7bef936c5b9d P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:14.494263 2611 raft_consensus.cc:515] T 6dff036b32904ddb9598be4b79504bb6 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:14.494887 2611 leader_election.cc:290] T 6dff036b32904ddb9598be4b79504bb6 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 election: Requested vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:52:14.495131 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1194e03172954aadb819cc9a6a5050fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:14.495479 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dff036b32904ddb9598be4b79504bb6" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
I20251029 02:52:14.495635 917 raft_consensus.cc:3060] T 6dff036b32904ddb9598be4b79504bb6 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:14.496795 762 raft_consensus.cc:2468] T 40e844fd7ad74680bcbe7bef936c5b9d P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:14.498013 847 leader_election.cc:304] T 40e844fd7ad74680bcbe7bef936c5b9d P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:14.498348 917 raft_consensus.cc:2468] T 6dff036b32904ddb9598be4b79504bb6 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b495a49349324f7d81908f2da700f0d9 in term 1.
I20251029 02:52:14.498790 711 leader_election.cc:304] T 6dff036b32904ddb9598be4b79504bb6 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:14.498919 1031 tablet_service.cc:1505] Processing CreateTablet for tablet 48e74fe2c98f4df5ac686a4405c4521b (DEFAULT_TABLE table=default.Table_a5ee816bf96c4eb891cb8ba5ed50f3ed [id=ec71481cc6464181926e080b7ba40da9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:14.499583 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48e74fe2c98f4df5ac686a4405c4521b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:14.500492 2611 raft_consensus.cc:2804] T 6dff036b32904ddb9598be4b79504bb6 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:14.500617 2611 raft_consensus.cc:697] T 6dff036b32904ddb9598be4b79504bb6 P b495a49349324f7d81908f2da700f0d9 [term 1 LEADER]: Becoming Leader. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Running, Role: LEADER
I20251029 02:52:14.500845 2611 consensus_queue.cc:237] T 6dff036b32904ddb9598be4b79504bb6 P b495a49349324f7d81908f2da700f0d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:14.509958 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_bf304b9f44274cf9b56f9e25fb0e9875"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:14.510691 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bf304b9f44274cf9b56f9e25fb0e9875 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:14.511384 1029 tablet_service.cc:1505] Processing CreateTablet for tablet 5b5f9dfe571c44af9c22e8de843c4207 (DEFAULT_TABLE table=default.Table_2c4f1597c5d34f11a1bd7ee6550803fc [id=ace38e96a5cb4d5bb821ea7d2cfb0a3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:14.511786 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b5f9dfe571c44af9c22e8de843c4207. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:14.513761 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 929) took 4260 ms. Trace:
I20251029 02:52:14.513926 1174 rpcz_store.cc:276] 1029 02:52:10.253596 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:10.253662 (+ 66us) service_pool.cc:225] Handling call
1029 02:52:14.467303 (+4213641us) ts_tablet_manager.cc:1151] Deleting tablet eeb75fc46e2a41f78252868ad5256966
1029 02:52:14.513419 (+ 46116us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:14.513701 (+ 282us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4213347}
I20251029 02:52:14.514945 896 tablet_service.cc:1505] Processing CreateTablet for tablet 2a92dd79bc6e42a2a5d85bc523fd5d23 (DEFAULT_TABLE table=default.Table_f861c96de6f04741b28da03e79652f8b [id=260e71eaf2894512bba512e34b3f8431]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:14.515022 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 937) took 4139 ms. Trace:
I20251029 02:52:14.515924 1176 ts_tablet_manager.cc:1929] T e7f846b294cc4eb687a26c8b5931ac19 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:14.516213 1174 rpcz_store.cc:276] 1029 02:52:10.375245 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:10.375304 (+ 59us) service_pool.cc:225] Handling call
1029 02:52:14.514762 (+4139458us) ts_tablet_manager.cc:1151] Deleting tablet 877e278deb4a4b01ac66b06be0890baf
1029 02:52:14.514772 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:14.514973 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4139212}
I20251029 02:52:14.516780 1176 log.cc:1199] T e7f846b294cc4eb687a26c8b5931ac19 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/e7f846b294cc4eb687a26c8b5931ac19
I20251029 02:52:14.517237 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a92dd79bc6e42a2a5d85bc523fd5d23. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:14.517292 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 943) took 3994 ms. Trace:
I20251029 02:52:14.517664 1174 rpcz_store.cc:276] 1029 02:52:10.523160 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:10.523211 (+ 51us) service_pool.cc:225] Handling call
1029 02:52:14.517068 (+3993857us) ts_tablet_manager.cc:1151] Deleting tablet 86fc22bb84794149b598b97df41eb881
1029 02:52:14.517077 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:14.517273 (+ 196us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3915332}
I20251029 02:52:14.518652 1174 tablet_replica.cc:333] T 49d9ce32e4f04f11be2ecefc5d523a8b P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:14.519367 1174 raft_consensus.cc:2243] T 49d9ce32e4f04f11be2ecefc5d523a8b P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:14.520293 1176 ts_tablet_manager.cc:1950] T e7f846b294cc4eb687a26c8b5931ac19 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:14.521756 525 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_cfbb812339124da1b8e6e8a95d816a5e" } modify_external_catalogs: true
I20251029 02:52:14.522411 1174 raft_consensus.cc:2272] T 49d9ce32e4f04f11be2ecefc5d523a8b P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:14.523330 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1011) took 2994 ms. Trace:
I20251029 02:52:14.525126 1174 ts_tablet_manager.cc:1916] T 49d9ce32e4f04f11be2ecefc5d523a8b P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:14.523995 1176 rpcz_store.cc:276] 1029 02:52:11.528315 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:11.528357 (+ 42us) service_pool.cc:225] Handling call
1029 02:52:14.452376 (+2924019us) ts_tablet_manager.cc:1151] Deleting tablet e7f846b294cc4eb687a26c8b5931ac19
1029 02:52:14.452384 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:14.482142 (+ 29758us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:14.515067 (+ 32925us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:14.523128 (+ 8061us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2873396}
I20251029 02:52:14.525687 2302 raft_consensus.cc:2804] T 40e844fd7ad74680bcbe7bef936c5b9d P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:14.527272 1176 tablet_replica.cc:333] T 061f3d38c4604fa4b57e74ff6f6fc7da P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:14.527532 878 tablet_service.cc:1505] Processing CreateTablet for tablet 44a0c058700040f8b5c30a621103a524 (DEFAULT_TABLE table=default.Table_8aa16ebd94174be79b8fd9d2206599a1 [id=300df93a696b407cbf826cfb9ca66919]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:14.527981 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44a0c058700040f8b5c30a621103a524. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:14.515491 1167 tablet_bootstrap.cc:492] T 46cd75ff18b14212b556b9bcb3012dd9 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:14.529105 1176 raft_consensus.cc:2243] T 061f3d38c4604fa4b57e74ff6f6fc7da P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:14.529621 2302 raft_consensus.cc:697] T 40e844fd7ad74680bcbe7bef936c5b9d P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:52:14.530465 2302 consensus_queue.cc:237] T 40e844fd7ad74680bcbe7bef936c5b9d P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:14.531939 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_fd79bf1c016a4671837a7237e117dcf5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:14.532692 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fd79bf1c016a4671837a7237e117dcf5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:14.533193 1176 raft_consensus.cc:2272] T 061f3d38c4604fa4b57e74ff6f6fc7da P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:14.535293 2452 tablet.cc:1722] T 44b518c3a0964a6fa863b2784d80c8a4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 02:52:14.536731 2453 tablet.cc:1722] T 44b518c3a0964a6fa863b2784d80c8a4 P c756528642a543eeacd93cef3e335a7d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 02:52:14.540198 1176 ts_tablet_manager.cc:1916] T 061f3d38c4604fa4b57e74ff6f6fc7da P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 02:52:14.543538 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet eeb75fc46e2a41f78252868ad5256966 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eeb75fc46e2a41f78252868ad5256966 already in progress: creating tablet
I20251029 02:52:14.543615 2201 raft_consensus.cc:493] T 6789bc16664e4b37b86702449ab27855 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:14.543767 2201 raft_consensus.cc:515] T 6789bc16664e4b37b86702449ab27855 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:14.544381 2201 leader_election.cc:290] T 6789bc16664e4b37b86702449ab27855 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
W20251029 02:52:14.544325 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 877e278deb4a4b01ac66b06be0890baf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 877e278deb4a4b01ac66b06be0890baf already in progress: creating tablet
W20251029 02:52:14.545161 512 catalog_manager.cc:4952] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 86fc22bb84794149b598b97df41eb881 because the tablet was not found. No further retry: Not found: Tablet not found: 86fc22bb84794149b598b97df41eb881
I20251029 02:52:14.545395 762 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6789bc16664e4b37b86702449ab27855" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:14.545576 762 raft_consensus.cc:2468] T 6789bc16664e4b37b86702449ab27855 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:14.545984 847 leader_election.cc:304] T 6789bc16664e4b37b86702449ab27855 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:14.546234 2201 raft_consensus.cc:2804] T 6789bc16664e4b37b86702449ab27855 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:14.546326 2201 raft_consensus.cc:493] T 6789bc16664e4b37b86702449ab27855 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:14.546393 2201 raft_consensus.cc:3060] T 6789bc16664e4b37b86702449ab27855 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:14.549213 2201 raft_consensus.cc:515] T 6789bc16664e4b37b86702449ab27855 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:14.550031 2201 leader_election.cc:290] T 6789bc16664e4b37b86702449ab27855 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:14.550662 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6789bc16664e4b37b86702449ab27855" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:14.550848 773 raft_consensus.cc:3060] T 6789bc16664e4b37b86702449ab27855 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:14.553496 773 raft_consensus.cc:2468] T 6789bc16664e4b37b86702449ab27855 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:14.554221 1175 ts_tablet_manager.cc:1929] T 44c0488eba0543078ff4f0b3d69366f2 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:14.554325 1175 log.cc:1199] T 44c0488eba0543078ff4f0b3d69366f2 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/44c0488eba0543078ff4f0b3d69366f2
I20251029 02:52:14.554545 847 leader_election.cc:304] T 6789bc16664e4b37b86702449ab27855 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:14.554736 1175 ts_tablet_manager.cc:1950] T 44c0488eba0543078ff4f0b3d69366f2 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:14.555075 2205 raft_consensus.cc:2804] T 6789bc16664e4b37b86702449ab27855 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:14.555630 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 872) took 5179 ms. Trace:
I20251029 02:52:14.555704 1175 rpcz_store.cc:276] 1029 02:52:09.376041 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:09.376082 (+ 41us) service_pool.cc:225] Handling call
1029 02:52:14.395638 (+5019556us) ts_tablet_manager.cc:1151] Deleting tablet 44c0488eba0543078ff4f0b3d69366f2
1029 02:52:14.395648 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:14.475215 (+ 79567us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:14.554099 (+ 78884us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:14.555611 (+ 1512us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5019419}
I20251029 02:52:14.556022 1175 tablet_replica.cc:333] T afb19ba39399410bb90b6225d7b5b5b1 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:14.556186 1175 raft_consensus.cc:2243] T afb19ba39399410bb90b6225d7b5b5b1 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:14.556319 1175 raft_consensus.cc:2272] T afb19ba39399410bb90b6225d7b5b5b1 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:14.556793 1175 ts_tablet_manager.cc:1916] T afb19ba39399410bb90b6225d7b5b5b1 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:14.556903 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet f93b0c39d7174dabbe018b1546dbc198 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:14 UTC) from {username='slave'} at 127.0.0.1:37626
W20251029 02:52:14.557212 2630 log.cc:927] Time spent T d041e87ceb11441983d0e38b7929c3a4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Append to log took a long time: real 0.086s user 0.000s sys 0.000s
I20251029 02:52:14.558169 1038 tablet_service.cc:1552] Processing DeleteTablet for tablet b81f26fb2739487dbc9af7b6e9848a4a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:14 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:14.558632 1024 tablet_service.cc:1505] Processing CreateTablet for tablet d9ee15c3f45a490bb7a1925e3542b0bb (DEFAULT_TABLE table=default.Table_eff289b2377f44aeb3411568c4854be1 [id=c3d49bcba6da4fbf939a0baa1137e5b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:14.558735 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 44c0488eba0543078ff4f0b3d69366f2 (table default.Table_bc32d7beda4f4957817b20124bf48e9c [id=df809b5c66d64eb1846fa5eb9da11781]) successfully deleted
I20251029 02:52:14.559036 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9ee15c3f45a490bb7a1925e3542b0bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:14.563449 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet 3c3ef62323d643dda28f0ca269274600 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:14 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:14.563623 1038 tablet_service.cc:1505] Processing CreateTablet for tablet 1194e03172954aadb819cc9a6a5050fe (DEFAULT_TABLE table=default.Table_13076d1361cd4ef9ae70f064a6b8679c [id=8c1e0a13dd3440598a3e1845bdc6bb8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:14.564025 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1194e03172954aadb819cc9a6a5050fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:14.564376 1036 tablet_service.cc:1505] Processing CreateTablet for tablet c977cf36300648d6a6a2236d358fc02d (DEFAULT_TABLE table=default.Table_31f9807cb2934a47a4fb2f67f9507053 [id=c3b6a55d9eaa4248aadb99c5ae1a9be5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:14.566044 1052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "40e844fd7ad74680bcbe7bef936c5b9d" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:52:14.566255 1052 raft_consensus.cc:3060] T 40e844fd7ad74680bcbe7bef936c5b9d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:14.566387 2201 raft_consensus.cc:493] T cd597c319af645909b8f3d1212fcb702 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:14.566543 2201 raft_consensus.cc:515] T cd597c319af645909b8f3d1212fcb702 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:14.567140 2201 leader_election.cc:290] T cd597c319af645909b8f3d1212fcb702 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:14.567968 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cd597c319af645909b8f3d1212fcb702" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:14.568176 773 raft_consensus.cc:2468] T cd597c319af645909b8f3d1212fcb702 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:14.568693 1040 tablet_service.cc:1505] Processing CreateTablet for tablet 3cb36eaa1ffe4db2928e87764eb65a35 (DEFAULT_TABLE table=default.Table_08c8b82cbf9a424c8514bea1a6324fb1 [id=de904db373d742ea9fcdd7aac63357bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:14.569057 847 leader_election.cc:304] T cd597c319af645909b8f3d1212fcb702 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:14.569339 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c977cf36300648d6a6a2236d358fc02d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:14.571355 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3cb36eaa1ffe4db2928e87764eb65a35. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:14.572472 1039 tablet_service.cc:1505] Processing CreateTablet for tablet 8c028683f8db4c02a10a37780e20f6dc (DEFAULT_TABLE table=default.Table_2f9f5508a47d486ca4004b3c2b50013d [id=379b8725da8749ab8f218856073e0ce5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:14.572819 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c028683f8db4c02a10a37780e20f6dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:14.573195 2201 raft_consensus.cc:2804] T cd597c319af645909b8f3d1212fcb702 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:14.573287 2201 raft_consensus.cc:493] T cd597c319af645909b8f3d1212fcb702 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:14.573362 2201 raft_consensus.cc:3060] T cd597c319af645909b8f3d1212fcb702 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:14.575647 1030 tablet_service.cc:1505] Processing CreateTablet for tablet 6959a3652c4d4c2d846cd68c3c03dd4b (DEFAULT_TABLE table=default.Table_8dbb47d3c55c40a1b33c0567d0eff749 [id=2093f9393d8a474dbe9a51ae8e336b3b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:14.576001 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6959a3652c4d4c2d846cd68c3c03dd4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:14.576367 1042 tablet_service.cc:1505] Processing CreateTablet for tablet 2fccba4e76b44949a37f04466ef00dfc (DEFAULT_TABLE table=default.Table_4ca293107d23430cbad5c6e53b07da9f [id=a4b75e76165e4a1a8d0c0ca8c97e856d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:14.576519 2425 raft_consensus.cc:493] T 72f9d4408f384e7093316d85f092e1b2 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:14.576716 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2fccba4e76b44949a37f04466ef00dfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:14.576655 2425 raft_consensus.cc:515] T 72f9d4408f384e7093316d85f092e1b2 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:14.577256 2425 leader_election.cc:290] T 72f9d4408f384e7093316d85f092e1b2 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:14.578258 1061 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6789bc16664e4b37b86702449ab27855" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:14.578341 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "72f9d4408f384e7093316d85f092e1b2" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:14.578578 773 raft_consensus.cc:2468] T 72f9d4408f384e7093316d85f092e1b2 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:14.579126 847 leader_election.cc:304] T 72f9d4408f384e7093316d85f092e1b2 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:14.579396 2425 raft_consensus.cc:2804] T 72f9d4408f384e7093316d85f092e1b2 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:14.579463 2425 raft_consensus.cc:493] T 72f9d4408f384e7093316d85f092e1b2 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:14.579520 2425 raft_consensus.cc:3060] T 72f9d4408f384e7093316d85f092e1b2 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:14.580552 1061 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6789bc16664e4b37b86702449ab27855" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:52:14.581836 1033 tablet_service.cc:1505] Processing CreateTablet for tablet 9dd2cb0e1747472084e49a2d83766761 (DEFAULT_TABLE table=default.Table_80d1a54a30fa423f809bef46b9301f29 [id=dbb0431e6f784b9992013b534e99a221]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:14.582149 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9dd2cb0e1747472084e49a2d83766761. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:14.583755 2425 raft_consensus.cc:515] T 72f9d4408f384e7093316d85f092e1b2 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:14.584355 2425 leader_election.cc:290] T 72f9d4408f384e7093316d85f092e1b2 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:14.590692 1053 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cd597c319af645909b8f3d1212fcb702" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:14.593604 1028 tablet_service.cc:1505] Processing CreateTablet for tablet 98509b563a7e4fb6be3cb8a939398de6 (DEFAULT_TABLE table=default.Table_dcf576a6745749cc85aeddaab539d832 [id=013f84786dc8453ea00661a2503cbc99]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:14.593757 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "72f9d4408f384e7093316d85f092e1b2" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:14.593990 773 raft_consensus.cc:3060] T 72f9d4408f384e7093316d85f092e1b2 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:14.594075 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98509b563a7e4fb6be3cb8a939398de6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:14.594568 1035 tablet_service.cc:1505] Processing CreateTablet for tablet e91d6bdca3ad40c48d93e6196644462c (DEFAULT_TABLE table=default.Table_fcfe86e2824842a2aa6e4188c33b4a74 [id=cc3e1f3858f04176acb8c39c65396a5e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:14.594918 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e91d6bdca3ad40c48d93e6196644462c. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:14.599597 849 leader_election.cc:343] T 6789bc16664e4b37b86702449ab27855 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
W20251029 02:52:14.599978 849 leader_election.cc:343] T 6789bc16664e4b37b86702449ab27855 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
W20251029 02:52:14.600250 849 leader_election.cc:343] T cd597c319af645909b8f3d1212fcb702 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:14.600894 1053 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "72f9d4408f384e7093316d85f092e1b2" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:52:14.600736 2425 consensus_queue.cc:1048] T f281191e7c6143f983f11d87f1c9b64e P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251029 02:52:14.601495 1166 tablet_bootstrap.cc:654] T 7bb85a12f573494cbb3118d1947eda27 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
W20251029 02:52:14.602471 849 leader_election.cc:343] T 72f9d4408f384e7093316d85f092e1b2 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:14.603749 1061 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "72f9d4408f384e7093316d85f092e1b2" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:14.604023 2425 raft_consensus.cc:493] T 6142018e44af4d32818adefa2ee43d02 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:14.604146 2425 raft_consensus.cc:515] T 6142018e44af4d32818adefa2ee43d02 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:14.604676 2425 leader_election.cc:290] T 6142018e44af4d32818adefa2ee43d02 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:14.605227 1034 tablet_service.cc:1552] Processing DeleteTablet for tablet 0e9a9dd2f6e4424cb1500682d5a49cc4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:14 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:14.605427 762 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6142018e44af4d32818adefa2ee43d02" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:14.605609 762 raft_consensus.cc:2468] T 6142018e44af4d32818adefa2ee43d02 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:14.606178 1034 tablet_service.cc:1505] Processing CreateTablet for tablet 2a92dd79bc6e42a2a5d85bc523fd5d23 (DEFAULT_TABLE table=default.Table_f861c96de6f04741b28da03e79652f8b [id=260e71eaf2894512bba512e34b3f8431]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:14.606266 847 leader_election.cc:304] T 6142018e44af4d32818adefa2ee43d02 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:14.606851 2425 raft_consensus.cc:2804] T 6142018e44af4d32818adefa2ee43d02 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:14.607132 2425 raft_consensus.cc:493] T 6142018e44af4d32818adefa2ee43d02 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:14.607368 2425 raft_consensus.cc:3060] T 6142018e44af4d32818adefa2ee43d02 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:14.609679 2201 raft_consensus.cc:515] T cd597c319af645909b8f3d1212fcb702 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:14.612247 2201 leader_election.cc:290] T cd597c319af645909b8f3d1212fcb702 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:14.612896 762 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cd597c319af645909b8f3d1212fcb702" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:14.613051 1172 tablet_bootstrap.cc:492] T fcb3db8399aa4cf188b07dda220f7f5b P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:14.613092 762 raft_consensus.cc:3060] T cd597c319af645909b8f3d1212fcb702 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:14.613175 1172 ts_tablet_manager.cc:1403] T fcb3db8399aa4cf188b07dda220f7f5b P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.192s user 0.009s sys 0.001s
I20251029 02:52:14.613847 1172 raft_consensus.cc:359] T fcb3db8399aa4cf188b07dda220f7f5b P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:14.614007 1172 raft_consensus.cc:385] T fcb3db8399aa4cf188b07dda220f7f5b P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:14.614053 1172 raft_consensus.cc:740] T fcb3db8399aa4cf188b07dda220f7f5b P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:14.615691 762 raft_consensus.cc:2468] T cd597c319af645909b8f3d1212fcb702 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:14.616600 762 raft_consensus.cc:1275] T 378f549ff7364d018d8b9a5ef01ac878 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Refusing update from remote peer c756528642a543eeacd93cef3e335a7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251029 02:52:14.617367 847 leader_election.cc:304] T cd597c319af645909b8f3d1212fcb702 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:14.617710 2426 raft_consensus.cc:2804] T cd597c319af645909b8f3d1212fcb702 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:14.618930 2302 consensus_queue.cc:1048] T 378f549ff7364d018d8b9a5ef01ac878 P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20251029 02:52:14.632383 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a92dd79bc6e42a2a5d85bc523fd5d23. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:14.633857 2469 tablet.cc:1722] T aacb6f4b9284457bb6607e3a85d5c809 P b495a49349324f7d81908f2da700f0d9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20251029 02:52:14.639911 2637 raft_consensus.cc:493] T ad214c870ee649279ed7321c5b69cbee P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:14.640098 2637 raft_consensus.cc:515] T ad214c870ee649279ed7321c5b69cbee P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:14.634722 773 raft_consensus.cc:2468] T 72f9d4408f384e7093316d85f092e1b2 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:14.640733 2637 leader_election.cc:290] T ad214c870ee649279ed7321c5b69cbee P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:52:14.641381 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ad214c870ee649279ed7321c5b69cbee" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:14.642030 711 leader_election.cc:343] T ad214c870ee649279ed7321c5b69cbee P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:14.632310 1025 tablet_service.cc:1505] Processing CreateTablet for tablet 44a0c058700040f8b5c30a621103a524 (DEFAULT_TABLE table=default.Table_8aa16ebd94174be79b8fd9d2206599a1 [id=300df93a696b407cbf826cfb9ca66919]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:14.643095 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44a0c058700040f8b5c30a621103a524. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:14.639803 762 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "58dde9397b434b089ade9968f369006a" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
W20251029 02:52:14.643657 989 consensus_peers.cc:597] T e65eaaae43ac435ebd665546de9d68dd P 0cfd4bee16f84b7c8b0ff3ce14db20d5 -> Peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Couldn't send request to peer b495a49349324f7d81908f2da700f0d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e65eaaae43ac435ebd665546de9d68dd. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:14.644101 762 raft_consensus.cc:2393] T 58dde9397b434b089ade9968f369006a P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in current term 1: Already voted for candidate c756528642a543eeacd93cef3e335a7d in this term.
I20251029 02:52:14.648612 847 leader_election.cc:304] T 72f9d4408f384e7093316d85f092e1b2 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5
I20251029 02:52:14.657580 1176 ts_tablet_manager.cc:1929] T 061f3d38c4604fa4b57e74ff6f6fc7da P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:14.662600 2469 tablet.cc:1722] T 378f549ff7364d018d8b9a5ef01ac878 P b495a49349324f7d81908f2da700f0d9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 02:52:14.663343 1176 log.cc:1199] T 061f3d38c4604fa4b57e74ff6f6fc7da P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/061f3d38c4604fa4b57e74ff6f6fc7da
I20251029 02:52:14.659960 2201 raft_consensus.cc:2804] T 72f9d4408f384e7093316d85f092e1b2 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:14.667725 2611 raft_consensus.cc:493] T 589a521f5cc24e6289c37b9cf256bce5 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:14.668378 2611 raft_consensus.cc:515] T 589a521f5cc24e6289c37b9cf256bce5 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:14.679052 1053 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6142018e44af4d32818adefa2ee43d02" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:14.670867 2201 raft_consensus.cc:697] T 72f9d4408f384e7093316d85f092e1b2 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:52:14.684893 2201 consensus_queue.cc:237] T 72f9d4408f384e7093316d85f092e1b2 P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
W20251029 02:52:14.692567 712 leader_election.cc:343] T 23b9b2ea814645eba63e5f6b12cfb966 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:14.693472 712 leader_election.cc:304] T 23b9b2ea814645eba63e5f6b12cfb966 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:14.698148 526 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_345b52c4609e43c78920c2f13d1e7b3f" } new_table_name: "default.Table_a4ec7e45e44448f388d60e853f432b31" modify_external_catalogs: true
I20251029 02:52:14.701359 520 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_7dd1cd855bb74080b6e242847e5faac6" } modify_external_catalogs: true
I20251029 02:52:14.701536 520 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_7dd1cd855bb74080b6e242847e5faac6" } modify_external_catalogs: true
I20251029 02:52:14.704142 523 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet c1222e30bc4141b3ae686d80053c4ef5 (table default.Table_7125184a813c44d997ab9d948a1156bd [id=31801defa6db4ca7a08fbf7f029b65c1]). Expected version 3 got 0
I20251029 02:52:14.705106 523 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_c0a7b43b54fb41c6ab5347fdb4ea3cf2" } modify_external_catalogs: true
I20251029 02:52:14.705300 523 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_c0a7b43b54fb41c6ab5347fdb4ea3cf2" } modify_external_catalogs: true
I20251029 02:52:14.708107 519 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_3d4aa5bec3c749139b0692e6cc06b9db" } new_table_name: "default.Table_447e2f76846e454885124e9615bc17c4" modify_external_catalogs: true
I20251029 02:52:14.614269 1172 consensus_queue.cc:260] T fcb3db8399aa4cf188b07dda220f7f5b P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:14.708981 2637 raft_consensus.cc:493] T fcb3db8399aa4cf188b07dda220f7f5b P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:14.709115 2637 raft_consensus.cc:515] T fcb3db8399aa4cf188b07dda220f7f5b P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:14.709939 2637 leader_election.cc:290] T fcb3db8399aa4cf188b07dda220f7f5b P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:14.711256 1172 ts_tablet_manager.cc:1434] T fcb3db8399aa4cf188b07dda220f7f5b P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.098s user 0.001s sys 0.001s
I20251029 02:52:14.711616 1172 tablet_bootstrap.cc:492] T 628a7d0419cb434f8171faf500d585d8 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:14.713299 1172 tablet_bootstrap.cc:654] T 628a7d0419cb434f8171faf500d585d8 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
W20251029 02:52:14.714284 526 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e426b37e4f304fe28e6136f32f1236ad consensus state has no leader
I20251029 02:52:14.718124 526 catalog_manager.cc:5649] T 6dff036b32904ddb9598be4b79504bb6 P b495a49349324f7d81908f2da700f0d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to b495a49349324f7d81908f2da700f0d9 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "b495a49349324f7d81908f2da700f0d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } health_report { overall_health: HEALTHY } } }
I20251029 02:52:14.718873 526 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet 6dff036b32904ddb9598be4b79504bb6 (table default.Table_0ae102f09fdc41bcbead0148653df9e2 [id=65ec61bf400a4628b98e64e7d43c025c]). Expected version 2 got 0
I20251029 02:52:14.721009 520 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 946f65014d97425aab3851009b742807
I20251029 02:52:14.721590 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_ca88b32b14364d2aab4e1b1480676c12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:14.722396 743 tablet_service.cc:1552] Processing DeleteTablet for tablet 946f65014d97425aab3851009b742807 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:14 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:14.723764 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_7511abddcde942a4adb0a23b3a345d06"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:14.724445 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7511abddcde942a4adb0a23b3a345d06 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:14.726191 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet b6d4f7bf702f4834bc5f4ef2b50b92a7 (table default.Table_c70eb9f284cd49109fd09e430a1cea6c [id=be817eb6ec33406eb316d36575c134fb]) successfully deleted
I20251029 02:52:14.726840 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet e7f846b294cc4eb687a26c8b5931ac19 (table default.Table_049332132e3142ca86d49635656135f5 [id=74c72bc1206b4247b1b1c994ccc402f8]) successfully deleted
I20251029 02:52:14.729948 525 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_cfbb812339124da1b8e6e8a95d816a5e" } modify_external_catalogs: true
I20251029 02:52:14.732002 520 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_e280ab7143204fbd8daa4f2c898617f9" } new_table_name: "default.Table_cf853e7441e7423fb014a38610fc6de6" modify_external_catalogs: true
I20251029 02:52:14.733296 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_75325372892242518c3f0a4b8b657c48"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:14.733915 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_75325372892242518c3f0a4b8b657c48 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:14.729568 524 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_d7efa6c76da9426a89093166507d7bd3" } modify_external_catalogs: true
I20251029 02:52:14.735580 524 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_d7efa6c76da9426a89093166507d7bd3" } modify_external_catalogs: true
I20251029 02:52:14.737421 1175 ts_tablet_manager.cc:1929] T afb19ba39399410bb90b6225d7b5b5b1 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:14.737603 1175 log.cc:1199] T afb19ba39399410bb90b6225d7b5b5b1 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/afb19ba39399410bb90b6225d7b5b5b1
I20251029 02:52:14.738188 1175 ts_tablet_manager.cc:1950] T afb19ba39399410bb90b6225d7b5b5b1 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
W20251029 02:52:14.756127 519 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9099fb43dbec46cd83f2cb3f4796ecc6 consensus state has no leader
I20251029 02:52:14.685760 2302 raft_consensus.cc:493] T 6f8cf09f23984b879b60ab6b27c7d52e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:14.685703 2453 tablet.cc:1722] T 378f549ff7364d018d8b9a5ef01ac878 P c756528642a543eeacd93cef3e335a7d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 02:52:14.756863 2302 raft_consensus.cc:515] T 6f8cf09f23984b879b60ab6b27c7d52e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:14.757434 2302 leader_election.cc:290] T 6f8cf09f23984b879b60ab6b27c7d52e P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:14.757984 1167 tablet_bootstrap.cc:654] T 46cd75ff18b14212b556b9bcb3012dd9 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
W20251029 02:52:14.759683 510 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.62:42343: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20251029 02:52:14.744158 849 leader_election.cc:343] T 72f9d4408f384e7093316d85f092e1b2 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:14.760689 2425 raft_consensus.cc:515] T 6142018e44af4d32818adefa2ee43d02 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
W20251029 02:52:14.761082 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ca88b32b14364d2aab4e1b1480676c12 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:14.761780 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 873) took 5385 ms. Trace:
I20251029 02:52:14.761914 2425 leader_election.cc:290] T 6142018e44af4d32818adefa2ee43d02 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:14.761945 1175 rpcz_store.cc:276] 1029 02:52:09.376350 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:09.376459 (+ 109us) service_pool.cc:225] Handling call
1029 02:52:14.555976 (+5179517us) ts_tablet_manager.cc:1151] Deleting tablet afb19ba39399410bb90b6225d7b5b5b1
1029 02:52:14.555986 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:14.608447 (+ 52461us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:14.737295 (+128848us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:14.761753 (+ 24458us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5179395}
I20251029 02:52:14.762431 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 874) took 5347 ms. Trace:
I20251029 02:52:14.762482 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet afb19ba39399410bb90b6225d7b5b5b1 (table default.Table_d4caf12f7662463c824210f3c9cd7e0f [id=7dfb670d18824f5dbdd321940fea965e]) successfully deleted
I20251029 02:52:14.762655 523 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet df669cf8da7f4c98a3f16423c55c15eb
I20251029 02:52:14.764463 878 tablet_service.cc:1552] Processing DeleteTablet for tablet 40e844fd7ad74680bcbe7bef936c5b9d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:59 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:14.764463 896 tablet_service.cc:1552] Processing DeleteTablet for tablet 946f65014d97425aab3851009b742807 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:14 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:14.764866 881 tablet_service.cc:1552] Processing DeleteTablet for tablet df669cf8da7f4c98a3f16423c55c15eb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:14 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:14.766630 747 tablet_service.cc:1552] Processing DeleteTablet for tablet df669cf8da7f4c98a3f16423c55c15eb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:14 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:14.766806 2611 leader_election.cc:290] T 589a521f5cc24e6289c37b9cf256bce5 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:52:14.766600 1175 rpcz_store.cc:276] 1029 02:52:09.415465 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:09.415537 (+ 72us) service_pool.cc:225] Handling call
1029 02:52:14.762300 (+5346763us) ts_tablet_manager.cc:1151] Deleting tablet 591a6c86fc984bac9fb9eb62e35dc6c1
1029 02:52:14.762312 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:14.762415 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5346561}
W20251029 02:52:14.767444 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 591a6c86fc984bac9fb9eb62e35dc6c1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 591a6c86fc984bac9fb9eb62e35dc6c1 already in progress: creating tablet
I20251029 02:52:14.767771 1175 tablet_replica.cc:333] T 1c53fc1fff1a4bb988beb46ff91fd3be P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:14.767900 524 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet fe9e660de57a46c7ba1d82f25e12b4b0
I20251029 02:52:14.768050 1175 raft_consensus.cc:2243] T 1c53fc1fff1a4bb988beb46ff91fd3be P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:14.768261 1175 raft_consensus.cc:2272] T 1c53fc1fff1a4bb988beb46ff91fd3be P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:14.768678 1054 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dff036b32904ddb9598be4b79504bb6" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:14.769362 1175 ts_tablet_manager.cc:1916] T 1c53fc1fff1a4bb988beb46ff91fd3be P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:14.769784 524 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_482767bf59bf442a99a8675f678afd06" } new_table_name: "default.Table_0b08a2dc1da148a9973a790b325f302a" modify_external_catalogs: true
I20251029 02:52:14.770193 881 tablet_service.cc:1552] Processing DeleteTablet for tablet fe9e660de57a46c7ba1d82f25e12b4b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:14 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:14.772311 1058 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ad214c870ee649279ed7321c5b69cbee" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:14.772887 747 tablet_service.cc:1552] Processing DeleteTablet for tablet fe9e660de57a46c7ba1d82f25e12b4b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:14 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:14.775013 1033 tablet_service.cc:1552] Processing DeleteTablet for tablet 946f65014d97425aab3851009b742807 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:14 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:14.775194 525 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet e1d00ea3dd3744ffa5258bbe0095851c
W20251029 02:52:14.776803 520 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d24fe99a1fb54476b90da2c22ff5d383 consensus state has no leader
I20251029 02:52:14.777557 881 tablet_service.cc:1552] Processing DeleteTablet for tablet e1d00ea3dd3744ffa5258bbe0095851c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:14 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:14.777575 519 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_ef95e34ffbdf43f9bbe0074cee266ac1" } new_table_name: "default.Table_230b5a9dc63d41fb95e7f9d048bb5d6f" modify_external_catalogs: true
W20251029 02:52:14.755411 847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:38481, user_credentials={real_user=slave}} blocked reactor thread for 80438.2us
I20251029 02:52:14.755548 2426 raft_consensus.cc:697] T cd597c319af645909b8f3d1212fcb702 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:52:14.778929 2426 consensus_queue.cc:237] T cd597c319af645909b8f3d1212fcb702 P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:14.780388 747 tablet_service.cc:1552] Processing DeleteTablet for tablet e1d00ea3dd3744ffa5258bbe0095851c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:14 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:14.782071 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6142018e44af4d32818adefa2ee43d02" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:14.782274 772 raft_consensus.cc:3060] T 6142018e44af4d32818adefa2ee43d02 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:14.782816 523 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_0c7029666c80452b8fae6945da658ec7" } modify_external_catalogs: true
I20251029 02:52:14.783056 523 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_0c7029666c80452b8fae6945da658ec7" } modify_external_catalogs: true
I20251029 02:52:14.785040 1176 ts_tablet_manager.cc:1950] T 061f3d38c4604fa4b57e74ff6f6fc7da P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:14.785883 1053 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cd597c319af645909b8f3d1212fcb702" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:52:14.785977 1167 tablet_bootstrap.cc:492] T 46cd75ff18b14212b556b9bcb3012dd9 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:14.786093 1167 ts_tablet_manager.cc:1403] T 46cd75ff18b14212b556b9bcb3012dd9 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.273s user 0.003s sys 0.003s
I20251029 02:52:14.792757 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fcb3db8399aa4cf188b07dda220f7f5b" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:14.793401 711 leader_election.cc:343] T fcb3db8399aa4cf188b07dda220f7f5b P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:14.793642 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "589a521f5cc24e6289c37b9cf256bce5" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:14.794322 711 leader_election.cc:343] T 589a521f5cc24e6289c37b9cf256bce5 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
W20251029 02:52:14.796176 849 leader_election.cc:343] T 6142018e44af4d32818adefa2ee43d02 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
W20251029 02:52:14.796521 849 leader_election.cc:343] T cd597c319af645909b8f3d1212fcb702 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
W20251029 02:52:14.796838 712 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:45003, user_credentials={real_user=slave}} blocked reactor thread for 104386us
I20251029 02:52:14.802419 1054 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6dff036b32904ddb9598be4b79504bb6" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:52:14.802973 1167 raft_consensus.cc:359] T 46cd75ff18b14212b556b9bcb3012dd9 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:14.804039 1167 raft_consensus.cc:385] T 46cd75ff18b14212b556b9bcb3012dd9 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:14.804167 1167 raft_consensus.cc:740] T 46cd75ff18b14212b556b9bcb3012dd9 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:14.797020 2611 raft_consensus.cc:2749] T 23b9b2ea814645eba63e5f6b12cfb966 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:14.807845 1167 consensus_queue.cc:260] T 46cd75ff18b14212b556b9bcb3012dd9 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:14.808475 1167 ts_tablet_manager.cc:1434] T 46cd75ff18b14212b556b9bcb3012dd9 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.022s user 0.001s sys 0.001s
I20251029 02:52:14.808808 1167 tablet_bootstrap.cc:492] T d29cc2a0c8b84e3ebadc7f571d666847 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:14.810292 1167 tablet_bootstrap.cc:654] T d29cc2a0c8b84e3ebadc7f571d666847 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:14.813179 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1012) took 3284 ms. Trace:
I20251029 02:52:14.813289 1176 rpcz_store.cc:276] 1029 02:52:11.528515 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:11.528553 (+ 38us) service_pool.cc:225] Handling call
1029 02:52:14.526779 (+2998226us) ts_tablet_manager.cc:1151] Deleting tablet 061f3d38c4604fa4b57e74ff6f6fc7da
1029 02:52:14.526792 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:14.555070 (+ 28278us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:14.656073 (+101003us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:14.813157 (+157084us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2943017}
I20251029 02:52:14.813606 1176 tablet_replica.cc:333] T 30d03e1e42d34242bd354f9e9ec4f127 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:14.813851 1176 raft_consensus.cc:2243] T 30d03e1e42d34242bd354f9e9ec4f127 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:14.814215 1176 raft_consensus.cc:2272] T 30d03e1e42d34242bd354f9e9ec4f127 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:14.814818 1176 ts_tablet_manager.cc:1916] T 30d03e1e42d34242bd354f9e9ec4f127 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 02:52:14.815937 712 leader_election.cc:343] T 6dff036b32904ddb9598be4b79504bb6 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
W20251029 02:52:14.816318 712 leader_election.cc:343] T ad214c870ee649279ed7321c5b69cbee P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:14.816490 712 leader_election.cc:304] T ad214c870ee649279ed7321c5b69cbee P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:14.816828 1053 raft_consensus.cc:1275] T 378f549ff7364d018d8b9a5ef01ac878 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Refusing update from remote peer c756528642a543eeacd93cef3e335a7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251029 02:52:14.817801 2611 raft_consensus.cc:2749] T ad214c870ee649279ed7321c5b69cbee P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:14.827646 2469 tablet.cc:1722] T c1222e30bc4141b3ae686d80053c4ef5 P b495a49349324f7d81908f2da700f0d9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251029 02:52:14.829667 2453 tablet.cc:1722] T c1222e30bc4141b3ae686d80053c4ef5 P c756528642a543eeacd93cef3e335a7d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251029 02:52:14.837658 2205 raft_consensus.cc:697] T 6789bc16664e4b37b86702449ab27855 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:52:14.838191 2205 consensus_queue.cc:237] T 6789bc16664e4b37b86702449ab27855 P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:14.838723 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f8cf09f23984b879b60ab6b27c7d52e" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:14.839229 773 raft_consensus.cc:2468] T 6f8cf09f23984b879b60ab6b27c7d52e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:14.846287 1172 tablet_bootstrap.cc:492] T 628a7d0419cb434f8171faf500d585d8 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:14.846618 1172 ts_tablet_manager.cc:1403] T 628a7d0419cb434f8171faf500d585d8 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.135s user 0.007s sys 0.000s
I20251029 02:52:14.849555 1172 raft_consensus.cc:359] T 628a7d0419cb434f8171faf500d585d8 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:14.850584 1052 raft_consensus.cc:2468] T 40e844fd7ad74680bcbe7bef936c5b9d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
W20251029 02:52:14.853328 524 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cb9856e5fe244d3e91f30f5e1d276d18 consensus state has no leader
I20251029 02:52:14.853801 526 catalog_manager.cc:6194] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: default.Table_8c5d559a2d7f427094b719d1c0b6cff4 [id=0e6e6f5dc35241c5a04c9d6cf4c39b26] alter complete (version 1)
W20251029 02:52:14.854413 519 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 636e9fe4eeab45749368cd1eb9d48b03 consensus state has no leader
I20251029 02:52:14.857129 523 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet d48d9178108b4eb59062d29b40ba0c11
I20251029 02:52:14.849824 1172 raft_consensus.cc:385] T 628a7d0419cb434f8171faf500d585d8 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:14.858038 881 tablet_service.cc:1552] Processing DeleteTablet for tablet d48d9178108b4eb59062d29b40ba0c11 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:14 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:14.858225 1172 raft_consensus.cc:740] T 628a7d0419cb434f8171faf500d585d8 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:14.858361 747 tablet_service.cc:1552] Processing DeleteTablet for tablet d48d9178108b4eb59062d29b40ba0c11 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:14 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:14.858562 1172 consensus_queue.cc:260] T 628a7d0419cb434f8171faf500d585d8 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:14.861142 772 raft_consensus.cc:2468] T 6142018e44af4d32818adefa2ee43d02 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:14.865335 747 tablet_service.cc:1552] Processing DeleteTablet for tablet fcb3db8399aa4cf188b07dda220f7f5b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:59 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:14.867053 1172 ts_tablet_manager.cc:1434] T 628a7d0419cb434f8171faf500d585d8 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.018s user 0.002s sys 0.000s
I20251029 02:52:14.867085 1040 tablet_service.cc:1552] Processing DeleteTablet for tablet df669cf8da7f4c98a3f16423c55c15eb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:14 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:14.867862 1040 tablet_service.cc:1552] Processing DeleteTablet for tablet fe9e660de57a46c7ba1d82f25e12b4b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:14 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:14.868602 1040 tablet_service.cc:1552] Processing DeleteTablet for tablet e1d00ea3dd3744ffa5258bbe0095851c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:14 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:14.876391 747 tablet_service.cc:1552] Processing DeleteTablet for tablet 628a7d0419cb434f8171faf500d585d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:02 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:14.876717 1172 tablet_bootstrap.cc:492] T f4729ecd182042c2a10f67ca25224101 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:14.878257 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet d48d9178108b4eb59062d29b40ba0c11 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:14 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:14.878700 989 leader_election.cc:304] T 58dde9397b434b089ade9968f369006a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5; no voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:14.879056 2188 raft_consensus.cc:2749] T 58dde9397b434b089ade9968f369006a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:14.880957 2452 tablet.cc:1722] T adf6e736cb9c4920807ee58682e134ef P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
W20251029 02:52:14.885511 712 leader_election.cc:343] T 6dff036b32904ddb9598be4b79504bb6 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:14.885689 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 061f3d38c4604fa4b57e74ff6f6fc7da (table default.Table_0f40dc35be6d46b3a991e93384fa1e1f [id=7f5f847a87ab464b932a46f3c04e43d6]) successfully deleted
I20251029 02:52:14.888888 1172 tablet_bootstrap.cc:654] T f4729ecd182042c2a10f67ca25224101 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:14.891788 2685 tablet.cc:1722] T adf6e736cb9c4920807ee58682e134ef P c756528642a543eeacd93cef3e335a7d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251029 02:52:14.896073 1172 tablet_bootstrap.cc:492] T f4729ecd182042c2a10f67ca25224101 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:14.897567 1172 ts_tablet_manager.cc:1403] T f4729ecd182042c2a10f67ca25224101 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.026s user 0.005s sys 0.000s
I20251029 02:52:14.899533 1172 raft_consensus.cc:359] T f4729ecd182042c2a10f67ca25224101 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:14.899865 2656 consensus_queue.cc:1048] T 27b08f88b99643d28ac609e45bf8104e P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:14.900456 1172 raft_consensus.cc:385] T f4729ecd182042c2a10f67ca25224101 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:14.900729 1172 raft_consensus.cc:740] T f4729ecd182042c2a10f67ca25224101 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:14.901229 1172 consensus_queue.cc:260] T f4729ecd182042c2a10f67ca25224101 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:14.901087 847 leader_election.cc:304] T 6f8cf09f23984b879b60ab6b27c7d52e P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:14.904210 2656 raft_consensus.cc:2804] T 6f8cf09f23984b879b60ab6b27c7d52e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:14.904294 2656 raft_consensus.cc:493] T 6f8cf09f23984b879b60ab6b27c7d52e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:14.904346 2656 raft_consensus.cc:3060] T 6f8cf09f23984b879b60ab6b27c7d52e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:14.907137 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_7533d0093ad94ff6b065550375df858a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:14.908092 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7533d0093ad94ff6b065550375df858a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:14.910022 847 leader_election.cc:304] T 6142018e44af4d32818adefa2ee43d02 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:14.910596 2201 raft_consensus.cc:2804] T 6142018e44af4d32818adefa2ee43d02 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:14.910714 2201 raft_consensus.cc:697] T 6142018e44af4d32818adefa2ee43d02 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:52:14.911005 2201 consensus_queue.cc:237] T 6142018e44af4d32818adefa2ee43d02 P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:14.912228 1172 ts_tablet_manager.cc:1434] T f4729ecd182042c2a10f67ca25224101 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20251029 02:52:14.913326 1172 tablet_bootstrap.cc:492] T 055184c641ba45a588bca611fb24d130 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:14.913549 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_7eb1d904f38342dc85bb06681099d741"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:14.914354 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7eb1d904f38342dc85bb06681099d741 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:14.917232 1172 tablet_bootstrap.cc:654] T 055184c641ba45a588bca611fb24d130 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:14.918591 524 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_a0c3ca13b0804fa399653c68e8bf4335" } new_table_name: "default.Table_32cbbd63da094e50be86cd48ec2be3bc" modify_external_catalogs: true
I20251029 02:52:14.923243 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_a7e37e7a8bba4557a83b0fe3f498df17"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:14.923995 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a7e37e7a8bba4557a83b0fe3f498df17 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:14.925794 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_e20d6df47ebc45ff8cd74e180a2fe940"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:14.926541 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e20d6df47ebc45ff8cd74e180a2fe940 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:14.927922 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_b652acd864e34c469594fa7f6ac8c433"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:14.928565 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b652acd864e34c469594fa7f6ac8c433 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:14.930770 1053 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f8cf09f23984b879b60ab6b27c7d52e" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:14.932775 521 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_3fae9f42aaba465bb13918661650e73a" } modify_external_catalogs: true
I20251029 02:52:14.933009 521 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_3fae9f42aaba465bb13918661650e73a" } modify_external_catalogs: true
I20251029 02:52:14.934648 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_6facbfaed7644f2f9c16f0fe7316e2fc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:14.935250 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6facbfaed7644f2f9c16f0fe7316e2fc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:14.937505 2611 raft_consensus.cc:493] T 84993b2f63e7421290a161dde4bb0607 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:14.937680 2611 raft_consensus.cc:515] T 84993b2f63e7421290a161dde4bb0607 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:14.938472 2611 leader_election.cc:290] T 84993b2f63e7421290a161dde4bb0607 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:52:14.938743 1053 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6142018e44af4d32818adefa2ee43d02" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:52:14.939118 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "84993b2f63e7421290a161dde4bb0607" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:14.939635 711 leader_election.cc:343] T 84993b2f63e7421290a161dde4bb0607 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
W20251029 02:52:14.942986 2647 log.cc:927] Time spent T adf6e736cb9c4920807ee58682e134ef P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Append to log took a long time: real 0.157s user 0.000s sys 0.000s
I20251029 02:52:14.943140 1172 tablet_bootstrap.cc:492] T 055184c641ba45a588bca611fb24d130 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:14.943305 1172 ts_tablet_manager.cc:1403] T 055184c641ba45a588bca611fb24d130 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.030s user 0.002s sys 0.004s
I20251029 02:52:14.944149 1172 raft_consensus.cc:359] T 055184c641ba45a588bca611fb24d130 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:14.944352 1172 raft_consensus.cc:385] T 055184c641ba45a588bca611fb24d130 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:14.944396 1172 raft_consensus.cc:740] T 055184c641ba45a588bca611fb24d130 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:14.946290 1172 consensus_queue.cc:260] T 055184c641ba45a588bca611fb24d130 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:14.947489 1172 ts_tablet_manager.cc:1434] T 055184c641ba45a588bca611fb24d130 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20251029 02:52:14.947759 1172 tablet_bootstrap.cc:492] T defb795026a44e12afb2defb9c9c6db2 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:14.948887 1175 ts_tablet_manager.cc:1929] T 1c53fc1fff1a4bb988beb46ff91fd3be P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251029 02:52:14.948979 1175 log.cc:1199] T 1c53fc1fff1a4bb988beb46ff91fd3be P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/1c53fc1fff1a4bb988beb46ff91fd3be
I20251029 02:52:14.948951 1174 ts_tablet_manager.cc:1929] T 49d9ce32e4f04f11be2ecefc5d523a8b P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:14.949077 1174 log.cc:1199] T 49d9ce32e4f04f11be2ecefc5d523a8b P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/49d9ce32e4f04f11be2ecefc5d523a8b
I20251029 02:52:14.949326 1175 ts_tablet_manager.cc:1950] T 1c53fc1fff1a4bb988beb46ff91fd3be P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:14.949455 1174 ts_tablet_manager.cc:1950] T 49d9ce32e4f04f11be2ecefc5d523a8b P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:14.952097 1176 ts_tablet_manager.cc:1929] T 30d03e1e42d34242bd354f9e9ec4f127 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:14.952211 1176 log.cc:1199] T 30d03e1e42d34242bd354f9e9ec4f127 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/30d03e1e42d34242bd354f9e9ec4f127
I20251029 02:52:14.952534 1176 ts_tablet_manager.cc:1950] T 30d03e1e42d34242bd354f9e9ec4f127 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:14.955181 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 944) took 4430 ms. Trace:
I20251029 02:52:14.955272 1174 rpcz_store.cc:276] 1029 02:52:10.524433 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:10.527207 (+ 2774us) service_pool.cc:225] Handling call
1029 02:52:14.518485 (+3991278us) ts_tablet_manager.cc:1151] Deleting tablet 49d9ce32e4f04f11be2ecefc5d523a8b
1029 02:52:14.518562 (+ 77us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:14.850289 (+331727us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:14.948775 (+ 98486us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:14.955159 (+ 6384us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3916683}
I20251029 02:52:14.955621 1174 tablet_replica.cc:333] T 13ee62fc2339410b9db70dfd3d3c6fa6 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:14.955796 1174 raft_consensus.cc:2243] T 13ee62fc2339410b9db70dfd3d3c6fa6 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:14.955974 1174 raft_consensus.cc:2272] T 13ee62fc2339410b9db70dfd3d3c6fa6 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:14.956475 1174 ts_tablet_manager.cc:1916] T 13ee62fc2339410b9db70dfd3d3c6fa6 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:14.957252 2656 raft_consensus.cc:515] T 6f8cf09f23984b879b60ab6b27c7d52e P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:14.958045 2656 leader_election.cc:290] T 6f8cf09f23984b879b60ab6b27c7d52e P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:14.957988 522 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_654ba01aef8c4aacb0f0aae654401c0a" } new_table_name: "default.Table_50f871884d474551aadff4efdf1ca9e7" modify_external_catalogs: true
I20251029 02:52:14.959205 1172 tablet_bootstrap.cc:654] T defb795026a44e12afb2defb9c9c6db2 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:14.973991 917 raft_consensus.cc:1275] T 6dff036b32904ddb9598be4b79504bb6 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Refusing update from remote peer b495a49349324f7d81908f2da700f0d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251029 02:52:14.974581 2611 consensus_queue.cc:1048] T 6dff036b32904ddb9598be4b79504bb6 P b495a49349324f7d81908f2da700f0d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:14.976112 1172 tablet_bootstrap.cc:492] T defb795026a44e12afb2defb9c9c6db2 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:14.976516 1172 ts_tablet_manager.cc:1403] T defb795026a44e12afb2defb9c9c6db2 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.029s user 0.001s sys 0.006s
I20251029 02:52:14.983359 1172 raft_consensus.cc:359] T defb795026a44e12afb2defb9c9c6db2 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:14.958487 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 49d9ce32e4f04f11be2ecefc5d523a8b (table default.Table_af1ab75f523d41338d4471824997f935 [id=468c76c8c99a4aa59c8b1f6c8e4bfa30]) successfully deleted
I20251029 02:52:14.987404 1172 raft_consensus.cc:385] T defb795026a44e12afb2defb9c9c6db2 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:14.987828 1172 raft_consensus.cc:740] T defb795026a44e12afb2defb9c9c6db2 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:14.988224 1172 consensus_queue.cc:260] T defb795026a44e12afb2defb9c9c6db2 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:14.989315 1172 ts_tablet_manager.cc:1434] T defb795026a44e12afb2defb9c9c6db2 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20251029 02:52:14.989686 1172 tablet_bootstrap.cc:492] T 5815626b05204b578cc4a110f246bf10 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:14.991890 1172 tablet_bootstrap.cc:654] T 5815626b05204b578cc4a110f246bf10 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:14.996804 1053 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f8cf09f23984b879b60ab6b27c7d52e" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:52:14.997825 2655 consensus_queue.cc:1048] T 378f549ff7364d018d8b9a5ef01ac878 P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251029 02:52:14.998136 849 leader_election.cc:343] T 6f8cf09f23984b879b60ab6b27c7d52e P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
W20251029 02:52:14.998405 849 consensus_peers.cc:597] T ae4231daa2594258aa51419c97a2f50a P c756528642a543eeacd93cef3e335a7d -> Peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Couldn't send request to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251029 02:52:15.012809 849 consensus_peers.cc:597] T 27b08f88b99643d28ac609e45bf8104e P c756528642a543eeacd93cef3e335a7d -> Peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Couldn't send request to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:15.018877 2611 raft_consensus.cc:493] T 748777c6cd6e4f49ab49912b719e48bd P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:15.019085 2611 raft_consensus.cc:515] T 748777c6cd6e4f49ab49912b719e48bd P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:15.019793 2611 leader_election.cc:290] T 748777c6cd6e4f49ab49912b719e48bd P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:52:15.020650 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "748777c6cd6e4f49ab49912b719e48bd" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:15.021193 711 leader_election.cc:343] T 748777c6cd6e4f49ab49912b719e48bd P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
W20251029 02:52:15.026804 849 leader_election.cc:343] T 6142018e44af4d32818adefa2ee43d02 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
W20251029 02:52:15.027206 849 leader_election.cc:343] T 6f8cf09f23984b879b60ab6b27c7d52e P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:15.030361 2699 tablet.cc:1722] T 378f549ff7364d018d8b9a5ef01ac878 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 02:52:15.033207 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1013) took 3504 ms. Trace:
I20251029 02:52:15.033344 1176 rpcz_store.cc:276] 1029 02:52:11.528705 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:11.528781 (+ 76us) service_pool.cc:225] Handling call
1029 02:52:14.813559 (+3284778us) ts_tablet_manager.cc:1151] Deleting tablet 30d03e1e42d34242bd354f9e9ec4f127
1029 02:52:14.813568 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:14.854196 (+ 40628us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:14.951983 (+ 97787us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:15.033182 (+ 81199us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3226136}
I20251029 02:52:15.033830 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1031) took 3488 ms. Trace:
I20251029 02:52:15.033907 1176 rpcz_store.cc:276] 1029 02:52:11.545052 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:11.587445 (+ 42393us) service_pool.cc:225] Handling call
1029 02:52:15.033695 (+3446250us) ts_tablet_manager.cc:1151] Deleting tablet 4b841723726e40bda0187fb84bf36a6a
1029 02:52:15.033707 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:15.033815 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3446064}
I20251029 02:52:15.034197 1176 tablet_replica.cc:333] T 2e97d1bc06a24e8d8ba11acb6fd280d8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:15.034407 1176 raft_consensus.cc:2243] T 2e97d1bc06a24e8d8ba11acb6fd280d8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:15.035382 1176 raft_consensus.cc:2272] T 2e97d1bc06a24e8d8ba11acb6fd280d8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:15.036068 1176 ts_tablet_manager.cc:1916] T 2e97d1bc06a24e8d8ba11acb6fd280d8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:15.049852 2469 tablet.cc:1722] T 44b518c3a0964a6fa863b2784d80c8a4 P b495a49349324f7d81908f2da700f0d9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 02:52:15.051546 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 875) took 5459 ms. Trace:
I20251029 02:52:15.051641 1175 rpcz_store.cc:276] 1029 02:52:09.591936 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:09.592002 (+ 66us) service_pool.cc:225] Handling call
1029 02:52:14.767714 (+5175712us) ts_tablet_manager.cc:1151] Deleting tablet 1c53fc1fff1a4bb988beb46ff91fd3be
1029 02:52:14.767727 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:14.850492 (+ 82765us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:14.948749 (+ 98257us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:15.051523 (+102774us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5175507}
I20251029 02:52:15.051990 1175 tablet_replica.cc:333] T d041e87ceb11441983d0e38b7929c3a4 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:15.052191 1175 raft_consensus.cc:2243] T d041e87ceb11441983d0e38b7929c3a4 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:15.052328 1175 raft_consensus.cc:2272] T d041e87ceb11441983d0e38b7929c3a4 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:15.053573 1175 ts_tablet_manager.cc:1916] T d041e87ceb11441983d0e38b7929c3a4 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:15.055601 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 1c53fc1fff1a4bb988beb46ff91fd3be (table default.Table_dd4934aabcd442039db111954d819f6d [id=9abdb27ccc0a48e991b7a4619808fdf8]) successfully deleted
W20251029 02:52:15.065125 847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:38481, user_credentials={real_user=slave}} blocked reactor thread for 155117us
I20251029 02:52:15.065490 2425 consensus_queue.cc:1048] T f281191e7c6143f983f11d87f1c9b64e P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:15.066617 766 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6f8cf09f23984b879b60ab6b27c7d52e" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:15.066808 766 raft_consensus.cc:3060] T 6f8cf09f23984b879b60ab6b27c7d52e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
W20251029 02:52:15.070600 513 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 04528b4b700744aab24f6b1d5cb2787a (table default.Table_0057d5992fe1462eb1839b7acf7486de [id=7aac9c86fb794cd59ab1efb5e34741a5], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 04528b4b700744aab24f6b1d5cb2787a consensus state has no leader
I20251029 02:52:15.085686 1166 tablet_bootstrap.cc:492] T 7bb85a12f573494cbb3118d1947eda27 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:15.085850 1166 ts_tablet_manager.cc:1403] T 7bb85a12f573494cbb3118d1947eda27 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.775s user 0.004s sys 0.011s
I20251029 02:52:15.086813 1166 raft_consensus.cc:359] T 7bb85a12f573494cbb3118d1947eda27 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:15.087090 1166 raft_consensus.cc:385] T 7bb85a12f573494cbb3118d1947eda27 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:15.087174 1166 raft_consensus.cc:740] T 7bb85a12f573494cbb3118d1947eda27 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:15.087432 1166 consensus_queue.cc:260] T 7bb85a12f573494cbb3118d1947eda27 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:15.088683 1166 ts_tablet_manager.cc:1434] T 7bb85a12f573494cbb3118d1947eda27 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20251029 02:52:15.093204 2611 raft_consensus.cc:493] T c081d2eacfac4ca8a0048e2a1c4f8f83 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:15.093359 2611 raft_consensus.cc:515] T c081d2eacfac4ca8a0048e2a1c4f8f83 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:15.094043 2611 leader_election.cc:290] T c081d2eacfac4ca8a0048e2a1c4f8f83 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:52:15.094928 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c081d2eacfac4ca8a0048e2a1c4f8f83" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:15.095491 711 leader_election.cc:343] T c081d2eacfac4ca8a0048e2a1c4f8f83 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:15.102653 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 30d03e1e42d34242bd354f9e9ec4f127 (table default.Table_435f2dfe42134760abc9e8880fa19f32 [id=c6e1c357fbfc49be972748da564cc9a9]) successfully deleted
W20251029 02:52:15.102973 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 4b841723726e40bda0187fb84bf36a6a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4b841723726e40bda0187fb84bf36a6a already in progress: creating tablet
I20251029 02:52:15.103799 2655 consensus_queue.cc:1048] T ae4231daa2594258aa51419c97a2f50a P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:15.103907 2686 tablet.cc:1722] T 6dff036b32904ddb9598be4b79504bb6 P b495a49349324f7d81908f2da700f0d9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 02:52:15.107426 1167 tablet_bootstrap.cc:492] T d29cc2a0c8b84e3ebadc7f571d666847 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:15.107816 1167 ts_tablet_manager.cc:1403] T d29cc2a0c8b84e3ebadc7f571d666847 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.299s user 0.016s sys 0.024s
I20251029 02:52:15.108995 1167 raft_consensus.cc:359] T d29cc2a0c8b84e3ebadc7f571d666847 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:15.114951 1167 raft_consensus.cc:385] T d29cc2a0c8b84e3ebadc7f571d666847 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:15.115087 1167 raft_consensus.cc:740] T d29cc2a0c8b84e3ebadc7f571d666847 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:15.115374 1167 consensus_queue.cc:260] T d29cc2a0c8b84e3ebadc7f571d666847 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:15.115949 1167 ts_tablet_manager.cc:1434] T d29cc2a0c8b84e3ebadc7f571d666847 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20251029 02:52:15.116375 1167 tablet_bootstrap.cc:492] T 748777c6cd6e4f49ab49912b719e48bd P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:15.117864 1167 tablet_bootstrap.cc:654] T 748777c6cd6e4f49ab49912b719e48bd P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:15.124049 2453 tablet.cc:1722] T 6dff036b32904ddb9598be4b79504bb6 P c756528642a543eeacd93cef3e335a7d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 02:52:15.137800 1062 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "748777c6cd6e4f49ab49912b719e48bd" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:15.138231 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c081d2eacfac4ca8a0048e2a1c4f8f83" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:15.137804 1053 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "589a521f5cc24e6289c37b9cf256bce5" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:15.139791 1055 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "84993b2f63e7421290a161dde4bb0607" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:15.140162 1052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fcb3db8399aa4cf188b07dda220f7f5b" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:15.141875 2656 consensus_queue.cc:1048] T 58dde9397b434b089ade9968f369006a P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:15.144253 1167 tablet_bootstrap.cc:492] T 748777c6cd6e4f49ab49912b719e48bd P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:15.144819 1167 ts_tablet_manager.cc:1403] T 748777c6cd6e4f49ab49912b719e48bd P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.029s user 0.001s sys 0.006s
I20251029 02:52:15.145830 1167 raft_consensus.cc:359] T 748777c6cd6e4f49ab49912b719e48bd P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:15.146061 1167 raft_consensus.cc:385] T 748777c6cd6e4f49ab49912b719e48bd P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:15.146142 1167 raft_consensus.cc:740] T 748777c6cd6e4f49ab49912b719e48bd P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:15.146441 1167 consensus_queue.cc:260] T 748777c6cd6e4f49ab49912b719e48bd P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:15.147285 1167 ts_tablet_manager.cc:1434] T 748777c6cd6e4f49ab49912b719e48bd P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251029 02:52:15.147629 1167 tablet_bootstrap.cc:492] T c081d2eacfac4ca8a0048e2a1c4f8f83 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:15.149485 2637 raft_consensus.cc:493] T f4729ecd182042c2a10f67ca25224101 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:15.149670 2637 raft_consensus.cc:515] T f4729ecd182042c2a10f67ca25224101 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:15.150470 2637 leader_election.cc:290] T f4729ecd182042c2a10f67ca25224101 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:52:15.158783 766 raft_consensus.cc:2468] T 6f8cf09f23984b879b60ab6b27c7d52e P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:15.160349 2656 raft_consensus.cc:493] T 20a762344c5f46f88481c4ac7ad6e43b P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:15.160480 2656 raft_consensus.cc:515] T 20a762344c5f46f88481c4ac7ad6e43b P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:15.161015 847 leader_election.cc:304] T 6f8cf09f23984b879b60ab6b27c7d52e P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5
I20251029 02:52:15.161167 2656 leader_election.cc:290] T 20a762344c5f46f88481c4ac7ad6e43b P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:15.161253 2426 raft_consensus.cc:2804] T 6f8cf09f23984b879b60ab6b27c7d52e P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:15.161350 2426 raft_consensus.cc:697] T 6f8cf09f23984b879b60ab6b27c7d52e P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:52:15.161566 2426 consensus_queue.cc:237] T 6f8cf09f23984b879b60ab6b27c7d52e P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:15.162958 1166 tablet_bootstrap.cc:492] T 6789bc16664e4b37b86702449ab27855 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:15.165091 1166 tablet_bootstrap.cc:654] T 6789bc16664e4b37b86702449ab27855 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:15.165982 1167 tablet_bootstrap.cc:654] T c081d2eacfac4ca8a0048e2a1c4f8f83 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:15.166229 1061 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "20a762344c5f46f88481c4ac7ad6e43b" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:15.168800 2637 raft_consensus.cc:493] T 628a7d0419cb434f8171faf500d585d8 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:15.169024 2637 raft_consensus.cc:515] T 628a7d0419cb434f8171faf500d585d8 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:15.169799 2637 leader_election.cc:290] T 628a7d0419cb434f8171faf500d585d8 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
W20251029 02:52:15.166704 849 leader_election.cc:343] T 20a762344c5f46f88481c4ac7ad6e43b P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
W20251029 02:52:15.174180 847 leader_election.cc:343] T 20a762344c5f46f88481c4ac7ad6e43b P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 20a762344c5f46f88481c4ac7ad6e43b
I20251029 02:52:15.174325 847 leader_election.cc:304] T 20a762344c5f46f88481c4ac7ad6e43b P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9
I20251029 02:52:15.174564 2656 consensus_queue.cc:1048] T 7bb85a12f573494cbb3118d1947eda27 P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:15.174971 2656 raft_consensus.cc:2749] T 20a762344c5f46f88481c4ac7ad6e43b P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 02:52:15.180080 989 consensus_peers.cc:597] T 2f198c3259864f7ab3625cb7bfae20b0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 -> Peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Couldn't send request to peer b495a49349324f7d81908f2da700f0d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2f198c3259864f7ab3625cb7bfae20b0. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:15.181180 2703 tablet.cc:1722] T adf6e736cb9c4920807ee58682e134ef P b495a49349324f7d81908f2da700f0d9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251029 02:52:15.193347 1052 raft_consensus.cc:3060] T 7bb85a12f573494cbb3118d1947eda27 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:15.197538 2699 tablet.cc:1722] T c1222e30bc4141b3ae686d80053c4ef5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251029 02:52:15.202332 1020 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:37622 (request call id 153) took 1206 ms. Trace:
I20251029 02:52:15.202790 1020 rpcz_store.cc:276] 1029 02:52:13.996150 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:13.996355 (+ 205us) service_pool.cc:225] Handling call
1029 02:52:14.332889 (+336534us) spinlock_profiling.cc:243] Waited 154 ms on lock 0x616000115c54. stack: 00007f34717829f6 00007f34717827b0 00007f347075e3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f347d61c200 00007f34814cc66f 00007f347b8f2c36 00007f347b8f2965 00007f347872127e 00007f347872015e 00007f3478726395 00007f3478726cbf 00007f3478726ae8 0000000000a8867c
1029 02:52:14.353922 (+ 21033us) spinlock_profiling.cc:243] Waited 20.3 ms on lock 0x6160002a2854. stack: 00007f34717829f6 00007f34717827b0 00007f347075e3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f347d61c200 00007f34814cc66f 00007f347b8f2c36 00007f347b8f2965 00007f347872127e 00007f347872015e 00007f3478726395 00007f3478726cbf 00007f3478726ae8 0000000000a8867c
1029 02:52:15.202300 (+848378us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":400696576}
I20251029 02:52:15.203536 2205 consensus_queue.cc:1048] T 58dde9397b434b089ade9968f369006a P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:15.204416 1172 tablet_bootstrap.cc:492] T 5815626b05204b578cc4a110f246bf10 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:15.204591 1172 ts_tablet_manager.cc:1403] T 5815626b05204b578cc4a110f246bf10 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.215s user 0.036s sys 0.023s
I20251029 02:52:15.205750 1174 ts_tablet_manager.cc:1929] T 13ee62fc2339410b9db70dfd3d3c6fa6 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:15.205866 1174 log.cc:1199] T 13ee62fc2339410b9db70dfd3d3c6fa6 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/13ee62fc2339410b9db70dfd3d3c6fa6
I20251029 02:52:15.206249 1174 ts_tablet_manager.cc:1950] T 13ee62fc2339410b9db70dfd3d3c6fa6 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:15.206828 2205 consensus_queue.cc:1048] T 7bb85a12f573494cbb3118d1947eda27 P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:15.212257 2637 raft_consensus.cc:493] T c9e923724ca9402682f13b252dbc37aa P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0cfd4bee16f84b7c8b0ff3ce14db20d5)
I20251029 02:52:15.212705 2637 raft_consensus.cc:515] T c9e923724ca9402682f13b252dbc37aa P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:15.213382 2637 leader_election.cc:290] T c9e923724ca9402682f13b252dbc37aa P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:15.205439 1172 raft_consensus.cc:359] T 5815626b05204b578cc4a110f246bf10 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:15.214891 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4729ecd182042c2a10f67ca25224101" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:15.215343 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "628a7d0419cb434f8171faf500d585d8" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:15.216612 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 948) took 4668 ms. Trace:
I20251029 02:52:15.216802 1018 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:37622 (request call id 152) took 1224 ms. Trace:
I20251029 02:52:15.216917 1018 rpcz_store.cc:276] 1029 02:52:13.992397 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:13.992739 (+ 342us) service_pool.cc:225] Handling call
1029 02:52:14.353465 (+360726us) spinlock_profiling.cc:243] Waited 301 ms on lock 0x616000115c54. stack: 00007f34717829f6 00007f34717827b0 00007f347075e3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f347d61c200 00007f34814cc66f 00007f347b8f2c36 00007f347b8f2965 00007f347872127e 00007f347872015e 00007f3478726395 00007f3478726cbf 00007f3478726ae8 0000000000a8867c
1029 02:52:14.562753 (+209288us) spinlock_profiling.cc:243] Waited 3.55 ms on lock 0x6160000e7830. stack: 00007f34717829f6 00007f34717827b0 00007f347075e3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f347e9a4829 00007f34814cc478 00007f347b8f2c36 00007f347b8f2965 00007f347872127e 00007f347872015e 00007f3478726395 00007f3478726cbf 00007f3478726ae8 0000000000a8867c
1029 02:52:14.643703 (+ 80950us) spinlock_profiling.cc:243] Waited 55.3 ms on lock 0x6160000f5254. stack: 00007f34717829f6 00007f34717827b0 00007f347075e3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f347d61c200 00007f34814cc66f 00007f347b8f2c36 00007f347b8f2965 00007f347872127e 00007f347872015e 00007f3478726395 00007f3478726cbf 00007f3478726ae8 0000000000a8867c
1029 02:52:15.216774 (+573071us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":829357824}
I20251029 02:52:15.216747 1174 rpcz_store.cc:276] 1029 02:52:10.548522 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:10.548738 (+ 216us) service_pool.cc:225] Handling call
1029 02:52:14.955572 (+4406834us) ts_tablet_manager.cc:1151] Deleting tablet 13ee62fc2339410b9db70dfd3d3c6fa6
1029 02:52:14.955581 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:15.151544 (+195963us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:15.205638 (+ 54094us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:15.216595 (+ 10957us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4353764}
I20251029 02:52:15.217994 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 13ee62fc2339410b9db70dfd3d3c6fa6 (table default.Table_cb4697efc8e049be929f4e7a004879d1 [id=b54d7930e083434680f3934fc4023653]) successfully deleted
I20251029 02:52:15.223062 1176 ts_tablet_manager.cc:1929] T 2e97d1bc06a24e8d8ba11acb6fd280d8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:15.223188 1176 log.cc:1199] T 2e97d1bc06a24e8d8ba11acb6fd280d8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/2e97d1bc06a24e8d8ba11acb6fd280d8
I20251029 02:52:15.223585 1176 ts_tablet_manager.cc:1950] T 2e97d1bc06a24e8d8ba11acb6fd280d8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
W20251029 02:52:15.234097 524 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 09567e75ea1140e5ba506a49fbb1d77e consensus state has no leader
I20251029 02:52:15.237238 517 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_17c5072bc19c40b0ac7348fdf90a1e8d" } modify_external_catalogs: true
I20251029 02:52:15.237432 517 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_17c5072bc19c40b0ac7348fdf90a1e8d" } modify_external_catalogs: true
I20251029 02:52:15.240149 518 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_05649dba28434c1a832deacbffb54eb1" } modify_external_catalogs: true
I20251029 02:52:15.240375 518 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_05649dba28434c1a832deacbffb54eb1" } modify_external_catalogs: true
I20251029 02:52:15.242242 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_024e9ca04c014a27bc2b62227d4828d0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:15.242866 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_024e9ca04c014a27bc2b62227d4828d0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:15.244035 521 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 078ace5eada044c88d2462efb2b57aa8
I20251029 02:52:15.244594 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_d8c5abd34fad48fb8ec28152a4b37e38"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:15.245157 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d8c5abd34fad48fb8ec28152a4b37e38 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:15.246803 525 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_97a40219642245b8958e117857092c0a" } modify_external_catalogs: true
I20251029 02:52:15.247018 525 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_97a40219642245b8958e117857092c0a" } modify_external_catalogs: true
I20251029 02:52:15.247604 1166 tablet_bootstrap.cc:492] T 6789bc16664e4b37b86702449ab27855 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:15.247730 1166 ts_tablet_manager.cc:1403] T 6789bc16664e4b37b86702449ab27855 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.085s user 0.003s sys 0.000s
I20251029 02:52:15.248343 519 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet f4729ecd182042c2a10f67ca25224101 (table default.Table_973ef856041949598a23da3dab8b40db [id=1dd2d5488bc647c8b82c53886583b28d]). Expected version 4 got 0
I20251029 02:52:15.248430 1166 raft_consensus.cc:359] T 6789bc16664e4b37b86702449ab27855 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:15.248579 1166 raft_consensus.cc:385] T 6789bc16664e4b37b86702449ab27855 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:15.248622 1166 raft_consensus.cc:740] T 6789bc16664e4b37b86702449ab27855 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:15.248778 1166 consensus_queue.cc:260] T 6789bc16664e4b37b86702449ab27855 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:15.249078 519 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_ab2b0e5b127542b6ab30655507c3e682" } new_table_name: "default.Table_06ebf0bbf7f648bca4075740bb6ff66a" modify_external_catalogs: true
I20251029 02:52:15.249322 1166 ts_tablet_manager.cc:1434] T 6789bc16664e4b37b86702449ab27855 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251029 02:52:15.249572 1166 tablet_bootstrap.cc:492] T 27b08f88b99643d28ac609e45bf8104e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:15.262068 881 tablet_service.cc:1505] Processing CreateTablet for tablet ce30385a98564c73a400e4f271fc574b (DEFAULT_TABLE table=default.Table_0b7ca974a3574a30afa0150eea5fe8bb [id=7e56f55353df4df1b3f4efd53194a4d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.262303 878 tablet_service.cc:1505] Processing CreateTablet for tablet 4c418961f37044c887d278f4e0fe2248 (DEFAULT_TABLE table=default.Table_75bd13ed4bc340139c8219e1de2c4402 [id=f54e15b78cfa4edf8f6f9c07d1f67808]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.262068 896 tablet_service.cc:1505] Processing CreateTablet for tablet 3bd0abf208c54372a8c9504a60988435 (DEFAULT_TABLE table=default.Table_cba115e38a014588b60259fe951f80c9 [id=f8d4856bfa984c39b7800647a4cda5ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.262583 894 tablet_service.cc:1505] Processing CreateTablet for tablet 2354963cd2d444a6a35794d15055122e (DEFAULT_TABLE table=default.Table_863430b577d14e0696d54412522d528f [id=2b2a81028628473bb4c6f6bbad35242d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.262758 885 tablet_service.cc:1505] Processing CreateTablet for tablet cfb266fc49a743e189817bb1330c45a1 (DEFAULT_TABLE table=default.Table_3db2442366f54d5e9b10cff1fddc2091 [id=a398873d2a4847c4a6850b1433d71ffd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.262774 898 tablet_service.cc:1505] Processing CreateTablet for tablet 449fb79ceff04b63a47293cabb499e69 (DEFAULT_TABLE table=default.Table_dd83c199ef234bc2914714ced85fd52f [id=d002212fc3b045309f00258f9b9ae474]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.263396 900 tablet_service.cc:1505] Processing CreateTablet for tablet 380668181abb42e79b46fdb6a3b9e6ad (DEFAULT_TABLE table=default.Table_ce5d7c1b786d4a58aa81cf3ac8e03aa0 [id=06650901dbc4450380d56b8364a44176]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.263722 879 tablet_service.cc:1505] Processing CreateTablet for tablet 46e054ff563f448d86d8b4f84810291b (DEFAULT_TABLE table=default.Table_4ef22a62446e4805a5a9f18357f4142d [id=5b96d586c2564d91b275e847c0b7e8d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.264039 895 tablet_service.cc:1505] Processing CreateTablet for tablet a702bdbac8ca4fbaaedbe108c4765897 (DEFAULT_TABLE table=default.Table_4aceae27cbe54148b726e134b4d22799 [id=2004671094534df7b0f1bad9c01f0196]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.264088 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_b2d77ffcda024d1e80fd8c298ddfd3e6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:15.264387 892 tablet_service.cc:1505] Processing CreateTablet for tablet 435e0ba5652140f4a5672c898646e82a (DEFAULT_TABLE table=default.Table_28d4296f305b4867bf3ef9aac130e5ad [id=83060e839b94481c955bf41ed480ce83]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 02:52:15.264617 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b2d77ffcda024d1e80fd8c298ddfd3e6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:15.264721 891 tablet_service.cc:1505] Processing CreateTablet for tablet daf79b0746fa4033a4297bb30e9a4bba (DEFAULT_TABLE table=default.Table_94100a9aca744e9192f154cb3e467ee1 [id=8e5f615405a9434d911b7571d6d37bce]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.265061 882 tablet_service.cc:1505] Processing CreateTablet for tablet 98ad4304204942c7beb788877dbec0ff (DEFAULT_TABLE table=default.Table_fe999fb018f84d38b900fd1852effeaf [id=93bf60b259564351b6c5be9e2a8a12ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.265385 893 tablet_service.cc:1505] Processing CreateTablet for tablet e9564575e0914392954a92666dd88333 (DEFAULT_TABLE table=default.Table_f88524c2f85e4721931d212083db66ba [id=8120745ce6934d46a69a80ddd04dfe19]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.265703 884 tablet_service.cc:1505] Processing CreateTablet for tablet e77750c41f884e00832e668416917f6a (DEFAULT_TABLE table=default.Table_a2f9a06d661940bbbc587bf4efb7ec28 [id=c0873cb878464260baa238ea4f50720f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.266024 899 tablet_service.cc:1505] Processing CreateTablet for tablet efc45a3b149b4f1d987aa8250ed7191a (DEFAULT_TABLE table=default.Table_7e47b9d463974ab595d765fc15c82ca0 [id=e36e172f28524f8e9bc1f6b3d9d42007]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.266258 888 tablet_service.cc:1552] Processing DeleteTablet for tablet 078ace5eada044c88d2462efb2b57aa8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:14 UTC) from {username='slave'} at 127.0.0.1:35702
W20251029 02:52:15.268177 522 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 79b65cab235141f2916b10723c5a57cd consensus state has no leader
I20251029 02:52:15.268558 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_db4e930bec75474ba682f938dd32ec41"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:15.269197 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_db4e930bec75474ba682f938dd32ec41 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:15.271158 1167 tablet_bootstrap.cc:492] T c081d2eacfac4ca8a0048e2a1c4f8f83 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:15.271299 1167 ts_tablet_manager.cc:1403] T c081d2eacfac4ca8a0048e2a1c4f8f83 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.124s user 0.003s sys 0.003s
I20251029 02:52:15.271512 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1014) took 3742 ms. Trace:
I20251029 02:52:15.271598 1176 rpcz_store.cc:276] 1029 02:52:11.528973 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:11.529042 (+ 69us) service_pool.cc:225] Handling call
1029 02:52:15.034150 (+3505108us) ts_tablet_manager.cc:1151] Deleting tablet 2e97d1bc06a24e8d8ba11acb6fd280d8
1029 02:52:15.034161 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:15.154709 (+120548us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:15.222961 (+ 68252us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:15.271489 (+ 48528us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3443505}
I20251029 02:52:15.272382 1176 tablet_replica.cc:333] T 8048506ced4540a5aee9f79bfc4599bf P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:15.272570 1176 raft_consensus.cc:2243] T 8048506ced4540a5aee9f79bfc4599bf P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:15.272729 1176 raft_consensus.cc:2272] T 8048506ced4540a5aee9f79bfc4599bf P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:15.273257 1176 ts_tablet_manager.cc:1916] T 8048506ced4540a5aee9f79bfc4599bf P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:15.274298 524 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet 6f8cf09f23984b879b60ab6b27c7d52e (table default.Table_96e017f445bc4f1fbebddf65c722b8de [id=98f28a15eced4860ac13a60abdc8e0e2]). Expected version 3 got 0
W20251029 02:52:15.220643 711 leader_election.cc:343] T f4729ecd182042c2a10f67ca25224101 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:15.274627 524 catalog_manager.cc:5649] T cd597c319af645909b8f3d1212fcb702 P c756528642a543eeacd93cef3e335a7d reported cstate change: term changed from 0 to 1, leader changed from <none> to c756528642a543eeacd93cef3e335a7d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "c756528642a543eeacd93cef3e335a7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } health_report { overall_health: UNKNOWN } } }
I20251029 02:52:15.274899 1172 raft_consensus.cc:385] T 5815626b05204b578cc4a110f246bf10 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:15.274979 1172 raft_consensus.cc:740] T 5815626b05204b578cc4a110f246bf10 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
W20251029 02:52:15.274948 711 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:41701, user_credentials={real_user=slave}} blocked reactor thread for 54652.3us
W20251029 02:52:15.275156 711 leader_election.cc:343] T 628a7d0419cb434f8171faf500d585d8 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:15.275174 524 catalog_manager.cc:5649] T 6789bc16664e4b37b86702449ab27855 P c756528642a543eeacd93cef3e335a7d reported cstate change: term changed from 0 to 1, leader changed from <none> to c756528642a543eeacd93cef3e335a7d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "c756528642a543eeacd93cef3e335a7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } health_report { overall_health: UNKNOWN } } }
I20251029 02:52:15.275206 1172 consensus_queue.cc:260] T 5815626b05204b578cc4a110f246bf10 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:15.275511 524 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet 6789bc16664e4b37b86702449ab27855 (table default.Table_9a7d7f4f9073437097aca9f7d6dc665e [id=ab8aeaf90f6c4bc2ad5a98780945c40a]). Expected version 2 got 0
I20251029 02:52:15.275949 747 tablet_service.cc:1505] Processing CreateTablet for tablet ce30385a98564c73a400e4f271fc574b (DEFAULT_TABLE table=default.Table_0b7ca974a3574a30afa0150eea5fe8bb [id=7e56f55353df4df1b3f4efd53194a4d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.276290 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce30385a98564c73a400e4f271fc574b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.276810 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce30385a98564c73a400e4f271fc574b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.222739 1175 ts_tablet_manager.cc:1929] T d041e87ceb11441983d0e38b7929c3a4 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:15.277069 1175 log.cc:1199] T d041e87ceb11441983d0e38b7929c3a4 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/d041e87ceb11441983d0e38b7929c3a4
I20251029 02:52:15.277261 745 tablet_service.cc:1505] Processing CreateTablet for tablet a702bdbac8ca4fbaaedbe108c4765897 (DEFAULT_TABLE table=default.Table_4aceae27cbe54148b726e134b4d22799 [id=2004671094534df7b0f1bad9c01f0196]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.277361 744 tablet_service.cc:1505] Processing CreateTablet for tablet 46e054ff563f448d86d8b4f84810291b (DEFAULT_TABLE table=default.Table_4ef22a62446e4805a5a9f18357f4142d [id=5b96d586c2564d91b275e847c0b7e8d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.277190 754 tablet_service.cc:1505] Processing CreateTablet for tablet 98ad4304204942c7beb788877dbec0ff (DEFAULT_TABLE table=default.Table_fe999fb018f84d38b900fd1852effeaf [id=93bf60b259564351b6c5be9e2a8a12ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.277449 1175 ts_tablet_manager.cc:1950] T d041e87ceb11441983d0e38b7929c3a4 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:15.277592 756 tablet_service.cc:1505] Processing CreateTablet for tablet 380668181abb42e79b46fdb6a3b9e6ad (DEFAULT_TABLE table=default.Table_ce5d7c1b786d4a58aa81cf3ac8e03aa0 [id=06650901dbc4450380d56b8364a44176]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.277607 738 tablet_service.cc:1505] Processing CreateTablet for tablet cfb266fc49a743e189817bb1330c45a1 (DEFAULT_TABLE table=default.Table_3db2442366f54d5e9b10cff1fddc2091 [id=a398873d2a4847c4a6850b1433d71ffd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.278710 739 tablet_service.cc:1505] Processing CreateTablet for tablet 4c418961f37044c887d278f4e0fe2248 (DEFAULT_TABLE table=default.Table_75bd13ed4bc340139c8219e1de2c4402 [id=f54e15b78cfa4edf8f6f9c07d1f67808]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.278789 742 tablet_service.cc:1505] Processing CreateTablet for tablet 449fb79ceff04b63a47293cabb499e69 (DEFAULT_TABLE table=default.Table_dd83c199ef234bc2914714ced85fd52f [id=d002212fc3b045309f00258f9b9ae474]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.277669 743 tablet_service.cc:1505] Processing CreateTablet for tablet 3bd0abf208c54372a8c9504a60988435 (DEFAULT_TABLE table=default.Table_cba115e38a014588b60259fe951f80c9 [id=f8d4856bfa984c39b7800647a4cda5ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.277675 751 tablet_service.cc:1505] Processing CreateTablet for tablet 2354963cd2d444a6a35794d15055122e (DEFAULT_TABLE table=default.Table_863430b577d14e0696d54412522d528f [id=2b2a81028628473bb4c6f6bbad35242d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.279021 740 tablet_service.cc:1505] Processing CreateTablet for tablet 435e0ba5652140f4a5672c898646e82a (DEFAULT_TABLE table=default.Table_28d4296f305b4867bf3ef9aac130e5ad [id=83060e839b94481c955bf41ed480ce83]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.279325 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46e054ff563f448d86d8b4f84810291b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.279377 749 tablet_service.cc:1505] Processing CreateTablet for tablet daf79b0746fa4033a4297bb30e9a4bba (DEFAULT_TABLE table=default.Table_94100a9aca744e9192f154cb3e467ee1 [id=8e5f615405a9434d911b7571d6d37bce]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.279688 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c418961f37044c887d278f4e0fe2248. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.282763 755 tablet_service.cc:1505] Processing CreateTablet for tablet e77750c41f884e00832e668416917f6a (DEFAULT_TABLE table=default.Table_a2f9a06d661940bbbc587bf4efb7ec28 [id=c0873cb878464260baa238ea4f50720f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.282863 748 tablet_service.cc:1505] Processing CreateTablet for tablet e9564575e0914392954a92666dd88333 (DEFAULT_TABLE table=default.Table_f88524c2f85e4721931d212083db66ba [id=8120745ce6934d46a69a80ddd04dfe19]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.282763 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a702bdbac8ca4fbaaedbe108c4765897. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.283123 750 tablet_service.cc:1505] Processing CreateTablet for tablet efc45a3b149b4f1d987aa8250ed7191a (DEFAULT_TABLE table=default.Table_7e47b9d463974ab595d765fc15c82ca0 [id=e36e172f28524f8e9bc1f6b3d9d42007]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.283397 741 tablet_service.cc:1552] Processing DeleteTablet for tablet 078ace5eada044c88d2462efb2b57aa8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:14 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:15.283699 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bd0abf208c54372a8c9504a60988435. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.283758 1172 ts_tablet_manager.cc:1434] T 5815626b05204b578cc4a110f246bf10 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.079s user 0.002s sys 0.000s
I20251029 02:52:15.286769 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2354963cd2d444a6a35794d15055122e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.290840 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cfb266fc49a743e189817bb1330c45a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.291445 1167 raft_consensus.cc:359] T c081d2eacfac4ca8a0048e2a1c4f8f83 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:15.291584 1167 raft_consensus.cc:385] T c081d2eacfac4ca8a0048e2a1c4f8f83 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:15.291625 1167 raft_consensus.cc:740] T c081d2eacfac4ca8a0048e2a1c4f8f83 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:15.291793 1167 consensus_queue.cc:260] T c081d2eacfac4ca8a0048e2a1c4f8f83 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:15.292230 1167 ts_tablet_manager.cc:1434] T c081d2eacfac4ca8a0048e2a1c4f8f83 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.021s user 0.002s sys 0.000s
I20251029 02:52:15.292483 1167 tablet_bootstrap.cc:492] T 84993b2f63e7421290a161dde4bb0607 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:15.293809 1167 tablet_bootstrap.cc:654] T 84993b2f63e7421290a161dde4bb0607 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:15.295162 898 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 449fb79ceff04b63a47293cabb499e69. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.296377 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 380668181abb42e79b46fdb6a3b9e6ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.298733 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46e054ff563f448d86d8b4f84810291b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.299356 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 435e0ba5652140f4a5672c898646e82a. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:15.254441 712 leader_election.cc:343] T 748777c6cd6e4f49ab49912b719e48bd P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:15.299574 712 leader_election.cc:304] T 748777c6cd6e4f49ab49912b719e48bd P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:15.299641 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet daf79b0746fa4033a4297bb30e9a4bba. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.299871 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98ad4304204942c7beb788877dbec0ff. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:15.299868 712 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:45003, user_credentials={real_user=slave}} blocked reactor thread for 45456.9us
W20251029 02:52:15.300056 712 leader_election.cc:343] T c081d2eacfac4ca8a0048e2a1c4f8f83 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:15.300102 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9564575e0914392954a92666dd88333. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.300132 712 leader_election.cc:304] T c081d2eacfac4ca8a0048e2a1c4f8f83 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
W20251029 02:52:15.300426 712 leader_election.cc:343] T 589a521f5cc24e6289c37b9cf256bce5 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:15.300491 712 leader_election.cc:304] T 589a521f5cc24e6289c37b9cf256bce5 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
W20251029 02:52:15.300735 712 leader_election.cc:343] T 84993b2f63e7421290a161dde4bb0607 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:15.300796 712 leader_election.cc:304] T 84993b2f63e7421290a161dde4bb0607 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:15.301047 2637 raft_consensus.cc:2749] T 748777c6cd6e4f49ab49912b719e48bd P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:15.301187 2637 raft_consensus.cc:2749] T c081d2eacfac4ca8a0048e2a1c4f8f83 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:15.301275 2637 raft_consensus.cc:2749] T 589a521f5cc24e6289c37b9cf256bce5 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:15.301344 2637 raft_consensus.cc:2749] T 84993b2f63e7421290a161dde4bb0607 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:15.302821 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e77750c41f884e00832e668416917f6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.303344 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet efc45a3b149b4f1d987aa8250ed7191a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.303493 1174 tablet_replica.cc:333] T 0f80b840ed954230bfbfe7e40127cd2f P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:15.303720 1174 raft_consensus.cc:2243] T 0f80b840ed954230bfbfe7e40127cd2f P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:15.303910 1174 raft_consensus.cc:2272] T 0f80b840ed954230bfbfe7e40127cd2f P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:15.304626 1174 ts_tablet_manager.cc:1916] T 0f80b840ed954230bfbfe7e40127cd2f P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:15.306288 1166 tablet_bootstrap.cc:654] T 27b08f88b99643d28ac609e45bf8104e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:15.306668 2201 raft_consensus.cc:493] T 46cd75ff18b14212b556b9bcb3012dd9 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:15.306799 2201 raft_consensus.cc:515] T 46cd75ff18b14212b556b9bcb3012dd9 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:15.307325 2201 leader_election.cc:290] T 46cd75ff18b14212b556b9bcb3012dd9 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:15.307991 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "46cd75ff18b14212b556b9bcb3012dd9" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:15.308255 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 435e0ba5652140f4a5672c898646e82a. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:15.308476 849 leader_election.cc:343] T 46cd75ff18b14212b556b9bcb3012dd9 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:15.310680 769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "46cd75ff18b14212b556b9bcb3012dd9" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:15.310873 769 raft_consensus.cc:2468] T 46cd75ff18b14212b556b9bcb3012dd9 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:15.311276 847 leader_election.cc:304] T 46cd75ff18b14212b556b9bcb3012dd9 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5
I20251029 02:52:15.311565 2201 raft_consensus.cc:2804] T 46cd75ff18b14212b556b9bcb3012dd9 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:15.311625 2201 raft_consensus.cc:493] T 46cd75ff18b14212b556b9bcb3012dd9 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:15.311678 2201 raft_consensus.cc:3060] T 46cd75ff18b14212b556b9bcb3012dd9 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:15.316642 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 380668181abb42e79b46fdb6a3b9e6ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.322324 2656 raft_consensus.cc:493] T 748777c6cd6e4f49ab49912b719e48bd P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:15.322484 2656 raft_consensus.cc:515] T 748777c6cd6e4f49ab49912b719e48bd P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:15.322783 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2354963cd2d444a6a35794d15055122e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.324142 1058 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4729ecd182042c2a10f67ca25224101" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:15.324659 1058 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "628a7d0419cb434f8171faf500d585d8" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:15.326483 525 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 7ff2399833e546448b85b9caeb5200fb
I20251029 02:52:15.326882 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cfb266fc49a743e189817bb1330c45a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.326975 522 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_771ea2fd4396483ca70f00c7b9fb4c5f" } new_table_name: "default.Table_c0448c09f5bf43269d40a54911bbc375" modify_external_catalogs: true
I20251029 02:52:15.327072 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 449fb79ceff04b63a47293cabb499e69. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.327196 525 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_094b4e33c9654fc7b1b4bf460be40fcd" } new_table_name: "default.Table_9751e24147fb4fb0b7b7030ecd636514" modify_external_catalogs: true
W20251029 02:52:15.328658 519 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 266aaea633bf4c059b9c02f092a474be consensus state has no leader
I20251029 02:52:15.328994 519 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_e0978085f4ba4e798435ad96267706ac" } modify_external_catalogs: true
I20251029 02:52:15.329159 519 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_e0978085f4ba4e798435ad96267706ac" } modify_external_catalogs: true
I20251029 02:52:15.330613 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98ad4304204942c7beb788877dbec0ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.330693 518 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet f66a7af499b446c7a33965c9e37ce460
I20251029 02:52:15.331043 517 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 2c4e78fbd7e645eb8b3be8ac4bfbb425
I20251029 02:52:15.331295 518 master_service.cc:954] ReplaceTablet: received request to replace tablet f93b0c39d7174dabbe018b1546dbc198 from {username='slave'} at 127.0.0.1:35314
W20251029 02:52:15.331813 711 leader_election.cc:343] T c9e923724ca9402682f13b252dbc37aa P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Not found: Tablet not found: c9e923724ca9402682f13b252dbc37aa
I20251029 02:52:15.331859 737 tablet_service.cc:1552] Processing DeleteTablet for tablet 7ff2399833e546448b85b9caeb5200fb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:15 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:15.333922 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bd0abf208c54372a8c9504a60988435. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.335001 737 tablet_service.cc:1552] Processing DeleteTablet for tablet 2c4e78fbd7e645eb8b3be8ac4bfbb425 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:15 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:15.335565 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 2e97d1bc06a24e8d8ba11acb6fd280d8 (table default.Table_24be2b894899463a810d2fe7511d4d6b [id=43e922b1efe44ad9882b5084b22d1560]) successfully deleted
I20251029 02:52:15.338240 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet daf79b0746fa4033a4297bb30e9a4bba. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.338426 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e77750c41f884e00832e668416917f6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.335002 741 tablet_service.cc:1552] Processing DeleteTablet for tablet f66a7af499b446c7a33965c9e37ce460 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:15 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:15.339004 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9564575e0914392954a92666dd88333. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.341820 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet efc45a3b149b4f1d987aa8250ed7191a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.342386 1172 tablet_bootstrap.cc:492] T c45ffc304e73490387ce800b702b5881 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:15.343973 1172 tablet_bootstrap.cc:654] T c45ffc304e73490387ce800b702b5881 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:15.345259 1174 ts_tablet_manager.cc:1929] T 0f80b840ed954230bfbfe7e40127cd2f P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:15.345371 1174 log.cc:1199] T 0f80b840ed954230bfbfe7e40127cd2f P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/0f80b840ed954230bfbfe7e40127cd2f
W20251029 02:52:15.345388 712 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:45003, user_credentials={real_user=slave}} blocked reactor thread for 44649.8us
W20251029 02:52:15.345619 712 consensus_peers.cc:597] T 6dff036b32904ddb9598be4b79504bb6 P b495a49349324f7d81908f2da700f0d9 -> Peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Couldn't send request to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:15.345708 1174 ts_tablet_manager.cc:1950] T 0f80b840ed954230bfbfe7e40127cd2f P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
W20251029 02:52:15.345803 712 leader_election.cc:343] T fcb3db8399aa4cf188b07dda220f7f5b P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:15.345889 712 leader_election.cc:304] T fcb3db8399aa4cf188b07dda220f7f5b P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:15.346419 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 949) took 4797 ms. Trace:
I20251029 02:52:15.346568 2611 raft_consensus.cc:2749] T fcb3db8399aa4cf188b07dda220f7f5b P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:15.346493 1174 rpcz_store.cc:276] 1029 02:52:10.548650 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:10.548913 (+ 263us) service_pool.cc:225] Handling call
1029 02:52:15.274828 (+4725915us) ts_tablet_manager.cc:1151] Deleting tablet 0f80b840ed954230bfbfe7e40127cd2f
1029 02:52:15.303448 (+ 28620us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:15.342088 (+ 38640us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:15.345160 (+ 3072us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:15.346398 (+ 1238us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4672971}
I20251029 02:52:15.346807 1174 tablet_replica.cc:333] T d61beda662d844cdaf67fd9a2e382ea2 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:15.346973 1174 raft_consensus.cc:2243] T d61beda662d844cdaf67fd9a2e382ea2 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:15.347107 1174 raft_consensus.cc:2272] T d61beda662d844cdaf67fd9a2e382ea2 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:15.347568 1174 ts_tablet_manager.cc:1916] T d61beda662d844cdaf67fd9a2e382ea2 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:15.347797 1166 tablet_bootstrap.cc:492] T 27b08f88b99643d28ac609e45bf8104e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:15.347945 1166 ts_tablet_manager.cc:1403] T 27b08f88b99643d28ac609e45bf8104e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.098s user 0.003s sys 0.000s
I20251029 02:52:15.348690 1166 raft_consensus.cc:359] T 27b08f88b99643d28ac609e45bf8104e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:15.350581 1166 raft_consensus.cc:385] T 27b08f88b99643d28ac609e45bf8104e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:15.350893 1166 raft_consensus.cc:740] T 27b08f88b99643d28ac609e45bf8104e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:15.351120 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 888) took 5471 ms. Trace:
I20251029 02:52:15.351210 1175 rpcz_store.cc:276] 1029 02:52:09.879924 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:09.882556 (+ 2632us) service_pool.cc:225] Handling call
1029 02:52:15.051942 (+5169386us) ts_tablet_manager.cc:1151] Deleting tablet d041e87ceb11441983d0e38b7929c3a4
1029 02:52:15.051952 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:15.151531 (+ 99579us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:15.222638 (+ 71107us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:15.351096 (+128458us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5169212}
I20251029 02:52:15.351544 1175 tablet_replica.cc:333] T c9e923724ca9402682f13b252dbc37aa P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:15.351780 1175 raft_consensus.cc:2243] T c9e923724ca9402682f13b252dbc37aa P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:15.351979 1175 raft_consensus.cc:2272] T c9e923724ca9402682f13b252dbc37aa P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:15.352519 1175 ts_tablet_manager.cc:1916] T c9e923724ca9402682f13b252dbc37aa P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:15.354677 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c418961f37044c887d278f4e0fe2248. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.355988 1166 consensus_queue.cc:260] T 27b08f88b99643d28ac609e45bf8104e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
W20251029 02:52:15.365435 522 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2a61593bfbfe4430ac85f8e5c65e56c3 consensus state has no leader
I20251029 02:52:15.365976 1040 tablet_service.cc:1505] Processing CreateTablet for tablet ce30385a98564c73a400e4f271fc574b (DEFAULT_TABLE table=default.Table_0b7ca974a3574a30afa0150eea5fe8bb [id=7e56f55353df4df1b3f4efd53194a4d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.366396 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce30385a98564c73a400e4f271fc574b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.374001 519 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 59fd5608e4c04fa293ca04595a20d858
W20251029 02:52:15.373982 712 leader_election.cc:343] T f4729ecd182042c2a10f67ca25224101 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:15.374095 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet d041e87ceb11441983d0e38b7929c3a4 (table default.Table_6a78c7ef966c490aaa5e73f56d6d194e [id=c54b92541c3047c0887c36d31743e71c]) successfully deleted
I20251029 02:52:15.374120 712 leader_election.cc:304] T f4729ecd182042c2a10f67ca25224101 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:15.374365 2722 raft_consensus.cc:2749] T f4729ecd182042c2a10f67ca25224101 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:15.374902 1166 ts_tablet_manager.cc:1434] T 27b08f88b99643d28ac609e45bf8104e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.027s user 0.001s sys 0.000s
I20251029 02:52:15.377251 749 tablet_service.cc:1552] Processing DeleteTablet for tablet 59fd5608e4c04fa293ca04595a20d858 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:15 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:15.379802 1166 tablet_bootstrap.cc:492] T 20a762344c5f46f88481c4ac7ad6e43b P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:15.381585 1166 tablet_bootstrap.cc:654] T 20a762344c5f46f88481c4ac7ad6e43b P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:15.382396 1040 tablet_service.cc:1505] Processing CreateTablet for tablet 3bd0abf208c54372a8c9504a60988435 (DEFAULT_TABLE table=default.Table_cba115e38a014588b60259fe951f80c9 [id=f8d4856bfa984c39b7800647a4cda5ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.395678 1037 tablet_service.cc:1505] Processing CreateTablet for tablet 926b02920aef457e84253c9bd0497be6 (DEFAULT_TABLE table=default.Table_69fcb07379dc4054b90c1c6d3e9c7985 [id=749e47c05af74ee990ad3513703f3dff]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.395982 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3bd0abf208c54372a8c9504a60988435. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.396178 1037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 926b02920aef457e84253c9bd0497be6. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:15.396481 712 leader_election.cc:343] T 628a7d0419cb434f8171faf500d585d8 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:15.396641 712 leader_election.cc:304] T 628a7d0419cb434f8171faf500d585d8 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
W20251029 02:52:15.397178 712 leader_election.cc:343] T c9e923724ca9402682f13b252dbc37aa P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Not found: Tablet not found: c9e923724ca9402682f13b252dbc37aa
I20251029 02:52:15.397284 712 leader_election.cc:304] T c9e923724ca9402682f13b252dbc37aa P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
W20251029 02:52:15.397478 712 raft_consensus.cc:2686] T c9e923724ca9402682f13b252dbc37aa P b495a49349324f7d81908f2da700f0d9: unable to run election callback: Service unavailable: Thread pool token was shut down
I20251029 02:52:15.382912 1035 tablet_service.cc:1505] Processing CreateTablet for tablet 4c418961f37044c887d278f4e0fe2248 (DEFAULT_TABLE table=default.Table_75bd13ed4bc340139c8219e1de2c4402 [id=f54e15b78cfa4edf8f6f9c07d1f67808]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.398885 1035 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c418961f37044c887d278f4e0fe2248. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.402616 2722 raft_consensus.cc:2749] T 628a7d0419cb434f8171faf500d585d8 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:15.403024 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a702bdbac8ca4fbaaedbe108c4765897. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.383351 1034 tablet_service.cc:1505] Processing CreateTablet for tablet 2354963cd2d444a6a35794d15055122e (DEFAULT_TABLE table=default.Table_863430b577d14e0696d54412522d528f [id=2b2a81028628473bb4c6f6bbad35242d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.403323 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2354963cd2d444a6a35794d15055122e. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:15.406987 847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:38481, user_credentials={real_user=slave}} blocked reactor thread for 95724.8us
I20251029 02:52:15.408000 1174 ts_tablet_manager.cc:1929] T d61beda662d844cdaf67fd9a2e382ea2 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:15.408099 1174 log.cc:1199] T d61beda662d844cdaf67fd9a2e382ea2 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/d61beda662d844cdaf67fd9a2e382ea2
I20251029 02:52:15.383446 1036 tablet_service.cc:1505] Processing CreateTablet for tablet 380668181abb42e79b46fdb6a3b9e6ad (DEFAULT_TABLE table=default.Table_ce5d7c1b786d4a58aa81cf3ac8e03aa0 [id=06650901dbc4450380d56b8364a44176]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.408387 1174 ts_tablet_manager.cc:1950] T d61beda662d844cdaf67fd9a2e382ea2 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:15.408499 1036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 380668181abb42e79b46fdb6a3b9e6ad. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:15.268934 2714 log.cc:927] Time spent T 7bb85a12f573494cbb3118d1947eda27 P b495a49349324f7d81908f2da700f0d9: Append to log took a long time: real 0.070s user 0.000s sys 0.000s
I20251029 02:52:15.383930 1039 tablet_service.cc:1505] Processing CreateTablet for tablet 46e054ff563f448d86d8b4f84810291b (DEFAULT_TABLE table=default.Table_4ef22a62446e4805a5a9f18357f4142d [id=5b96d586c2564d91b275e847c0b7e8d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.411285 1176 ts_tablet_manager.cc:1929] T 8048506ced4540a5aee9f79bfc4599bf P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:15.411425 1176 log.cc:1199] T 8048506ced4540a5aee9f79bfc4599bf P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/8048506ced4540a5aee9f79bfc4599bf
I20251029 02:52:15.384290 1030 tablet_service.cc:1505] Processing CreateTablet for tablet a702bdbac8ca4fbaaedbe108c4765897 (DEFAULT_TABLE table=default.Table_4aceae27cbe54148b726e134b4d22799 [id=2004671094534df7b0f1bad9c01f0196]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.414644 1058 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "748777c6cd6e4f49ab49912b719e48bd" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:15.384579 1029 tablet_service.cc:1505] Processing CreateTablet for tablet 449fb79ceff04b63a47293cabb499e69 (DEFAULT_TABLE table=default.Table_dd83c199ef234bc2914714ced85fd52f [id=d002212fc3b045309f00258f9b9ae474]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.385439 1038 tablet_service.cc:1505] Processing CreateTablet for tablet 435e0ba5652140f4a5672c898646e82a (DEFAULT_TABLE table=default.Table_28d4296f305b4867bf3ef9aac130e5ad [id=83060e839b94481c955bf41ed480ce83]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.384579 1025 tablet_service.cc:1505] Processing CreateTablet for tablet cfb266fc49a743e189817bb1330c45a1 (DEFAULT_TABLE table=default.Table_3db2442366f54d5e9b10cff1fddc2091 [id=a398873d2a4847c4a6850b1433d71ffd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.385756 1031 tablet_service.cc:1505] Processing CreateTablet for tablet daf79b0746fa4033a4297bb30e9a4bba (DEFAULT_TABLE table=default.Table_94100a9aca744e9192f154cb3e467ee1 [id=8e5f615405a9434d911b7571d6d37bce]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.386200 1042 tablet_service.cc:1505] Processing CreateTablet for tablet 98ad4304204942c7beb788877dbec0ff (DEFAULT_TABLE table=default.Table_fe999fb018f84d38b900fd1852effeaf [id=93bf60b259564351b6c5be9e2a8a12ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.393854 1024 tablet_service.cc:1505] Processing CreateTablet for tablet e9564575e0914392954a92666dd88333 (DEFAULT_TABLE table=default.Table_f88524c2f85e4721931d212083db66ba [id=8120745ce6934d46a69a80ddd04dfe19]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.394856 1028 tablet_service.cc:1505] Processing CreateTablet for tablet e77750c41f884e00832e668416917f6a (DEFAULT_TABLE table=default.Table_a2f9a06d661940bbbc587bf4efb7ec28 [id=c0873cb878464260baa238ea4f50720f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.418041 526 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet 27b08f88b99643d28ac609e45bf8104e (table default.Table_a9e50760813844af85a85c5f8d22f8e0 [id=74a2a41c908146ec8e16c6a3227b94f2]). Expected version 1 got 0
I20251029 02:52:15.418421 526 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet 055184c641ba45a588bca611fb24d130 (table default.Table_b96dfc0b5a574a6ab3f16335d08a7f50 [id=aec3db766dee438c89fb2f1939ee7b7b]). Expected version 1 got 0
I20251029 02:52:15.418655 526 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet f4729ecd182042c2a10f67ca25224101 (table default.Table_973ef856041949598a23da3dab8b40db [id=1dd2d5488bc647c8b82c53886583b28d]). Expected version 4 got 0
I20251029 02:52:15.419548 523 catalog_manager.cc:6194] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: default.Table_9e67f2d45e8f44c6bff1b72166c092a2 [id=e0c0cf3d456444c191a32bb180e83798] alter complete (version 2)
I20251029 02:52:15.422099 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 0f80b840ed954230bfbfe7e40127cd2f (table default.Table_5902a11255a049be9dd07acf8b50adda [id=5bd71ce398e84db28f54262cd3c5528b]) successfully deleted
I20251029 02:52:15.395131 1033 tablet_service.cc:1505] Processing CreateTablet for tablet efc45a3b149b4f1d987aa8250ed7191a (DEFAULT_TABLE table=default.Table_7e47b9d463974ab595d765fc15c82ca0 [id=e36e172f28524f8e9bc1f6b3d9d42007]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.411422 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46e054ff563f448d86d8b4f84810291b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.424480 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9564575e0914392954a92666dd88333. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.425647 1175 ts_tablet_manager.cc:1929] T c9e923724ca9402682f13b252dbc37aa P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251029 02:52:15.425742 1175 log.cc:1199] T c9e923724ca9402682f13b252dbc37aa P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/c9e923724ca9402682f13b252dbc37aa
I20251029 02:52:15.426105 1175 ts_tablet_manager.cc:1950] T c9e923724ca9402682f13b252dbc37aa P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:15.426640 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet daf79b0746fa4033a4297bb30e9a4bba. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.427204 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a702bdbac8ca4fbaaedbe108c4765897. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.429033 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 889) took 5540 ms. Trace:
I20251029 02:52:15.429114 1175 rpcz_store.cc:276] 1029 02:52:09.888421 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:09.888480 (+ 59us) service_pool.cc:225] Handling call
1029 02:52:15.351490 (+5463010us) ts_tablet_manager.cc:1151] Deleting tablet c9e923724ca9402682f13b252dbc37aa
1029 02:52:15.351502 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:15.390965 (+ 39463us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:15.425545 (+ 34580us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:15.429008 (+ 3463us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5462760}
I20251029 02:52:15.429420 1175 tablet_replica.cc:333] T 59eee86f249e4d6a9d09d30276dcd659 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:15.429595 1175 raft_consensus.cc:2243] T 59eee86f249e4d6a9d09d30276dcd659 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:15.429728 1175 raft_consensus.cc:2272] T 59eee86f249e4d6a9d09d30276dcd659 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:15.430215 1175 ts_tablet_manager.cc:1916] T 59eee86f249e4d6a9d09d30276dcd659 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:15.430617 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 449fb79ceff04b63a47293cabb499e69. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.430773 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet c9e923724ca9402682f13b252dbc37aa (table default.Table_443f33587b8f428484b99e07355d69ca [id=03a25420ce6a46fc93a9fb753246bc7b]) successfully deleted
I20251029 02:52:15.430817 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98ad4304204942c7beb788877dbec0ff. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:15.431981 849 leader_election.cc:343] T 748777c6cd6e4f49ab49912b719e48bd P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:15.433527 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet efc45a3b149b4f1d987aa8250ed7191a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.434787 748 tablet_service.cc:1505] Processing CreateTablet for tablet 926b02920aef457e84253c9bd0497be6 (DEFAULT_TABLE table=default.Table_69fcb07379dc4054b90c1c6d3e9c7985 [id=749e47c05af74ee990ad3513703f3dff]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.435197 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 926b02920aef457e84253c9bd0497be6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.435349 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cfb266fc49a743e189817bb1330c45a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.436460 526 catalog_manager.cc:6194] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: default.Table_7125184a813c44d997ab9d948a1156bd [id=31801defa6db4ca7a08fbf7f029b65c1] alter complete (version 3)
I20251029 02:52:15.438602 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e77750c41f884e00832e668416917f6a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.442401 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 435e0ba5652140f4a5672c898646e82a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.444221 2649 consensus_queue.cc:1048] T 40e844fd7ad74680bcbe7bef936c5b9d P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:15.444725 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_2d6bca94a5a14e4ea414e1863b335737"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:15.444888 2655 consensus_queue.cc:1048] T 40e844fd7ad74680bcbe7bef936c5b9d P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251029 02:52:15.445276 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2d6bca94a5a14e4ea414e1863b335737 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:15.446657 523 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_b2aa73be4c2a45c6ac352144abf10116" } new_table_name: "default.Table_c9eb146faf5c4b05832bc07f428f85e2" modify_external_catalogs: true
I20251029 02:52:15.448006 740 tablet_service.cc:1505] Processing CreateTablet for tablet 8616fa24a54147b684f50d373a383c2b (DEFAULT_TABLE table=default.Table_5c2a494d41d0490e9c6eb102db44abbe [id=9124af15f5dc476f9bb0c954b326ed00]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.448243 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_07fdb04dcb96444cb939ea7d0843af6b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:15.448421 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8616fa24a54147b684f50d373a383c2b. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:15.448694 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_07fdb04dcb96444cb939ea7d0843af6b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:52:15.451308 847 consensus_peers.cc:597] T d041e87ceb11441983d0e38b7929c3a4 P c756528642a543eeacd93cef3e335a7d -> Peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Couldn't send request to peer b495a49349324f7d81908f2da700f0d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d041e87ceb11441983d0e38b7929c3a4. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:15.459748 755 tablet_service.cc:1505] Processing CreateTablet for tablet 62f4a04568804c3da03393c08676c5da (DEFAULT_TABLE table=default.Table_0cd699a8b604487f89d72e6d948f2bc2 [id=002eddc907784bc4a201dc84341d1949]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.460484 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62f4a04568804c3da03393c08676c5da. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.462922 750 tablet_service.cc:1505] Processing CreateTablet for tablet 06b1df46901343b6b28be4c17dd61bed (DEFAULT_TABLE table=default.Table_4e9d94609b4a4d1f94615dcc9830d717 [id=761ebdd0095c414488aa221cbc99d3bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.465590 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06b1df46901343b6b28be4c17dd61bed. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.467646 1172 tablet_bootstrap.cc:492] T c45ffc304e73490387ce800b702b5881 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:15.468560 1172 ts_tablet_manager.cc:1403] T c45ffc304e73490387ce800b702b5881 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.126s user 0.008s sys 0.002s
I20251029 02:52:15.468109 739 tablet_service.cc:1505] Processing CreateTablet for tablet 0bc634974f1f4cbba8092f1ddff4c0ff (DEFAULT_TABLE table=default.Table_15d2f76cb37642f5a8f3611fbcdd5259 [id=e6cdfdeb066c41cf93aa24d534aa0e12]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.469147 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0bc634974f1f4cbba8092f1ddff4c0ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.469419 751 tablet_service.cc:1505] Processing CreateTablet for tablet c525515e5b3148af80468d7af5fb8b22 (DEFAULT_TABLE table=default.Table_3eb06cb0b47545f2854d25ae8849681f [id=561d9f68e7e04c508ae00e8184a3c2cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.469781 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c525515e5b3148af80468d7af5fb8b22. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.472919 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_357785b823ab474dbb17df63a96f7554"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:15.473285 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_2b0686bf14ba41edafd8823ec10bcae6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:15.473511 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_357785b823ab474dbb17df63a96f7554 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:52:15.473773 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2b0686bf14ba41edafd8823ec10bcae6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:15.472990 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_bdb0ff1c8e1440b89844dc9a076cf5de"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:15.474970 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bdb0ff1c8e1440b89844dc9a076cf5de in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:15.476429 517 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_fcfe86e2824842a2aa6e4188c33b4a74" } modify_external_catalogs: true
I20251029 02:52:15.476456 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_acc016af001143dc8e6dce38c782721f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:15.476622 517 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_fcfe86e2824842a2aa6e4188c33b4a74" } modify_external_catalogs: true
W20251029 02:52:15.476879 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_acc016af001143dc8e6dce38c782721f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:15.477829 518 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_a3886404687640698bd3302e39ee6854" } new_table_name: "default.Table_6e84a60e0ca74916b5b79e569e41e3df" modify_external_catalogs: true
I20251029 02:52:15.481344 756 tablet_service.cc:1505] Processing CreateTablet for tablet 35d8d0f427f943878baf9047793b4180 (DEFAULT_TABLE table=default.Table_44608f8fced543919b531792404e12a6 [id=a5882cf7ac804f86b303193b71df9fe9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.482448 738 tablet_service.cc:1505] Processing CreateTablet for tablet 8fa6a27df46b4cf786470792431cefbc (DEFAULT_TABLE table=default.Table_16b74e4bf0c240649a053af2967ae6bb [id=46a0947dfcd046298db276a6e24935e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.482767 2656 leader_election.cc:290] T 748777c6cd6e4f49ab49912b719e48bd P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:15.482990 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8fa6a27df46b4cf786470792431cefbc. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:15.485533 849 consensus_peers.cc:597] T e7f846b294cc4eb687a26c8b5931ac19 P c756528642a543eeacd93cef3e335a7d -> Peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Couldn't send request to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e7f846b294cc4eb687a26c8b5931ac19. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:15.486383 1172 raft_consensus.cc:359] T c45ffc304e73490387ce800b702b5881 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:15.487135 885 tablet_service.cc:1552] Processing DeleteTablet for tablet 7ff2399833e546448b85b9caeb5200fb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:15 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:15.486569 1172 raft_consensus.cc:385] T c45ffc304e73490387ce800b702b5881 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:15.487381 769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "748777c6cd6e4f49ab49912b719e48bd" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:15.487452 1172 raft_consensus.cc:740] T c45ffc304e73490387ce800b702b5881 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:15.487531 769 raft_consensus.cc:2468] T 748777c6cd6e4f49ab49912b719e48bd P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:15.487713 1172 consensus_queue.cc:260] T c45ffc304e73490387ce800b702b5881 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:15.488217 847 leader_election.cc:304] T 748777c6cd6e4f49ab49912b719e48bd P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5
I20251029 02:52:15.489151 749 tablet_service.cc:1505] Processing CreateTablet for tablet eade12ecfabf48008db86f98767ff31c (DEFAULT_TABLE table=default.Table_81de1a2ccffd4a439e22e14cbb0c7cc3 [id=8347e0af97e7498eac14286866eed42a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.489521 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eade12ecfabf48008db86f98767ff31c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.491667 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35d8d0f427f943878baf9047793b4180. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.492902 2656 raft_consensus.cc:2804] T 748777c6cd6e4f49ab49912b719e48bd P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:15.492973 2656 raft_consensus.cc:493] T 748777c6cd6e4f49ab49912b719e48bd P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:15.493026 2656 raft_consensus.cc:3060] T 748777c6cd6e4f49ab49912b719e48bd P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:15.495292 745 tablet_service.cc:1505] Processing CreateTablet for tablet a3c8b25e6f4849d6aadca322d5365af1 (DEFAULT_TABLE table=default.Table_69cd5bad2bb64a15908025872e382616 [id=084131cd2315487699ac340402b408d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.495636 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3c8b25e6f4849d6aadca322d5365af1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.498677 1172 ts_tablet_manager.cc:1434] T c45ffc304e73490387ce800b702b5881 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20251029 02:52:15.498785 744 tablet_service.cc:1505] Processing CreateTablet for tablet e841d3cc59f1475eac9a521648aee14b (DEFAULT_TABLE table=default.Table_97f3a6c87c274315b8c9651e5e1c1714 [id=34d2e2aaa3fa4efab5980c09ee510b89]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.499173 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e841d3cc59f1475eac9a521648aee14b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.499508 1172 tablet_bootstrap.cc:492] T f93b0c39d7174dabbe018b1546dbc198 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:15.503540 742 tablet_service.cc:1505] Processing CreateTablet for tablet 004eb4e26cd24e49a1931443d6ac7cd0 (DEFAULT_TABLE table=default.Table_f352e3374c0142cb90edcd42caf06ef4 [id=06595399c8e848dfb99b1c0c4a2240eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.503888 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 004eb4e26cd24e49a1931443d6ac7cd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.507108 754 tablet_service.cc:1505] Processing CreateTablet for tablet 818262aee43c4343b864e01f836e76b8 (DEFAULT_TABLE table=default.Table_6fa2b2dfa5c240ba95baf05031690542 [id=d93d8bc567dc49c68409f850376be70e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.507462 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 818262aee43c4343b864e01f836e76b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.512212 738 tablet_service.cc:1505] Processing CreateTablet for tablet 96e7e1b0f14e4e738e7b2d0eaf8d53bb (DEFAULT_TABLE table=default.Table_93495adef2ff4b039f9aebdb6c748b49 [id=19e9b81045c64937adfd882d4bde9f85]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.512629 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96e7e1b0f14e4e738e7b2d0eaf8d53bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.517431 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 947) took 4969 ms. Trace:
I20251029 02:52:15.517539 1174 rpcz_store.cc:276] 1029 02:52:10.548397 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:10.549907 (+ 1510us) service_pool.cc:225] Handling call
1029 02:52:15.346759 (+4796852us) ts_tablet_manager.cc:1151] Deleting tablet d61beda662d844cdaf67fd9a2e382ea2
1029 02:52:15.346769 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:15.404949 (+ 58180us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:15.407916 (+ 2967us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:15.517405 (+109489us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4744863}
I20251029 02:52:15.517874 1174 tablet_replica.cc:333] T 28f7ceb91a34450cad4b5b45f5285b41 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:15.513430 744 tablet_service.cc:1505] Processing CreateTablet for tablet 0573fe69797f48539df5caf4b3777b4d (DEFAULT_TABLE table=default.Table_f90425a19e964b75b94c68952d26ae1e [id=48211a42281c40a99c8cf27174cf6679]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.518065 1174 raft_consensus.cc:2243] T 28f7ceb91a34450cad4b5b45f5285b41 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:15.518237 1174 raft_consensus.cc:2272] T 28f7ceb91a34450cad4b5b45f5285b41 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:15.518801 1174 ts_tablet_manager.cc:1916] T 28f7ceb91a34450cad4b5b45f5285b41 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:15.519083 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0573fe69797f48539df5caf4b3777b4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.521862 1172 tablet_bootstrap.cc:654] T f93b0c39d7174dabbe018b1546dbc198 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:15.529129 739 tablet_service.cc:1505] Processing CreateTablet for tablet 78c7cbd6bde646c69eae1698d1b0f7a9 (DEFAULT_TABLE table=default.Table_7ca8e3e58d62408c8010ea1aeb204a94 [id=9965df24e91c4add863ba6182c7a2277]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.529560 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78c7cbd6bde646c69eae1698d1b0f7a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.530257 894 tablet_service.cc:1552] Processing DeleteTablet for tablet f66a7af499b446c7a33965c9e37ce460 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:15 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:15.532058 748 tablet_service.cc:1505] Processing CreateTablet for tablet 15d41a61ba5943fcbc085943031907c8 (DEFAULT_TABLE table=default.Table_0f200bab93784e9d97c7d001330e303a [id=0809df5782144d1da622b5726db932f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.532545 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15d41a61ba5943fcbc085943031907c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.541225 750 tablet_service.cc:1505] Processing CreateTablet for tablet eb67e9b7430b4b918fd29d3c6faf6ca9 (DEFAULT_TABLE table=default.Table_6aa358a4fc5c4bce9803079e4b61fdff [id=966a15d73ffd4bfab746e01a73224eec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.541369 2656 raft_consensus.cc:515] T 748777c6cd6e4f49ab49912b719e48bd P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:15.541786 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb67e9b7430b4b918fd29d3c6faf6ca9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.542102 2656 leader_election.cc:290] T 748777c6cd6e4f49ab49912b719e48bd P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:15.542589 769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "748777c6cd6e4f49ab49912b719e48bd" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:15.542755 769 raft_consensus.cc:3060] T 748777c6cd6e4f49ab49912b719e48bd P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:15.542725 1058 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "748777c6cd6e4f49ab49912b719e48bd" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
W20251029 02:52:15.543279 849 leader_election.cc:343] T 748777c6cd6e4f49ab49912b719e48bd P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:15.543375 1175 ts_tablet_manager.cc:1929] T 59eee86f249e4d6a9d09d30276dcd659 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:15.543485 1175 log.cc:1199] T 59eee86f249e4d6a9d09d30276dcd659 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/59eee86f249e4d6a9d09d30276dcd659
I20251029 02:52:15.543850 1175 ts_tablet_manager.cc:1950] T 59eee86f249e4d6a9d09d30276dcd659 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:15.544610 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 890) took 5649 ms. Trace:
I20251029 02:52:15.544687 1175 rpcz_store.cc:276] 1029 02:52:09.894989 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:09.951434 (+ 56445us) service_pool.cc:225] Handling call
1029 02:52:15.429377 (+5477943us) ts_tablet_manager.cc:1151] Deleting tablet 59eee86f249e4d6a9d09d30276dcd659
1029 02:52:15.429387 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:15.517444 (+ 88057us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:15.543268 (+ 25824us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:15.544591 (+ 1323us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5477572}
I20251029 02:52:15.545771 769 raft_consensus.cc:2468] T 748777c6cd6e4f49ab49912b719e48bd P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:15.546450 847 leader_election.cc:304] T 748777c6cd6e4f49ab49912b719e48bd P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5
I20251029 02:52:15.547005 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 59eee86f249e4d6a9d09d30276dcd659 (table default.Table_8f1e631e578a4010927f77c77cc29cb0 [id=edac8dce6fa747bda675efdf71adb1d6]) successfully deleted
I20251029 02:52:15.547217 2656 raft_consensus.cc:2804] T 748777c6cd6e4f49ab49912b719e48bd P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:15.547372 1172 tablet_bootstrap.cc:492] T f93b0c39d7174dabbe018b1546dbc198 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:15.547472 2656 raft_consensus.cc:697] T 748777c6cd6e4f49ab49912b719e48bd P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:52:15.547432 755 tablet_service.cc:1505] Processing CreateTablet for tablet 1169fbf6f22b458099f1b0c2cc784d90 (DEFAULT_TABLE table=default.Table_742111c9d1cc4930b57efd12f4190058 [id=00827045f0bf4e89b7aafb89fe9226f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.547830 1172 ts_tablet_manager.cc:1403] T f93b0c39d7174dabbe018b1546dbc198 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.048s user 0.004s sys 0.000s
I20251029 02:52:15.547938 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1169fbf6f22b458099f1b0c2cc784d90. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.548869 1172 raft_consensus.cc:359] T f93b0c39d7174dabbe018b1546dbc198 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:15.549101 1172 raft_consensus.cc:385] T f93b0c39d7174dabbe018b1546dbc198 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:15.549188 1172 raft_consensus.cc:740] T f93b0c39d7174dabbe018b1546dbc198 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:15.549546 1172 consensus_queue.cc:260] T f93b0c39d7174dabbe018b1546dbc198 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:15.550298 2718 raft_consensus.cc:493] T c32c28b0e74a4af3a0e69bbf24ab1a1f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b495a49349324f7d81908f2da700f0d9)
I20251029 02:52:15.550603 2718 raft_consensus.cc:515] T c32c28b0e74a4af3a0e69bbf24ab1a1f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:15.551043 1172 ts_tablet_manager.cc:1434] T f93b0c39d7174dabbe018b1546dbc198 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20251029 02:52:15.551273 1172 tablet_bootstrap.cc:492] T b0fec06ad37d4c12a7045d424b88cc5e P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:15.551396 2718 leader_election.cc:290] T c32c28b0e74a4af3a0e69bbf24ab1a1f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:15.555006 2656 consensus_queue.cc:237] T 748777c6cd6e4f49ab49912b719e48bd P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:15.558434 756 tablet_service.cc:1505] Processing CreateTablet for tablet 3d06c88ce8394f37ac8af308b7be042b (DEFAULT_TABLE table=default.Table_e7bb3e262a894c2f84b6edfc3917b30d [id=2cca3b7c50b243b5883b8ac1f353f253]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.558836 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d06c88ce8394f37ac8af308b7be042b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.563520 2201 raft_consensus.cc:515] T 46cd75ff18b14212b556b9bcb3012dd9 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:15.566704 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 910) took 5522 ms. Trace:
I20251029 02:52:15.566843 1175 rpcz_store.cc:276] 1029 02:52:10.044691 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:10.044916 (+ 225us) service_pool.cc:225] Handling call
1029 02:52:15.544977 (+5500061us) ts_tablet_manager.cc:1151] Deleting tablet 23856678b7aa4f9ebf65fb4564a4bf91
1029 02:52:15.544987 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:15.566677 (+ 21690us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5499826}
W20251029 02:52:15.567296 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 23856678b7aa4f9ebf65fb4564a4bf91 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 23856678b7aa4f9ebf65fb4564a4bf91 already in progress: creating tablet
I20251029 02:52:15.570832 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 912) took 5525 ms. Trace:
I20251029 02:52:15.570936 1175 rpcz_store.cc:276] 1029 02:52:10.044862 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:10.050581 (+ 5719us) service_pool.cc:225] Handling call
1029 02:52:15.570676 (+5520095us) ts_tablet_manager.cc:1151] Deleting tablet 43a76b984da3414da701221f87349337
1029 02:52:15.570691 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:15.570812 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5516325}
I20251029 02:52:15.571297 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 919) took 5404 ms. Trace:
I20251029 02:52:15.571363 1175 rpcz_store.cc:276] 1029 02:52:10.167273 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:10.180807 (+ 13534us) service_pool.cc:225] Handling call
1029 02:52:15.571199 (+5390392us) ts_tablet_manager.cc:1151] Deleting tablet 4660fd0534df49e0af43c36fb1122d84
1029 02:52:15.571208 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:15.571279 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5390168}
I20251029 02:52:15.571659 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 922) took 5320 ms. Trace:
I20251029 02:52:15.571718 1172 tablet_bootstrap.cc:654] T b0fec06ad37d4c12a7045d424b88cc5e P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:15.572479 526 catalog_manager.cc:6194] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: default.Table_0a039bfb2e1f458f89ffc3ee93300b8f [id=64650da66d464074828f2dc955045f68] alter complete (version 3)
W20251029 02:52:15.574138 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 43a76b984da3414da701221f87349337 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 43a76b984da3414da701221f87349337 already in progress: creating tablet
W20251029 02:52:15.574393 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 4660fd0534df49e0af43c36fb1122d84 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4660fd0534df49e0af43c36fb1122d84 already in progress: creating tablet
I20251029 02:52:15.583639 885 tablet_service.cc:1505] Processing CreateTablet for tablet 35d8d0f427f943878baf9047793b4180 (DEFAULT_TABLE table=default.Table_44608f8fced543919b531792404e12a6 [id=a5882cf7ac804f86b303193b71df9fe9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.584103 893 tablet_service.cc:1505] Processing CreateTablet for tablet 06b1df46901343b6b28be4c17dd61bed (DEFAULT_TABLE table=default.Table_4e9d94609b4a4d1f94615dcc9830d717 [id=761ebdd0095c414488aa221cbc99d3bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.584367 886 tablet_service.cc:1505] Processing CreateTablet for tablet eade12ecfabf48008db86f98767ff31c (DEFAULT_TABLE table=default.Table_81de1a2ccffd4a439e22e14cbb0c7cc3 [id=8347e0af97e7498eac14286866eed42a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.584497 888 tablet_service.cc:1505] Processing CreateTablet for tablet 0bc634974f1f4cbba8092f1ddff4c0ff (DEFAULT_TABLE table=default.Table_15d2f76cb37642f5a8f3611fbcdd5259 [id=e6cdfdeb066c41cf93aa24d534aa0e12]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.584784 887 tablet_service.cc:1505] Processing CreateTablet for tablet a3c8b25e6f4849d6aadca322d5365af1 (DEFAULT_TABLE table=default.Table_69cd5bad2bb64a15908025872e382616 [id=084131cd2315487699ac340402b408d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.584863 883 tablet_service.cc:1505] Processing CreateTablet for tablet c525515e5b3148af80468d7af5fb8b22 (DEFAULT_TABLE table=default.Table_3eb06cb0b47545f2854d25ae8849681f [id=561d9f68e7e04c508ae00e8184a3c2cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.585602 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet d61beda662d844cdaf67fd9a2e382ea2 (table default.Table_ca4e3b5646244cafa5a0429d5a8f2ac6 [id=481e8e175f364d31bb7dbbaa2df2dad1]) successfully deleted
I20251029 02:52:15.586232 881 tablet_service.cc:1505] Processing CreateTablet for tablet 8616fa24a54147b684f50d373a383c2b (DEFAULT_TABLE table=default.Table_5c2a494d41d0490e9c6eb102db44abbe [id=9124af15f5dc476f9bb0c954b326ed00]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.586681 896 tablet_service.cc:1505] Processing CreateTablet for tablet 62f4a04568804c3da03393c08676c5da (DEFAULT_TABLE table=default.Table_0cd699a8b604487f89d72e6d948f2bc2 [id=002eddc907784bc4a201dc84341d1949]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.587080 890 tablet_service.cc:1505] Processing CreateTablet for tablet 8fa6a27df46b4cf786470792431cefbc (DEFAULT_TABLE table=default.Table_16b74e4bf0c240649a053af2967ae6bb [id=46a0947dfcd046298db276a6e24935e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.588459 2655 raft_consensus.cc:493] T c32c28b0e74a4af3a0e69bbf24ab1a1f P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b495a49349324f7d81908f2da700f0d9)
I20251029 02:52:15.588609 2655 raft_consensus.cc:515] T c32c28b0e74a4af3a0e69bbf24ab1a1f P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
W20251029 02:52:15.589293 523 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d07bfb380e884843ae611380a1d10b36 consensus state has no leader
I20251029 02:52:15.589557 523 master_service.cc:954] ReplaceTablet: received request to replace tablet 18632de3bbbd491a86460497544580c1 from {username='slave'} at 127.0.0.1:35314
I20251029 02:52:15.591063 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_2a4d6457e4db4e9280cd96080b7223b0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:15.591681 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2a4d6457e4db4e9280cd96080b7223b0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:15.592147 894 tablet_service.cc:1505] Processing CreateTablet for tablet 926b02920aef457e84253c9bd0497be6 (DEFAULT_TABLE table=default.Table_69fcb07379dc4054b90c1c6d3e9c7985 [id=749e47c05af74ee990ad3513703f3dff]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.592214 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_68c016f54d7f45818aa4eb2f91c27539"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:15.592643 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_68c016f54d7f45818aa4eb2f91c27539 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:15.592929 878 tablet_service.cc:1552] Processing DeleteTablet for tablet 59fd5608e4c04fa293ca04595a20d858 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:15 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:15.593334 878 tablet_service.cc:1505] Processing CreateTablet for tablet e841d3cc59f1475eac9a521648aee14b (DEFAULT_TABLE table=default.Table_97f3a6c87c274315b8c9651e5e1c1714 [id=34d2e2aaa3fa4efab5980c09ee510b89]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.593451 521 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_27e1e56b7ff0486facdfdd318dc845fc" } modify_external_catalogs: true
I20251029 02:52:15.593641 521 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_27e1e56b7ff0486facdfdd318dc845fc" } modify_external_catalogs: true
I20251029 02:52:15.594039 2201 leader_election.cc:290] T 46cd75ff18b14212b556b9bcb3012dd9 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:15.594623 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "46cd75ff18b14212b556b9bcb3012dd9" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:15.594789 773 raft_consensus.cc:3060] T 46cd75ff18b14212b556b9bcb3012dd9 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:15.595135 522 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_95133cb1a69941d0b8a6321091752822" } modify_external_catalogs: true
I20251029 02:52:15.595314 522 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_95133cb1a69941d0b8a6321091752822" } modify_external_catalogs: true
I20251029 02:52:15.595474 1058 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "46cd75ff18b14212b556b9bcb3012dd9" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
W20251029 02:52:15.595971 849 leader_election.cc:343] T 46cd75ff18b14212b556b9bcb3012dd9 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:15.596714 520 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_69cd5bad2bb64a15908025872e382616" } modify_external_catalogs: true
I20251029 02:52:15.596915 520 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_69cd5bad2bb64a15908025872e382616" } modify_external_catalogs: true
I20251029 02:52:15.597538 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35d8d0f427f943878baf9047793b4180. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.598109 517 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet e91d6bdca3ad40c48d93e6196644462c
I20251029 02:52:15.598948 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06b1df46901343b6b28be4c17dd61bed. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.599118 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eade12ecfabf48008db86f98767ff31c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.599771 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0bc634974f1f4cbba8092f1ddff4c0ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.600008 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3c8b25e6f4849d6aadca322d5365af1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.600559 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c525515e5b3148af80468d7af5fb8b22. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.600708 896 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62f4a04568804c3da03393c08676c5da. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.601029 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8fa6a27df46b4cf786470792431cefbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.601581 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8616fa24a54147b684f50d373a383c2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.571724 1175 rpcz_store.cc:276] 1029 02:52:10.250880 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:10.250936 (+ 56us) service_pool.cc:225] Handling call
1029 02:52:15.571549 (+5320613us) ts_tablet_manager.cc:1151] Deleting tablet eeb75fc46e2a41f78252868ad5256966
1029 02:52:15.571560 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:15.571644 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5320390}
I20251029 02:52:15.604594 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 930) took 5236 ms. Trace:
I20251029 02:52:15.604799 738 tablet_service.cc:1552] Processing DeleteTablet for tablet e91d6bdca3ad40c48d93e6196644462c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:15 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:15.605180 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 926b02920aef457e84253c9bd0497be6. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.604809 1175 rpcz_store.cc:276] 1029 02:52:10.368177 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:10.368228 (+ 51us) service_pool.cc:225] Handling call
1029 02:52:15.604463 (+5236235us) ts_tablet_manager.cc:1151] Deleting tablet 877e278deb4a4b01ac66b06be0890baf
1029 02:52:15.604474 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:15.604571 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5235883}
I20251029 02:52:15.609017 773 raft_consensus.cc:2468] T 46cd75ff18b14212b556b9bcb3012dd9 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:15.609030 884 tablet_service.cc:1552] Processing DeleteTablet for tablet 2c4e78fbd7e645eb8b3be8ac4bfbb425 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:15 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:15.609447 847 leader_election.cc:304] T 46cd75ff18b14212b556b9bcb3012dd9 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5
I20251029 02:52:15.609778 2425 raft_consensus.cc:2804] T 46cd75ff18b14212b556b9bcb3012dd9 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:15.609872 2425 raft_consensus.cc:697] T 46cd75ff18b14212b556b9bcb3012dd9 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:52:15.610080 2425 consensus_queue.cc:237] T 46cd75ff18b14212b556b9bcb3012dd9 P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:15.610687 884 tablet_service.cc:1505] Processing CreateTablet for tablet 004eb4e26cd24e49a1931443d6ac7cd0 (DEFAULT_TABLE table=default.Table_f352e3374c0142cb90edcd42caf06ef4 [id=06595399c8e848dfb99b1c0c4a2240eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.611130 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 004eb4e26cd24e49a1931443d6ac7cd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.611327 899 tablet_service.cc:1552] Processing DeleteTablet for tablet 46cd75ff18b14212b556b9bcb3012dd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:05 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:15.613737 2722 raft_consensus.cc:493] T 055184c641ba45a588bca611fb24d130 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:15.613781 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_9ad0c74be72449c7b3868451dd9ac8d2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:15.613866 2722 raft_consensus.cc:515] T 055184c641ba45a588bca611fb24d130 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
W20251029 02:52:15.614375 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9ad0c74be72449c7b3868451dd9ac8d2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:15.614500 2722 leader_election.cc:290] T 055184c641ba45a588bca611fb24d130 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:52:15.615358 525 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_21ec6b8b68ee4e62ab9808a41c4551c4" } new_table_name: "default.Table_0a33656128144bd98100260c9f0df989" modify_external_catalogs: true
W20251029 02:52:15.616675 518 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7c6b9031ad71444d95df0c33135636b3 consensus state has no leader
W20251029 02:52:15.617167 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet eeb75fc46e2a41f78252868ad5256966 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eeb75fc46e2a41f78252868ad5256966 already in progress: creating tablet
W20251029 02:52:15.617465 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 877e278deb4a4b01ac66b06be0890baf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 877e278deb4a4b01ac66b06be0890baf already in progress: creating tablet
I20251029 02:52:15.619836 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_d7867a1cdf554a9c9f4578435d47cf14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:15.620321 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d7867a1cdf554a9c9f4578435d47cf14 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:15.620344 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e841d3cc59f1475eac9a521648aee14b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.625100 2201 raft_consensus.cc:493] T 97b2ecae572a4b7ea1b9595074556708 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b495a49349324f7d81908f2da700f0d9)
I20251029 02:52:15.625262 2201 raft_consensus.cc:515] T 97b2ecae572a4b7ea1b9595074556708 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:15.626482 2201 leader_election.cc:290] T 97b2ecae572a4b7ea1b9595074556708 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
W20251029 02:52:15.627585 849 leader_election.cc:343] T 97b2ecae572a4b7ea1b9595074556708 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Not found: Tablet not found: 97b2ecae572a4b7ea1b9595074556708
I20251029 02:52:15.628533 900 tablet_service.cc:1505] Processing CreateTablet for tablet 96e7e1b0f14e4e738e7b2d0eaf8d53bb (DEFAULT_TABLE table=default.Table_93495adef2ff4b039f9aebdb6c748b49 [id=19e9b81045c64937adfd882d4bde9f85]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.628974 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96e7e1b0f14e4e738e7b2d0eaf8d53bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.639091 1175 tablet_replica.cc:333] T 67651306f3b44e7eb14dcee9a2007a81 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:15.639320 1175 raft_consensus.cc:2243] T 67651306f3b44e7eb14dcee9a2007a81 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus shutting down.
W20251029 02:52:15.639385 847 leader_election.cc:343] T 97b2ecae572a4b7ea1b9595074556708 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 97b2ecae572a4b7ea1b9595074556708
I20251029 02:52:15.639485 1175 raft_consensus.cc:2272] T 67651306f3b44e7eb14dcee9a2007a81 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:15.639533 847 leader_election.cc:304] T 97b2ecae572a4b7ea1b9595074556708 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9
I20251029 02:52:15.640062 1175 ts_tablet_manager.cc:1916] T 67651306f3b44e7eb14dcee9a2007a81 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:15.640077 2426 raft_consensus.cc:2749] T 97b2ecae572a4b7ea1b9595074556708 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251029 02:52:15.648804 1175 ts_tablet_manager.cc:1929] T 67651306f3b44e7eb14dcee9a2007a81 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:15.649123 1175 log.cc:1199] T 67651306f3b44e7eb14dcee9a2007a81 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/67651306f3b44e7eb14dcee9a2007a81
I20251029 02:52:15.649619 1175 ts_tablet_manager.cc:1950] T 67651306f3b44e7eb14dcee9a2007a81 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:15.650498 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 936) took 5104 ms. Trace:
I20251029 02:52:15.650822 899 tablet_service.cc:1505] Processing CreateTablet for tablet 818262aee43c4343b864e01f836e76b8 (DEFAULT_TABLE table=default.Table_6fa2b2dfa5c240ba95baf05031690542 [id=d93d8bc567dc49c68409f850376be70e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.651650 899 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 818262aee43c4343b864e01f836e76b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.650693 1175 rpcz_store.cc:276] 1029 02:52:10.545894 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:10.545970 (+ 76us) service_pool.cc:225] Handling call
1029 02:52:15.608626 (+5062656us) ts_tablet_manager.cc:1151] Deleting tablet 67651306f3b44e7eb14dcee9a2007a81
1029 02:52:15.639021 (+ 30395us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:15.644723 (+ 5702us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:15.647838 (+ 3115us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:15.650481 (+ 2643us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5054477}
I20251029 02:52:15.654151 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 67651306f3b44e7eb14dcee9a2007a81 (table default.Table_ff796c36eb2c4bb1809f39e1cb68c223 [id=95bbd9adfe994fecab10c9f23c0bedb0]) successfully deleted
I20251029 02:52:15.654233 1175 tablet_replica.cc:333] T 7bb85a12f573494cbb3118d1947eda27 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:15.654414 1175 raft_consensus.cc:2243] T 7bb85a12f573494cbb3118d1947eda27 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:15.654645 1175 raft_consensus.cc:2272] T 7bb85a12f573494cbb3118d1947eda27 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:15.655670 1175 ts_tablet_manager.cc:1916] T 7bb85a12f573494cbb3118d1947eda27 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:15.663066 879 tablet_service.cc:1505] Processing CreateTablet for tablet 0573fe69797f48539df5caf4b3777b4d (DEFAULT_TABLE table=default.Table_f90425a19e964b75b94c68952d26ae1e [id=48211a42281c40a99c8cf27174cf6679]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.663460 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0573fe69797f48539df5caf4b3777b4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.666288 2722 raft_consensus.cc:493] T 32519fbffcef44a890161e5f3043806c P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:15.666437 2722 raft_consensus.cc:515] T 32519fbffcef44a890161e5f3043806c P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:15.666617 523 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 18632de3bbbd491a86460497544580c1
I20251029 02:52:15.667075 523 catalog_manager.cc:6719] ReplaceTablet: tablet 18632de3bbbd491a86460497544580c1 deleted and replaced by tablet c3dabe9d8e534953838ce5f0e3259e9d
I20251029 02:52:15.667124 2722 leader_election.cc:290] T 32519fbffcef44a890161e5f3043806c P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:15.667407 523 master_service.cc:954] ReplaceTablet: received request to replace tablet f1e975b267464e64a5bbe24ef4e308a3 from {username='slave'} at 127.0.0.1:35314
W20251029 02:52:15.669070 525 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0de19a308a534cdaa38f46ab43789322 consensus state has no leader
I20251029 02:52:15.669482 525 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_b717a166304f42e4886196617960e361" } new_table_name: "default.Table_f47c6012c3a343d3baa413d0cd44571c" modify_external_catalogs: true
I20251029 02:52:15.669644 1172 tablet_bootstrap.cc:492] T b0fec06ad37d4c12a7045d424b88cc5e P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:15.669756 1172 ts_tablet_manager.cc:1403] T b0fec06ad37d4c12a7045d424b88cc5e P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.119s user 0.003s sys 0.007s
I20251029 02:52:15.670434 1172 raft_consensus.cc:359] T b0fec06ad37d4c12a7045d424b88cc5e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:15.670611 1172 raft_consensus.cc:385] T b0fec06ad37d4c12a7045d424b88cc5e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:15.670655 1172 raft_consensus.cc:740] T b0fec06ad37d4c12a7045d424b88cc5e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:15.670866 1172 consensus_queue.cc:260] T b0fec06ad37d4c12a7045d424b88cc5e P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:15.671432 1172 ts_tablet_manager.cc:1434] T b0fec06ad37d4c12a7045d424b88cc5e P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 02:52:15.671540 524 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_957971e954a244309b62d05bb172dc9d" } new_table_name: "default.Table_abd23446915f4d56a53e0bf265ffb5a7" modify_external_catalogs: true
I20251029 02:52:15.671708 1172 tablet_bootstrap.cc:492] T cc802bda2f6a4e439b605f2cafa37967 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:15.672147 520 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet a3c8b25e6f4849d6aadca322d5365af1
I20251029 02:52:15.672806 520 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_6d9b81aa4e3a4880ac256ecf0fe9f010" } modify_external_catalogs: true
I20251029 02:52:15.672991 520 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_6d9b81aa4e3a4880ac256ecf0fe9f010" } modify_external_catalogs: true
I20251029 02:52:15.673082 1172 tablet_bootstrap.cc:654] T cc802bda2f6a4e439b605f2cafa37967 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:15.673322 522 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 6dc30c614be64b8ba629d9884d450290
I20251029 02:52:15.673890 522 master_service.cc:954] ReplaceTablet: received request to replace tablet 688af6580c4d4637a39f817b4e324618 from {username='slave'} at 127.0.0.1:35314
I20251029 02:52:15.675542 521 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet e60b029eb3ec466cbdced20d536f4f47
I20251029 02:52:15.676231 526 catalog_manager.cc:6194] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: default.Table_0ae102f09fdc41bcbead0148653df9e2 [id=65ec61bf400a4628b98e64e7d43c025c] alter complete (version 2)
I20251029 02:52:15.676287 891 tablet_service.cc:1505] Processing CreateTablet for tablet 78c7cbd6bde646c69eae1698d1b0f7a9 (DEFAULT_TABLE table=default.Table_7ca8e3e58d62408c8010ea1aeb204a94 [id=9965df24e91c4add863ba6182c7a2277]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.679559 890 tablet_service.cc:1505] Processing CreateTablet for tablet eb67e9b7430b4b918fd29d3c6faf6ca9 (DEFAULT_TABLE table=default.Table_6aa358a4fc5c4bce9803079e4b61fdff [id=966a15d73ffd4bfab746e01a73224eec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.679982 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78c7cbd6bde646c69eae1698d1b0f7a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.680526 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb67e9b7430b4b918fd29d3c6faf6ca9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.668545 1175 ts_tablet_manager.cc:1929] T 7bb85a12f573494cbb3118d1947eda27 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:15.683283 1175 log.cc:1199] T 7bb85a12f573494cbb3118d1947eda27 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/7bb85a12f573494cbb3118d1947eda27
I20251029 02:52:15.683691 1175 ts_tablet_manager.cc:1950] T 7bb85a12f573494cbb3118d1947eda27 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:15.688057 2655 leader_election.cc:290] T c32c28b0e74a4af3a0e69bbf24ab1a1f P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:15.688445 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 937) took 5141 ms. Trace:
I20251029 02:52:15.688524 1175 rpcz_store.cc:276] 1029 02:52:10.546544 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:10.546593 (+ 49us) service_pool.cc:225] Handling call
1029 02:52:15.654186 (+5107593us) ts_tablet_manager.cc:1151] Deleting tablet 7bb85a12f573494cbb3118d1947eda27
1029 02:52:15.654197 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:15.661596 (+ 7399us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:15.668449 (+ 6853us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:15.688424 (+ 19975us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5099911}
I20251029 02:52:15.688897 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 938) took 5122 ms. Trace:
I20251029 02:52:15.688944 754 tablet_service.cc:1552] Processing DeleteTablet for tablet 6dc30c614be64b8ba629d9884d450290 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:15 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:15.688978 1175 rpcz_store.cc:276] 1029 02:52:10.566754 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:10.566809 (+ 55us) service_pool.cc:225] Handling call
1029 02:52:15.688756 (+5121947us) ts_tablet_manager.cc:1151] Deleting tablet 0700b93cc398403d8b3602e84182a3ce
1029 02:52:15.688764 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:15.688881 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5121770}
I20251029 02:52:15.689174 750 tablet_service.cc:1552] Processing DeleteTablet for tablet e60b029eb3ec466cbdced20d536f4f47 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:15 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:15.689599 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 7bb85a12f573494cbb3118d1947eda27 (table default.Table_0100f32eabf24d88a739a3cdac3597e1 [id=f22450cafadd404d8d224cdee646b5a2]) successfully deleted
W20251029 02:52:15.689888 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 0700b93cc398403d8b3602e84182a3ce because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0700b93cc398403d8b3602e84182a3ce already in progress: creating tablet
I20251029 02:52:15.690191 748 tablet_service.cc:1552] Processing DeleteTablet for tablet a3c8b25e6f4849d6aadca322d5365af1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:15 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:15.690869 1058 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c32c28b0e74a4af3a0e69bbf24ab1a1f" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:15.691105 1058 raft_consensus.cc:2468] T c32c28b0e74a4af3a0e69bbf24ab1a1f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:15.691366 1175 tablet_replica.cc:333] T 58dde9397b434b089ade9968f369006a P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:15.691538 1175 raft_consensus.cc:2243] T 58dde9397b434b089ade9968f369006a P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:15.691677 1175 raft_consensus.cc:2272] T 58dde9397b434b089ade9968f369006a P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:15.693508 1175 ts_tablet_manager.cc:1916] T 58dde9397b434b089ade9968f369006a P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 02:52:15.694679 847 leader_election.cc:343] T c32c28b0e74a4af3a0e69bbf24ab1a1f P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: c32c28b0e74a4af3a0e69bbf24ab1a1f
I20251029 02:52:15.695135 849 leader_election.cc:304] T c32c28b0e74a4af3a0e69bbf24ab1a1f P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters: b495a49349324f7d81908f2da700f0d9
I20251029 02:52:15.695617 2655 raft_consensus.cc:2804] T c32c28b0e74a4af3a0e69bbf24ab1a1f P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251029 02:52:15.695696 2655 raft_consensus.cc:493] T c32c28b0e74a4af3a0e69bbf24ab1a1f P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election (detected failure of leader b495a49349324f7d81908f2da700f0d9)
I20251029 02:52:15.695765 2655 raft_consensus.cc:3060] T c32c28b0e74a4af3a0e69bbf24ab1a1f P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Advancing to term 2
I20251029 02:52:15.700124 755 tablet_service.cc:1552] Processing DeleteTablet for tablet 18632de3bbbd491a86460497544580c1 with delete_type TABLET_DATA_DELETED (replaced by tablet c3dabe9d8e534953838ce5f0e3259e9d) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:15.705718 526 catalog_manager.cc:5649] T 6f8cf09f23984b879b60ab6b27c7d52e P c756528642a543eeacd93cef3e335a7d reported cstate change: term changed from 0 to 1, leader changed from <none> to c756528642a543eeacd93cef3e335a7d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "c756528642a543eeacd93cef3e335a7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } health_report { overall_health: UNKNOWN } } }
I20251029 02:52:15.709478 526 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet 6f8cf09f23984b879b60ab6b27c7d52e (table default.Table_96e017f445bc4f1fbebddf65c722b8de [id=98f28a15eced4860ac13a60abdc8e0e2]). Expected version 3 got 0
I20251029 02:52:15.709565 895 tablet_service.cc:1505] Processing CreateTablet for tablet 1169fbf6f22b458099f1b0c2cc784d90 (DEFAULT_TABLE table=default.Table_742111c9d1cc4930b57efd12f4190058 [id=00827045f0bf4e89b7aafb89fe9226f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.709973 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1169fbf6f22b458099f1b0c2cc784d90. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.710579 526 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet 6789bc16664e4b37b86702449ab27855 (table default.Table_9a7d7f4f9073437097aca9f7d6dc665e [id=ab8aeaf90f6c4bc2ad5a98780945c40a]). Expected version 2 got 0
I20251029 02:52:15.714825 526 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet 748777c6cd6e4f49ab49912b719e48bd (table default.Table_1e69e259edba466f99ed7a41dcf6367f [id=a2f3b36dc78a4e94bd054a6a5712b384]). Expected version 1 got 0
I20251029 02:52:15.716024 526 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet 27b08f88b99643d28ac609e45bf8104e (table default.Table_a9e50760813844af85a85c5f8d22f8e0 [id=74a2a41c908146ec8e16c6a3227b94f2]). Expected version 1 got 0
I20251029 02:52:15.722369 2201 raft_consensus.cc:493] T 8048506ced4540a5aee9f79bfc4599bf P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b495a49349324f7d81908f2da700f0d9)
I20251029 02:52:15.722842 2201 raft_consensus.cc:515] T 8048506ced4540a5aee9f79bfc4599bf P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:15.724138 2201 leader_election.cc:290] T 8048506ced4540a5aee9f79bfc4599bf P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:15.725602 1058 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "8048506ced4540a5aee9f79bfc4599bf" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:52:15.725046 847 leader_election.cc:343] T 8048506ced4540a5aee9f79bfc4599bf P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 8048506ced4540a5aee9f79bfc4599bf
W20251029 02:52:15.729385 849 leader_election.cc:343] T 8048506ced4540a5aee9f79bfc4599bf P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: Tablet not RUNNING: STOPPED
I20251029 02:52:15.730683 885 tablet_service.cc:1505] Processing CreateTablet for tablet 3d06c88ce8394f37ac8af308b7be042b (DEFAULT_TABLE table=default.Table_e7bb3e262a894c2f84b6edfc3917b30d [id=2cca3b7c50b243b5883b8ac1f353f253]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.733883 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d06c88ce8394f37ac8af308b7be042b. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:15.739032 2720 log.cc:927] Time spent T 7bb85a12f573494cbb3118d1947eda27 P c756528642a543eeacd93cef3e335a7d: Append to log took a long time: real 0.160s user 0.001s sys 0.000s
I20251029 02:52:15.739655 1175 ts_tablet_manager.cc:1929] T 58dde9397b434b089ade9968f369006a P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:15.741565 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "055184c641ba45a588bca611fb24d130" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:15.745498 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "32519fbffcef44a890161e5f3043806c" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:15.746917 524 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9efc87a35b96414fa38d7e849185278c consensus state has no leader
I20251029 02:52:15.747249 849 leader_election.cc:304] T 8048506ced4540a5aee9f79bfc4599bf P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9
I20251029 02:52:15.747341 523 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet f1e975b267464e64a5bbe24ef4e308a3
I20251029 02:52:15.747762 523 catalog_manager.cc:6719] ReplaceTablet: tablet f1e975b267464e64a5bbe24ef4e308a3 deleted and replaced by tablet b00b7696d0dc46a2aec79ebf3c66f39f
I20251029 02:52:15.748237 522 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 688af6580c4d4637a39f817b4e324618
I20251029 02:52:15.748672 522 catalog_manager.cc:6719] ReplaceTablet: tablet 688af6580c4d4637a39f817b4e324618 deleted and replaced by tablet e9387edc3c434f5881a3a43e79a3af58
I20251029 02:52:15.748834 1175 log.cc:1199] T 58dde9397b434b089ade9968f369006a P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/58dde9397b434b089ade9968f369006a
W20251029 02:52:15.749122 525 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b6436a020e3542fbb25ac2846ed42bef consensus state has no leader
I20251029 02:52:15.749302 2201 raft_consensus.cc:2749] T 8048506ced4540a5aee9f79bfc4599bf P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251029 02:52:15.751839 1175 ts_tablet_manager.cc:1950] T 58dde9397b434b089ade9968f369006a P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:15.753201 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 941) took 5175 ms. Trace:
W20251029 02:52:15.746289 711 leader_election.cc:343] T 055184c641ba45a588bca611fb24d130 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:15.753291 1175 rpcz_store.cc:276] 1029 02:52:10.577856 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:10.577951 (+ 95us) service_pool.cc:225] Handling call
1029 02:52:15.689194 (+5111243us) ts_tablet_manager.cc:1151] Deleting tablet 58dde9397b434b089ade9968f369006a
1029 02:52:15.691311 (+ 2117us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:15.719795 (+ 28484us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:15.739534 (+ 19739us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:15.753179 (+ 13645us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5111063}
W20251029 02:52:15.753590 711 leader_election.cc:343] T 32519fbffcef44a890161e5f3043806c P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:15.754171 755 tablet_service.cc:1552] Processing DeleteTablet for tablet f1e975b267464e64a5bbe24ef4e308a3 with delete_type TABLET_DATA_DELETED (replaced by tablet b00b7696d0dc46a2aec79ebf3c66f39f) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:15.754361 748 tablet_service.cc:1552] Processing DeleteTablet for tablet 688af6580c4d4637a39f817b4e324618 with delete_type TABLET_DATA_DELETED (replaced by tablet e9387edc3c434f5881a3a43e79a3af58) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:15.754743 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 58dde9397b434b089ade9968f369006a (table default.Table_b9f2a2e3d9f8446fbd46ca6c48155ccf [id=233cd457a662483a84d93d4243a978b1]) successfully deleted
I20251029 02:52:15.759994 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 940) took 5182 ms. Trace:
I20251029 02:52:15.760593 1175 rpcz_store.cc:276] 1029 02:52:10.577725 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:10.577911 (+ 186us) service_pool.cc:225] Handling call
1029 02:52:15.759831 (+5181920us) ts_tablet_manager.cc:1151] Deleting tablet ef2e9dd8f90a47389d9062d1f99a1da1
1029 02:52:15.759844 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:15.759971 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5181562}
W20251029 02:52:15.761332 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet ef2e9dd8f90a47389d9062d1f99a1da1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ef2e9dd8f90a47389d9062d1f99a1da1 already in progress: creating tablet
I20251029 02:52:15.761831 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 939) took 5184 ms. Trace:
I20251029 02:52:15.762539 1175 rpcz_store.cc:276] 1029 02:52:10.577596 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:10.578012 (+ 416us) service_pool.cc:225] Handling call
1029 02:52:15.761718 (+5183706us) ts_tablet_manager.cc:1151] Deleting tablet bfb9115f324f44c98f876a2d56bb23a1
1029 02:52:15.761727 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:15.761813 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5183085}
W20251029 02:52:15.767027 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet bfb9115f324f44c98f876a2d56bb23a1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bfb9115f324f44c98f876a2d56bb23a1 already in progress: creating tablet
I20251029 02:52:15.767577 517 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet c45ffc304e73490387ce800b702b5881 (table default.Table_075076404d494b29b2cb90e9f10219e5 [id=97e8edf0d4df452d96c9e13e55aa0736]). Expected version 1 got 0
I20251029 02:52:15.767627 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 942) took 5150 ms. Trace:
I20251029 02:52:15.767715 1175 rpcz_store.cc:276] 1029 02:52:10.617562 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:10.618414 (+ 852us) service_pool.cc:225] Handling call
1029 02:52:15.767478 (+5149064us) ts_tablet_manager.cc:1151] Deleting tablet 862d268a526545ba8f25bb91c2992321
1029 02:52:15.767492 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:15.767607 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5148745}
W20251029 02:52:15.768186 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 862d268a526545ba8f25bb91c2992321 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 862d268a526545ba8f25bb91c2992321 already in progress: creating tablet
I20251029 02:52:15.769804 748 tablet_service.cc:1552] Processing DeleteTablet for tablet 5815626b05204b578cc4a110f246bf10 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:08 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:15.770102 755 tablet_service.cc:1552] Processing DeleteTablet for tablet b0fec06ad37d4c12a7045d424b88cc5e with delete_type TABLET_DATA_DELETED (replaced by tablet 4eec207d19f540518bb567d7ab21d048) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:15.770598 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 943) took 5123 ms. Trace:
I20251029 02:52:15.771502 2179 raft_consensus.cc:493] T 27b08f88b99643d28ac609e45bf8104e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:15.771632 2179 raft_consensus.cc:515] T 27b08f88b99643d28ac609e45bf8104e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:15.772182 2179 leader_election.cc:290] T 27b08f88b99643d28ac609e45bf8104e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:15.772403 1175 rpcz_store.cc:276] 1029 02:52:10.646878 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:10.646940 (+ 62us) service_pool.cc:225] Handling call
1029 02:52:15.770447 (+5123507us) ts_tablet_manager.cc:1151] Deleting tablet 34d141bee474452693dba1b9d7f2b023
1029 02:52:15.770461 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:15.770581 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5123280}
W20251029 02:52:15.773013 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 34d141bee474452693dba1b9d7f2b023 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 34d141bee474452693dba1b9d7f2b023 already in progress: creating tablet
I20251029 02:52:15.773120 1175 tablet_replica.cc:333] T 40e844fd7ad74680bcbe7bef936c5b9d P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:15.773300 1175 raft_consensus.cc:2243] T 40e844fd7ad74680bcbe7bef936c5b9d P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:15.773468 1175 raft_consensus.cc:2272] T 40e844fd7ad74680bcbe7bef936c5b9d P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:15.785530 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c32c28b0e74a4af3a0e69bbf24ab1a1f" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:15.799317 773 raft_consensus.cc:1275] T 6789bc16664e4b37b86702449ab27855 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Refusing update from remote peer c756528642a543eeacd93cef3e335a7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251029 02:52:15.800164 2201 consensus_queue.cc:1048] T 6789bc16664e4b37b86702449ab27855 P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:15.802728 2722 raft_consensus.cc:493] T c45ffc304e73490387ce800b702b5881 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:15.802873 2722 raft_consensus.cc:515] T c45ffc304e73490387ce800b702b5881 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:15.803536 2722 leader_election.cc:290] T c45ffc304e73490387ce800b702b5881 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:52:15.804003 526 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_1455335d0ba84ae08eff9eca1366a8f0" } new_table_name: "default.Table_1dc2e51e31fa4ba09d5e2583a17d110b" modify_external_catalogs: true
I20251029 02:52:15.805334 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_01a3b212de724677af552f1859bf6481"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:15.805814 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_01a3b212de724677af552f1859bf6481 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:15.806728 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_a92bbb1e05604a78a67943dc495cc225"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:15.807161 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a92bbb1e05604a78a67943dc495cc225 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:15.808173 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_6c87fb0ffa1c4d818b953a2b0dace9a3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:15.808564 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6c87fb0ffa1c4d818b953a2b0dace9a3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:15.809577 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_ff6546625fd1420abd9ab22239938a2d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:15.809981 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ff6546625fd1420abd9ab22239938a2d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:15.811802 1174 ts_tablet_manager.cc:1929] T 28f7ceb91a34450cad4b5b45f5285b41 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:15.811905 1174 log.cc:1199] T 28f7ceb91a34450cad4b5b45f5285b41 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/28f7ceb91a34450cad4b5b45f5285b41
I20251029 02:52:15.813884 2722 raft_consensus.cc:493] T b0fec06ad37d4c12a7045d424b88cc5e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:15.814060 2722 raft_consensus.cc:515] T b0fec06ad37d4c12a7045d424b88cc5e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:15.814756 2722 leader_election.cc:290] T b0fec06ad37d4c12a7045d424b88cc5e P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:52:15.411850 1176 ts_tablet_manager.cc:1950] T 8048506ced4540a5aee9f79bfc4599bf P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:15.815810 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_bb58177f826141a1bfdc278c16bcb22a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:15.816049 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1015) took 4286 ms. Trace:
I20251029 02:52:15.816128 1176 rpcz_store.cc:276] 1029 02:52:11.529106 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:11.529186 (+ 80us) service_pool.cc:225] Handling call
1029 02:52:15.272331 (+3743145us) ts_tablet_manager.cc:1151] Deleting tablet 8048506ced4540a5aee9f79bfc4599bf
1029 02:52:15.272341 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:15.303165 (+ 30824us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:15.306469 (+ 3304us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:15.816031 (+509562us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3677481}
W20251029 02:52:15.816268 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bb58177f826141a1bfdc278c16bcb22a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:15.816402 1176 tablet_replica.cc:333] T 738046ab57334f2dab9f4aab979c5b42 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:15.816558 1176 raft_consensus.cc:2243] T 738046ab57334f2dab9f4aab979c5b42 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:15.816725 1176 raft_consensus.cc:2272] T 738046ab57334f2dab9f4aab979c5b42 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:15.817253 1176 ts_tablet_manager.cc:1916] T 738046ab57334f2dab9f4aab979c5b42 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:15.817368 518 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_a53e7fa11e1f41af89d9421d2c107641" } modify_external_catalogs: true
I20251029 02:52:15.817504 518 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_a53e7fa11e1f41af89d9421d2c107641" } modify_external_catalogs: true
I20251029 02:52:15.819015 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_f3927a91da1e440ab76a73168740c06d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:15.819200 878 tablet_service.cc:1505] Processing CreateTablet for tablet 15d41a61ba5943fcbc085943031907c8 (DEFAULT_TABLE table=default.Table_0f200bab93784e9d97c7d001330e303a [id=0809df5782144d1da622b5726db932f4]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 02:52:15.819442 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f3927a91da1e440ab76a73168740c06d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:15.819676 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15d41a61ba5943fcbc085943031907c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.823284 521 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_5c2a494d41d0490e9c6eb102db44abbe" } new_table_name: "default.Table_6870fd296d904fdcaeb6e01ed44d4f3d" modify_external_catalogs: true
I20251029 02:52:15.825068 2722 raft_consensus.cc:493] T 5815626b05204b578cc4a110f246bf10 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:15.825227 2722 raft_consensus.cc:515] T 5815626b05204b578cc4a110f246bf10 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:15.825978 2722 leader_election.cc:290] T 5815626b05204b578cc4a110f246bf10 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
W20251029 02:52:15.828296 526 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c91ecaea96224e33be4d728678729886 consensus state has no leader
I20251029 02:52:15.834815 1027 tablet_service.cc:1552] Processing DeleteTablet for tablet 078ace5eada044c88d2462efb2b57aa8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:14 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:15.835189 1027 tablet_service.cc:1552] Processing DeleteTablet for tablet 7ff2399833e546448b85b9caeb5200fb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:15 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:15.835516 1027 tablet_service.cc:1552] Processing DeleteTablet for tablet f66a7af499b446c7a33965c9e37ce460 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:15 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:15.835790 1027 tablet_service.cc:1552] Processing DeleteTablet for tablet 2c4e78fbd7e645eb8b3be8ac4bfbb425 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:15 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:15.838919 1027 tablet_service.cc:1552] Processing DeleteTablet for tablet 59fd5608e4c04fa293ca04595a20d858 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:15 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:15.839258 1027 tablet_service.cc:1552] Processing DeleteTablet for tablet 40e844fd7ad74680bcbe7bef936c5b9d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:59 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:15.839715 1027 tablet_service.cc:1505] Processing CreateTablet for tablet 35d8d0f427f943878baf9047793b4180 (DEFAULT_TABLE table=default.Table_44608f8fced543919b531792404e12a6 [id=a5882cf7ac804f86b303193b71df9fe9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.840090 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35d8d0f427f943878baf9047793b4180. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.848251 1166 tablet_bootstrap.cc:492] T 20a762344c5f46f88481c4ac7ad6e43b P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:15.848467 1166 ts_tablet_manager.cc:1403] T 20a762344c5f46f88481c4ac7ad6e43b P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.469s user 0.003s sys 0.000s
I20251029 02:52:15.849002 1021 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:37622 (request call id 155) took 1075 ms. Trace:
I20251029 02:52:15.849334 1166 raft_consensus.cc:359] T 20a762344c5f46f88481c4ac7ad6e43b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:15.851511 1172 tablet_bootstrap.cc:492] T cc802bda2f6a4e439b605f2cafa37967 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:15.853786 1172 ts_tablet_manager.cc:1403] T cc802bda2f6a4e439b605f2cafa37967 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.182s user 0.003s sys 0.014s
I20251029 02:52:15.854607 1172 raft_consensus.cc:359] T cc802bda2f6a4e439b605f2cafa37967 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:15.854842 1172 raft_consensus.cc:385] T cc802bda2f6a4e439b605f2cafa37967 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:15.854943 1172 raft_consensus.cc:740] T cc802bda2f6a4e439b605f2cafa37967 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:15.855288 1172 consensus_queue.cc:260] T cc802bda2f6a4e439b605f2cafa37967 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:15.856182 1172 ts_tablet_manager.cc:1434] T cc802bda2f6a4e439b605f2cafa37967 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251029 02:52:15.847694 1175 ts_tablet_manager.cc:1916] T 40e844fd7ad74680bcbe7bef936c5b9d P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:15.852248 1040 tablet_service.cc:1505] Processing CreateTablet for tablet 62f4a04568804c3da03393c08676c5da (DEFAULT_TABLE table=default.Table_0cd699a8b604487f89d72e6d948f2bc2 [id=002eddc907784bc4a201dc84341d1949]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.858848 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 62f4a04568804c3da03393c08676c5da. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.860322 1034 tablet_service.cc:1505] Processing CreateTablet for tablet 004eb4e26cd24e49a1931443d6ac7cd0 (DEFAULT_TABLE table=default.Table_f352e3374c0142cb90edcd42caf06ef4 [id=06595399c8e848dfb99b1c0c4a2240eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.860683 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 004eb4e26cd24e49a1931443d6ac7cd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.852761 1039 tablet_service.cc:1505] Processing CreateTablet for tablet 06b1df46901343b6b28be4c17dd61bed (DEFAULT_TABLE table=default.Table_4e9d94609b4a4d1f94615dcc9830d717 [id=761ebdd0095c414488aa221cbc99d3bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.853587 1031 tablet_service.cc:1505] Processing CreateTablet for tablet 0bc634974f1f4cbba8092f1ddff4c0ff (DEFAULT_TABLE table=default.Table_15d2f76cb37642f5a8f3611fbcdd5259 [id=e6cdfdeb066c41cf93aa24d534aa0e12]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.853873 1025 tablet_service.cc:1505] Processing CreateTablet for tablet c525515e5b3148af80468d7af5fb8b22 (DEFAULT_TABLE table=default.Table_3eb06cb0b47545f2854d25ae8849681f [id=561d9f68e7e04c508ae00e8184a3c2cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.854140 1028 tablet_service.cc:1505] Processing CreateTablet for tablet 8fa6a27df46b4cf786470792431cefbc (DEFAULT_TABLE table=default.Table_16b74e4bf0c240649a053af2967ae6bb [id=46a0947dfcd046298db276a6e24935e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.854398 1038 tablet_service.cc:1505] Processing CreateTablet for tablet eade12ecfabf48008db86f98767ff31c (DEFAULT_TABLE table=default.Table_81de1a2ccffd4a439e22e14cbb0c7cc3 [id=8347e0af97e7498eac14286866eed42a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.854686 1024 tablet_service.cc:1505] Processing CreateTablet for tablet a3c8b25e6f4849d6aadca322d5365af1 (DEFAULT_TABLE table=default.Table_69cd5bad2bb64a15908025872e382616 [id=084131cd2315487699ac340402b408d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.854964 1042 tablet_service.cc:1505] Processing CreateTablet for tablet e841d3cc59f1475eac9a521648aee14b (DEFAULT_TABLE table=default.Table_97f3a6c87c274315b8c9651e5e1c1714 [id=34d2e2aaa3fa4efab5980c09ee510b89]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.862841 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06b1df46901343b6b28be4c17dd61bed. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.863129 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c525515e5b3148af80468d7af5fb8b22. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.866696 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8fa6a27df46b4cf786470792431cefbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.867334 755 tablet_service.cc:1552] Processing DeleteTablet for tablet cc802bda2f6a4e439b605f2cafa37967 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:57 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:15.867976 1034 tablet_service.cc:1505] Processing CreateTablet for tablet 818262aee43c4343b864e01f836e76b8 (DEFAULT_TABLE table=default.Table_6fa2b2dfa5c240ba95baf05031690542 [id=d93d8bc567dc49c68409f850376be70e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.868309 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 818262aee43c4343b864e01f836e76b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.870947 1038 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eade12ecfabf48008db86f98767ff31c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.872421 523 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_c32e2f0d85f9486d8957cedb7e09dda5" } new_table_name: "default.Table_4d227f8ec9fc44d1849f330845d894fd" modify_external_catalogs: true
I20251029 02:52:15.873062 1033 tablet_service.cc:1505] Processing CreateTablet for tablet 96e7e1b0f14e4e738e7b2d0eaf8d53bb (DEFAULT_TABLE table=default.Table_93495adef2ff4b039f9aebdb6c748b49 [id=19e9b81045c64937adfd882d4bde9f85]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.873430 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96e7e1b0f14e4e738e7b2d0eaf8d53bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.875082 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3c8b25e6f4849d6aadca322d5365af1. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.877513 1166 raft_consensus.cc:385] T 20a762344c5f46f88481c4ac7ad6e43b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:15.877599 1166 raft_consensus.cc:740] T 20a762344c5f46f88481c4ac7ad6e43b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:15.878758 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e841d3cc59f1475eac9a521648aee14b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.879254 1166 consensus_queue.cc:260] T 20a762344c5f46f88481c4ac7ad6e43b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:15.879465 525 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_fa832508491b47ddbcd9e89058c75ad5" } modify_external_catalogs: true
I20251029 02:52:15.879653 525 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_fa832508491b47ddbcd9e89058c75ad5" } modify_external_catalogs: true
I20251029 02:52:15.880215 522 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_f8f6696a8a1c4fcca702c03886a45f62" } modify_external_catalogs: true
I20251029 02:52:15.880342 522 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_f8f6696a8a1c4fcca702c03886a45f62" } modify_external_catalogs: true
I20251029 02:52:15.880728 526 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_4ef22a62446e4805a5a9f18357f4142d" } modify_external_catalogs: true
I20251029 02:52:15.880829 526 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_4ef22a62446e4805a5a9f18357f4142d" } modify_external_catalogs: true
I20251029 02:52:15.881721 1040 tablet_service.cc:1505] Processing CreateTablet for tablet 78c7cbd6bde646c69eae1698d1b0f7a9 (DEFAULT_TABLE table=default.Table_7ca8e3e58d62408c8010ea1aeb204a94 [id=9965df24e91c4add863ba6182c7a2277]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.881989 1039 tablet_service.cc:1505] Processing CreateTablet for tablet 15d41a61ba5943fcbc085943031907c8 (DEFAULT_TABLE table=default.Table_0f200bab93784e9d97c7d001330e303a [id=0809df5782144d1da622b5726db932f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.882063 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78c7cbd6bde646c69eae1698d1b0f7a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.882566 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15d41a61ba5943fcbc085943031907c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.883356 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0bc634974f1f4cbba8092f1ddff4c0ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.885468 1172 tablet_bootstrap.cc:492] T e3000183230e4350b47f5085952f2db6 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:15.887099 1172 tablet_bootstrap.cc:654] T e3000183230e4350b47f5085952f2db6 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:15.887091 1033 tablet_service.cc:1505] Processing CreateTablet for tablet 0573fe69797f48539df5caf4b3777b4d (DEFAULT_TABLE table=default.Table_f90425a19e964b75b94c68952d26ae1e [id=48211a42281c40a99c8cf27174cf6679]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.887482 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0573fe69797f48539df5caf4b3777b4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.887920 1166 ts_tablet_manager.cc:1434] T 20a762344c5f46f88481c4ac7ad6e43b P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.039s user 0.001s sys 0.001s
I20251029 02:52:15.889063 1166 tablet_bootstrap.cc:492] T 32519fbffcef44a890161e5f3043806c P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:15.891469 1166 tablet_bootstrap.cc:654] T 32519fbffcef44a890161e5f3043806c P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:15.898554 1029 tablet_service.cc:1505] Processing CreateTablet for tablet eb67e9b7430b4b918fd29d3c6faf6ca9 (DEFAULT_TABLE table=default.Table_6aa358a4fc5c4bce9803079e4b61fdff [id=966a15d73ffd4bfab746e01a73224eec]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.899142 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb67e9b7430b4b918fd29d3c6faf6ca9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.903324 1036 tablet_service.cc:1552] Processing DeleteTablet for tablet 18632de3bbbd491a86460497544580c1 with delete_type TABLET_DATA_DELETED (replaced by tablet c3dabe9d8e534953838ce5f0e3259e9d) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:15.903525 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet a3c8b25e6f4849d6aadca322d5365af1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:15 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:15.903301 1039 tablet_service.cc:1505] Processing CreateTablet for tablet 3d06c88ce8394f37ac8af308b7be042b (DEFAULT_TABLE table=default.Table_e7bb3e262a894c2f84b6edfc3917b30d [id=2cca3b7c50b243b5883b8ac1f353f253]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.904011 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d06c88ce8394f37ac8af308b7be042b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.904042 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_5e4e40ac95f149f6b0116c0cb7c5fa72"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:15.904160 1024 tablet_service.cc:1505] Processing CreateTablet for tablet 1169fbf6f22b458099f1b0c2cc784d90 (DEFAULT_TABLE table=default.Table_742111c9d1cc4930b57efd12f4190058 [id=00827045f0bf4e89b7aafb89fe9226f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.904479 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1169fbf6f22b458099f1b0c2cc784d90. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:15.904577 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5e4e40ac95f149f6b0116c0cb7c5fa72 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:52:15.905937 521 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8616fa24a54147b684f50d373a383c2b consensus state has no leader
I20251029 02:52:15.906387 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_397408db7cb64d52830da673241064ac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:15.906957 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_397408db7cb64d52830da673241064ac in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:15.908396 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_9dbf9a53f1e34611976c67bff0ac7f9f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:15.908833 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9dbf9a53f1e34611976c67bff0ac7f9f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:15.908871 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_255cf3364f1e4be9a543378e563a730f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:15.913450 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_255cf3364f1e4be9a543378e563a730f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:15.914376 1166 tablet_bootstrap.cc:492] T 32519fbffcef44a890161e5f3043806c P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:15.914573 1166 ts_tablet_manager.cc:1403] T 32519fbffcef44a890161e5f3043806c P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.026s user 0.001s sys 0.004s
W20251029 02:52:15.917770 523 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6c10980d0c1042539ac8fb6406403ce0 consensus state has no leader
I20251029 02:52:15.918670 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet 6dc30c614be64b8ba629d9884d450290 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:15 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:15.919078 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet e60b029eb3ec466cbdced20d536f4f47 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:15 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:15.919417 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet f1e975b267464e64a5bbe24ef4e308a3 with delete_type TABLET_DATA_DELETED (replaced by tablet b00b7696d0dc46a2aec79ebf3c66f39f) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:15.919764 1035 tablet_service.cc:1552] Processing DeleteTablet for tablet 688af6580c4d4637a39f817b4e324618 with delete_type TABLET_DATA_DELETED (replaced by tablet e9387edc3c434f5881a3a43e79a3af58) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:15.916270 1025 tablet_service.cc:1552] Processing DeleteTablet for tablet e91d6bdca3ad40c48d93e6196644462c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:15 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:15.858038 1027 tablet_service.cc:1505] Processing CreateTablet for tablet 8616fa24a54147b684f50d373a383c2b (DEFAULT_TABLE table=default.Table_5c2a494d41d0490e9c6eb102db44abbe [id=9124af15f5dc476f9bb0c954b326ed00]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.922874 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8616fa24a54147b684f50d373a383c2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.927073 1166 raft_consensus.cc:359] T 32519fbffcef44a890161e5f3043806c P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:15.928248 1166 raft_consensus.cc:385] T 32519fbffcef44a890161e5f3043806c P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:15.928552 1166 raft_consensus.cc:740] T 32519fbffcef44a890161e5f3043806c P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:15.928967 755 tablet_service.cc:1505] Processing CreateTablet for tablet 2925722be8a045c689c69ac729fb3799 (DEFAULT_TABLE table=default.Table_d8c5abd34fad48fb8ec28152a4b37e38 [id=f6160584ba214beb8beacc856d5a9ca6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.929023 1166 consensus_queue.cc:260] T 32519fbffcef44a890161e5f3043806c P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:15.929366 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2925722be8a045c689c69ac729fb3799. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.930794 1166 ts_tablet_manager.cc:1434] T 32519fbffcef44a890161e5f3043806c P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20251029 02:52:15.931063 1166 tablet_bootstrap.cc:492] T 372d93914abc41b38a702742b7db0bc2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:15.932647 1166 tablet_bootstrap.cc:654] T 372d93914abc41b38a702742b7db0bc2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:15.935508 2722 raft_consensus.cc:493] T f93b0c39d7174dabbe018b1546dbc198 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:15.935647 2722 raft_consensus.cc:515] T f93b0c39d7174dabbe018b1546dbc198 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:15.936249 2722 leader_election.cc:290] T f93b0c39d7174dabbe018b1546dbc198 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:52:15.937911 748 tablet_service.cc:1505] Processing CreateTablet for tablet 9542585fd9824a55b3ad2f120065a02d (DEFAULT_TABLE table=default.Table_7eb1d904f38342dc85bb06681099d741 [id=cbadf4b0377b4c3dbf106b533a18acdf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.938271 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9542585fd9824a55b3ad2f120065a02d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.940850 1176 ts_tablet_manager.cc:1929] T 738046ab57334f2dab9f4aab979c5b42 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:15.940984 1176 log.cc:1199] T 738046ab57334f2dab9f4aab979c5b42 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/738046ab57334f2dab9f4aab979c5b42
I20251029 02:52:15.941393 1176 ts_tablet_manager.cc:1950] T 738046ab57334f2dab9f4aab979c5b42 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:15.941623 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "27b08f88b99643d28ac609e45bf8104e" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:15.943049 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 8048506ced4540a5aee9f79bfc4599bf (table default.Table_f2a72498cf4540949baff93eba1f5ef0 [id=a047685192f34bf1a1f3d1361cdaf272]) successfully deleted
I20251029 02:52:15.946627 1174 ts_tablet_manager.cc:1950] T 28f7ceb91a34450cad4b5b45f5285b41 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:15.849174 1021 rpcz_store.cc:276] 1029 02:52:14.773147 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:14.773557 (+ 410us) service_pool.cc:225] Handling call
1029 02:52:14.937920 (+164363us) spinlock_profiling.cc:243] Waited 22.2 ms on lock 0x616000116254. stack: 00007f34717829f6 00007f34717827b0 00007f347075e3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f347d61c200 00007f34814cc66f 00007f347b8f2c36 00007f347b8f2965 00007f347872127e 00007f347872015e 00007f3478726395 00007f3478726cbf 00007f3478726ae8 0000000000a8867c
1029 02:52:15.053774 (+115854us) spinlock_profiling.cc:243] Waited 115 ms on lock 0x6160000e4a54. stack: 00007f34717829f6 00007f34717827b0 00007f347075e3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f347d61c200 00007f34814cc66f 00007f347b8f2c36 00007f347b8f2965 00007f347872127e 00007f347872015e 00007f3478726395 00007f3478726cbf 00007f3478726ae8 0000000000a8867c
1029 02:52:15.848983 (+795209us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":316315008}
I20251029 02:52:15.947450 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 945) took 5399 ms. Trace:
I20251029 02:52:15.947528 1174 rpcz_store.cc:276] 1029 02:52:10.548151 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:10.550203 (+ 2052us) service_pool.cc:225] Handling call
1029 02:52:15.517824 (+4967621us) ts_tablet_manager.cc:1151] Deleting tablet 28f7ceb91a34450cad4b5b45f5285b41
1029 02:52:15.517835 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:15.570931 (+ 53096us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:15.811682 (+240751us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:15.947428 (+135746us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4915884}
I20251029 02:52:15.947890 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 950) took 5386 ms. Trace:
I20251029 02:52:15.947959 1174 rpcz_store.cc:276] 1029 02:52:10.561502 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:10.562617 (+ 1115us) service_pool.cc:225] Handling call
1029 02:52:15.947774 (+5385157us) ts_tablet_manager.cc:1151] Deleting tablet 0700b93cc398403d8b3602e84182a3ce
1029 02:52:15.947784 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:15.947874 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5345785}
I20251029 02:52:15.948263 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 951) took 5374 ms. Trace:
I20251029 02:52:15.948334 1174 rpcz_store.cc:276] 1029 02:52:10.573694 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:10.577251 (+ 3557us) service_pool.cc:225] Handling call
1029 02:52:15.948164 (+5370913us) ts_tablet_manager.cc:1151] Deleting tablet bfb9115f324f44c98f876a2d56bb23a1
1029 02:52:15.948173 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:15.948245 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5346136}
I20251029 02:52:15.948599 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 952) took 5371 ms. Trace:
I20251029 02:52:15.948661 1174 rpcz_store.cc:276] 1029 02:52:10.577206 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:10.581626 (+ 4420us) service_pool.cc:225] Handling call
1029 02:52:15.948513 (+5366887us) ts_tablet_manager.cc:1151] Deleting tablet ef2e9dd8f90a47389d9062d1f99a1da1
1029 02:52:15.948522 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:15.948585 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5346449}
I20251029 02:52:15.948905 1174 tablet_replica.cc:333] T 0d84dc8dc465496f9adec798b2e03e13 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:15.949260 750 tablet_service.cc:1505] Processing CreateTablet for tablet b3dc6e55bb2e47bab7df5df62a1bcb51 (DEFAULT_TABLE table=default.Table_ca88b32b14364d2aab4e1b1480676c12 [id=8aacd9b9d75346419887f8be922803ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.949613 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3dc6e55bb2e47bab7df5df62a1bcb51. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:15.954317 2722 raft_consensus.cc:493] T defb795026a44e12afb2defb9c9c6db2 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:15.954437 2722 raft_consensus.cc:515] T defb795026a44e12afb2defb9c9c6db2 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:15.955129 2722 leader_election.cc:290] T defb795026a44e12afb2defb9c9c6db2 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:52:15.969342 519 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_99567b9d8c154303bae22fda47db1d0a" } new_table_name: "default.Table_bfdf938b673d4a4594ad4017beecfeee" modify_external_catalogs: true
I20251029 02:52:15.977023 523 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_f8a743f1d5174309921ff1ff7f6b10af" } new_table_name: "default.Table_e3da94b6b2134ddab176d5c3d16d81aa" modify_external_catalogs: true
I20251029 02:52:15.977510 522 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 19e15a6df77f4e0c9b40ae4f49fdd06f
I20251029 02:52:15.977510 525 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 2849e856dd714da88fca5a9fafe74c22
I20251029 02:52:15.985013 750 tablet_service.cc:1552] Processing DeleteTablet for tablet 2849e856dd714da88fca5a9fafe74c22 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:15 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:15.985608 755 tablet_service.cc:1552] Processing DeleteTablet for tablet 19e15a6df77f4e0c9b40ae4f49fdd06f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:15 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:15.994015 755 tablet_service.cc:1505] Processing CreateTablet for tablet 4905b28faec74d30859461d770282f41 (DEFAULT_TABLE table=default.Table_7511abddcde942a4adb0a23b3a345d06 [id=d9f9ae6423a34675968f74ae094aee89]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:15.994655 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4905b28faec74d30859461d770282f41. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:16.010578 1175 ts_tablet_manager.cc:1929] T 40e844fd7ad74680bcbe7bef936c5b9d P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:16.010835 1175 log.cc:1199] T 40e844fd7ad74680bcbe7bef936c5b9d P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/40e844fd7ad74680bcbe7bef936c5b9d
I20251029 02:52:16.011282 1175 ts_tablet_manager.cc:1950] T 40e844fd7ad74680bcbe7bef936c5b9d P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:16.012176 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 944) took 5333 ms. Trace:
I20251029 02:52:16.012341 1175 rpcz_store.cc:276] 1029 02:52:10.678618 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:10.678698 (+ 80us) service_pool.cc:225] Handling call
1029 02:52:15.773072 (+5094374us) ts_tablet_manager.cc:1151] Deleting tablet 40e844fd7ad74680bcbe7bef936c5b9d
1029 02:52:15.773082 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:15.895778 (+122696us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:16.010447 (+114669us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:16.012151 (+ 1704us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5073598}
I20251029 02:52:16.012850 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 946) took 5229 ms. Trace:
I20251029 02:52:16.012988 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 40e844fd7ad74680bcbe7bef936c5b9d (table default.Table_a66eb4754b854a998e458627d738338b [id=c47f0d4579d74e31be083ac638480e38]) successfully deleted
I20251029 02:52:16.013067 1175 rpcz_store.cc:276] 1029 02:52:10.783404 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:10.783461 (+ 57us) service_pool.cc:225] Handling call
1029 02:52:16.012726 (+5229265us) ts_tablet_manager.cc:1151] Deleting tablet 5a1aa131cf4e4edca5bb2515edd7556a
1029 02:52:16.012736 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:16.012836 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5229073}
I20251029 02:52:16.013389 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 948) took 5212 ms. Trace:
I20251029 02:52:16.013458 1175 rpcz_store.cc:276] 1029 02:52:10.800575 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:10.805101 (+ 4526us) service_pool.cc:225] Handling call
1029 02:52:16.013297 (+5208196us) ts_tablet_manager.cc:1151] Deleting tablet 8378d67f1da743cca91803e37ea54281
1029 02:52:16.013305 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:16.013375 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5208005}
W20251029 02:52:16.013567 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 5a1aa131cf4e4edca5bb2515edd7556a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5a1aa131cf4e4edca5bb2515edd7556a already in progress: creating tablet
W20251029 02:52:16.014016 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 8378d67f1da743cca91803e37ea54281 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8378d67f1da743cca91803e37ea54281 already in progress: creating tablet
I20251029 02:52:16.014541 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1016) took 4485 ms. Trace:
I20251029 02:52:16.014631 1176 rpcz_store.cc:276] 1029 02:52:11.529200 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:11.543144 (+ 13944us) service_pool.cc:225] Handling call
1029 02:52:15.816361 (+4273217us) ts_tablet_manager.cc:1151] Deleting tablet 738046ab57334f2dab9f4aab979c5b42
1029 02:52:15.816369 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:15.936397 (+120028us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:15.940748 (+ 4351us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:16.014497 (+ 73749us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4213746}
I20251029 02:52:16.014894 1176 tablet_replica.cc:333] T ba699862e6f646c6beaa39f080d0faba P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:16.015136 1176 raft_consensus.cc:2243] T ba699862e6f646c6beaa39f080d0faba P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:16.015297 1176 raft_consensus.cc:2272] T ba699862e6f646c6beaa39f080d0faba P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:16.015894 1176 ts_tablet_manager.cc:1916] T ba699862e6f646c6beaa39f080d0faba P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:16.016158 2426 consensus_queue.cc:1048] T 6142018e44af4d32818adefa2ee43d02 P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:16.017509 2655 raft_consensus.cc:515] T c32c28b0e74a4af3a0e69bbf24ab1a1f P c756528642a543eeacd93cef3e335a7d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:16.018088 916 raft_consensus.cc:2393] T c32c28b0e74a4af3a0e69bbf24ab1a1f P c756528642a543eeacd93cef3e335a7d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in current term 2: Already voted for candidate c756528642a543eeacd93cef3e335a7d in this term.
I20251029 02:52:16.020370 2655 leader_election.cc:290] T c32c28b0e74a4af3a0e69bbf24ab1a1f P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 election: Requested vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
W20251029 02:52:16.022219 847 leader_election.cc:343] T c32c28b0e74a4af3a0e69bbf24ab1a1f P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: c32c28b0e74a4af3a0e69bbf24ab1a1f
I20251029 02:52:16.024586 750 tablet_service.cc:1505] Processing CreateTablet for tablet 9426756257c64db982d1e879c0c0f8a9 (DEFAULT_TABLE table=default.Table_75325372892242518c3f0a4b8b657c48 [id=9812d79d53084361ba6cf509fb6eb0f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:16.024984 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9426756257c64db982d1e879c0c0f8a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:16.034255 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_26c8d3fab4e146319800a03dc27ced50"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:16.034848 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_26c8d3fab4e146319800a03dc27ced50 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:16.035633 525 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet 6789bc16664e4b37b86702449ab27855 (table default.Table_9a7d7f4f9073437097aca9f7d6dc665e [id=ab8aeaf90f6c4bc2ad5a98780945c40a]). Expected version 2 got 0
I20251029 02:52:16.036298 522 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_cba115e38a014588b60259fe951f80c9" } new_table_name: "default.Table_922c0c122cf148fe8a96313411028e9b" modify_external_catalogs: true
I20251029 02:52:16.037034 526 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 46e054ff563f448d86d8b4f84810291b
I20251029 02:52:16.040270 750 tablet_service.cc:1552] Processing DeleteTablet for tablet 46e054ff563f448d86d8b4f84810291b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:15 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:16.042553 518 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 5bcadc0b68e84ae290690edef392a45b
I20251029 02:52:16.044677 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_1aa03bfd183b40188824e6368f5746ee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:16.045481 849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:45003, user_credentials={real_user=slave}} blocked reactor thread for 213156us
I20251029 02:52:16.048087 750 tablet_service.cc:1552] Processing DeleteTablet for tablet 5bcadc0b68e84ae290690edef392a45b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:15 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:16.053089 520 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 60f0eadc074a4872b6638314fed98e0e
I20251029 02:52:16.052027 1174 raft_consensus.cc:2243] T 0d84dc8dc465496f9adec798b2e03e13 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:16.057106 1060 raft_consensus.cc:3060] T 6789bc16664e4b37b86702449ab27855 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:16.057492 750 tablet_service.cc:1552] Processing DeleteTablet for tablet 60f0eadc074a4872b6638314fed98e0e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:15 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:16.059005 1174 raft_consensus.cc:2272] T 0d84dc8dc465496f9adec798b2e03e13 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:16.059576 1174 ts_tablet_manager.cc:1916] T 0d84dc8dc465496f9adec798b2e03e13 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:16.060195 1060 raft_consensus.cc:1275] T 6789bc16664e4b37b86702449ab27855 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Refusing update from remote peer c756528642a543eeacd93cef3e335a7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251029 02:52:16.060858 748 tablet_service.cc:1505] Processing CreateTablet for tablet d9b3adb851bc46feb77e4b484eb0b8d4 (DEFAULT_TABLE table=default.Table_024e9ca04c014a27bc2b62227d4828d0 [id=b7453480e4d84319a2ce51432f83659b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:16.061282 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9b3adb851bc46feb77e4b484eb0b8d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:16.065604 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c32c28b0e74a4af3a0e69bbf24ab1a1f" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:52:16.065917 1060 raft_consensus.cc:3060] T c32c28b0e74a4af3a0e69bbf24ab1a1f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Advancing to term 2
W20251029 02:52:16.068550 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1aa03bfd183b40188824e6368f5746ee in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:16.069730 1060 raft_consensus.cc:2468] T c32c28b0e74a4af3a0e69bbf24ab1a1f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 2.
I20251029 02:52:16.070819 750 tablet_service.cc:1505] Processing CreateTablet for tablet b64880e732f54510a3954248cc940fbc (DEFAULT_TABLE table=default.Table_4098c28776dd4f41864c61d2299b90cd [id=47ee5993b95a477db52f9c51d034e549]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 02:52:16.071170 512 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 44a0c058700040f8b5c30a621103a524 (table default.Table_8aa16ebd94174be79b8fd9d2206599a1 [id=300df93a696b407cbf826cfb9ca66919], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 44a0c058700040f8b5c30a621103a524 consensus state has no leader
I20251029 02:52:16.071283 2205 raft_consensus.cc:493] T af4860c6b72f4d988b206b5ef2f22906 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b495a49349324f7d81908f2da700f0d9)
I20251029 02:52:16.071430 2205 raft_consensus.cc:515] T af4860c6b72f4d988b206b5ef2f22906 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:16.071738 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b64880e732f54510a3954248cc940fbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:16.071941 2205 leader_election.cc:290] T af4860c6b72f4d988b206b5ef2f22906 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
W20251029 02:52:16.073282 847 leader_election.cc:343] T af4860c6b72f4d988b206b5ef2f22906 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: af4860c6b72f4d988b206b5ef2f22906
I20251029 02:52:16.076858 1176 ts_tablet_manager.cc:1929] T ba699862e6f646c6beaa39f080d0faba P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:16.076988 1176 log.cc:1199] T ba699862e6f646c6beaa39f080d0faba P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/ba699862e6f646c6beaa39f080d0faba
I20251029 02:52:16.077421 1176 ts_tablet_manager.cc:1950] T ba699862e6f646c6beaa39f080d0faba P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:16.077759 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1020) took 4534 ms. Trace:
I20251029 02:52:16.077847 1176 rpcz_store.cc:276] 1029 02:52:11.543708 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:11.543863 (+ 155us) service_pool.cc:225] Handling call
1029 02:52:16.014855 (+4470992us) ts_tablet_manager.cc:1151] Deleting tablet ba699862e6f646c6beaa39f080d0faba
1029 02:52:16.014863 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:16.057033 (+ 42170us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:16.076732 (+ 19699us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:16.077742 (+ 1010us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4412203}
I20251029 02:52:16.078334 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1037) took 4532 ms. Trace:
I20251029 02:52:16.082008 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_00585488392141a58ecfc968a6f3ca61"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:16.082623 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_00585488392141a58ecfc968a6f3ca61 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:16.083930 520 catalog_manager.cc:5649] T 748777c6cd6e4f49ab49912b719e48bd P c756528642a543eeacd93cef3e335a7d reported cstate change: term changed from 0 to 1, leader changed from <none> to c756528642a543eeacd93cef3e335a7d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "c756528642a543eeacd93cef3e335a7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } health_report { overall_health: UNKNOWN } } }
I20251029 02:52:16.084357 520 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet 748777c6cd6e4f49ab49912b719e48bd (table default.Table_1e69e259edba466f99ed7a41dcf6367f [id=a2f3b36dc78a4e94bd054a6a5712b384]). Expected version 1 got 0
I20251029 02:52:16.088106 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "055184c641ba45a588bca611fb24d130" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:16.090804 1058 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "32519fbffcef44a890161e5f3043806c" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:16.091022 1058 raft_consensus.cc:2468] T 32519fbffcef44a890161e5f3043806c P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b495a49349324f7d81908f2da700f0d9 in term 0.
I20251029 02:52:16.091886 1174 ts_tablet_manager.cc:1929] T 0d84dc8dc465496f9adec798b2e03e13 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:16.092010 1174 log.cc:1199] T 0d84dc8dc465496f9adec798b2e03e13 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/0d84dc8dc465496f9adec798b2e03e13
I20251029 02:52:16.092310 1172 tablet_bootstrap.cc:492] T e3000183230e4350b47f5085952f2db6 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:16.092442 1172 ts_tablet_manager.cc:1403] T e3000183230e4350b47f5085952f2db6 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.207s user 0.005s sys 0.000s
I20251029 02:52:16.093194 1172 raft_consensus.cc:359] T e3000183230e4350b47f5085952f2db6 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:16.093839 1058 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c45ffc304e73490387ce800b702b5881" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:16.097384 2740 raft_consensus.cc:493] T 217144957930495aa3e9dfcfde268e71 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0cfd4bee16f84b7c8b0ff3ce14db20d5)
I20251029 02:52:16.097519 2740 raft_consensus.cc:515] T 217144957930495aa3e9dfcfde268e71 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:16.098258 2740 leader_election.cc:290] T 217144957930495aa3e9dfcfde268e71 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:16.098404 2740 consensus_queue.cc:1048] T 6789bc16664e4b37b86702449ab27855 P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251029 02:52:16.098202 849 consensus_peers.cc:597] T cd597c319af645909b8f3d1212fcb702 P c756528642a543eeacd93cef3e335a7d -> Peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Couldn't send request to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:16.099099 849 leader_election.cc:304] T c32c28b0e74a4af3a0e69bbf24ab1a1f P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters: b495a49349324f7d81908f2da700f0d9
I20251029 02:52:16.099576 1172 raft_consensus.cc:385] T e3000183230e4350b47f5085952f2db6 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:16.100051 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af4860c6b72f4d988b206b5ef2f22906" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:16.100348 1060 raft_consensus.cc:2468] T af4860c6b72f4d988b206b5ef2f22906 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:16.102039 1172 raft_consensus.cc:740] T e3000183230e4350b47f5085952f2db6 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:16.102110 2718 raft_consensus.cc:493] T 6789bc16664e4b37b86702449ab27855 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:16.102232 2718 raft_consensus.cc:515] T 6789bc16664e4b37b86702449ab27855 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:16.102800 2718 leader_election.cc:290] T 6789bc16664e4b37b86702449ab27855 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:16.078423 1176 rpcz_store.cc:276] 1029 02:52:11.545437 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:11.602662 (+ 57225us) service_pool.cc:225] Handling call
1029 02:52:16.078163 (+4475501us) ts_tablet_manager.cc:1151] Deleting tablet 509288c033d54e4b9da3f26a30dc46a0
1029 02:52:16.078175 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:16.078314 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4475340}
I20251029 02:52:16.104151 1172 consensus_queue.cc:260] T e3000183230e4350b47f5085952f2db6 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:16.107054 1172 ts_tablet_manager.cc:1434] T e3000183230e4350b47f5085952f2db6 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20251029 02:52:16.107939 1058 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0fec06ad37d4c12a7045d424b88cc5e" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:16.108012 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1038) took 4562 ms. Trace:
I20251029 02:52:16.108088 1176 rpcz_store.cc:276] 1029 02:52:11.545508 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:11.602827 (+ 57319us) service_pool.cc:225] Handling call
1029 02:52:16.103795 (+4500968us) ts_tablet_manager.cc:1151] Deleting tablet c172417d1d444c439e9cda611a2ed551
1029 02:52:16.107852 (+ 4057us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:16.107997 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4500854}
I20251029 02:52:16.108282 1052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f93b0c39d7174dabbe018b1546dbc198" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:16.108582 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5815626b05204b578cc4a110f246bf10" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:16.110966 2763 tablet.cc:1722] T 6789bc16664e4b37b86702449ab27855 P b495a49349324f7d81908f2da700f0d9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 02:52:16.112303 754 tablet_service.cc:1505] Processing CreateTablet for tablet 09bbf13b23334bbf88f2eb2e327b20f2 (DEFAULT_TABLE table=default.Table_bf304b9f44274cf9b56f9e25fb0e9875 [id=1e7048e956fa484093370764b98d30cb]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 02:52:16.113804 989 consensus_peers.cc:597] T 44c0488eba0543078ff4f0b3d69366f2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 -> Peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Couldn't send request to peer b495a49349324f7d81908f2da700f0d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 44c0488eba0543078ff4f0b3d69366f2. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:16.114497 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "defb795026a44e12afb2defb9c9c6db2" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:16.114558 749 tablet_service.cc:1505] Processing CreateTablet for tablet 1537b318704f48ccb01c48a4e783da94 (DEFAULT_TABLE table=default.Table_a7e37e7a8bba4557a83b0fe3f498df17 [id=1c093c8c547144d2a413329c199d2962]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 02:52:16.099781 847 leader_election.cc:343] T 217144957930495aa3e9dfcfde268e71 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 217144957930495aa3e9dfcfde268e71
I20251029 02:52:16.100140 1174 ts_tablet_manager.cc:1950] T 0d84dc8dc465496f9adec798b2e03e13 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:16.116663 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 946) took 5568 ms. Trace:
I20251029 02:52:16.116796 1174 rpcz_store.cc:276] 1029 02:52:10.548265 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:10.550642 (+ 2377us) service_pool.cc:225] Handling call
1029 02:52:15.948842 (+5398200us) ts_tablet_manager.cc:1151] Deleting tablet 0d84dc8dc465496f9adec798b2e03e13
1029 02:52:15.948874 (+ 32us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:16.058768 (+109894us) spinlock_profiling.cc:243] Waited 4.06 ms on lock 0x616000230e54. stack: 00007f1e09cad9f6 00007f1e09cad7b0 00007f1e08c893ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f1e15b78100 00007f1e16ec5ae3 00007f1e19a86287 00007f1e19ab92d4 00007f1e19a99d70 00007f1e19a99afd 0000000000a8867c 00007f1e09cf3d7f 00007f1e09cf704b 00007f1e09cf6ec8
1029 02:52:16.062425 (+ 3657us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:16.091769 (+ 29344us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:16.116645 (+ 24876us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":98849,"spinlock_wait_cycles":9343360,"tablet-delete.queue_time_us":5346693}
I20251029 02:52:16.118464 756 tablet_service.cc:1505] Processing CreateTablet for tablet 6d57c43e61f3487099a9713f8885a85a (DEFAULT_TABLE table=default.Table_6facbfaed7644f2f9c16f0fe7316e2fc [id=4037bd8625d146409a7878c4e5b6643c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:16.119352 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_e6af8601f9a34c6183e494c964553b5e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:16.119604 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09bbf13b23334bbf88f2eb2e327b20f2. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:16.119865 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e6af8601f9a34c6183e494c964553b5e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:16.121891 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_b997b92037ef41199e6a64c11b0bfb81"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:16.122359 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b997b92037ef41199e6a64c11b0bfb81 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:16.122788 1166 tablet_bootstrap.cc:492] T 372d93914abc41b38a702742b7db0bc2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:16.122915 1166 ts_tablet_manager.cc:1403] T 372d93914abc41b38a702742b7db0bc2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.192s user 0.007s sys 0.001s
W20251029 02:52:16.101593 1167 thread.cc:641] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.803s user 0.011s sys 0.007s
W20251029 02:52:16.124300 1167 thread.cc:608] log-alloc [worker] (thread pool) Time spent starting thread: real 0.826s user 0.011s sys 0.007s
I20251029 02:52:16.125109 1167 tablet_bootstrap.cc:492] T 84993b2f63e7421290a161dde4bb0607 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:16.125248 1167 ts_tablet_manager.cc:1403] T 84993b2f63e7421290a161dde4bb0607 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.833s user 0.013s sys 0.007s
I20251029 02:52:16.125533 918 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f93b0c39d7174dabbe018b1546dbc198" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:16.125662 914 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "defb795026a44e12afb2defb9c9c6db2" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:16.126593 1166 raft_consensus.cc:359] T 372d93914abc41b38a702742b7db0bc2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:16.126636 1167 raft_consensus.cc:359] T 84993b2f63e7421290a161dde4bb0607 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:16.126935 1167 raft_consensus.cc:385] T 84993b2f63e7421290a161dde4bb0607 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:16.127005 1167 raft_consensus.cc:740] T 84993b2f63e7421290a161dde4bb0607 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:16.127342 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1537b318704f48ccb01c48a4e783da94. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:16.127206 1167 consensus_queue.cc:260] T 84993b2f63e7421290a161dde4bb0607 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:16.127735 1172 tablet_bootstrap.cc:492] T 786d9148fb1d417a8c2c8b8c53665899 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:16.127812 1167 ts_tablet_manager.cc:1434] T 84993b2f63e7421290a161dde4bb0607 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251029 02:52:16.105237 2762 tablet.cc:1722] T 6789bc16664e4b37b86702449ab27855 P c756528642a543eeacd93cef3e335a7d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 02:52:16.110808 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c45ffc304e73490387ce800b702b5881" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:16.113137 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0fec06ad37d4c12a7045d424b88cc5e" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:16.113499 919 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5815626b05204b578cc4a110f246bf10" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:16.128530 1167 tablet_bootstrap.cc:492] T ad214c870ee649279ed7321c5b69cbee P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:16.129408 1174 tablet_replica.cc:333] T 738046ab57334f2dab9f4aab979c5b42 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:16.129580 1174 raft_consensus.cc:2243] T 738046ab57334f2dab9f4aab979c5b42 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:16.129886 1174 raft_consensus.cc:2272] T 738046ab57334f2dab9f4aab979c5b42 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:16.130541 1174 ts_tablet_manager.cc:1916] T 738046ab57334f2dab9f4aab979c5b42 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:16.134387 849 leader_election.cc:304] T af4860c6b72f4d988b206b5ef2f22906 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters: b495a49349324f7d81908f2da700f0d9
I20251029 02:52:16.134614 2756 raft_consensus.cc:2804] T c32c28b0e74a4af3a0e69bbf24ab1a1f P c756528642a543eeacd93cef3e335a7d [term 2 FOLLOWER]: Leader election won for term 2
I20251029 02:52:16.135694 2756 raft_consensus.cc:697] T c32c28b0e74a4af3a0e69bbf24ab1a1f P c756528642a543eeacd93cef3e335a7d [term 2 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:52:16.135761 2740 raft_consensus.cc:2804] T af4860c6b72f4d988b206b5ef2f22906 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251029 02:52:16.135942 2740 raft_consensus.cc:493] T af4860c6b72f4d988b206b5ef2f22906 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election (detected failure of leader b495a49349324f7d81908f2da700f0d9)
I20251029 02:52:16.135986 2756 consensus_queue.cc:237] T c32c28b0e74a4af3a0e69bbf24ab1a1f P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:16.136072 2740 raft_consensus.cc:3060] T af4860c6b72f4d988b206b5ef2f22906 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Advancing to term 2
I20251029 02:52:16.136754 1172 tablet_bootstrap.cc:654] T 786d9148fb1d417a8c2c8b8c53665899 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:16.134693 2656 consensus_queue.cc:1048] T 72f9d4408f384e7093316d85f092e1b2 P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:16.138085 524 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_ae6101d366dc45db81fc1af337fb9d31" } new_table_name: "default.Table_d07c65e3c15d47aa9b724a151431aa52" modify_external_catalogs: true
I20251029 02:52:16.138893 2740 raft_consensus.cc:515] T af4860c6b72f4d988b206b5ef2f22906 P c756528642a543eeacd93cef3e335a7d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:16.139748 2740 leader_election.cc:290] T af4860c6b72f4d988b206b5ef2f22906 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 election: Requested vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
W20251029 02:52:16.141367 847 leader_election.cc:343] T af4860c6b72f4d988b206b5ef2f22906 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: af4860c6b72f4d988b206b5ef2f22906
W20251029 02:52:16.142807 711 leader_election.cc:343] T f93b0c39d7174dabbe018b1546dbc198 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
W20251029 02:52:16.144325 711 leader_election.cc:343] T defb795026a44e12afb2defb9c9c6db2 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
W20251029 02:52:16.145280 711 leader_election.cc:343] T c45ffc304e73490387ce800b702b5881 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:16.145534 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d57c43e61f3487099a9713f8885a85a. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:16.145721 711 leader_election.cc:343] T b0fec06ad37d4c12a7045d424b88cc5e P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
W20251029 02:52:16.149621 711 leader_election.cc:343] T 5815626b05204b578cc4a110f246bf10 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:16.128928 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1039) took 4583 ms. Trace:
I20251029 02:52:16.150986 1166 raft_consensus.cc:385] T 372d93914abc41b38a702742b7db0bc2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:16.151078 1166 raft_consensus.cc:740] T 372d93914abc41b38a702742b7db0bc2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:16.151000 1176 rpcz_store.cc:276] 1029 02:52:11.545578 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:11.602946 (+ 57368us) service_pool.cc:225] Handling call
1029 02:52:16.108371 (+4505425us) ts_tablet_manager.cc:1151] Deleting tablet c7a3e6d8e3fb45898d8a5e094dbf0fd7
1029 02:52:16.128742 (+ 20371us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:16.128908 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4505330}
I20251029 02:52:16.151298 1166 consensus_queue.cc:260] T 372d93914abc41b38a702742b7db0bc2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:16.151798 1166 ts_tablet_manager.cc:1434] T 372d93914abc41b38a702742b7db0bc2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.026s user 0.001s sys 0.000s
I20251029 02:52:16.152769 1167 tablet_bootstrap.cc:654] T ad214c870ee649279ed7321c5b69cbee P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:16.153592 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 957) took 5253 ms. Trace:
I20251029 02:52:16.153669 1175 rpcz_store.cc:276] 1029 02:52:10.899870 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:10.899924 (+ 54us) service_pool.cc:225] Handling call
1029 02:52:16.013634 (+5113710us) ts_tablet_manager.cc:1151] Deleting tablet 12432c84ca104b0e99926a247318906a
1029 02:52:16.013641 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:16.153569 (+139928us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5113549}
W20251029 02:52:16.154114 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 12432c84ca104b0e99926a247318906a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 12432c84ca104b0e99926a247318906a already in progress: creating tablet
I20251029 02:52:16.154177 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 958) took 5253 ms. Trace:
I20251029 02:52:16.154244 1175 rpcz_store.cc:276] 1029 02:52:10.900814 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:10.900860 (+ 46us) service_pool.cc:225] Handling call
1029 02:52:16.154054 (+5253194us) ts_tablet_manager.cc:1151] Deleting tablet e74eda48dca647efacabe525ef316944
1029 02:52:16.154064 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:16.154163 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5253056}
I20251029 02:52:16.154656 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 967) took 5180 ms. Trace:
I20251029 02:52:16.154723 1175 rpcz_store.cc:276] 1029 02:52:10.974042 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:10.974112 (+ 70us) service_pool.cc:225] Handling call
1029 02:52:16.154569 (+5180457us) ts_tablet_manager.cc:1151] Deleting tablet 4b841723726e40bda0187fb84bf36a6a
1029 02:52:16.154577 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:16.154642 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5138223}
I20251029 02:52:16.155313 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 973) took 4950 ms. Trace:
I20251029 02:52:16.155469 1175 rpcz_store.cc:276] 1029 02:52:11.204793 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:11.204849 (+ 56us) service_pool.cc:225] Handling call
1029 02:52:16.155104 (+4950255us) ts_tablet_manager.cc:1151] Deleting tablet 509288c033d54e4b9da3f26a30dc46a0
1029 02:52:16.155113 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:16.155288 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4950064}
W20251029 02:52:16.155604 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet e74eda48dca647efacabe525ef316944 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e74eda48dca647efacabe525ef316944 already in progress: creating tablet
W20251029 02:52:16.155846 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 4b841723726e40bda0187fb84bf36a6a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4b841723726e40bda0187fb84bf36a6a already in progress: creating tablet
I20251029 02:52:16.156075 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 974) took 4948 ms. Trace:
W20251029 02:52:16.156162 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 509288c033d54e4b9da3f26a30dc46a0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 509288c033d54e4b9da3f26a30dc46a0 already in progress: creating tablet
I20251029 02:52:16.156152 1175 rpcz_store.cc:276] 1029 02:52:11.207829 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:11.207885 (+ 56us) service_pool.cc:225] Handling call
1029 02:52:16.155949 (+4948064us) ts_tablet_manager.cc:1151] Deleting tablet c172417d1d444c439e9cda611a2ed551
1029 02:52:16.155962 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:16.156056 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4947889}
I20251029 02:52:16.156699 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 976) took 4908 ms. Trace:
W20251029 02:52:16.142014 989 consensus_peers.cc:597] T afb19ba39399410bb90b6225d7b5b5b1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 -> Peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Couldn't send request to peer b495a49349324f7d81908f2da700f0d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: afb19ba39399410bb90b6225d7b5b5b1. This is attempt 1: this message will repeat every 5th retry.
W20251029 02:52:16.158313 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet c172417d1d444c439e9cda611a2ed551 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c172417d1d444c439e9cda611a2ed551 already in progress: creating tablet
W20251029 02:52:16.159178 989 leader_election.cc:343] T c32c28b0e74a4af3a0e69bbf24ab1a1f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: c32c28b0e74a4af3a0e69bbf24ab1a1f
W20251029 02:52:16.159469 989 consensus_peers.cc:597] T 59eee86f249e4d6a9d09d30276dcd659 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 -> Peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Couldn't send request to peer b495a49349324f7d81908f2da700f0d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 59eee86f249e4d6a9d09d30276dcd659. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:16.166031 2655 raft_consensus.cc:493] T c081d2eacfac4ca8a0048e2a1c4f8f83 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:16.165947 2774 tablet.cc:1722] T 731c0acfbe204e8199ad246e38ae2ccd P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 02:52:16.166354 2655 raft_consensus.cc:515] T c081d2eacfac4ca8a0048e2a1c4f8f83 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:16.167339 2655 leader_election.cc:290] T c081d2eacfac4ca8a0048e2a1c4f8f83 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:16.181159 885 tablet_service.cc:1552] Processing DeleteTablet for tablet e91d6bdca3ad40c48d93e6196644462c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:15 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:16.181655 885 tablet_service.cc:1552] Processing DeleteTablet for tablet 18632de3bbbd491a86460497544580c1 with delete_type TABLET_DATA_DELETED (replaced by tablet c3dabe9d8e534953838ce5f0e3259e9d) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:16.184496 885 tablet_service.cc:1552] Processing DeleteTablet for tablet a3c8b25e6f4849d6aadca322d5365af1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:15 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:16.187460 885 tablet_service.cc:1552] Processing DeleteTablet for tablet 6dc30c614be64b8ba629d9884d450290 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:15 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:16.187613 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1040) took 4641 ms. Trace:
I20251029 02:52:16.187723 1176 rpcz_store.cc:276] 1029 02:52:11.545647 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:11.603046 (+ 57399us) service_pool.cc:225] Handling call
1029 02:52:16.151324 (+4548278us) ts_tablet_manager.cc:1151] Deleting tablet 7dbc9c8b110a48f0a801ad468738c3a3
1029 02:52:16.187477 (+ 36153us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:16.187594 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4548165}
I20251029 02:52:16.188076 1166 tablet_bootstrap.cc:492] T 72f9d4408f384e7093316d85f092e1b2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:16.188109 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1041) took 4642 ms. Trace:
I20251029 02:52:16.188241 1176 rpcz_store.cc:276] 1029 02:52:11.545722 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:11.603179 (+ 57457us) service_pool.cc:225] Handling call
1029 02:52:16.187978 (+4584799us) ts_tablet_manager.cc:1151] Deleting tablet 96f7876e5d6d4a0db236df5e7f266fd2
1029 02:52:16.187989 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:16.188091 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4584695}
I20251029 02:52:16.188580 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1042) took 4642 ms. Trace:
I20251029 02:52:16.188661 1176 rpcz_store.cc:276] 1029 02:52:11.545786 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:11.603289 (+ 57503us) service_pool.cc:225] Handling call
1029 02:52:16.188469 (+4585180us) ts_tablet_manager.cc:1151] Deleting tablet ecf3b900206b4c86a7adbc83a769d90e
1029 02:52:16.188481 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:16.188565 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4585081}
I20251029 02:52:16.189021 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1021) took 4645 ms. Trace:
I20251029 02:52:16.189112 1176 rpcz_store.cc:276] 1029 02:52:11.543788 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:11.543977 (+ 189us) service_pool.cc:225] Handling call
1029 02:52:16.188909 (+4644932us) ts_tablet_manager.cc:1151] Deleting tablet 86fc22bb84794149b598b97df41eb881
1029 02:52:16.188919 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:16.189005 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4582280}
I20251029 02:52:16.189419 1176 tablet_replica.cc:333] T c32c28b0e74a4af3a0e69bbf24ab1a1f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:16.189661 1176 raft_consensus.cc:2243] T c32c28b0e74a4af3a0e69bbf24ab1a1f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 2 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:16.189854 1176 raft_consensus.cc:2272] T c32c28b0e74a4af3a0e69bbf24ab1a1f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:16.190452 1166 tablet_bootstrap.cc:654] T 72f9d4408f384e7093316d85f092e1b2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:16.190496 1176 ts_tablet_manager.cc:1916] T c32c28b0e74a4af3a0e69bbf24ab1a1f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:16.191708 885 tablet_service.cc:1552] Processing DeleteTablet for tablet e60b029eb3ec466cbdced20d536f4f47 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:15 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:16.192111 885 tablet_service.cc:1552] Processing DeleteTablet for tablet f1e975b267464e64a5bbe24ef4e308a3 with delete_type TABLET_DATA_DELETED (replaced by tablet b00b7696d0dc46a2aec79ebf3c66f39f) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:16.192876 885 tablet_service.cc:1552] Processing DeleteTablet for tablet 688af6580c4d4637a39f817b4e324618 with delete_type TABLET_DATA_DELETED (replaced by tablet e9387edc3c434f5881a3a43e79a3af58) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:16.192898 1058 raft_consensus.cc:3060] T 27b08f88b99643d28ac609e45bf8104e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:16.193257 885 tablet_service.cc:1552] Processing DeleteTablet for tablet d29cc2a0c8b84e3ebadc7f571d666847 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:00 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:16.193619 885 tablet_service.cc:1552] Processing DeleteTablet for tablet c081d2eacfac4ca8a0048e2a1c4f8f83 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:04 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:16.194149 885 tablet_service.cc:1505] Processing CreateTablet for tablet 2925722be8a045c689c69ac729fb3799 (DEFAULT_TABLE table=default.Table_d8c5abd34fad48fb8ec28152a4b37e38 [id=f6160584ba214beb8beacc856d5a9ca6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:16.194591 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2925722be8a045c689c69ac729fb3799. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:16.198799 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "af4860c6b72f4d988b206b5ef2f22906" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:52:16.199221 1060 raft_consensus.cc:3060] T af4860c6b72f4d988b206b5ef2f22906 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Advancing to term 2
I20251029 02:52:16.203838 886 tablet_service.cc:1505] Processing CreateTablet for tablet b3dc6e55bb2e47bab7df5df62a1bcb51 (DEFAULT_TABLE table=default.Table_ca88b32b14364d2aab4e1b1480676c12 [id=8aacd9b9d75346419887f8be922803ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:16.204236 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3dc6e55bb2e47bab7df5df62a1bcb51. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:16.206786 519 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 70850519135644d1ac3d596ecf67221f consensus state has no leader
W20251029 02:52:16.207120 523 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 753446e48bed460cb1f9e03868b0d3a4 consensus state has no leader
I20251029 02:52:16.207438 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_6206526535a84403a4b5e8c8ad05887b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:16.208990 519 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet 6789bc16664e4b37b86702449ab27855 (table default.Table_9a7d7f4f9073437097aca9f7d6dc665e [id=ab8aeaf90f6c4bc2ad5a98780945c40a]). Expected version 2 got 0
W20251029 02:52:16.209224 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6206526535a84403a4b5e8c8ad05887b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:16.209888 519 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_b652acd864e34c469594fa7f6ac8c433" } new_table_name: "default.Table_df32b9165e994364ba2a30400ae6e538" modify_external_catalogs: true
I20251029 02:52:16.214314 1059 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c081d2eacfac4ca8a0048e2a1c4f8f83" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:16.219197 886 tablet_service.cc:1552] Processing DeleteTablet for tablet 19e15a6df77f4e0c9b40ae4f49fdd06f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:15 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:16.222823 885 tablet_service.cc:1505] Processing CreateTablet for tablet 9542585fd9824a55b3ad2f120065a02d (DEFAULT_TABLE table=default.Table_7eb1d904f38342dc85bb06681099d741 [id=cbadf4b0377b4c3dbf106b533a18acdf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:16.223379 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9542585fd9824a55b3ad2f120065a02d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:16.225310 884 tablet_service.cc:1552] Processing DeleteTablet for tablet 2849e856dd714da88fca5a9fafe74c22 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:15 UTC) from {username='slave'} at 127.0.0.1:35702
W20251029 02:52:16.225625 522 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3bd0abf208c54372a8c9504a60988435 consensus state has no leader
I20251029 02:52:16.225920 884 tablet_service.cc:1505] Processing CreateTablet for tablet 4905b28faec74d30859461d770282f41 (DEFAULT_TABLE table=default.Table_7511abddcde942a4adb0a23b3a345d06 [id=d9f9ae6423a34675968f74ae094aee89]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:16.226260 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4905b28faec74d30859461d770282f41. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:16.231909 524 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d22b01dc872b4ebb8f65b586b29eea00 consensus state has no leader
I20251029 02:52:16.232900 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_1d64090271db4acc8f23e7b4c0d4947c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:16.233453 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1d64090271db4acc8f23e7b4c0d4947c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:16.234361 884 tablet_service.cc:1552] Processing DeleteTablet for tablet 46e054ff563f448d86d8b4f84810291b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:15 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:16.234668 517 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_9c149fba4ea94e5dabe5d6584e7c471a" } modify_external_catalogs: true
I20251029 02:52:16.234756 884 tablet_service.cc:1552] Processing DeleteTablet for tablet 5bcadc0b68e84ae290690edef392a45b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:15 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:16.234818 517 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_9c149fba4ea94e5dabe5d6584e7c471a" } modify_external_catalogs: true
I20251029 02:52:16.235037 884 tablet_service.cc:1552] Processing DeleteTablet for tablet 60f0eadc074a4872b6638314fed98e0e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:15 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:16.235491 884 tablet_service.cc:1505] Processing CreateTablet for tablet 9426756257c64db982d1e879c0c0f8a9 (DEFAULT_TABLE table=default.Table_75325372892242518c3f0a4b8b657c48 [id=9812d79d53084361ba6cf509fb6eb0f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:16.235951 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_07b6fa17dfc04d919f5a344a8b225333"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:16.236100 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9426756257c64db982d1e879c0c0f8a9. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:16.236366 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_07b6fa17dfc04d919f5a344a8b225333 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:16.237370 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_b5a3de99a1d84f62b07ded51499e5b82"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:16.237757 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b5a3de99a1d84f62b07ded51499e5b82 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:16.239310 520 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_7fdd50180d294d44bc4c64f2cc5dc57a" } modify_external_catalogs: true
I20251029 02:52:16.239440 520 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_7fdd50180d294d44bc4c64f2cc5dc57a" } modify_external_catalogs: true
I20251029 02:52:16.241819 1060 raft_consensus.cc:2468] T af4860c6b72f4d988b206b5ef2f22906 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 2.
I20251029 02:52:16.243774 524 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_d5b1b53f211a44bf944f53dfde769c5d" } modify_external_catalogs: true
I20251029 02:52:16.243948 524 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_d5b1b53f211a44bf944f53dfde769c5d" } modify_external_catalogs: true
I20251029 02:52:16.245337 525 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_0cd699a8b604487f89d72e6d948f2bc2" } modify_external_catalogs: true
I20251029 02:52:16.245481 525 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_0cd699a8b604487f89d72e6d948f2bc2" } modify_external_catalogs: true
W20251029 02:52:16.248390 519 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ba51eb6947da4a4aa4b4fb37ecf05975 consensus state has no leader
W20251029 02:52:16.252035 989 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:38481, user_credentials={real_user=slave}} blocked reactor thread for 91418.6us
I20251029 02:52:16.253053 886 tablet_service.cc:1505] Processing CreateTablet for tablet b64880e732f54510a3954248cc940fbc (DEFAULT_TABLE table=default.Table_4098c28776dd4f41864c61d2299b90cd [id=47ee5993b95a477db52f9c51d034e549]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:16.253479 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b64880e732f54510a3954248cc940fbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:16.256186 522 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_6fa2b2dfa5c240ba95baf05031690542" } modify_external_catalogs: true
I20251029 02:52:16.256340 522 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_6fa2b2dfa5c240ba95baf05031690542" } modify_external_catalogs: true
W20251029 02:52:16.256578 849 leader_election.cc:343] T 217144957930495aa3e9dfcfde268e71 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Not found: Tablet not found: 217144957930495aa3e9dfcfde268e71
I20251029 02:52:16.256692 849 leader_election.cc:304] T 217144957930495aa3e9dfcfde268e71 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9
I20251029 02:52:16.256922 2740 raft_consensus.cc:2749] T 217144957930495aa3e9dfcfde268e71 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251029 02:52:16.257108 849 consensus_peers.cc:597] T 6f8cf09f23984b879b60ab6b27c7d52e P c756528642a543eeacd93cef3e335a7d -> Peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Couldn't send request to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251029 02:52:16.257253 849 consensus_peers.cc:597] T 6142018e44af4d32818adefa2ee43d02 P c756528642a543eeacd93cef3e335a7d -> Peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Couldn't send request to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251029 02:52:16.258299 849 leader_election.cc:343] T c081d2eacfac4ca8a0048e2a1c4f8f83 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:16.261227 517 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 7fba607c566c4fa1a803cba8d86ca646
I20251029 02:52:16.262853 891 tablet_service.cc:1505] Processing CreateTablet for tablet d9b3adb851bc46feb77e4b484eb0b8d4 (DEFAULT_TABLE table=default.Table_024e9ca04c014a27bc2b62227d4828d0 [id=b7453480e4d84319a2ce51432f83659b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:16.263262 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9b3adb851bc46feb77e4b484eb0b8d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:16.272207 1167 tablet_bootstrap.cc:492] T ad214c870ee649279ed7321c5b69cbee P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:16.272410 1167 ts_tablet_manager.cc:1403] T ad214c870ee649279ed7321c5b69cbee P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.144s user 0.003s sys 0.001s
I20251029 02:52:16.273856 1167 raft_consensus.cc:359] T ad214c870ee649279ed7321c5b69cbee P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:16.274065 1167 raft_consensus.cc:385] T ad214c870ee649279ed7321c5b69cbee P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:16.274124 1167 raft_consensus.cc:740] T ad214c870ee649279ed7321c5b69cbee P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:16.274318 1167 consensus_queue.cc:260] T ad214c870ee649279ed7321c5b69cbee P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:16.279008 891 tablet_service.cc:1505] Processing CreateTablet for tablet 09bbf13b23334bbf88f2eb2e327b20f2 (DEFAULT_TABLE table=default.Table_bf304b9f44274cf9b56f9e25fb0e9875 [id=1e7048e956fa484093370764b98d30cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:16.279570 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09bbf13b23334bbf88f2eb2e327b20f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:16.286660 900 tablet_service.cc:1505] Processing CreateTablet for tablet 6d57c43e61f3487099a9713f8885a85a (DEFAULT_TABLE table=default.Table_6facbfaed7644f2f9c16f0fe7316e2fc [id=4037bd8625d146409a7878c4e5b6643c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:16.173400 755 tablet_service.cc:1505] Processing CreateTablet for tablet 242e56760d8940cfb06baeacc3c6c60d (DEFAULT_TABLE table=default.Table_590e7ccd21564447951d30fbf609598e [id=22ec2407ea2341458628db861b630a2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:16.287022 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d57c43e61f3487099a9713f8885a85a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:16.287262 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 242e56760d8940cfb06baeacc3c6c60d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:16.291540 524 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 90c2ed692a954a25bdf79b9884f6c5ce
I20251029 02:52:16.292694 751 tablet_service.cc:1552] Processing DeleteTablet for tablet 90c2ed692a954a25bdf79b9884f6c5ce with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:16 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:16.291543 520 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 6408901d1e5f4ac8a7bff0de708660eb
I20251029 02:52:16.294068 751 tablet_service.cc:1552] Processing DeleteTablet for tablet 6408901d1e5f4ac8a7bff0de708660eb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:16 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:16.295445 879 tablet_service.cc:1505] Processing CreateTablet for tablet 5bb7a9e5f5b6424fbae965d050f0d1c8 (DEFAULT_TABLE table=default.Table_442850f577aa437a820314bbf3df9ac4 [id=1bbf331b6fea4c989aa693d404e31d53]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:16.295806 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5bb7a9e5f5b6424fbae965d050f0d1c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:16.174209 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c081d2eacfac4ca8a0048e2a1c4f8f83" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:16.175601 756 tablet_service.cc:1505] Processing CreateTablet for tablet 9fcf4a80ebb446f89159659a663eede4 (DEFAULT_TABLE table=default.Table_98b6a6ee7ae34368bef7d883e216978d [id=603ef089eb46405aa4a7981b936fef8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:16.180536 750 tablet_service.cc:1505] Processing CreateTablet for tablet 5bb7a9e5f5b6424fbae965d050f0d1c8 (DEFAULT_TABLE table=default.Table_442850f577aa437a820314bbf3df9ac4 [id=1bbf331b6fea4c989aa693d404e31d53]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:16.188081 1172 tablet_bootstrap.cc:492] T 786d9148fb1d417a8c2c8b8c53665899 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:16.296669 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fcf4a80ebb446f89159659a663eede4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:16.296772 1172 ts_tablet_manager.cc:1403] T 786d9148fb1d417a8c2c8b8c53665899 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.169s user 0.002s sys 0.002s
I20251029 02:52:16.297492 1172 raft_consensus.cc:359] T 786d9148fb1d417a8c2c8b8c53665899 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:16.297641 1172 raft_consensus.cc:385] T 786d9148fb1d417a8c2c8b8c53665899 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:16.297688 1172 raft_consensus.cc:740] T 786d9148fb1d417a8c2c8b8c53665899 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:16.190294 749 tablet_service.cc:1505] Processing CreateTablet for tablet 4d8c8eb1fade468b85447b0f8b98a50f (DEFAULT_TABLE table=default.Table_db4e930bec75474ba682f938dd32ec41 [id=2b9a7fb76d634f25a4ef6599331c2480]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:16.298075 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d8c8eb1fade468b85447b0f8b98a50f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:16.301173 1172 consensus_queue.cc:260] T 786d9148fb1d417a8c2c8b8c53665899 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:16.301826 1172 ts_tablet_manager.cc:1434] T 786d9148fb1d417a8c2c8b8c53665899 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251029 02:52:16.302338 879 tablet_service.cc:1505] Processing CreateTablet for tablet 9fcf4a80ebb446f89159659a663eede4 (DEFAULT_TABLE table=default.Table_98b6a6ee7ae34368bef7d883e216978d [id=603ef089eb46405aa4a7981b936fef8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:16.302765 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fcf4a80ebb446f89159659a663eede4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:16.306795 900 tablet_service.cc:1505] Processing CreateTablet for tablet 242e56760d8940cfb06baeacc3c6c60d (DEFAULT_TABLE table=default.Table_590e7ccd21564447951d30fbf609598e [id=22ec2407ea2341458628db861b630a2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:16.307199 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 242e56760d8940cfb06baeacc3c6c60d. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:16.307482 849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:45003, user_credentials={real_user=slave}} blocked reactor thread for 48884.2us
I20251029 02:52:16.193866 754 tablet_service.cc:1505] Processing CreateTablet for tablet b2daeaaa083246ae9431296a8565eef8 (DEFAULT_TABLE table=default.Table_30bf322e8193402e80523ebfba5c8e43 [id=d78df9db30d94001bb0d191bd96da198]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:16.211166 2722 raft_consensus.cc:493] T e3000183230e4350b47f5085952f2db6 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:16.307938 2722 raft_consensus.cc:515] T e3000183230e4350b47f5085952f2db6 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:16.308009 849 leader_election.cc:304] T af4860c6b72f4d988b206b5ef2f22906 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters: b495a49349324f7d81908f2da700f0d9
I20251029 02:52:16.308112 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2daeaaa083246ae9431296a8565eef8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:16.308539 2722 leader_election.cc:290] T e3000183230e4350b47f5085952f2db6 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:52:16.212711 739 tablet_service.cc:1505] Processing CreateTablet for tablet ba51eb6947da4a4aa4b4fb37ecf05975 (DEFAULT_TABLE table=default.Table_b652acd864e34c469594fa7f6ac8c433 [id=c9a5c95591d04c269c1b2d838d77379a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:16.308840 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba51eb6947da4a4aa4b4fb37ecf05975. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:16.314512 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5bb7a9e5f5b6424fbae965d050f0d1c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:16.221678 738 tablet_service.cc:1505] Processing CreateTablet for tablet 77ba190fdaaa44c8ae3b40d1acf73314 (DEFAULT_TABLE table=default.Table_7533d0093ad94ff6b065550375df858a [id=8bd9b71555d145dba4a06b2bf9bd88d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:16.224188 744 tablet_service.cc:1505] Processing CreateTablet for tablet f145da2e92b14c5fa32aa2ea1fc78b2f (DEFAULT_TABLE table=default.Table_fd79bf1c016a4671837a7237e117dcf5 [id=daeab145500a47dbaaf7c8655c1b503a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:16.262243 740 tablet_service.cc:1552] Processing DeleteTablet for tablet 7fba607c566c4fa1a803cba8d86ca646 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:16 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:16.315323 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 738046ab57334f2dab9f4aab979c5b42 (table default.Table_31d42031714e415fab63cc5b9933fe0c [id=33b26e2658ff4270a5f1306b6cde34db]) successfully deleted
I20251029 02:52:16.315410 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77ba190fdaaa44c8ae3b40d1acf73314. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:16.316255 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet ba699862e6f646c6beaa39f080d0faba (table default.Table_84e9fe6549cc42ed94c0f4c0b489f726 [id=feb1f56ae44249c6adce65637017bb43]) successfully deleted
W20251029 02:52:16.317031 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 509288c033d54e4b9da3f26a30dc46a0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 509288c033d54e4b9da3f26a30dc46a0 already in progress: creating tablet
I20251029 02:52:16.318338 1167 ts_tablet_manager.cc:1434] T ad214c870ee649279ed7321c5b69cbee P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.046s user 0.002s sys 0.000s
I20251029 02:52:16.318713 1167 tablet_bootstrap.cc:492] T eca016aed05e4ce09e1a2bf96e2286f5 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
W20251029 02:52:16.318711 712 leader_election.cc:343] T 055184c641ba45a588bca611fb24d130 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:16.318814 712 leader_election.cc:304] T 055184c641ba45a588bca611fb24d130 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:16.319227 712 leader_election.cc:304] T 32519fbffcef44a890161e5f3043806c P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters: c756528642a543eeacd93cef3e335a7d
I20251029 02:52:16.319453 2786 raft_consensus.cc:2749] T 055184c641ba45a588bca611fb24d130 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:16.319730 2786 raft_consensus.cc:2804] T 32519fbffcef44a890161e5f3043806c P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:16.319792 2786 raft_consensus.cc:493] T 32519fbffcef44a890161e5f3043806c P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:16.319849 2786 raft_consensus.cc:3060] T 32519fbffcef44a890161e5f3043806c P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:16.320310 1167 tablet_bootstrap.cc:654] T eca016aed05e4ce09e1a2bf96e2286f5 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:16.302868 776 raft_consensus.cc:2468] T c081d2eacfac4ca8a0048e2a1c4f8f83 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:16.192548 748 tablet_service.cc:1505] Processing CreateTablet for tablet f6fdaac654d4434d81963746f53205b2 (DEFAULT_TABLE table=default.Table_b2d77ffcda024d1e80fd8c298ddfd3e6 [id=c309ea497e8b4c8dbafeb7162d58236d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:16.306245 1175 rpcz_store.cc:276] 1029 02:52:11.248687 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:11.248777 (+ 90us) service_pool.cc:225] Handling call
1029 02:52:16.156557 (+4907780us) ts_tablet_manager.cc:1151] Deleting tablet 7dbc9c8b110a48f0a801ad468738c3a3
1029 02:52:16.156568 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:16.156681 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4907583}
I20251029 02:52:16.217242 742 tablet_service.cc:1505] Processing CreateTablet for tablet 16b1cc4db77d426ab0419ec13bd14c8b (DEFAULT_TABLE table=default.Table_e20d6df47ebc45ff8cd74e180a2fe940 [id=269a2458193e4140938bb7384b1a0fa3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:16.314870 1172 tablet_bootstrap.cc:492] T 506f8c112a3b43e1a3d6bbafb8cb81e8 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:16.322435 2786 raft_consensus.cc:515] T 32519fbffcef44a890161e5f3043806c P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
W20251029 02:52:16.323261 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 7dbc9c8b110a48f0a801ad468738c3a3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7dbc9c8b110a48f0a801ad468738c3a3 already in progress: creating tablet
I20251029 02:52:16.324800 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f145da2e92b14c5fa32aa2ea1fc78b2f. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:16.326740 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet c172417d1d444c439e9cda611a2ed551 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c172417d1d444c439e9cda611a2ed551 already in progress: creating tablet
I20251029 02:52:16.327203 524 master_service.cc:954] ReplaceTablet: received request to replace tablet c73059366f4b45528f8b907263c4d085 from {username='slave'} at 127.0.0.1:35314
I20251029 02:52:16.328306 2786 leader_election.cc:290] T 32519fbffcef44a890161e5f3043806c P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 election: Requested vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:16.332260 1172 tablet_bootstrap.cc:654] T 506f8c112a3b43e1a3d6bbafb8cb81e8 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:16.334079 847 leader_election.cc:304] T c081d2eacfac4ca8a0048e2a1c4f8f83 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5
I20251029 02:52:16.335071 2201 raft_consensus.cc:2804] T c081d2eacfac4ca8a0048e2a1c4f8f83 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:16.335224 2201 raft_consensus.cc:493] T c081d2eacfac4ca8a0048e2a1c4f8f83 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:16.335371 2201 raft_consensus.cc:3060] T c081d2eacfac4ca8a0048e2a1c4f8f83 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:16.338842 2201 raft_consensus.cc:515] T c081d2eacfac4ca8a0048e2a1c4f8f83 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:16.340400 2201 leader_election.cc:290] T c081d2eacfac4ca8a0048e2a1c4f8f83 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:16.341984 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c081d2eacfac4ca8a0048e2a1c4f8f83" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:16.342464 776 raft_consensus.cc:3060] T c081d2eacfac4ca8a0048e2a1c4f8f83 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:16.350955 887 tablet_service.cc:1505] Processing CreateTablet for tablet 4d8c8eb1fade468b85447b0f8b98a50f (DEFAULT_TABLE table=default.Table_db4e930bec75474ba682f938dd32ec41 [id=2b9a7fb76d634f25a4ef6599331c2480]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:16.351426 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d8c8eb1fade468b85447b0f8b98a50f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:16.354244 776 raft_consensus.cc:2468] T c081d2eacfac4ca8a0048e2a1c4f8f83 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:16.355409 847 leader_election.cc:304] T c081d2eacfac4ca8a0048e2a1c4f8f83 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:16.356117 1172 tablet_bootstrap.cc:492] T 506f8c112a3b43e1a3d6bbafb8cb81e8 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:16.356412 1172 ts_tablet_manager.cc:1403] T 506f8c112a3b43e1a3d6bbafb8cb81e8 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.042s user 0.007s sys 0.000s
I20251029 02:52:16.356819 2747 raft_consensus.cc:2804] T c081d2eacfac4ca8a0048e2a1c4f8f83 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:16.356984 2747 raft_consensus.cc:697] T c081d2eacfac4ca8a0048e2a1c4f8f83 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:52:16.357261 2747 consensus_queue.cc:237] T c081d2eacfac4ca8a0048e2a1c4f8f83 P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:16.358368 1172 raft_consensus.cc:359] T 506f8c112a3b43e1a3d6bbafb8cb81e8 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:16.359197 1172 raft_consensus.cc:385] T 506f8c112a3b43e1a3d6bbafb8cb81e8 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:16.359367 1172 raft_consensus.cc:740] T 506f8c112a3b43e1a3d6bbafb8cb81e8 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:16.359886 1172 consensus_queue.cc:260] T 506f8c112a3b43e1a3d6bbafb8cb81e8 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:16.360730 1166 tablet_bootstrap.cc:492] T 72f9d4408f384e7093316d85f092e1b2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:16.360707 887 tablet_service.cc:1505] Processing CreateTablet for tablet f6fdaac654d4434d81963746f53205b2 (DEFAULT_TABLE table=default.Table_b2d77ffcda024d1e80fd8c298ddfd3e6 [id=c309ea497e8b4c8dbafeb7162d58236d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:16.360846 1172 ts_tablet_manager.cc:1434] T 506f8c112a3b43e1a3d6bbafb8cb81e8 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251029 02:52:16.361018 1166 ts_tablet_manager.cc:1403] T 72f9d4408f384e7093316d85f092e1b2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.173s user 0.002s sys 0.002s
I20251029 02:52:16.361104 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6fdaac654d4434d81963746f53205b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:16.361518 1172 tablet_bootstrap.cc:492] T 24f07b1efac64b099b0b5ed467463749 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:16.367987 1166 raft_consensus.cc:359] T 72f9d4408f384e7093316d85f092e1b2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:16.368723 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_e80c2a94126d48cf94c30936cdca8281"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:16.369354 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e80c2a94126d48cf94c30936cdca8281 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:16.371253 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16b1cc4db77d426ab0419ec13bd14c8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:16.374629 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6fdaac654d4434d81963746f53205b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:16.380756 2740 consensus_queue.cc:1048] T 748777c6cd6e4f49ab49912b719e48bd P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:16.372553 1176 ts_tablet_manager.cc:1929] T c32c28b0e74a4af3a0e69bbf24ab1a1f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:16.383630 1176 log.cc:1199] T c32c28b0e74a4af3a0e69bbf24ab1a1f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/c32c28b0e74a4af3a0e69bbf24ab1a1f
I20251029 02:52:16.383415 1166 raft_consensus.cc:385] T 72f9d4408f384e7093316d85f092e1b2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:16.384018 1176 ts_tablet_manager.cc:1950] T c32c28b0e74a4af3a0e69bbf24ab1a1f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:16.383945 1166 raft_consensus.cc:740] T 72f9d4408f384e7093316d85f092e1b2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:16.384348 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1023) took 4840 ms. Trace:
I20251029 02:52:16.384426 1176 rpcz_store.cc:276] 1029 02:52:11.544097 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:11.544139 (+ 42us) service_pool.cc:225] Handling call
1029 02:52:16.189368 (+4645229us) ts_tablet_manager.cc:1151] Deleting tablet c32c28b0e74a4af3a0e69bbf24ab1a1f
1029 02:52:16.189381 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:16.369199 (+179818us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:16.372451 (+ 3252us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:16.384329 (+ 11878us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4582707}
I20251029 02:52:16.384806 1176 tablet_replica.cc:333] T 1275e192871440408a8345fef9b56260 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:16.384992 1176 raft_consensus.cc:2243] T 1275e192871440408a8345fef9b56260 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:16.385293 1176 raft_consensus.cc:2272] T 1275e192871440408a8345fef9b56260 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:16.385318 1166 consensus_queue.cc:260] T 72f9d4408f384e7093316d85f092e1b2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:16.385849 1176 ts_tablet_manager.cc:1916] T 1275e192871440408a8345fef9b56260 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:16.387635 1166 ts_tablet_manager.cc:1434] T 72f9d4408f384e7093316d85f092e1b2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.026s user 0.000s sys 0.002s
I20251029 02:52:16.388204 888 tablet_service.cc:1505] Processing CreateTablet for tablet b2daeaaa083246ae9431296a8565eef8 (DEFAULT_TABLE table=default.Table_30bf322e8193402e80523ebfba5c8e43 [id=d78df9db30d94001bb0d191bd96da198]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:16.388644 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2daeaaa083246ae9431296a8565eef8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:16.392870 517 catalog_manager.cc:6194] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: default.Table_9a7d7f4f9073437097aca9f7d6dc665e [id=ab8aeaf90f6c4bc2ad5a98780945c40a] alter complete (version 2)
I20251029 02:52:16.393396 1176 ts_tablet_manager.cc:1929] T 1275e192871440408a8345fef9b56260 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:16.393507 1176 log.cc:1199] T 1275e192871440408a8345fef9b56260 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/1275e192871440408a8345fef9b56260
I20251029 02:52:16.393869 1176 ts_tablet_manager.cc:1950] T 1275e192871440408a8345fef9b56260 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:16.394191 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1024) took 4849 ms. Trace:
I20251029 02:52:16.394270 1176 rpcz_store.cc:276] 1029 02:52:11.544331 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:11.544374 (+ 43us) service_pool.cc:225] Handling call
1029 02:52:16.384758 (+4840384us) ts_tablet_manager.cc:1151] Deleting tablet 1275e192871440408a8345fef9b56260
1029 02:52:16.384768 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:16.389680 (+ 4912us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:16.393311 (+ 3631us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:16.394176 (+ 865us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4777653}
I20251029 02:52:16.394603 1176 tablet_replica.cc:333] T dabb8d760aae4aa09eb83cdb586c9788 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:16.394768 1176 raft_consensus.cc:2243] T dabb8d760aae4aa09eb83cdb586c9788 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:16.394937 1176 raft_consensus.cc:2272] T dabb8d760aae4aa09eb83cdb586c9788 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:16.395771 1176 ts_tablet_manager.cc:1916] T dabb8d760aae4aa09eb83cdb586c9788 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:16.395934 888 tablet_service.cc:1505] Processing CreateTablet for tablet ba51eb6947da4a4aa4b4fb37ecf05975 (DEFAULT_TABLE table=default.Table_b652acd864e34c469594fa7f6ac8c433 [id=c9a5c95591d04c269c1b2d838d77379a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:16.396318 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba51eb6947da4a4aa4b4fb37ecf05975. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:16.404767 2774 tablet.cc:1722] T 6789bc16664e4b37b86702449ab27855 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 02:52:16.408699 1176 ts_tablet_manager.cc:1929] T dabb8d760aae4aa09eb83cdb586c9788 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:16.408824 1176 log.cc:1199] T dabb8d760aae4aa09eb83cdb586c9788 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/dabb8d760aae4aa09eb83cdb586c9788
I20251029 02:52:16.409229 1176 ts_tablet_manager.cc:1950] T dabb8d760aae4aa09eb83cdb586c9788 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:16.409528 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1022) took 4865 ms. Trace:
I20251029 02:52:16.409603 1176 rpcz_store.cc:276] 1029 02:52:11.543882 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:11.543924 (+ 42us) service_pool.cc:225] Handling call
1029 02:52:16.394556 (+4850632us) ts_tablet_manager.cc:1151] Deleting tablet dabb8d760aae4aa09eb83cdb586c9788
1029 02:52:16.394566 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:16.401153 (+ 6587us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:16.408596 (+ 7443us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:16.409512 (+ 916us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4244,"tablet-delete.queue_time_us":4787927}
I20251029 02:52:16.409921 1176 tablet_replica.cc:333] T 13ee62fc2339410b9db70dfd3d3c6fa6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:16.410113 1176 raft_consensus.cc:2243] T 13ee62fc2339410b9db70dfd3d3c6fa6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:16.410184 1172 tablet_bootstrap.cc:654] T 24f07b1efac64b099b0b5ed467463749 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:16.410269 1166 tablet_bootstrap.cc:492] T ae4231daa2594258aa51419c97a2f50a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:16.410426 1176 raft_consensus.cc:2272] T 13ee62fc2339410b9db70dfd3d3c6fa6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:16.410907 891 tablet_service.cc:1505] Processing CreateTablet for tablet 1537b318704f48ccb01c48a4e783da94 (DEFAULT_TABLE table=default.Table_a7e37e7a8bba4557a83b0fe3f498df17 [id=1c093c8c547144d2a413329c199d2962]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:16.410988 1176 ts_tablet_manager.cc:1916] T 13ee62fc2339410b9db70dfd3d3c6fa6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:16.412125 517 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_81009c7f23bd483fac10943bb273ae87" } new_table_name: "default.Table_a75c04341d7244f3ba161a938bb401b3" modify_external_catalogs: true
I20251029 02:52:16.412920 519 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_1b4647439c30401a9285f7281255f7dd" } new_table_name: "default.Table_3117f7f41f3f40778cc552f49250a0a9" modify_external_catalogs: true
I20251029 02:52:16.415263 525 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 62f4a04568804c3da03393c08676c5da
I20251029 02:52:16.416544 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1537b318704f48ccb01c48a4e783da94. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:16.411839 849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:45003, user_credentials={real_user=slave}} blocked reactor thread for 103852us
I20251029 02:52:16.412708 894 tablet_service.cc:1505] Processing CreateTablet for tablet 77ba190fdaaa44c8ae3b40d1acf73314 (DEFAULT_TABLE table=default.Table_7533d0093ad94ff6b065550375df858a [id=8bd9b71555d145dba4a06b2bf9bd88d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:16.416873 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77ba190fdaaa44c8ae3b40d1acf73314. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:16.416885 522 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 818262aee43c4343b864e01f836e76b8
I20251029 02:52:16.413197 878 tablet_service.cc:1552] Processing DeleteTablet for tablet 7fba607c566c4fa1a803cba8d86ca646 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:16 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:16.413264 895 tablet_service.cc:1505] Processing CreateTablet for tablet f145da2e92b14c5fa32aa2ea1fc78b2f (DEFAULT_TABLE table=default.Table_fd79bf1c016a4671837a7237e117dcf5 [id=daeab145500a47dbaaf7c8655c1b503a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:16.421612 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f145da2e92b14c5fa32aa2ea1fc78b2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:16.422941 748 tablet_service.cc:1552] Processing DeleteTablet for tablet 818262aee43c4343b864e01f836e76b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:16 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:16.423207 1176 ts_tablet_manager.cc:1929] T 13ee62fc2339410b9db70dfd3d3c6fa6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:16.423312 1176 log.cc:1199] T 13ee62fc2339410b9db70dfd3d3c6fa6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/13ee62fc2339410b9db70dfd3d3c6fa6
I20251029 02:52:16.423606 750 tablet_service.cc:1552] Processing DeleteTablet for tablet e3000183230e4350b47f5085952f2db6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:01 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:16.423708 738 tablet_service.cc:1552] Processing DeleteTablet for tablet 62f4a04568804c3da03393c08676c5da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:16 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:16.426596 1058 raft_consensus.cc:3060] T 72f9d4408f384e7093316d85f092e1b2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:16.427970 1061 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c081d2eacfac4ca8a0048e2a1c4f8f83" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:52:16.414215 2756 raft_consensus.cc:2804] T af4860c6b72f4d988b206b5ef2f22906 P c756528642a543eeacd93cef3e335a7d [term 2 FOLLOWER]: Leader election won for term 2
I20251029 02:52:16.429742 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 975) took 5181 ms. Trace:
I20251029 02:52:16.429833 1175 rpcz_store.cc:276] 1029 02:52:11.248554 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:11.254789 (+ 6235us) service_pool.cc:225] Handling call
1029 02:52:16.321802 (+5067013us) ts_tablet_manager.cc:1151] Deleting tablet c7a3e6d8e3fb45898d8a5e094dbf0fd7
1029 02:52:16.408582 (+ 86780us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:16.429718 (+ 21136us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5065762}
W20251029 02:52:16.430397 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet c7a3e6d8e3fb45898d8a5e094dbf0fd7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c7a3e6d8e3fb45898d8a5e094dbf0fd7 already in progress: creating tablet
I20251029 02:52:16.431043 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 977) took 5140 ms. Trace:
W20251029 02:52:16.433005 519 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3e48adc758b24fa0b3163410f7af7505 consensus state has no leader
I20251029 02:52:16.434875 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_e6fd174aebf746e28bfa0c2ef9a699c2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:16.434938 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_0599b691b9134e749b5d3b952686b828"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:16.435485 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0599b691b9134e749b5d3b952686b828 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:52:16.435508 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e6fd174aebf746e28bfa0c2ef9a699c2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:16.436501 1176 ts_tablet_manager.cc:1950] T 13ee62fc2339410b9db70dfd3d3c6fa6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:16.436905 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1019) took 4893 ms. Trace:
I20251029 02:52:16.436981 1176 rpcz_store.cc:276] 1029 02:52:11.543625 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:11.543721 (+ 96us) service_pool.cc:225] Handling call
1029 02:52:16.409874 (+4866153us) ts_tablet_manager.cc:1151] Deleting tablet 13ee62fc2339410b9db70dfd3d3c6fa6
1029 02:52:16.409883 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:16.420021 (+ 10138us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:16.423097 (+ 3076us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:16.436888 (+ 13791us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8310,"tablet-delete.queue_time_us":4807256}
I20251029 02:52:16.437165 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_dc881c73ba4f46bf92afb390195d2b04"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:16.437296 1176 tablet_replica.cc:333] T af4860c6b72f4d988b206b5ef2f22906 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:16.437493 1176 raft_consensus.cc:2243] T af4860c6b72f4d988b206b5ef2f22906 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 2 FOLLOWER]: Raft consensus shutting down.
W20251029 02:52:16.437670 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dc881c73ba4f46bf92afb390195d2b04 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:16.439015 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_1cda56d119414969969012027b7e7671"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:16.439301 1176 raft_consensus.cc:2272] T af4860c6b72f4d988b206b5ef2f22906 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 2 FOLLOWER]: Raft consensus is shut down!
W20251029 02:52:16.439466 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1cda56d119414969969012027b7e7671 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:16.439872 1176 ts_tablet_manager.cc:1916] T af4860c6b72f4d988b206b5ef2f22906 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:16.439885 519 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_16b74e4bf0c240649a053af2967ae6bb" } new_table_name: "default.Table_09c621ba9fce4850ae1f0019e3c2abf8" modify_external_catalogs: true
I20251029 02:52:16.441443 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_a9246d0b85c04d90a2531912329b8366"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:16.442016 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a9246d0b85c04d90a2531912329b8366 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:16.455796 1176 ts_tablet_manager.cc:1929] T af4860c6b72f4d988b206b5ef2f22906 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:16.455937 1176 log.cc:1199] T af4860c6b72f4d988b206b5ef2f22906 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/af4860c6b72f4d988b206b5ef2f22906
I20251029 02:52:16.456405 1176 ts_tablet_manager.cc:1950] T af4860c6b72f4d988b206b5ef2f22906 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:16.456769 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1018) took 4913 ms. Trace:
I20251029 02:52:16.456864 1176 rpcz_store.cc:276] 1029 02:52:11.543544 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:11.551739 (+ 8195us) service_pool.cc:225] Handling call
1029 02:52:16.437251 (+4885512us) ts_tablet_manager.cc:1151] Deleting tablet af4860c6b72f4d988b206b5ef2f22906
1029 02:52:16.437259 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:16.447530 (+ 10271us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:16.455684 (+ 8154us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:16.456748 (+ 1064us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4785078}
I20251029 02:52:16.457062 2718 raft_consensus.cc:493] T 372d93914abc41b38a702742b7db0bc2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:16.457213 1176 tablet_replica.cc:333] T 9bb2166a590d4210b02dd779ef6654ad P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:16.457177 2718 raft_consensus.cc:515] T 372d93914abc41b38a702742b7db0bc2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:16.457370 1176 raft_consensus.cc:2243] T 9bb2166a590d4210b02dd779ef6654ad P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:16.457638 1176 raft_consensus.cc:2272] T 9bb2166a590d4210b02dd779ef6654ad P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:16.457933 2740 raft_consensus.cc:493] T 84993b2f63e7421290a161dde4bb0607 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:16.458070 2740 raft_consensus.cc:515] T 84993b2f63e7421290a161dde4bb0607 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:16.458216 1176 ts_tablet_manager.cc:1916] T 9bb2166a590d4210b02dd779ef6654ad P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:16.458209 2655 raft_consensus.cc:493] T 061f3d38c4604fa4b57e74ff6f6fc7da P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0cfd4bee16f84b7c8b0ff3ce14db20d5)
I20251029 02:52:16.458323 2655 raft_consensus.cc:515] T 061f3d38c4604fa4b57e74ff6f6fc7da P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:16.458585 2740 leader_election.cc:290] T 84993b2f63e7421290a161dde4bb0607 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:16.458829 2655 leader_election.cc:290] T 061f3d38c4604fa4b57e74ff6f6fc7da P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:16.459496 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 90c2ed692a954a25bdf79b9884f6c5ce with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:16 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:16.459812 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 6408901d1e5f4ac8a7bff0de708660eb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:16 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:16.460091 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 84993b2f63e7421290a161dde4bb0607 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:57 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:16.460160 2718 leader_election.cc:290] T 372d93914abc41b38a702742b7db0bc2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
W20251029 02:52:16.460714 847 leader_election.cc:343] T 061f3d38c4604fa4b57e74ff6f6fc7da P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 061f3d38c4604fa4b57e74ff6f6fc7da
W20251029 02:52:16.461694 847 consensus_peers.cc:597] T 7bb85a12f573494cbb3118d1947eda27 P c756528642a543eeacd93cef3e335a7d -> Peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Couldn't send request to peer b495a49349324f7d81908f2da700f0d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7bb85a12f573494cbb3118d1947eda27. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:16.446048 1172 tablet_bootstrap.cc:492] T 24f07b1efac64b099b0b5ed467463749 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:16.460212 766 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "84993b2f63e7421290a161dde4bb0607" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:16.463167 1175 rpcz_store.cc:276] 1029 02:52:11.290913 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:11.296696 (+ 5783us) service_pool.cc:225] Handling call
1029 02:52:16.430842 (+5134146us) ts_tablet_manager.cc:1151] Deleting tablet 96f7876e5d6d4a0db236df5e7f266fd2
1029 02:52:16.430858 (+ 16us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:16.431018 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5133920}
I20251029 02:52:16.463290 1172 ts_tablet_manager.cc:1403] T 24f07b1efac64b099b0b5ed467463749 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.102s user 0.002s sys 0.003s
I20251029 02:52:16.463300 766 raft_consensus.cc:2468] T 84993b2f63e7421290a161dde4bb0607 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:16.464130 2756 raft_consensus.cc:697] T af4860c6b72f4d988b206b5ef2f22906 P c756528642a543eeacd93cef3e335a7d [term 2 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:52:16.464308 1172 raft_consensus.cc:359] T 24f07b1efac64b099b0b5ed467463749 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:16.464495 1172 raft_consensus.cc:385] T 24f07b1efac64b099b0b5ed467463749 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:16.464540 1172 raft_consensus.cc:740] T 24f07b1efac64b099b0b5ed467463749 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:16.464448 2756 consensus_queue.cc:237] T af4860c6b72f4d988b206b5ef2f22906 P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:16.464740 1172 consensus_queue.cc:260] T 24f07b1efac64b099b0b5ed467463749 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:16.465226 1172 ts_tablet_manager.cc:1434] T 24f07b1efac64b099b0b5ed467463749 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 02:52:16.465502 1172 tablet_bootstrap.cc:492] T 7816556521144c928d7a02694e22e79f P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:16.468358 847 leader_election.cc:304] T 84993b2f63e7421290a161dde4bb0607 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:16.468894 2740 raft_consensus.cc:2804] T 84993b2f63e7421290a161dde4bb0607 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:16.468992 2740 raft_consensus.cc:493] T 84993b2f63e7421290a161dde4bb0607 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:16.469048 2740 raft_consensus.cc:3060] T 84993b2f63e7421290a161dde4bb0607 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:16.469995 2655 consensus_queue.cc:1048] T 6f8cf09f23984b879b60ab6b27c7d52e P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:16.470743 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 62f4a04568804c3da03393c08676c5da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:16 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:16.471041 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 818262aee43c4343b864e01f836e76b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:16 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:16.471565 2740 raft_consensus.cc:515] T 84993b2f63e7421290a161dde4bb0607 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:16.472095 2740 leader_election.cc:290] T 84993b2f63e7421290a161dde4bb0607 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:16.472833 1176 ts_tablet_manager.cc:1929] T 9bb2166a590d4210b02dd779ef6654ad P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:16.472958 1176 log.cc:1199] T 9bb2166a590d4210b02dd779ef6654ad P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/9bb2166a590d4210b02dd779ef6654ad
I20251029 02:52:16.473347 1176 ts_tablet_manager.cc:1950] T 9bb2166a590d4210b02dd779ef6654ad P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:16.473636 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1017) took 4930 ms. Trace:
W20251029 02:52:16.473713 849 leader_election.cc:343] T c081d2eacfac4ca8a0048e2a1c4f8f83 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:16.473721 1176 rpcz_store.cc:276] 1029 02:52:11.543446 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:11.552972 (+ 9526us) service_pool.cc:225] Handling call
1029 02:52:16.457144 (+4904172us) ts_tablet_manager.cc:1151] Deleting tablet 9bb2166a590d4210b02dd779ef6654ad
1029 02:52:16.457155 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:16.467860 (+ 10705us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:16.472698 (+ 4838us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:16.473621 (+ 923us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4804873}
I20251029 02:52:16.474112 1176 tablet_replica.cc:333] T 1bdb67125493430398ff5c80b73de2ac P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:16.474305 1176 raft_consensus.cc:2243] T 1bdb67125493430398ff5c80b73de2ac P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:16.474927 2649 consensus_queue.cc:1048] T 72f9d4408f384e7093316d85f092e1b2 P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:16.475485 888 tablet_service.cc:1505] Processing CreateTablet for tablet 16b1cc4db77d426ab0419ec13bd14c8b (DEFAULT_TABLE table=default.Table_e20d6df47ebc45ff8cd74e180a2fe940 [id=269a2458193e4140938bb7384b1a0fa3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:16.475935 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16b1cc4db77d426ab0419ec13bd14c8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:16.476114 1172 tablet_bootstrap.cc:654] T 7816556521144c928d7a02694e22e79f P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:16.476056 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "84993b2f63e7421290a161dde4bb0607" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:52:16.477128 847 consensus_peers.cc:597] T 58dde9397b434b089ade9968f369006a P c756528642a543eeacd93cef3e335a7d -> Peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Couldn't send request to peer b495a49349324f7d81908f2da700f0d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 58dde9397b434b089ade9968f369006a. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:16.477365 766 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "84993b2f63e7421290a161dde4bb0607" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:16.477557 766 raft_consensus.cc:3060] T 84993b2f63e7421290a161dde4bb0607 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
W20251029 02:52:16.479475 849 leader_election.cc:343] T 84993b2f63e7421290a161dde4bb0607 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
W20251029 02:52:16.479755 849 consensus_peers.cc:597] T 748777c6cd6e4f49ab49912b719e48bd P c756528642a543eeacd93cef3e335a7d -> Peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Couldn't send request to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251029 02:52:16.479856 849 leader_election.cc:343] T 061f3d38c4604fa4b57e74ff6f6fc7da P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Not found: Tablet not found: 061f3d38c4604fa4b57e74ff6f6fc7da
I20251029 02:52:16.479933 849 leader_election.cc:304] T 061f3d38c4604fa4b57e74ff6f6fc7da P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9
I20251029 02:52:16.480268 766 raft_consensus.cc:2468] T 84993b2f63e7421290a161dde4bb0607 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:16.480398 2426 raft_consensus.cc:2749] T 061f3d38c4604fa4b57e74ff6f6fc7da P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251029 02:52:16.477542 712 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:45003, user_credentials={real_user=slave}} blocked reactor thread for 158002us
W20251029 02:52:16.480718 712 leader_election.cc:343] T c45ffc304e73490387ce800b702b5881 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:16.480827 712 leader_election.cc:304] T c45ffc304e73490387ce800b702b5881 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:16.480901 1176 raft_consensus.cc:2272] T 1bdb67125493430398ff5c80b73de2ac P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:16.481096 2722 raft_consensus.cc:2749] T c45ffc304e73490387ce800b702b5881 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:16.481400 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "84993b2f63e7421290a161dde4bb0607" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
W20251029 02:52:16.481503 712 leader_election.cc:343] T b0fec06ad37d4c12a7045d424b88cc5e P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:16.481583 712 leader_election.cc:304] T b0fec06ad37d4c12a7045d424b88cc5e P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:16.481765 2722 raft_consensus.cc:2749] T b0fec06ad37d4c12a7045d424b88cc5e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 02:52:16.482096 712 leader_election.cc:343] T f93b0c39d7174dabbe018b1546dbc198 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:16.482201 712 leader_election.cc:304] T f93b0c39d7174dabbe018b1546dbc198 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:16.482393 2722 raft_consensus.cc:2749] T f93b0c39d7174dabbe018b1546dbc198 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:16.482409 847 leader_election.cc:304] T 84993b2f63e7421290a161dde4bb0607 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:16.482750 2649 raft_consensus.cc:2804] T 84993b2f63e7421290a161dde4bb0607 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
W20251029 02:52:16.482811 712 leader_election.cc:343] T 5815626b05204b578cc4a110f246bf10 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:16.482861 2649 raft_consensus.cc:697] T 84993b2f63e7421290a161dde4bb0607 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:52:16.482892 712 leader_election.cc:304] T 5815626b05204b578cc4a110f246bf10 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:16.483058 2722 raft_consensus.cc:2749] T 5815626b05204b578cc4a110f246bf10 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:16.483114 2649 consensus_queue.cc:237] T 84993b2f63e7421290a161dde4bb0607 P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
W20251029 02:52:16.483500 712 leader_election.cc:343] T defb795026a44e12afb2defb9c9c6db2 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:16.483585 712 leader_election.cc:304] T defb795026a44e12afb2defb9c9c6db2 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:16.483804 2722 raft_consensus.cc:2749] T defb795026a44e12afb2defb9c9c6db2 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:16.485186 766 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6789bc16664e4b37b86702449ab27855" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
W20251029 02:52:16.485507 849 leader_election.cc:343] T 84993b2f63e7421290a161dde4bb0607 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
W20251029 02:52:16.485848 849 consensus_peers.cc:597] T 1bdb67125493430398ff5c80b73de2ac P c756528642a543eeacd93cef3e335a7d -> Peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Couldn't send request to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:16.485993 520 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet 731c0acfbe204e8199ad246e38ae2ccd (table default.Table_9751e24147fb4fb0b7b7030ecd636514 [id=93e27f360fbb436aa652b936c9046510]). Expected version 1 got 0
I20251029 02:52:16.486532 520 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet 6789bc16664e4b37b86702449ab27855 (table default.Table_9a7d7f4f9073437097aca9f7d6dc665e [id=ab8aeaf90f6c4bc2ad5a98780945c40a]). Expected version 2 got 0
I20251029 02:52:16.486757 520 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet 32519fbffcef44a890161e5f3043806c (table default.Table_a78f5837df7b44e791631e421aa7de46 [id=b95fe1c7d6ff4519b90f993f1fa961fd]). Expected version 3 got 0
I20251029 02:52:16.487442 520 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet 27b08f88b99643d28ac609e45bf8104e (table default.Table_a9e50760813844af85a85c5f8d22f8e0 [id=74a2a41c908146ec8e16c6a3227b94f2]). Expected version 1 got 0
W20251029 02:52:16.488668 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 96f7876e5d6d4a0db236df5e7f266fd2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 96f7876e5d6d4a0db236df5e7f266fd2 already in progress: creating tablet
I20251029 02:52:16.490351 919 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "27b08f88b99643d28ac609e45bf8104e" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:16.490686 917 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e3000183230e4350b47f5085952f2db6" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:16.494315 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 28f7ceb91a34450cad4b5b45f5285b41 (table default.Table_5580c367c15f4f3a87626302cfde455f [id=89406c2781fd4674b634e3b1dbd06dc7]) successfully deleted
I20251029 02:52:16.494884 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "372d93914abc41b38a702742b7db0bc2" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:16.495064 916 raft_consensus.cc:2468] T 372d93914abc41b38a702742b7db0bc2 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 0.
I20251029 02:52:16.495594 1176 ts_tablet_manager.cc:1916] T 1bdb67125493430398ff5c80b73de2ac P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 02:52:16.496241 711 leader_election.cc:343] T e3000183230e4350b47f5085952f2db6 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:16.497099 913 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6789bc16664e4b37b86702449ab27855" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:16.498880 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "32519fbffcef44a890161e5f3043806c" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
W20251029 02:52:16.503926 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 0700b93cc398403d8b3602e84182a3ce because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0700b93cc398403d8b3602e84182a3ce already in progress: creating tablet
W20251029 02:52:16.504375 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet bfb9115f324f44c98f876a2d56bb23a1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bfb9115f324f44c98f876a2d56bb23a1 already in progress: creating tablet
W20251029 02:52:16.504710 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet ef2e9dd8f90a47389d9062d1f99a1da1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ef2e9dd8f90a47389d9062d1f99a1da1 already in progress: creating tablet
I20251029 02:52:16.505829 990 leader_election.cc:304] T c32c28b0e74a4af3a0e69bbf24ab1a1f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5; no voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d
W20251029 02:52:16.506064 990 raft_consensus.cc:2686] T c32c28b0e74a4af3a0e69bbf24ab1a1f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: unable to run election callback: Service unavailable: Thread pool token was shut down
W20251029 02:52:16.507714 711 leader_election.cc:343] T 32519fbffcef44a890161e5f3043806c P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:16.508432 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 0d84dc8dc465496f9adec798b2e03e13 (table default.Table_0d5730d385074a70b295b233e6ecd79e [id=423993f75a9948aebe1dee01d3e12702]) successfully deleted
I20251029 02:52:16.516687 990 leader_election.cc:304] T 27b08f88b99643d28ac609e45bf8104e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5; no voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:16.517189 2717 raft_consensus.cc:2749] T 27b08f88b99643d28ac609e45bf8104e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 02:52:16.521217 849 consensus_peers.cc:597] T 2e97d1bc06a24e8d8ba11acb6fd280d8 P c756528642a543eeacd93cef3e335a7d -> Peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Couldn't send request to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2e97d1bc06a24e8d8ba11acb6fd280d8. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:16.523685 2740 consensus_queue.cc:1048] T cd597c319af645909b8f3d1212fcb702 P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:16.525228 2426 raft_consensus.cc:493] T d29cc2a0c8b84e3ebadc7f571d666847 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:16.525357 2426 raft_consensus.cc:515] T d29cc2a0c8b84e3ebadc7f571d666847 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:16.525983 2426 leader_election.cc:290] T d29cc2a0c8b84e3ebadc7f571d666847 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:16.527662 1059 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d29cc2a0c8b84e3ebadc7f571d666847" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:52:16.528304 849 leader_election.cc:343] T d29cc2a0c8b84e3ebadc7f571d666847 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:16.528952 766 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d29cc2a0c8b84e3ebadc7f571d666847" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:16.529184 766 raft_consensus.cc:2468] T d29cc2a0c8b84e3ebadc7f571d666847 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:16.530277 847 leader_election.cc:304] T d29cc2a0c8b84e3ebadc7f571d666847 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5
I20251029 02:52:16.530826 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 978) took 5202 ms. Trace:
I20251029 02:52:16.530992 2426 raft_consensus.cc:2804] T d29cc2a0c8b84e3ebadc7f571d666847 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:16.531055 2426 raft_consensus.cc:493] T d29cc2a0c8b84e3ebadc7f571d666847 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:16.531107 2426 raft_consensus.cc:3060] T d29cc2a0c8b84e3ebadc7f571d666847 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:16.530962 1175 rpcz_store.cc:276] 1029 02:52:11.328759 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:11.328845 (+ 86us) service_pool.cc:225] Handling call
1029 02:52:16.530710 (+5201865us) ts_tablet_manager.cc:1151] Deleting tablet ecf3b900206b4c86a7adbc83a769d90e
1029 02:52:16.530720 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:16.530806 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5201661}
W20251029 02:52:16.531527 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet ecf3b900206b4c86a7adbc83a769d90e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ecf3b900206b4c86a7adbc83a769d90e already in progress: creating tablet
I20251029 02:52:16.531919 1175 tablet_replica.cc:333] T 1bdb67125493430398ff5c80b73de2ac P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:16.532155 1175 raft_consensus.cc:2243] T 1bdb67125493430398ff5c80b73de2ac P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:16.532362 1175 raft_consensus.cc:2272] T 1bdb67125493430398ff5c80b73de2ac P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:16.533046 1175 ts_tablet_manager.cc:1916] T 1bdb67125493430398ff5c80b73de2ac P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:16.534617 2426 raft_consensus.cc:515] T d29cc2a0c8b84e3ebadc7f571d666847 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:16.535590 2426 leader_election.cc:290] T d29cc2a0c8b84e3ebadc7f571d666847 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:16.536460 1176 ts_tablet_manager.cc:1929] T 1bdb67125493430398ff5c80b73de2ac P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251029 02:52:16.536571 1176 log.cc:1199] T 1bdb67125493430398ff5c80b73de2ac P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/1bdb67125493430398ff5c80b73de2ac
I20251029 02:52:16.536501 1059 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d29cc2a0c8b84e3ebadc7f571d666847" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:52:16.536726 1166 tablet_bootstrap.cc:654] T ae4231daa2594258aa51419c97a2f50a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:16.536963 1176 ts_tablet_manager.cc:1950] T 1bdb67125493430398ff5c80b73de2ac P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
W20251029 02:52:16.537237 849 leader_election.cc:343] T d29cc2a0c8b84e3ebadc7f571d666847 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:16.537326 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1058) took 4552 ms. Trace:
I20251029 02:52:16.537397 1176 rpcz_store.cc:276] 1029 02:52:11.984997 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:11.985050 (+ 53us) service_pool.cc:225] Handling call
1029 02:52:16.474050 (+4489000us) ts_tablet_manager.cc:1151] Deleting tablet 1bdb67125493430398ff5c80b73de2ac
1029 02:52:16.474063 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:16.506205 (+ 32142us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:16.536350 (+ 30145us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:16.537309 (+ 959us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4488847}
I20251029 02:52:16.537784 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1060) took 4552 ms. Trace:
I20251029 02:52:16.537868 1176 rpcz_store.cc:276] 1029 02:52:11.985696 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:11.985736 (+ 40us) service_pool.cc:225] Handling call
1029 02:52:16.537667 (+4551931us) ts_tablet_manager.cc:1151] Deleting tablet 81ee839d811a4f46b93a57ede397bd93
1029 02:52:16.537674 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:16.537765 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4551816}
I20251029 02:52:16.538157 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1061) took 4552 ms. Trace:
I20251029 02:52:16.538224 1176 rpcz_store.cc:276] 1029 02:52:11.985932 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:11.985963 (+ 31us) service_pool.cc:225] Handling call
1029 02:52:16.538055 (+4552092us) ts_tablet_manager.cc:1151] Deleting tablet 0f96af1cb0ea4b46bd846ae0739f4947
1029 02:52:16.538062 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:16.538146 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4551987}
I20251029 02:52:16.538407 766 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d29cc2a0c8b84e3ebadc7f571d666847" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:16.538641 766 raft_consensus.cc:3060] T d29cc2a0c8b84e3ebadc7f571d666847 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:16.542182 766 raft_consensus.cc:2468] T d29cc2a0c8b84e3ebadc7f571d666847 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:16.542702 847 leader_election.cc:304] T d29cc2a0c8b84e3ebadc7f571d666847 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5
I20251029 02:52:16.543406 2740 raft_consensus.cc:2804] T d29cc2a0c8b84e3ebadc7f571d666847 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:16.543545 2740 raft_consensus.cc:697] T d29cc2a0c8b84e3ebadc7f571d666847 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:52:16.543995 2740 consensus_queue.cc:237] T d29cc2a0c8b84e3ebadc7f571d666847 P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:16.546108 990 leader_election.cc:304] T 372d93914abc41b38a702742b7db0bc2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:16.546375 2717 raft_consensus.cc:2804] T 372d93914abc41b38a702742b7db0bc2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:16.546461 2717 raft_consensus.cc:493] T 372d93914abc41b38a702742b7db0bc2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:16.546550 2717 raft_consensus.cc:3060] T 372d93914abc41b38a702742b7db0bc2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:16.538481 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1062) took 4552 ms. Trace:
I20251029 02:52:16.546869 1176 rpcz_store.cc:276] 1029 02:52:11.986145 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:11.986176 (+ 31us) service_pool.cc:225] Handling call
1029 02:52:16.538389 (+4552213us) ts_tablet_manager.cc:1151] Deleting tablet a530fe8de8ae4be5b0fd9a5f7e94ea9e
1029 02:52:16.538396 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:16.538471 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4552096}
I20251029 02:52:16.547199 1172 tablet_bootstrap.cc:492] T 7816556521144c928d7a02694e22e79f P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:16.547261 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1063) took 4560 ms. Trace:
I20251029 02:52:16.547327 1172 ts_tablet_manager.cc:1403] T 7816556521144c928d7a02694e22e79f P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.082s user 0.005s sys 0.001s
I20251029 02:52:16.547324 1176 rpcz_store.cc:276] 1029 02:52:11.986374 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:11.986406 (+ 32us) service_pool.cc:225] Handling call
1029 02:52:16.547113 (+4560707us) ts_tablet_manager.cc:1151] Deleting tablet 426f9be3b0074f60989acc43ec1e9bd6
1029 02:52:16.547123 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:16.547246 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4560600}
I20251029 02:52:16.547569 1176 tablet_replica.cc:333] T 44c0488eba0543078ff4f0b3d69366f2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:16.547737 1176 raft_consensus.cc:2243] T 44c0488eba0543078ff4f0b3d69366f2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:16.548023 1176 raft_consensus.cc:2272] T 44c0488eba0543078ff4f0b3d69366f2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:16.548077 1172 raft_consensus.cc:359] T 7816556521144c928d7a02694e22e79f P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:16.548223 1172 raft_consensus.cc:385] T 7816556521144c928d7a02694e22e79f P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:16.548278 1172 raft_consensus.cc:740] T 7816556521144c928d7a02694e22e79f P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:16.548506 1172 consensus_queue.cc:260] T 7816556521144c928d7a02694e22e79f P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:16.548585 1176 ts_tablet_manager.cc:1916] T 44c0488eba0543078ff4f0b3d69366f2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:16.549158 1172 ts_tablet_manager.cc:1434] T 7816556521144c928d7a02694e22e79f P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 02:52:16.549413 1172 tablet_bootstrap.cc:492] T 688af6580c4d4637a39f817b4e324618 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:16.550711 2717 raft_consensus.cc:515] T 372d93914abc41b38a702742b7db0bc2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
W20251029 02:52:16.551669 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet c7a3e6d8e3fb45898d8a5e094dbf0fd7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c7a3e6d8e3fb45898d8a5e094dbf0fd7 already in progress: creating tablet
W20251029 02:52:16.552508 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 7dbc9c8b110a48f0a801ad468738c3a3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7dbc9c8b110a48f0a801ad468738c3a3 already in progress: creating tablet
W20251029 02:52:16.552853 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 96f7876e5d6d4a0db236df5e7f266fd2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 96f7876e5d6d4a0db236df5e7f266fd2 already in progress: creating tablet
W20251029 02:52:16.553179 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet ecf3b900206b4c86a7adbc83a769d90e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ecf3b900206b4c86a7adbc83a769d90e already in progress: creating tablet
W20251029 02:52:16.553484 513 catalog_manager.cc:4952] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 86fc22bb84794149b598b97df41eb881 because the tablet was not found. No further retry: Not found: Tablet not found: 86fc22bb84794149b598b97df41eb881
I20251029 02:52:16.555037 2717 leader_election.cc:290] T 372d93914abc41b38a702742b7db0bc2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Requested vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:52:16.556371 2763 tablet.cc:1722] T 731c0acfbe204e8199ad246e38ae2ccd P b495a49349324f7d81908f2da700f0d9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 02:52:16.557261 1027 tablet_service.cc:1505] Processing CreateTablet for tablet 2925722be8a045c689c69ac729fb3799 (DEFAULT_TABLE table=default.Table_d8c5abd34fad48fb8ec28152a4b37e38 [id=f6160584ba214beb8beacc856d5a9ca6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:16.557337 1040 tablet_service.cc:1505] Processing CreateTablet for tablet 9542585fd9824a55b3ad2f120065a02d (DEFAULT_TABLE table=default.Table_7eb1d904f38342dc85bb06681099d741 [id=cbadf4b0377b4c3dbf106b533a18acdf]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:16.557654 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2925722be8a045c689c69ac729fb3799. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:16.557855 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9542585fd9824a55b3ad2f120065a02d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:16.573823 1174 ts_tablet_manager.cc:1929] T 738046ab57334f2dab9f4aab979c5b42 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:16.573971 1174 log.cc:1199] T 738046ab57334f2dab9f4aab979c5b42 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/738046ab57334f2dab9f4aab979c5b42
I20251029 02:52:16.574322 1174 ts_tablet_manager.cc:1950] T 738046ab57334f2dab9f4aab979c5b42 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:16.574668 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 955) took 5934 ms. Trace:
I20251029 02:52:16.574764 1174 rpcz_store.cc:276] 1029 02:52:10.640465 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:10.640530 (+ 65us) service_pool.cc:225] Handling call
1029 02:52:16.129363 (+5488833us) ts_tablet_manager.cc:1151] Deleting tablet 738046ab57334f2dab9f4aab979c5b42
1029 02:52:16.129373 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:16.275188 (+145815us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:16.573711 (+298523us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:16.574645 (+ 934us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5488673}
I20251029 02:52:16.575168 1174 tablet_replica.cc:333] T e7f846b294cc4eb687a26c8b5931ac19 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:16.575392 1174 raft_consensus.cc:2243] T e7f846b294cc4eb687a26c8b5931ac19 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:16.575804 1174 raft_consensus.cc:2272] T e7f846b294cc4eb687a26c8b5931ac19 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:16.580871 1176 ts_tablet_manager.cc:1929] T 44c0488eba0543078ff4f0b3d69366f2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:16.580981 1176 log.cc:1199] T 44c0488eba0543078ff4f0b3d69366f2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/44c0488eba0543078ff4f0b3d69366f2
I20251029 02:52:16.581338 1176 ts_tablet_manager.cc:1950] T 44c0488eba0543078ff4f0b3d69366f2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:16.581617 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1072) took 4276 ms. Trace:
I20251029 02:52:16.581677 1176 rpcz_store.cc:276] 1029 02:52:12.305211 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:12.305258 (+ 47us) service_pool.cc:225] Handling call
1029 02:52:16.547527 (+4242269us) ts_tablet_manager.cc:1151] Deleting tablet 44c0488eba0543078ff4f0b3d69366f2
1029 02:52:16.547535 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:16.565573 (+ 18038us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:16.580756 (+ 15183us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:16.581604 (+ 848us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4241757}
I20251029 02:52:16.582002 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1073) took 4276 ms. Trace:
I20251029 02:52:16.582079 1176 rpcz_store.cc:276] 1029 02:52:12.305683 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:12.305756 (+ 73us) service_pool.cc:225] Handling call
1029 02:52:16.581905 (+4276149us) ts_tablet_manager.cc:1151] Deleting tablet 0d84dc8dc465496f9adec798b2e03e13
1029 02:52:16.581913 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:16.581987 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4275981}
I20251029 02:52:16.582362 1176 tablet_replica.cc:333] T 2f198c3259864f7ab3625cb7bfae20b0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:16.582594 1176 raft_consensus.cc:2243] T 2f198c3259864f7ab3625cb7bfae20b0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:16.582883 1176 raft_consensus.cc:2272] T 2f198c3259864f7ab3625cb7bfae20b0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:16.583436 1176 ts_tablet_manager.cc:1916] T 2f198c3259864f7ab3625cb7bfae20b0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:16.590982 1033 tablet_service.cc:1552] Processing DeleteTablet for tablet 2849e856dd714da88fca5a9fafe74c22 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:15 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:16.591311 1040 tablet_service.cc:1552] Processing DeleteTablet for tablet 19e15a6df77f4e0c9b40ae4f49fdd06f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:15 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:16.591933 1027 tablet_service.cc:1505] Processing CreateTablet for tablet b3dc6e55bb2e47bab7df5df62a1bcb51 (DEFAULT_TABLE table=default.Table_ca88b32b14364d2aab4e1b1480676c12 [id=8aacd9b9d75346419887f8be922803ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:16.595089 1033 tablet_service.cc:1505] Processing CreateTablet for tablet 9426756257c64db982d1e879c0c0f8a9 (DEFAULT_TABLE table=default.Table_75325372892242518c3f0a4b8b657c48 [id=9812d79d53084361ba6cf509fb6eb0f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:16.596153 1167 tablet_bootstrap.cc:492] T eca016aed05e4ce09e1a2bf96e2286f5 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:16.596621 1167 ts_tablet_manager.cc:1403] T eca016aed05e4ce09e1a2bf96e2286f5 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.278s user 0.003s sys 0.001s
I20251029 02:52:16.596725 1040 tablet_service.cc:1505] Processing CreateTablet for tablet 4905b28faec74d30859461d770282f41 (DEFAULT_TABLE table=default.Table_7511abddcde942a4adb0a23b3a345d06 [id=d9f9ae6423a34675968f74ae094aee89]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 02:52:16.621426 517 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e9387edc3c434f5881a3a43e79a3af58 has no consensus state
I20251029 02:52:16.623171 524 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet c73059366f4b45528f8b907263c4d085
I20251029 02:52:16.623682 524 catalog_manager.cc:6719] ReplaceTablet: tablet c73059366f4b45528f8b907263c4d085 deleted and replaced by tablet fb63ec8e38fe4cbb8df2065b760c9112
I20251029 02:52:16.627700 521 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_b99166d4afe04f44a80ecd035fa90d72" } new_table_name: "default.Table_61c29c73a4354026ba1dab6dd81536ea" modify_external_catalogs: true
I20251029 02:52:16.628417 517 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet 748777c6cd6e4f49ab49912b719e48bd (table default.Table_1e69e259edba466f99ed7a41dcf6367f [id=a2f3b36dc78a4e94bd054a6a5712b384]). Expected version 1 got 0
I20251029 02:52:16.629499 738 tablet_service.cc:1552] Processing DeleteTablet for tablet c73059366f4b45528f8b907263c4d085 with delete_type TABLET_DATA_DELETED (replaced by tablet fb63ec8e38fe4cbb8df2065b760c9112) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:16.633054 517 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_fe999fb018f84d38b900fd1852effeaf" } new_table_name: "default.Table_e48d1ae4ca5d49e0914d4353bb65cefb" modify_external_catalogs: true
I20251029 02:52:16.637816 738 tablet_service.cc:1552] Processing DeleteTablet for tablet 506f8c112a3b43e1a3d6bbafb8cb81e8 with delete_type TABLET_DATA_DELETED (replaced by tablet ef2e9dd8f90a47389d9062d1f99a1da1) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:16.657684 1166 tablet_bootstrap.cc:492] T ae4231daa2594258aa51419c97a2f50a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:16.658131 1166 ts_tablet_manager.cc:1403] T ae4231daa2594258aa51419c97a2f50a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.248s user 0.001s sys 0.003s
I20251029 02:52:16.662885 1024 tablet_service.cc:1552] Processing DeleteTablet for tablet 46e054ff563f448d86d8b4f84810291b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:15 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:16.663314 1024 tablet_service.cc:1552] Processing DeleteTablet for tablet 5bcadc0b68e84ae290690edef392a45b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:15 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:16.663758 1024 tablet_service.cc:1552] Processing DeleteTablet for tablet 60f0eadc074a4872b6638314fed98e0e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:15 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:16.664256 1024 tablet_service.cc:1505] Processing CreateTablet for tablet b64880e732f54510a3954248cc940fbc (DEFAULT_TABLE table=default.Table_4098c28776dd4f41864c61d2299b90cd [id=47ee5993b95a477db52f9c51d034e549]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:16.665032 524 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet 748777c6cd6e4f49ab49912b719e48bd (table default.Table_1e69e259edba466f99ed7a41dcf6367f [id=a2f3b36dc78a4e94bd054a6a5712b384]). Expected version 1 got 0
I20251029 02:52:16.665134 1039 tablet_service.cc:1505] Processing CreateTablet for tablet d9b3adb851bc46feb77e4b484eb0b8d4 (DEFAULT_TABLE table=default.Table_024e9ca04c014a27bc2b62227d4828d0 [id=b7453480e4d84319a2ce51432f83659b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:16.665207 524 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet 6f8cf09f23984b879b60ab6b27c7d52e (table default.Table_96e017f445bc4f1fbebddf65c722b8de [id=98f28a15eced4860ac13a60abdc8e0e2]). Expected version 3 got 0
I20251029 02:52:16.666008 1031 tablet_service.cc:1505] Processing CreateTablet for tablet 09bbf13b23334bbf88f2eb2e327b20f2 (DEFAULT_TABLE table=default.Table_bf304b9f44274cf9b56f9e25fb0e9875 [id=1e7048e956fa484093370764b98d30cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:16.668751 1166 raft_consensus.cc:359] T ae4231daa2594258aa51419c97a2f50a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:16.669739 1166 raft_consensus.cc:385] T ae4231daa2594258aa51419c97a2f50a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:16.670954 1166 raft_consensus.cc:740] T ae4231daa2594258aa51419c97a2f50a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:16.671370 1166 consensus_queue.cc:260] T ae4231daa2594258aa51419c97a2f50a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:16.676061 1166 ts_tablet_manager.cc:1434] T ae4231daa2594258aa51419c97a2f50a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.018s user 0.000s sys 0.002s
I20251029 02:52:16.676563 1166 tablet_bootstrap.cc:492] T 70b008011d5547bbaf3f45685394139f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
W20251029 02:52:16.677809 847 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:38481, user_credentials={real_user=slave}} blocked reactor thread for 83009us
I20251029 02:52:16.678722 1167 raft_consensus.cc:359] T eca016aed05e4ce09e1a2bf96e2286f5 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:16.678918 1167 raft_consensus.cc:385] T eca016aed05e4ce09e1a2bf96e2286f5 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:16.678979 1167 raft_consensus.cc:740] T eca016aed05e4ce09e1a2bf96e2286f5 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:16.679206 1167 consensus_queue.cc:260] T eca016aed05e4ce09e1a2bf96e2286f5 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:16.679750 1167 ts_tablet_manager.cc:1434] T eca016aed05e4ce09e1a2bf96e2286f5 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.083s user 0.002s sys 0.000s
I20251029 02:52:16.680446 1174 ts_tablet_manager.cc:1916] T e7f846b294cc4eb687a26c8b5931ac19 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:16.681628 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 738046ab57334f2dab9f4aab979c5b42 (table default.Table_31d42031714e415fab63cc5b9933fe0c [id=33b26e2658ff4270a5f1306b6cde34db]) successfully deleted
I20251029 02:52:16.681880 888 tablet_service.cc:1552] Processing DeleteTablet for tablet c73059366f4b45528f8b907263c4d085 with delete_type TABLET_DATA_DELETED (replaced by tablet fb63ec8e38fe4cbb8df2065b760c9112) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:16.681985 895 tablet_service.cc:1552] Processing DeleteTablet for tablet ad214c870ee649279ed7321c5b69cbee with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:05 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:16.682925 2656 raft_consensus.cc:493] T ad214c870ee649279ed7321c5b69cbee P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:16.683060 2656 raft_consensus.cc:515] T ad214c870ee649279ed7321c5b69cbee P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:16.683692 2656 leader_election.cc:290] T ad214c870ee649279ed7321c5b69cbee P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
W20251029 02:52:16.684474 849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:45003, user_credentials={real_user=slave}} blocked reactor thread for 58372.9us
I20251029 02:52:16.685353 1059 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ad214c870ee649279ed7321c5b69cbee" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:52:16.685926 849 consensus_peers.cc:597] T 46cd75ff18b14212b556b9bcb3012dd9 P c756528642a543eeacd93cef3e335a7d -> Peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Couldn't send request to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20251029 02:52:16.686100 849 leader_election.cc:343] T ad214c870ee649279ed7321c5b69cbee P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:16.686919 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ad214c870ee649279ed7321c5b69cbee" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:16.687134 776 raft_consensus.cc:2468] T ad214c870ee649279ed7321c5b69cbee P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
W20251029 02:52:16.687917 847 consensus_peers.cc:597] T 67651306f3b44e7eb14dcee9a2007a81 P c756528642a543eeacd93cef3e335a7d -> Peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Couldn't send request to peer b495a49349324f7d81908f2da700f0d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 67651306f3b44e7eb14dcee9a2007a81. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:16.689126 847 leader_election.cc:304] T ad214c870ee649279ed7321c5b69cbee P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5
I20251029 02:52:16.689597 2656 raft_consensus.cc:2804] T ad214c870ee649279ed7321c5b69cbee P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:16.689664 1172 tablet_bootstrap.cc:654] T 688af6580c4d4637a39f817b4e324618 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:16.689687 2656 raft_consensus.cc:493] T ad214c870ee649279ed7321c5b69cbee P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:16.689761 2656 raft_consensus.cc:3060] T ad214c870ee649279ed7321c5b69cbee P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:16.691736 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09bbf13b23334bbf88f2eb2e327b20f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:16.691939 1034 tablet_service.cc:1505] Processing CreateTablet for tablet 1537b318704f48ccb01c48a4e783da94 (DEFAULT_TABLE table=default.Table_a7e37e7a8bba4557a83b0fe3f498df17 [id=1c093c8c547144d2a413329c199d2962]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:16.692258 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1537b318704f48ccb01c48a4e783da94. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:16.692849 2656 raft_consensus.cc:515] T ad214c870ee649279ed7321c5b69cbee P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:16.693694 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3dc6e55bb2e47bab7df5df62a1bcb51. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:16.694608 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9b3adb851bc46feb77e4b484eb0b8d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:16.696934 1059 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ad214c870ee649279ed7321c5b69cbee" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
W20251029 02:52:16.697448 849 leader_election.cc:343] T ad214c870ee649279ed7321c5b69cbee P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:16.697932 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9426756257c64db982d1e879c0c0f8a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:16.698611 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4905b28faec74d30859461d770282f41. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:16.698637 1176 ts_tablet_manager.cc:1929] T 2f198c3259864f7ab3625cb7bfae20b0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:16.698734 1176 log.cc:1199] T 2f198c3259864f7ab3625cb7bfae20b0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/2f198c3259864f7ab3625cb7bfae20b0
I20251029 02:52:16.699285 1176 ts_tablet_manager.cc:1950] T 2f198c3259864f7ab3625cb7bfae20b0 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:16.699776 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1074) took 4393 ms. Trace:
I20251029 02:52:16.700294 1167 tablet_bootstrap.cc:492] T defb795026a44e12afb2defb9c9c6db2 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:16.700271 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "372d93914abc41b38a702742b7db0bc2" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
I20251029 02:52:16.700433 916 raft_consensus.cc:3060] T 372d93914abc41b38a702742b7db0bc2 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:16.701807 1167 tablet_bootstrap.cc:654] T defb795026a44e12afb2defb9c9c6db2 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:16.702615 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b64880e732f54510a3954248cc940fbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:16.703147 916 raft_consensus.cc:2468] T 372d93914abc41b38a702742b7db0bc2 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 1.
I20251029 02:52:16.703117 2722 raft_consensus.cc:493] T cc802bda2f6a4e439b605f2cafa37967 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:16.703267 2722 raft_consensus.cc:515] T cc802bda2f6a4e439b605f2cafa37967 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:16.699847 1176 rpcz_store.cc:276] 1029 02:52:12.306156 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:12.306203 (+ 47us) service_pool.cc:225] Handling call
1029 02:52:16.582315 (+4276112us) ts_tablet_manager.cc:1151] Deleting tablet 2f198c3259864f7ab3625cb7bfae20b0
1029 02:52:16.582325 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:16.694929 (+112604us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:16.698563 (+ 3634us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:16.699762 (+ 1199us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4275956}
I20251029 02:52:16.703853 2722 leader_election.cc:290] T cc802bda2f6a4e439b605f2cafa37967 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:52:16.704046 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1075) took 4377 ms. Trace:
I20251029 02:52:16.704121 1176 rpcz_store.cc:276] 1029 02:52:12.326671 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:12.328395 (+ 1724us) service_pool.cc:225] Handling call
1029 02:52:16.703922 (+4375527us) ts_tablet_manager.cc:1151] Deleting tablet 1150d2210a234f1aa554f486cdcabc4c
1029 02:52:16.703932 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:16.704029 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4375298}
I20251029 02:52:16.704384 919 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cc802bda2f6a4e439b605f2cafa37967" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:16.704761 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1076) took 4376 ms. Trace:
W20251029 02:52:16.704821 711 leader_election.cc:343] T cc802bda2f6a4e439b605f2cafa37967 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:16.704823 1176 rpcz_store.cc:276] 1029 02:52:12.328454 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:12.328507 (+ 53us) service_pool.cc:225] Handling call
1029 02:52:16.704660 (+4376153us) ts_tablet_manager.cc:1151] Deleting tablet fd1bf6b0f45349d0981f9ddd89269489
1029 02:52:16.704668 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:16.704747 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4375974}
I20251029 02:52:16.705097 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1077) took 4335 ms. Trace:
I20251029 02:52:16.705156 1176 rpcz_store.cc:276] 1029 02:52:12.369363 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:12.369423 (+ 60us) service_pool.cc:225] Handling call
1029 02:52:16.705012 (+4335589us) ts_tablet_manager.cc:1151] Deleting tablet 41cfa2d722034618a05de717b28a778d
1029 02:52:16.705020 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:16.705085 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4335404}
I20251029 02:52:16.705405 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1078) took 4327 ms. Trace:
I20251029 02:52:16.705466 1176 rpcz_store.cc:276] 1029 02:52:12.377720 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:12.377775 (+ 55us) service_pool.cc:225] Handling call
1029 02:52:16.705326 (+4327551us) ts_tablet_manager.cc:1151] Deleting tablet 99d60ef5325e45059026596a756448de
1029 02:52:16.705334 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:16.705393 (+ 59us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4327379}
I20251029 02:52:16.705729 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1080) took 4269 ms. Trace:
I20251029 02:52:16.705783 1176 rpcz_store.cc:276] 1029 02:52:12.436367 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:12.436420 (+ 53us) service_pool.cc:225] Handling call
1029 02:52:16.705648 (+4269228us) ts_tablet_manager.cc:1151] Deleting tablet 49aaa2d8986a44df979edc1c42612189
1029 02:52:16.705655 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:16.705716 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4267912}
I20251029 02:52:16.706013 1176 tablet_replica.cc:333] T f281191e7c6143f983f11d87f1c9b64e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:16.706171 1176 raft_consensus.cc:2243] T f281191e7c6143f983f11d87f1c9b64e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:16.706313 1176 raft_consensus.cc:2272] T f281191e7c6143f983f11d87f1c9b64e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:16.707041 1176 ts_tablet_manager.cc:1916] T f281191e7c6143f983f11d87f1c9b64e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:16.708428 990 leader_election.cc:304] T 372d93914abc41b38a702742b7db0bc2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:16.708731 2179 raft_consensus.cc:2804] T 372d93914abc41b38a702742b7db0bc2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:16.708835 2179 raft_consensus.cc:697] T 372d93914abc41b38a702742b7db0bc2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Becoming Leader. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Running, Role: LEADER
I20251029 02:52:16.709025 2717 raft_consensus.cc:493] T 20a762344c5f46f88481c4ac7ad6e43b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:16.709077 2179 consensus_queue.cc:237] T 372d93914abc41b38a702742b7db0bc2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:16.709131 2717 raft_consensus.cc:515] T 20a762344c5f46f88481c4ac7ad6e43b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:16.709642 2717 leader_election.cc:290] T 20a762344c5f46f88481c4ac7ad6e43b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:52:16.711624 919 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "20a762344c5f46f88481c4ac7ad6e43b" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:16.711799 919 raft_consensus.cc:2468] T 20a762344c5f46f88481c4ac7ad6e43b P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 0.
I20251029 02:52:16.713404 990 leader_election.cc:304] T 20a762344c5f46f88481c4ac7ad6e43b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:16.713752 2718 raft_consensus.cc:2804] T 20a762344c5f46f88481c4ac7ad6e43b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:16.713837 2718 raft_consensus.cc:493] T 20a762344c5f46f88481c4ac7ad6e43b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:16.713910 2718 raft_consensus.cc:3060] T 20a762344c5f46f88481c4ac7ad6e43b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:16.717144 2718 raft_consensus.cc:515] T 20a762344c5f46f88481c4ac7ad6e43b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:16.717856 1176 ts_tablet_manager.cc:1929] T f281191e7c6143f983f11d87f1c9b64e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:16.717903 2718 leader_election.cc:290] T 20a762344c5f46f88481c4ac7ad6e43b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Requested vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:52:16.717964 1176 log.cc:1199] T f281191e7c6143f983f11d87f1c9b64e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/f281191e7c6143f983f11d87f1c9b64e
I20251029 02:52:16.718287 1176 ts_tablet_manager.cc:1950] T f281191e7c6143f983f11d87f1c9b64e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:16.718609 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1083) took 4110 ms. Trace:
I20251029 02:52:16.718680 1176 rpcz_store.cc:276] 1029 02:52:12.607720 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:12.607792 (+ 72us) service_pool.cc:225] Handling call
1029 02:52:16.705977 (+4098185us) ts_tablet_manager.cc:1151] Deleting tablet f281191e7c6143f983f11d87f1c9b64e
1029 02:52:16.705984 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:16.710051 (+ 4067us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:16.717768 (+ 7717us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:16.718594 (+ 826us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4097989}
I20251029 02:52:16.719064 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1084) took 4060 ms. Trace:
I20251029 02:52:16.719355 919 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "20a762344c5f46f88481c4ac7ad6e43b" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
I20251029 02:52:16.719559 919 raft_consensus.cc:3060] T 20a762344c5f46f88481c4ac7ad6e43b P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:16.719774 2656 leader_election.cc:290] T ad214c870ee649279ed7321c5b69cbee P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:16.720386 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ad214c870ee649279ed7321c5b69cbee" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:16.720554 776 raft_consensus.cc:3060] T ad214c870ee649279ed7321c5b69cbee P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:16.723570 776 raft_consensus.cc:2468] T ad214c870ee649279ed7321c5b69cbee P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:16.722417 919 raft_consensus.cc:2468] T 20a762344c5f46f88481c4ac7ad6e43b P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 1.
I20251029 02:52:16.724287 847 leader_election.cc:304] T ad214c870ee649279ed7321c5b69cbee P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5
I20251029 02:52:16.719134 1176 rpcz_store.cc:276] 1029 02:52:12.658900 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:12.658956 (+ 56us) service_pool.cc:225] Handling call
1029 02:52:16.718935 (+4059979us) ts_tablet_manager.cc:1151] Deleting tablet 9280503662894544a1f227a62a9b8a01
1029 02:52:16.718944 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:16.719047 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4059802}
I20251029 02:52:16.726234 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1085) took 3987 ms. Trace:
I20251029 02:52:16.726207 990 leader_election.cc:304] T 20a762344c5f46f88481c4ac7ad6e43b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:16.726313 1176 rpcz_store.cc:276] 1029 02:52:12.738761 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:12.738818 (+ 57us) service_pool.cc:225] Handling call
1029 02:52:16.726117 (+3987299us) ts_tablet_manager.cc:1151] Deleting tablet fe5aff0f61a44b91bfb37b42457a91f0
1029 02:52:16.726126 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:16.726219 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3980883}
I20251029 02:52:16.726649 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1086) took 3721 ms. Trace:
I20251029 02:52:16.726720 1176 rpcz_store.cc:276] 1029 02:52:13.005239 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:13.007068 (+ 1829us) service_pool.cc:225] Handling call
1029 02:52:16.726550 (+3719482us) ts_tablet_manager.cc:1151] Deleting tablet ca71c4638a6b419b876e22af7e6cf00a
1029 02:52:16.726558 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:16.726636 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3719245}
I20251029 02:52:16.726819 2649 raft_consensus.cc:2804] T ad214c870ee649279ed7321c5b69cbee P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:16.726959 1176 tablet_replica.cc:333] T 44ae2b07edca44afbe1b1fb3a5b1115d P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:16.726934 2649 raft_consensus.cc:697] T ad214c870ee649279ed7321c5b69cbee P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:52:16.727213 2649 consensus_queue.cc:237] T ad214c870ee649279ed7321c5b69cbee P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:16.728359 1176 raft_consensus.cc:2243] T 44ae2b07edca44afbe1b1fb3a5b1115d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:16.728523 1176 raft_consensus.cc:2272] T 44ae2b07edca44afbe1b1fb3a5b1115d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:16.728679 2718 raft_consensus.cc:2804] T 20a762344c5f46f88481c4ac7ad6e43b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:16.729110 1176 ts_tablet_manager.cc:1916] T 44ae2b07edca44afbe1b1fb3a5b1115d P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:16.729727 2718 raft_consensus.cc:697] T 20a762344c5f46f88481c4ac7ad6e43b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Becoming Leader. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Running, Role: LEADER
I20251029 02:52:16.729955 2718 consensus_queue.cc:237] T 20a762344c5f46f88481c4ac7ad6e43b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:16.732753 1167 tablet_bootstrap.cc:492] T defb795026a44e12afb2defb9c9c6db2 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:16.732873 1167 ts_tablet_manager.cc:1403] T defb795026a44e12afb2defb9c9c6db2 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.033s user 0.003s sys 0.000s
I20251029 02:52:16.733546 1167 raft_consensus.cc:359] T defb795026a44e12afb2defb9c9c6db2 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:16.733697 1167 raft_consensus.cc:385] T defb795026a44e12afb2defb9c9c6db2 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:16.733770 1167 raft_consensus.cc:740] T defb795026a44e12afb2defb9c9c6db2 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:16.734210 1167 consensus_queue.cc:260] T defb795026a44e12afb2defb9c9c6db2 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:16.734916 1167 ts_tablet_manager.cc:1434] T defb795026a44e12afb2defb9c9c6db2 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251029 02:52:16.735231 1167 tablet_bootstrap.cc:492] T 32519fbffcef44a890161e5f3043806c P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:16.736828 1167 tablet_bootstrap.cc:654] T 32519fbffcef44a890161e5f3043806c P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:16.737392 1166 tablet_bootstrap.cc:654] T 70b008011d5547bbaf3f45685394139f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:16.739408 1039 tablet_service.cc:1505] Processing CreateTablet for tablet 4d8c8eb1fade468b85447b0f8b98a50f (DEFAULT_TABLE table=default.Table_db4e930bec75474ba682f938dd32ec41 [id=2b9a7fb76d634f25a4ef6599331c2480]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:16.739868 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d8c8eb1fade468b85447b0f8b98a50f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:16.739408 1027 tablet_service.cc:1505] Processing CreateTablet for tablet 9fcf4a80ebb446f89159659a663eede4 (DEFAULT_TABLE table=default.Table_98b6a6ee7ae34368bef7d883e216978d [id=603ef089eb46405aa4a7981b936fef8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:16.743047 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fcf4a80ebb446f89159659a663eede4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:16.746799 1031 tablet_service.cc:1505] Processing CreateTablet for tablet 5bb7a9e5f5b6424fbae965d050f0d1c8 (DEFAULT_TABLE table=default.Table_442850f577aa437a820314bbf3df9ac4 [id=1bbf331b6fea4c989aa693d404e31d53]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:16.747118 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5bb7a9e5f5b6424fbae965d050f0d1c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:16.748319 1040 tablet_service.cc:1505] Processing CreateTablet for tablet f6fdaac654d4434d81963746f53205b2 (DEFAULT_TABLE table=default.Table_b2d77ffcda024d1e80fd8c298ddfd3e6 [id=c309ea497e8b4c8dbafeb7162d58236d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:16.748764 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6fdaac654d4434d81963746f53205b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:16.750586 1024 tablet_service.cc:1505] Processing CreateTablet for tablet 242e56760d8940cfb06baeacc3c6c60d (DEFAULT_TABLE table=default.Table_590e7ccd21564447951d30fbf609598e [id=22ec2407ea2341458628db861b630a2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:16.754942 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 242e56760d8940cfb06baeacc3c6c60d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:16.766113 1042 tablet_service.cc:1505] Processing CreateTablet for tablet ba51eb6947da4a4aa4b4fb37ecf05975 (DEFAULT_TABLE table=default.Table_b652acd864e34c469594fa7f6ac8c433 [id=c9a5c95591d04c269c1b2d838d77379a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:16.766113 1034 tablet_service.cc:1505] Processing CreateTablet for tablet b2daeaaa083246ae9431296a8565eef8 (DEFAULT_TABLE table=default.Table_30bf322e8193402e80523ebfba5c8e43 [id=d78df9db30d94001bb0d191bd96da198]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:16.766592 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba51eb6947da4a4aa4b4fb37ecf05975. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:16.767392 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2daeaaa083246ae9431296a8565eef8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:16.771404 1176 ts_tablet_manager.cc:1929] T 44ae2b07edca44afbe1b1fb3a5b1115d P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:16.771591 1176 log.cc:1199] T 44ae2b07edca44afbe1b1fb3a5b1115d P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/44ae2b07edca44afbe1b1fb3a5b1115d
I20251029 02:52:16.772121 1176 ts_tablet_manager.cc:1950] T 44ae2b07edca44afbe1b1fb3a5b1115d P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:16.772526 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1087) took 3681 ms. Trace:
I20251029 02:52:16.772630 1176 rpcz_store.cc:276] 1029 02:52:13.091073 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:13.091453 (+ 380us) service_pool.cc:225] Handling call
1029 02:52:16.726916 (+3635463us) ts_tablet_manager.cc:1151] Deleting tablet 44ae2b07edca44afbe1b1fb3a5b1115d
1029 02:52:16.726925 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:16.753586 (+ 26661us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:16.757439 (+ 3853us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:16.772505 (+ 15066us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3635266}
I20251029 02:52:16.773041 1176 tablet_replica.cc:333] T e65eaaae43ac435ebd665546de9d68dd P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:16.773265 1176 raft_consensus.cc:2243] T e65eaaae43ac435ebd665546de9d68dd P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:16.773594 1176 raft_consensus.cc:2272] T e65eaaae43ac435ebd665546de9d68dd P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:16.774307 1176 ts_tablet_manager.cc:1916] T e65eaaae43ac435ebd665546de9d68dd P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:16.779341 1029 tablet_service.cc:1505] Processing CreateTablet for tablet 77ba190fdaaa44c8ae3b40d1acf73314 (DEFAULT_TABLE table=default.Table_7533d0093ad94ff6b065550375df858a [id=8bd9b71555d145dba4a06b2bf9bd88d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:16.779341 1028 tablet_service.cc:1505] Processing CreateTablet for tablet f145da2e92b14c5fa32aa2ea1fc78b2f (DEFAULT_TABLE table=default.Table_fd79bf1c016a4671837a7237e117dcf5 [id=daeab145500a47dbaaf7c8655c1b503a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:16.779834 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f145da2e92b14c5fa32aa2ea1fc78b2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:16.780018 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77ba190fdaaa44c8ae3b40d1acf73314. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:16.783914 1042 tablet_service.cc:1505] Processing CreateTablet for tablet 16b1cc4db77d426ab0419ec13bd14c8b (DEFAULT_TABLE table=default.Table_e20d6df47ebc45ff8cd74e180a2fe940 [id=269a2458193e4140938bb7384b1a0fa3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:16.784314 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16b1cc4db77d426ab0419ec13bd14c8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:16.784255 2649 raft_consensus.cc:493] T defb795026a44e12afb2defb9c9c6db2 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:16.784438 2649 raft_consensus.cc:515] T defb795026a44e12afb2defb9c9c6db2 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:16.785705 1059 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "defb795026a44e12afb2defb9c9c6db2" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:52:16.786717 849 leader_election.cc:343] T defb795026a44e12afb2defb9c9c6db2 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:16.787134 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "defb795026a44e12afb2defb9c9c6db2" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:16.787393 776 raft_consensus.cc:2468] T defb795026a44e12afb2defb9c9c6db2 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:16.787926 847 leader_election.cc:304] T defb795026a44e12afb2defb9c9c6db2 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5
I20251029 02:52:16.788553 2656 raft_consensus.cc:2804] T defb795026a44e12afb2defb9c9c6db2 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:16.788654 2656 raft_consensus.cc:493] T defb795026a44e12afb2defb9c9c6db2 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:16.788743 2656 raft_consensus.cc:3060] T defb795026a44e12afb2defb9c9c6db2 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:16.792549 1029 tablet_service.cc:1552] Processing DeleteTablet for tablet 6408901d1e5f4ac8a7bff0de708660eb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:16 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:16.792599 1025 tablet_service.cc:1552] Processing DeleteTablet for tablet 62f4a04568804c3da03393c08676c5da with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:16 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:16.792940 1025 tablet_service.cc:1552] Processing DeleteTablet for tablet 818262aee43c4343b864e01f836e76b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:16 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:16.797726 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet 90c2ed692a954a25bdf79b9884f6c5ce with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:16 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:16.802403 2656 raft_consensus.cc:515] T defb795026a44e12afb2defb9c9c6db2 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:16.803853 1059 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "defb795026a44e12afb2defb9c9c6db2" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
W20251029 02:52:16.804523 849 leader_election.cc:343] T defb795026a44e12afb2defb9c9c6db2 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:16.804811 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "defb795026a44e12afb2defb9c9c6db2" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:16.804982 776 raft_consensus.cc:3060] T defb795026a44e12afb2defb9c9c6db2 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:16.806387 1033 tablet_service.cc:1505] Processing CreateTablet for tablet 6d57c43e61f3487099a9713f8885a85a (DEFAULT_TABLE table=default.Table_6facbfaed7644f2f9c16f0fe7316e2fc [id=4037bd8625d146409a7878c4e5b6643c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:16.806799 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d57c43e61f3487099a9713f8885a85a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:16.809561 1176 ts_tablet_manager.cc:1929] T e65eaaae43ac435ebd665546de9d68dd P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:16.811056 1176 log.cc:1199] T e65eaaae43ac435ebd665546de9d68dd P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/e65eaaae43ac435ebd665546de9d68dd
I20251029 02:52:16.811465 1176 ts_tablet_manager.cc:1950] T e65eaaae43ac435ebd665546de9d68dd P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:16.811779 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1088) took 3720 ms. Trace:
I20251029 02:52:16.811849 1176 rpcz_store.cc:276] 1029 02:52:13.091157 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:13.092042 (+ 885us) service_pool.cc:225] Handling call
1029 02:52:16.772988 (+3680946us) ts_tablet_manager.cc:1151] Deleting tablet e65eaaae43ac435ebd665546de9d68dd
1029 02:52:16.773000 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:16.793001 (+ 20001us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:16.809452 (+ 16451us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:16.811765 (+ 2313us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3680769}
I20251029 02:52:16.812288 1176 tablet_replica.cc:333] T afb19ba39399410bb90b6225d7b5b5b1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:16.812489 1176 raft_consensus.cc:2243] T afb19ba39399410bb90b6225d7b5b5b1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:16.812772 1176 raft_consensus.cc:2272] T afb19ba39399410bb90b6225d7b5b5b1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:16.813344 1176 ts_tablet_manager.cc:1916] T afb19ba39399410bb90b6225d7b5b5b1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:16.816051 1167 tablet_bootstrap.cc:492] T 32519fbffcef44a890161e5f3043806c P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:16.816175 1167 ts_tablet_manager.cc:1403] T 32519fbffcef44a890161e5f3043806c P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.081s user 0.009s sys 0.017s
I20251029 02:52:16.816850 1167 raft_consensus.cc:359] T 32519fbffcef44a890161e5f3043806c P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:16.817040 1167 raft_consensus.cc:385] T 32519fbffcef44a890161e5f3043806c P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:16.817082 1167 raft_consensus.cc:740] T 32519fbffcef44a890161e5f3043806c P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:16.817314 1167 consensus_queue.cc:260] T 32519fbffcef44a890161e5f3043806c P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:16.818580 2656 leader_election.cc:290] T defb795026a44e12afb2defb9c9c6db2 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:16.818787 2649 leader_election.cc:290] T defb795026a44e12afb2defb9c9c6db2 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:16.818902 1167 ts_tablet_manager.cc:1434] T 32519fbffcef44a890161e5f3043806c P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251029 02:52:16.819175 1167 tablet_bootstrap.cc:492] T fcb3db8399aa4cf188b07dda220f7f5b P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:16.820724 1167 tablet_bootstrap.cc:654] T fcb3db8399aa4cf188b07dda220f7f5b P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
W20251029 02:52:16.826840 519 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8fa6a27df46b4cf786470792431cefbc consensus state has no leader
I20251029 02:52:16.829489 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_917b4493547e4d55be883cc4a51d45a1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:16.830089 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_917b4493547e4d55be883cc4a51d45a1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:16.831785 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_b3a7b65a549244a6af331747d706ecf3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:16.832654 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_2ef80417992a45b09f93d266693e6f5b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:16.833182 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2ef80417992a45b09f93d266693e6f5b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:16.834196 526 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet 748777c6cd6e4f49ab49912b719e48bd (table default.Table_1e69e259edba466f99ed7a41dcf6367f [id=a2f3b36dc78a4e94bd054a6a5712b384]). Expected version 1 got 0
I20251029 02:52:16.834739 523 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_90cad35ff5a14d0b87b2785627781b23" } modify_external_catalogs: true
I20251029 02:52:16.834869 523 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_90cad35ff5a14d0b87b2785627781b23" } modify_external_catalogs: true
W20251029 02:52:16.835029 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b3a7b65a549244a6af331747d706ecf3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:16.835749 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_f1a523ffbf3c47b0bfabb752b8c2f5a2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:16.836158 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f1a523ffbf3c47b0bfabb752b8c2f5a2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:16.840690 526 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_c4333d8c61d346b1ac1a809d3ccb1915" } modify_external_catalogs: true
I20251029 02:52:16.840854 526 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_c4333d8c61d346b1ac1a809d3ccb1915" } modify_external_catalogs: true
I20251029 02:52:16.842398 520 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_492f50b2191a46ee8e6bb05e69377006" } modify_external_catalogs: true
I20251029 02:52:16.842576 520 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_492f50b2191a46ee8e6bb05e69377006" } modify_external_catalogs: true
I20251029 02:52:16.846720 1027 tablet_service.cc:1552] Processing DeleteTablet for tablet 7fba607c566c4fa1a803cba8d86ca646 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:16 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:16.847448 1968 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:42343
I20251029 02:52:16.847448 2827 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:42343 every 8 connection(s)
I20251029 02:52:16.850289 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet c32c28b0e74a4af3a0e69bbf24ab1a1f (table default.Table_3d94f8b10e884dc0b338dcfbb9c82309 [id=3db1589f811041eaa1cd7412a26391bc]) successfully deleted
I20251029 02:52:16.850855 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 1275e192871440408a8345fef9b56260 (table default.Table_5265805d6c3746209f9466df88b164b9 [id=5c4f8f38dfc748dda3abf03f7882cf35]) successfully deleted
I20251029 02:52:16.851332 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet dabb8d760aae4aa09eb83cdb586c9788 (table default.Table_d897a86961894f1aaad67274b687080e [id=b3e544a043444b77a597c6c65c15b795]) successfully deleted
I20251029 02:52:16.853006 1968 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/master-0/data/info.pb
I20251029 02:52:16.854729 1172 tablet_bootstrap.cc:492] T 688af6580c4d4637a39f817b4e324618 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:16.854946 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 13ee62fc2339410b9db70dfd3d3c6fa6 (table default.Table_cb4697efc8e049be929f4e7a004879d1 [id=b54d7930e083434680f3934fc4023653]) successfully deleted
I20251029 02:52:16.854966 1172 ts_tablet_manager.cc:1403] T 688af6580c4d4637a39f817b4e324618 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.306s user 0.002s sys 0.003s
I20251029 02:52:16.855583 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet af4860c6b72f4d988b206b5ef2f22906 (table default.Table_f5db3e3913ca4ea98b02981cd04a5965 [id=7eb08c679e0e441bb1ad4be13b2494ef]) successfully deleted
I20251029 02:52:16.855912 1172 raft_consensus.cc:359] T 688af6580c4d4637a39f817b4e324618 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:16.856338 1172 raft_consensus.cc:385] T 688af6580c4d4637a39f817b4e324618 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:16.856479 1172 raft_consensus.cc:740] T 688af6580c4d4637a39f817b4e324618 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:16.856765 1172 consensus_queue.cc:260] T 688af6580c4d4637a39f817b4e324618 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:16.860217 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskYVokEG/build/asan/bin/kudu as pid 1968
W20251029 02:52:16.860639 847 consensus_peers.cc:597] T 1bdb67125493430398ff5c80b73de2ac P c756528642a543eeacd93cef3e335a7d -> Peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Couldn't send request to peer b495a49349324f7d81908f2da700f0d9. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:16.861358 1172 ts_tablet_manager.cc:1434] T 688af6580c4d4637a39f817b4e324618 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20251029 02:52:16.861732 1172 tablet_bootstrap.cc:492] T c7b78fd458e4414b9fc3aa064a80b206 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:16.863390 1172 tablet_bootstrap.cc:654] T c7b78fd458e4414b9fc3aa064a80b206 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:16.868382 1167 tablet_bootstrap.cc:492] T fcb3db8399aa4cf188b07dda220f7f5b P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:16.868894 1167 ts_tablet_manager.cc:1403] T fcb3db8399aa4cf188b07dda220f7f5b P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.050s user 0.002s sys 0.015s
I20251029 02:52:16.869752 1167 raft_consensus.cc:359] T fcb3db8399aa4cf188b07dda220f7f5b P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:16.870004 1167 raft_consensus.cc:385] T fcb3db8399aa4cf188b07dda220f7f5b P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:16.870107 1167 raft_consensus.cc:740] T fcb3db8399aa4cf188b07dda220f7f5b P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:16.870402 1167 consensus_queue.cc:260] T fcb3db8399aa4cf188b07dda220f7f5b P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:16.870923 1167 ts_tablet_manager.cc:1434] T fcb3db8399aa4cf188b07dda220f7f5b P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251029 02:52:16.871966 1167 tablet_bootstrap.cc:492] T 23b9b2ea814645eba63e5f6b12cfb966 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:16.874017 1167 tablet_bootstrap.cc:654] T 23b9b2ea814645eba63e5f6b12cfb966 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:16.875713 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 9bb2166a590d4210b02dd779ef6654ad (table default.Table_50252b3b3cc4403da5e259f9f91be915 [id=ff64a323de354792997beb99bae4a52c]) successfully deleted
I20251029 02:52:16.876312 1042 tablet_service.cc:1552] Processing DeleteTablet for tablet c73059366f4b45528f8b907263c4d085 with delete_type TABLET_DATA_DELETED (replaced by tablet fb63ec8e38fe4cbb8df2065b760c9112) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:16.893606 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 1bdb67125493430398ff5c80b73de2ac (table default.Table_2846af457a4b4e27a739af50c7f8c7f7 [id=8c3d3d88395e4712af0f62c4ad14edea]) successfully deleted
W20251029 02:52:16.894188 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 81ee839d811a4f46b93a57ede397bd93 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 81ee839d811a4f46b93a57ede397bd93 already in progress: creating tablet
W20251029 02:52:16.894307 521 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5c40f2f9cf2b41a3b129e7ef1bf41eb3 consensus state has no leader
W20251029 02:52:16.894732 517 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 98ad4304204942c7beb788877dbec0ff consensus state has no leader
W20251029 02:52:16.895557 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 0f96af1cb0ea4b46bd846ae0739f4947 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0f96af1cb0ea4b46bd846ae0739f4947 already in progress: creating tablet
W20251029 02:52:16.896581 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet a530fe8de8ae4be5b0fd9a5f7e94ea9e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a530fe8de8ae4be5b0fd9a5f7e94ea9e already in progress: creating tablet
W20251029 02:52:16.896970 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 426f9be3b0074f60989acc43ec1e9bd6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 426f9be3b0074f60989acc43ec1e9bd6 already in progress: creating tablet
I20251029 02:52:16.898609 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 44c0488eba0543078ff4f0b3d69366f2 (table default.Table_bc32d7beda4f4957817b20124bf48e9c [id=df809b5c66d64eb1846fa5eb9da11781]) successfully deleted
W20251029 02:52:16.899149 513 catalog_manager.cc:4952] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 0d84dc8dc465496f9adec798b2e03e13 because the tablet was not found. No further retry: Not found: Tablet not found: 0d84dc8dc465496f9adec798b2e03e13
I20251029 02:52:16.899693 517 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_1f3338455b1a4eef85ecea36443bc329" } modify_external_catalogs: true
I20251029 02:52:16.899874 517 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_1f3338455b1a4eef85ecea36443bc329" } modify_external_catalogs: true
I20251029 02:52:16.901556 2747 consensus_queue.cc:1048] T 46cd75ff18b14212b556b9bcb3012dd9 P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:16.903486 524 catalog_manager.cc:6194] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: default.Table_9751e24147fb4fb0b7b7030ecd636514 [id=93e27f360fbb436aa652b936c9046510] alter complete (version 1)
W20251029 02:52:16.904211 847 consensus_peers.cc:597] T 40e844fd7ad74680bcbe7bef936c5b9d P c756528642a543eeacd93cef3e335a7d -> Peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Couldn't send request to peer b495a49349324f7d81908f2da700f0d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 40e844fd7ad74680bcbe7bef936c5b9d. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:16.904534 524 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_f28c52f38da44fa6b73e9bca9b29908e" } modify_external_catalogs: true
I20251029 02:52:16.904721 524 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_f28c52f38da44fa6b73e9bca9b29908e" } modify_external_catalogs: true
I20251029 02:52:16.915553 1166 tablet_bootstrap.cc:492] T 70b008011d5547bbaf3f45685394139f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:16.915759 1166 ts_tablet_manager.cc:1403] T 70b008011d5547bbaf3f45685394139f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.239s user 0.005s sys 0.002s
I20251029 02:52:16.926138 522 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_e6af8601f9a34c6183e494c964553b5e" } new_table_name: "default.Table_8c6b8330348d403c92b6e2da82211db8" modify_external_catalogs: true
I20251029 02:52:16.928114 1176 ts_tablet_manager.cc:1929] T afb19ba39399410bb90b6225d7b5b5b1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:16.928246 1176 log.cc:1199] T afb19ba39399410bb90b6225d7b5b5b1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/afb19ba39399410bb90b6225d7b5b5b1
I20251029 02:52:16.928455 518 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_2c589dbe63b741eeb175624a88baf0a9" } new_table_name: "default.Table_a3ee7878447a4c47970b01ff8b52955f" modify_external_catalogs: true
I20251029 02:52:16.928617 1176 ts_tablet_manager.cc:1950] T afb19ba39399410bb90b6225d7b5b5b1 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:16.928752 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_fd4e20d865b640bd818b71d13226aa26"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:16.928989 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1089) took 3837 ms. Trace:
I20251029 02:52:16.929062 1176 rpcz_store.cc:276] 1029 02:52:13.091236 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:13.092256 (+ 1020us) service_pool.cc:225] Handling call
1029 02:52:16.812237 (+3719981us) ts_tablet_manager.cc:1151] Deleting tablet afb19ba39399410bb90b6225d7b5b5b1
1029 02:52:16.812249 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:16.897049 (+ 84800us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:16.927997 (+ 30948us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:16.928962 (+ 965us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3719736}
W20251029 02:52:16.929265 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fd4e20d865b640bd818b71d13226aa26 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:16.929461 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1107) took 3513 ms. Trace:
I20251029 02:52:16.929538 1176 rpcz_store.cc:276] 1029 02:52:13.416386 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:13.421028 (+ 4642us) service_pool.cc:225] Handling call
1029 02:52:16.929318 (+3508290us) ts_tablet_manager.cc:1151] Deleting tablet 31ca9012ddf6401ba78888a0c82b4ef2
1029 02:52:16.929329 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:16.929444 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3507302}
I20251029 02:52:16.929889 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1106) took 3513 ms. Trace:
I20251029 02:52:16.929953 1176 rpcz_store.cc:276] 1029 02:52:13.416325 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:13.422048 (+ 5723us) service_pool.cc:225] Handling call
1029 02:52:16.929757 (+3507709us) ts_tablet_manager.cc:1151] Deleting tablet 416e7c48daa849cd9635656529488922
1029 02:52:16.929767 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:16.929872 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3507551}
I20251029 02:52:16.930248 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1105) took 3514 ms. Trace:
I20251029 02:52:16.930316 1176 rpcz_store.cc:276] 1029 02:52:13.416250 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:13.425668 (+ 9418us) service_pool.cc:225] Handling call
1029 02:52:16.930147 (+3504479us) ts_tablet_manager.cc:1151] Deleting tablet eb3c20822d684bea89eb358251e21e33
1029 02:52:16.930157 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:16.930234 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3497422}
I20251029 02:52:16.930436 519 master_service.cc:954] ReplaceTablet: received request to replace tablet 98bbaec01186435b8bcc7257a1390185 from {username='slave'} at 127.0.0.1:35314
I20251029 02:52:16.941854 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1104) took 3525 ms. Trace:
I20251029 02:52:16.941980 1176 rpcz_store.cc:276] 1029 02:52:13.416151 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:13.426187 (+ 10036us) service_pool.cc:225] Handling call
1029 02:52:16.930503 (+3504316us) ts_tablet_manager.cc:1151] Deleting tablet 21c73cb2f1084827b581fb09cf7239a1
1029 02:52:16.936284 (+ 5781us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:16.941831 (+ 5547us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3497711}
I20251029 02:52:16.944731 1174 ts_tablet_manager.cc:1929] T e7f846b294cc4eb687a26c8b5931ac19 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:16.944823 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1103) took 3528 ms. Trace:
I20251029 02:52:16.944921 1176 rpcz_store.cc:276] 1029 02:52:13.416045 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:13.428059 (+ 12014us) service_pool.cc:225] Handling call
1029 02:52:16.942226 (+3514167us) ts_tablet_manager.cc:1151] Deleting tablet 15cadd681ceb4daba178a9cc2b7aec08
1029 02:52:16.942237 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:16.944806 (+ 2569us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3509371}
I20251029 02:52:16.945317 1176 tablet_replica.cc:333] T 82289e0f823c4d43bfd8c4e878572b2b P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:16.945515 1176 raft_consensus.cc:2243] T 82289e0f823c4d43bfd8c4e878572b2b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:16.945667 1176 pending_rounds.cc:70] T 82289e0f823c4d43bfd8c4e878572b2b P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Trying to abort 1 pending ops.
I20251029 02:52:16.945711 1176 pending_rounds.cc:77] T 82289e0f823c4d43bfd8c4e878572b2b P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7215949155507691520 op_type: ALTER_SCHEMA_OP alter_schema_request { tablet_id: "82289e0f823c4d43bfd8c4e878572b2b" schema { columns { id: 10 name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } schema_version: 1 new_table_name: "default.Table_e7bb3e262a894c2f84b6edfc3917b30d" dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" new_extra_config { } }
W20251029 02:52:16.946727 1176 tablet_service.cc:731] failed op from {username='slave'} at 127.0.0.1:37626: Aborted: Op aborted
I20251029 02:52:16.947270 1176 raft_consensus.cc:2272] T 82289e0f823c4d43bfd8c4e878572b2b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:16.947986 1176 ts_tablet_manager.cc:1916] T 82289e0f823c4d43bfd8c4e878572b2b P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:16.950971 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_3e74debf297b4d7cb2f35a8a9bdf9155"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:16.951706 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3e74debf297b4d7cb2f35a8a9bdf9155 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:16.952591 1166 raft_consensus.cc:359] T 70b008011d5547bbaf3f45685394139f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:16.952821 1166 raft_consensus.cc:385] T 70b008011d5547bbaf3f45685394139f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:16.952930 1166 raft_consensus.cc:740] T 70b008011d5547bbaf3f45685394139f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:16.953173 1166 consensus_queue.cc:260] T 70b008011d5547bbaf3f45685394139f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:16.959062 1166 ts_tablet_manager.cc:1434] T 70b008011d5547bbaf3f45685394139f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.043s user 0.001s sys 0.001s
I20251029 02:52:16.960980 1166 tablet_bootstrap.cc:492] T d31e548feae24b739087c1e211d514f9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:16.897627 1175 ts_tablet_manager.cc:1929] T 1bdb67125493430398ff5c80b73de2ac P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251029 02:52:16.906476 738 tablet_service.cc:1552] Processing DeleteTablet for tablet 24f07b1efac64b099b0b5ed467463749 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:59 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:16.906705 750 tablet_service.cc:1552] Processing DeleteTablet for tablet 7816556521144c928d7a02694e22e79f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:58 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:16.914398 2763 tablet.cc:1722] T 27b08f88b99643d28ac609e45bf8104e P b495a49349324f7d81908f2da700f0d9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 02:52:16.976568 1175 log.cc:1199] T 1bdb67125493430398ff5c80b73de2ac P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/1bdb67125493430398ff5c80b73de2ac
I20251029 02:52:16.976619 821 heartbeater.cc:344] Connected to a master server at 127.0.104.62:42343
I20251029 02:52:16.977197 1175 ts_tablet_manager.cc:1950] T 1bdb67125493430398ff5c80b73de2ac P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:16.977576 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 991) took 5291 ms. Trace:
I20251029 02:52:16.977726 1175 rpcz_store.cc:276] 1029 02:52:11.685870 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:11.686332 (+ 462us) service_pool.cc:225] Handling call
1029 02:52:16.531839 (+4845507us) ts_tablet_manager.cc:1151] Deleting tablet 1bdb67125493430398ff5c80b73de2ac
1029 02:52:16.531854 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:16.594400 (+ 62546us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:16.897518 (+303118us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:16.977557 (+ 80039us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4814250}
I20251029 02:52:16.978230 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 993) took 5248 ms. Trace:
I20251029 02:52:16.978293 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 1bdb67125493430398ff5c80b73de2ac (table default.Table_2846af457a4b4e27a739af50c7f8c7f7 [id=8c3d3d88395e4712af0f62c4ad14edea]) successfully deleted
I20251029 02:52:16.978350 1175 rpcz_store.cc:276] 1029 02:52:11.729635 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:11.729689 (+ 54us) service_pool.cc:225] Handling call
1029 02:52:16.978111 (+5248422us) ts_tablet_manager.cc:1151] Deleting tablet 81ee839d811a4f46b93a57ede397bd93
1029 02:52:16.978119 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:16.978215 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5189830}
I20251029 02:52:16.978698 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 996) took 5234 ms. Trace:
W20251029 02:52:16.978806 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 81ee839d811a4f46b93a57ede397bd93 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 81ee839d811a4f46b93a57ede397bd93 already in progress: creating tablet
I20251029 02:52:16.979348 776 raft_consensus.cc:2468] T defb795026a44e12afb2defb9c9c6db2 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:16.985085 1175 rpcz_store.cc:276] 1029 02:52:11.744047 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:11.744099 (+ 52us) service_pool.cc:225] Handling call
1029 02:52:16.978579 (+5234480us) ts_tablet_manager.cc:1151] Deleting tablet 426f9be3b0074f60989acc43ec1e9bd6
1029 02:52:16.978589 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:16.978685 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5190207}
I20251029 02:52:16.985607 1174 log.cc:1199] T e7f846b294cc4eb687a26c8b5931ac19 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/e7f846b294cc4eb687a26c8b5931ac19
I20251029 02:52:16.985556 847 leader_election.cc:304] T defb795026a44e12afb2defb9c9c6db2 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5
I20251029 02:52:16.986289 2655 raft_consensus.cc:2804] T defb795026a44e12afb2defb9c9c6db2 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:16.986415 2655 raft_consensus.cc:697] T defb795026a44e12afb2defb9c9c6db2 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:52:16.986696 2655 consensus_queue.cc:237] T defb795026a44e12afb2defb9c9c6db2 P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:16.946035 2830 tablet.cc:1722] T 731c0acfbe204e8199ad246e38ae2ccd P c756528642a543eeacd93cef3e335a7d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 02:52:16.987829 965 heartbeater.cc:344] Connected to a master server at 127.0.104.62:42343
I20251029 02:52:16.991402 1166 tablet_bootstrap.cc:654] T d31e548feae24b739087c1e211d514f9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:16.986683 2842 tablet.cc:1722] T 27b08f88b99643d28ac609e45bf8104e P c756528642a543eeacd93cef3e335a7d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20251029 02:52:17.001809 518 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a8cef3074aef43b2981eaffdc5a5b5bb consensus state has no leader
I20251029 02:52:17.002658 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_ae6b1788f8824096bd72ee8ec2ddcb15"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:17.003250 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ae6b1788f8824096bd72ee8ec2ddcb15 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:17.010900 1174 ts_tablet_manager.cc:1950] T e7f846b294cc4eb687a26c8b5931ac19 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:17.007458 821 heartbeater.cc:344] Connected to a master server at 127.0.104.62:42343
I20251029 02:52:17.011314 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 995) took 5267 ms. Trace:
W20251029 02:52:17.011369 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 426f9be3b0074f60989acc43ec1e9bd6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 426f9be3b0074f60989acc43ec1e9bd6 already in progress: creating tablet
I20251029 02:52:17.012369 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_8c6777c5195142f8b08409b19267c331"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:17.012517 1175 rpcz_store.cc:276] 1029 02:52:11.743938 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:11.744289 (+ 351us) service_pool.cc:225] Handling call
1029 02:52:17.011178 (+5266889us) ts_tablet_manager.cc:1151] Deleting tablet a530fe8de8ae4be5b0fd9a5f7e94ea9e
1029 02:52:17.011191 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:17.011298 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5222747}
W20251029 02:52:17.013080 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8c6777c5195142f8b08409b19267c331 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:17.013150 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 956) took 6365 ms. Trace:
W20251029 02:52:17.013742 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet a530fe8de8ae4be5b0fd9a5f7e94ea9e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a530fe8de8ae4be5b0fd9a5f7e94ea9e already in progress: creating tablet
I20251029 02:52:17.013963 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 994) took 5270 ms. Trace:
I20251029 02:52:17.014041 1175 rpcz_store.cc:276] 1029 02:52:11.743827 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:11.744391 (+ 564us) service_pool.cc:225] Handling call
1029 02:52:17.013841 (+5269450us) ts_tablet_manager.cc:1151] Deleting tablet 0f96af1cb0ea4b46bd846ae0739f4947
1029 02:52:17.013850 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:17.013947 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5225360}
W20251029 02:52:17.014412 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 0f96af1cb0ea4b46bd846ae0739f4947 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0f96af1cb0ea4b46bd846ae0739f4947 already in progress: creating tablet
I20251029 02:52:17.016801 1174 rpcz_store.cc:276] 1029 02:52:10.647391 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:10.647491 (+ 100us) service_pool.cc:225] Handling call
1029 02:52:16.575115 (+5927624us) ts_tablet_manager.cc:1151] Deleting tablet e7f846b294cc4eb687a26c8b5931ac19
1029 02:52:16.575126 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:16.801606 (+226480us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:16.944616 (+143010us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:17.013129 (+ 68513us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5927454}
I20251029 02:52:17.020056 1166 tablet_bootstrap.cc:492] T d31e548feae24b739087c1e211d514f9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:17.020291 1166 ts_tablet_manager.cc:1403] T d31e548feae24b739087c1e211d514f9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.059s user 0.006s sys 0.000s
I20251029 02:52:17.021225 1166 raft_consensus.cc:359] T d31e548feae24b739087c1e211d514f9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:17.022217 1166 raft_consensus.cc:385] T d31e548feae24b739087c1e211d514f9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:17.022311 1166 raft_consensus.cc:740] T d31e548feae24b739087c1e211d514f9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:17.023078 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet e7f846b294cc4eb687a26c8b5931ac19 (table default.Table_049332132e3142ca86d49635656135f5 [id=74c72bc1206b4247b1b1c994ccc402f8]) successfully deleted
I20251029 02:52:17.023631 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_52063b7a3e27479caf9d097f7eecd731"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:17.024109 1166 consensus_queue.cc:260] T d31e548feae24b739087c1e211d514f9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
W20251029 02:52:17.024358 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_52063b7a3e27479caf9d097f7eecd731 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:17.024727 1174 tablet_replica.cc:333] T 30d03e1e42d34242bd354f9e9ec4f127 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:17.024996 1174 raft_consensus.cc:2243] T 30d03e1e42d34242bd354f9e9ec4f127 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:17.025230 1174 raft_consensus.cc:2272] T 30d03e1e42d34242bd354f9e9ec4f127 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:17.022131 1176 ts_tablet_manager.cc:1929] T 82289e0f823c4d43bfd8c4e878572b2b P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251029 02:52:17.026679 1176 log.cc:1199] T 82289e0f823c4d43bfd8c4e878572b2b P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/82289e0f823c4d43bfd8c4e878572b2b
I20251029 02:52:17.026917 1174 ts_tablet_manager.cc:1916] T 30d03e1e42d34242bd354f9e9ec4f127 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:17.027065 1176 ts_tablet_manager.cc:1950] T 82289e0f823c4d43bfd8c4e878572b2b P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:17.027384 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1118) took 3401 ms. Trace:
I20251029 02:52:17.027462 1176 rpcz_store.cc:276] 1029 02:52:13.625707 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:13.630867 (+ 5160us) service_pool.cc:225] Handling call
1029 02:52:16.945270 (+3314403us) ts_tablet_manager.cc:1151] Deleting tablet 82289e0f823c4d43bfd8c4e878572b2b
1029 02:52:16.945281 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:16.993989 (+ 48708us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:17.021997 (+ 28008us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:17.027368 (+ 5371us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3314226}
I20251029 02:52:17.027842 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1119) took 3293 ms. Trace:
I20251029 02:52:17.027912 1176 rpcz_store.cc:276] 1029 02:52:13.734672 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:13.734724 (+ 52us) service_pool.cc:225] Handling call
1029 02:52:17.027717 (+3292993us) ts_tablet_manager.cc:1151] Deleting tablet dab6ed2266ed42508738399d388c4da3
1029 02:52:17.027726 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:17.027828 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3292818}
I20251029 02:52:17.028188 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1124) took 3227 ms. Trace:
I20251029 02:52:17.028250 1176 rpcz_store.cc:276] 1029 02:52:13.800554 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:13.800602 (+ 48us) service_pool.cc:225] Handling call
1029 02:52:17.028105 (+3227503us) ts_tablet_manager.cc:1151] Deleting tablet b4621bf9e6324eaeaa64a6ca51cd5ca9
1029 02:52:17.028113 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:17.028177 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3227339}
I20251029 02:52:17.028461 1176 tablet_replica.cc:333] T d041e87ceb11441983d0e38b7929c3a4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:17.033253 519 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_bdb0ff1c8e1440b89844dc9a076cf5de" } new_table_name: "default.Table_2af94979b80a40e3804eed706e1d4cbe" modify_external_catalogs: true
I20251029 02:52:17.034664 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_06ef187658064f3d9f471ac2b1d40245"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:17.036506 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_06ef187658064f3d9f471ac2b1d40245 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:17.028617 1176 raft_consensus.cc:2243] T d041e87ceb11441983d0e38b7929c3a4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:17.037179 1176 raft_consensus.cc:2272] T d041e87ceb11441983d0e38b7929c3a4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:17.037662 1176 ts_tablet_manager.cc:1916] T d041e87ceb11441983d0e38b7929c3a4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 02:52:17.039281 522 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d31ac6638b424e1786755ed2dd9e4b64 has no consensus state
I20251029 02:52:17.041042 2722 raft_consensus.cc:493] T 506f8c112a3b43e1a3d6bbafb8cb81e8 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:17.041193 2722 raft_consensus.cc:515] T 506f8c112a3b43e1a3d6bbafb8cb81e8 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:17.041795 2722 leader_election.cc:290] T 506f8c112a3b43e1a3d6bbafb8cb81e8 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:52:17.042299 1172 tablet_bootstrap.cc:492] T c7b78fd458e4414b9fc3aa064a80b206 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:17.042434 1172 ts_tablet_manager.cc:1403] T c7b78fd458e4414b9fc3aa064a80b206 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.181s user 0.001s sys 0.003s
I20251029 02:52:17.030620 1166 ts_tablet_manager.cc:1434] T d31e548feae24b739087c1e211d514f9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20251029 02:52:17.030761 2847 tablet.cc:1722] T 27b08f88b99643d28ac609e45bf8104e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 02:52:17.043139 1107 heartbeater.cc:344] Connected to a master server at 127.0.104.62:42343
I20251029 02:52:17.045111 1166 tablet_bootstrap.cc:492] T defb795026a44e12afb2defb9c9c6db2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:17.046478 1166 tablet_bootstrap.cc:654] T defb795026a44e12afb2defb9c9c6db2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:17.046685 919 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "506f8c112a3b43e1a3d6bbafb8cb81e8" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:17.047308 711 leader_election.cc:343] T 506f8c112a3b43e1a3d6bbafb8cb81e8 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:17.054234 1166 tablet_bootstrap.cc:492] T defb795026a44e12afb2defb9c9c6db2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:17.054502 1107 heartbeater.cc:344] Connected to a master server at 127.0.104.62:42343
I20251029 02:52:17.055789 1166 ts_tablet_manager.cc:1403] T defb795026a44e12afb2defb9c9c6db2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20251029 02:52:17.056457 1166 raft_consensus.cc:359] T defb795026a44e12afb2defb9c9c6db2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:17.059216 1166 raft_consensus.cc:385] T defb795026a44e12afb2defb9c9c6db2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:17.059294 1166 raft_consensus.cc:740] T defb795026a44e12afb2defb9c9c6db2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:17.059505 1166 consensus_queue.cc:260] T defb795026a44e12afb2defb9c9c6db2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:17.059979 1166 ts_tablet_manager.cc:1434] T defb795026a44e12afb2defb9c9c6db2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20251029 02:52:17.064568 1166 tablet_bootstrap.cc:492] T 84993b2f63e7421290a161dde4bb0607 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:17.066789 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 2f198c3259864f7ab3625cb7bfae20b0 (table default.Table_a3377614e4a54266a4457944d48b0060 [id=991a151cc5d84421bf50858b1fa864e6]) successfully deleted
W20251029 02:52:17.067198 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 1150d2210a234f1aa554f486cdcabc4c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1150d2210a234f1aa554f486cdcabc4c already in progress: creating tablet
W20251029 02:52:17.067572 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet fd1bf6b0f45349d0981f9ddd89269489 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fd1bf6b0f45349d0981f9ddd89269489 already in progress: creating tablet
W20251029 02:52:17.067914 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 41cfa2d722034618a05de717b28a778d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 41cfa2d722034618a05de717b28a778d already in progress: creating tablet
I20251029 02:52:17.068315 965 heartbeater.cc:344] Connected to a master server at 127.0.104.62:42343
W20251029 02:52:17.068367 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 99d60ef5325e45059026596a756448de because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 99d60ef5325e45059026596a756448de already in progress: creating tablet
W20251029 02:52:17.068836 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 49aaa2d8986a44df979edc1c42612189 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 49aaa2d8986a44df979edc1c42612189 already in progress: creating tablet
I20251029 02:52:17.069168 1166 tablet_bootstrap.cc:654] T 84993b2f63e7421290a161dde4bb0607 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
W20251029 02:52:17.071780 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 9280503662894544a1f227a62a9b8a01 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9280503662894544a1f227a62a9b8a01 already in progress: creating tablet
W20251029 02:52:17.072418 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet fe5aff0f61a44b91bfb37b42457a91f0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fe5aff0f61a44b91bfb37b42457a91f0 already in progress: creating tablet
W20251029 02:52:17.072757 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet ca71c4638a6b419b876e22af7e6cf00a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ca71c4638a6b419b876e22af7e6cf00a already in progress: creating tablet
I20251029 02:52:17.076655 1167 tablet_bootstrap.cc:492] T 23b9b2ea814645eba63e5f6b12cfb966 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:17.077533 1167 ts_tablet_manager.cc:1403] T 23b9b2ea814645eba63e5f6b12cfb966 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.206s user 0.007s sys 0.009s
I20251029 02:52:17.078440 1167 raft_consensus.cc:359] T 23b9b2ea814645eba63e5f6b12cfb966 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:17.079571 1167 raft_consensus.cc:385] T 23b9b2ea814645eba63e5f6b12cfb966 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:17.079761 1167 raft_consensus.cc:740] T 23b9b2ea814645eba63e5f6b12cfb966 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:17.080183 1167 consensus_queue.cc:260] T 23b9b2ea814645eba63e5f6b12cfb966 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:17.080366 1176 ts_tablet_manager.cc:1929] T d041e87ceb11441983d0e38b7929c3a4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:17.080497 1176 log.cc:1199] T d041e87ceb11441983d0e38b7929c3a4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/d041e87ceb11441983d0e38b7929c3a4
I20251029 02:52:17.080888 1176 ts_tablet_manager.cc:1950] T d041e87ceb11441983d0e38b7929c3a4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:17.081396 1167 ts_tablet_manager.cc:1434] T 23b9b2ea814645eba63e5f6b12cfb966 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251029 02:52:17.081772 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1121) took 3302 ms. Trace:
I20251029 02:52:17.081879 1176 rpcz_store.cc:276] 1029 02:52:13.779265 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:13.779311 (+ 46us) service_pool.cc:225] Handling call
1029 02:52:17.028408 (+3249097us) ts_tablet_manager.cc:1151] Deleting tablet d041e87ceb11441983d0e38b7929c3a4
1029 02:52:17.028415 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:17.040772 (+ 12357us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:17.080252 (+ 39480us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:17.081745 (+ 1493us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3226659}
I20251029 02:52:17.082264 1176 tablet_replica.cc:333] T 59eee86f249e4d6a9d09d30276dcd659 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:17.082527 1176 raft_consensus.cc:2243] T 59eee86f249e4d6a9d09d30276dcd659 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:17.082814 1176 raft_consensus.cc:2272] T 59eee86f249e4d6a9d09d30276dcd659 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:17.082851 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 44ae2b07edca44afbe1b1fb3a5b1115d (table default.Table_c2719a3251c84739a6681f80f1d8f94d [id=a746921b2aba4cd6af22a293991968e8]) successfully deleted
I20251029 02:52:17.083387 1176 ts_tablet_manager.cc:1916] T 59eee86f249e4d6a9d09d30276dcd659 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:17.083801 1172 raft_consensus.cc:359] T c7b78fd458e4414b9fc3aa064a80b206 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:17.083981 1172 raft_consensus.cc:385] T c7b78fd458e4414b9fc3aa064a80b206 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:17.084028 1172 raft_consensus.cc:740] T c7b78fd458e4414b9fc3aa064a80b206 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:17.084587 1172 consensus_queue.cc:260] T c7b78fd458e4414b9fc3aa064a80b206 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:17.085186 1172 ts_tablet_manager.cc:1434] T c7b78fd458e4414b9fc3aa064a80b206 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.043s user 0.002s sys 0.000s
I20251029 02:52:17.090948 1175 tablet_replica.cc:333] T 46cd75ff18b14212b556b9bcb3012dd9 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:17.091212 1175 raft_consensus.cc:2243] T 46cd75ff18b14212b556b9bcb3012dd9 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:17.091427 1175 raft_consensus.cc:2272] T 46cd75ff18b14212b556b9bcb3012dd9 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:17.091823 1166 tablet_bootstrap.cc:492] T 84993b2f63e7421290a161dde4bb0607 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:17.092038 1166 ts_tablet_manager.cc:1403] T 84993b2f63e7421290a161dde4bb0607 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.000s
I20251029 02:52:17.092991 1166 raft_consensus.cc:359] T 84993b2f63e7421290a161dde4bb0607 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:17.093228 1166 raft_consensus.cc:385] T 84993b2f63e7421290a161dde4bb0607 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:17.093339 1166 raft_consensus.cc:740] T 84993b2f63e7421290a161dde4bb0607 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:17.093623 1166 consensus_queue.cc:260] T 84993b2f63e7421290a161dde4bb0607 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:17.094257 1166 ts_tablet_manager.cc:1434] T 84993b2f63e7421290a161dde4bb0607 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 02:52:17.095039 1175 ts_tablet_manager.cc:1916] T 46cd75ff18b14212b556b9bcb3012dd9 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:17.096032 522 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_b15248a339e14b85944e649464498b84" } new_table_name: "default.Table_a9c2f9e099c14953bbce19d79154eb31" modify_external_catalogs: true
I20251029 02:52:17.100351 1058 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e3000183230e4350b47f5085952f2db6" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:17.101048 1058 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "32519fbffcef44a890161e5f3043806c" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:52:17.101187 1058 raft_consensus.cc:3060] T 32519fbffcef44a890161e5f3043806c P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:17.104833 1176 ts_tablet_manager.cc:1929] T 59eee86f249e4d6a9d09d30276dcd659 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:17.104938 1176 log.cc:1199] T 59eee86f249e4d6a9d09d30276dcd659 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/59eee86f249e4d6a9d09d30276dcd659
I20251029 02:52:17.105309 1176 ts_tablet_manager.cc:1950] T 59eee86f249e4d6a9d09d30276dcd659 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:17.105331 1058 raft_consensus.cc:2468] T 32519fbffcef44a890161e5f3043806c P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b495a49349324f7d81908f2da700f0d9 in term 1.
W20251029 02:52:17.106096 519 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e4d17459491f4f76b990b64b68fc06a5 has no consensus state
I20251029 02:52:17.106576 519 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_b74b95d8844643009cb9312d31770ded" } new_table_name: "default.Table_aae6e3dc36e548d6a9116e0535b7a32d" modify_external_catalogs: true
I20251029 02:52:17.106956 1058 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cc802bda2f6a4e439b605f2cafa37967" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:17.108153 1058 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "506f8c112a3b43e1a3d6bbafb8cb81e8" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:17.110611 989 leader_election.cc:304] T 6789bc16664e4b37b86702449ab27855 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5; no voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:17.111390 2717 raft_consensus.cc:2749] T 6789bc16664e4b37b86702449ab27855 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251029 02:52:17.117786 989 leader_election.cc:343] T 372d93914abc41b38a702742b7db0bc2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 372d93914abc41b38a702742b7db0bc2
W20251029 02:52:17.118469 512 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 636e9fe4eeab45749368cd1eb9d48b03 (table default.Table_230b5a9dc63d41fb95e7f9d048bb5d6f [id=7ef14a66267546ca92eb58eedf69e3ff], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 636e9fe4eeab45749368cd1eb9d48b03 consensus state has no leader
W20251029 02:52:17.119566 989 leader_election.cc:343] T 372d93914abc41b38a702742b7db0bc2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 372d93914abc41b38a702742b7db0bc2
W20251029 02:52:17.120051 519 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c92a3c77be8943fb929d321559e3a46c consensus state has no leader
W20251029 02:52:17.120239 989 leader_election.cc:343] T 20a762344c5f46f88481c4ac7ad6e43b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 20a762344c5f46f88481c4ac7ad6e43b
I20251029 02:52:17.121222 1172 tablet_bootstrap.cc:492] T abf18e06df9b482a8b25c078b9495272 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:17.123631 1172 tablet_bootstrap.cc:654] T abf18e06df9b482a8b25c078b9495272 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
W20251029 02:52:17.124423 989 leader_election.cc:343] T 20a762344c5f46f88481c4ac7ad6e43b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 20a762344c5f46f88481c4ac7ad6e43b
I20251029 02:52:17.127415 519 master_service.cc:954] ReplaceTablet: received request to replace tablet 4eec207d19f540518bb567d7ab21d048 from {username='slave'} at 127.0.0.1:35314
I20251029 02:52:17.128697 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet e65eaaae43ac435ebd665546de9d68dd (table default.Table_d9fe74a49f4843b089aae7fca9e4a826 [id=152882c742b2459ba863fd41083284c1]) successfully deleted
I20251029 02:52:17.131381 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet afb19ba39399410bb90b6225d7b5b5b1 (table default.Table_d4caf12f7662463c824210f3c9cd7e0f [id=7dfb670d18824f5dbdd321940fea965e]) successfully deleted
W20251029 02:52:17.131846 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 31ca9012ddf6401ba78888a0c82b4ef2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 31ca9012ddf6401ba78888a0c82b4ef2 already in progress: creating tablet
W20251029 02:52:17.134460 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 416e7c48daa849cd9635656529488922 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 416e7c48daa849cd9635656529488922 already in progress: creating tablet
I20251029 02:52:17.135968 1175 ts_tablet_manager.cc:1929] T 46cd75ff18b14212b556b9bcb3012dd9 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20251029 02:52:17.136334 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet eb3c20822d684bea89eb358251e21e33 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eb3c20822d684bea89eb358251e21e33 already in progress: creating tablet
W20251029 02:52:17.136678 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 21c73cb2f1084827b581fb09cf7239a1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 21c73cb2f1084827b581fb09cf7239a1 already in progress: creating tablet
W20251029 02:52:17.137019 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 15cadd681ceb4daba178a9cc2b7aec08 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 15cadd681ceb4daba178a9cc2b7aec08 already in progress: creating tablet
W20251029 02:52:17.137375 513 catalog_manager.cc:5064] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): alter failed for tablet 82289e0f823c4d43bfd8c4e878572b2b (table default.Table_e7bb3e262a894c2f84b6edfc3917b30d [id=2cca3b7c50b243b5883b8ac1f353f253]): Aborted: Op aborted
I20251029 02:52:17.138783 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_a68b9a5d4e444e579e08cc644d3aee8c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:17.139394 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a68b9a5d4e444e579e08cc644d3aee8c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:17.139364 2747 raft_consensus.cc:493] T 23b9b2ea814645eba63e5f6b12cfb966 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:17.139498 2747 raft_consensus.cc:515] T 23b9b2ea814645eba63e5f6b12cfb966 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:17.139715 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 82289e0f823c4d43bfd8c4e878572b2b (table default.Table_e7bb3e262a894c2f84b6edfc3917b30d [id=2cca3b7c50b243b5883b8ac1f353f253]) successfully deleted
I20251029 02:52:17.139912 1175 log.cc:1199] T 46cd75ff18b14212b556b9bcb3012dd9 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/46cd75ff18b14212b556b9bcb3012dd9
I20251029 02:52:17.140147 2747 leader_election.cc:290] T 23b9b2ea814645eba63e5f6b12cfb966 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:17.140264 1175 ts_tablet_manager.cc:1950] T 46cd75ff18b14212b556b9bcb3012dd9 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:17.140692 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "23b9b2ea814645eba63e5f6b12cfb966" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:17.140843 776 raft_consensus.cc:2468] T 23b9b2ea814645eba63e5f6b12cfb966 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:17.141242 847 leader_election.cc:304] T 23b9b2ea814645eba63e5f6b12cfb966 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:17.140561 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 999) took 5261 ms. Trace:
I20251029 02:52:17.141398 1058 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "23b9b2ea814645eba63e5f6b12cfb966" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:17.141503 1175 rpcz_store.cc:276] 1029 02:52:11.879603 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:11.880298 (+ 695us) service_pool.cc:225] Handling call
1029 02:52:17.038779 (+5158481us) ts_tablet_manager.cc:1151] Deleting tablet 46cd75ff18b14212b556b9bcb3012dd9
1029 02:52:17.090884 (+ 52105us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:17.114002 (+ 23118us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:17.135831 (+ 21829us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:17.140546 (+ 4715us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5158236}
W20251029 02:52:17.142019 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet dab6ed2266ed42508738399d388c4da3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dab6ed2266ed42508738399d388c4da3 already in progress: creating tablet
I20251029 02:52:17.142040 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 46cd75ff18b14212b556b9bcb3012dd9 (table default.Table_2ccb14f87f774da3b62823c798b31a7c [id=0829f9f13eea4c2081a9545b98e928bf]) successfully deleted
I20251029 02:52:17.142414 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 1004) took 4812 ms. Trace:
I20251029 02:52:17.142481 1175 rpcz_store.cc:276] 1029 02:52:12.330060 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:12.330128 (+ 68us) service_pool.cc:225] Handling call
1029 02:52:17.142297 (+4812169us) ts_tablet_manager.cc:1151] Deleting tablet 1150d2210a234f1aa554f486cdcabc4c
1029 02:52:17.142307 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:17.142398 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4811992}
W20251029 02:52:17.142910 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet b4621bf9e6324eaeaa64a6ca51cd5ca9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b4621bf9e6324eaeaa64a6ca51cd5ca9 already in progress: creating tablet
I20251029 02:52:17.143239 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet d041e87ceb11441983d0e38b7929c3a4 (table default.Table_6a78c7ef966c490aaa5e73f56d6d194e [id=c54b92541c3047c0887c36d31743e71c]) successfully deleted
I20251029 02:52:17.143493 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_46f6e47207184dedad719ac68774c0b7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:17.143975 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_46f6e47207184dedad719ac68774c0b7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:52:17.143599 712 leader_election.cc:343] T e3000183230e4350b47f5085952f2db6 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:17.144132 712 leader_election.cc:304] T e3000183230e4350b47f5085952f2db6 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:17.144515 712 leader_election.cc:304] T 32519fbffcef44a890161e5f3043806c P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters: c756528642a543eeacd93cef3e335a7d
I20251029 02:52:17.144805 2722 raft_consensus.cc:2749] T e3000183230e4350b47f5085952f2db6 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:17.144939 2722 raft_consensus.cc:2804] T 32519fbffcef44a890161e5f3043806c P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:17.145011 2722 raft_consensus.cc:697] T 32519fbffcef44a890161e5f3043806c P b495a49349324f7d81908f2da700f0d9 [term 1 LEADER]: Becoming Leader. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Running, Role: LEADER
I20251029 02:52:17.145226 2722 consensus_queue.cc:237] T 32519fbffcef44a890161e5f3043806c P b495a49349324f7d81908f2da700f0d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:17.146740 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_896799e8c5bd4815973929a8ba006a36"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:17.147202 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_896799e8c5bd4815973929a8ba006a36 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:17.154078 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_adbbcf20c8ea4fb691d28bdee6541552"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:17.154594 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_adbbcf20c8ea4fb691d28bdee6541552 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:52:17.163643 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 1150d2210a234f1aa554f486cdcabc4c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1150d2210a234f1aa554f486cdcabc4c already in progress: creating tablet
I20251029 02:52:17.164350 2717 raft_consensus.cc:493] T ae4231daa2594258aa51419c97a2f50a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:17.164496 2717 raft_consensus.cc:515] T ae4231daa2594258aa51419c97a2f50a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:17.165064 2717 leader_election.cc:290] T ae4231daa2594258aa51419c97a2f50a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:17.166005 919 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae4231daa2594258aa51419c97a2f50a" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:17.164099 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 1005) took 4833 ms. Trace:
I20251029 02:52:17.167809 1175 rpcz_store.cc:276] 1029 02:52:12.330930 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:12.331119 (+ 189us) service_pool.cc:225] Handling call
1029 02:52:17.163968 (+4832849us) ts_tablet_manager.cc:1151] Deleting tablet fd1bf6b0f45349d0981f9ddd89269489
1029 02:52:17.163979 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:17.164079 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4825622}
W20251029 02:52:17.169906 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet fd1bf6b0f45349d0981f9ddd89269489 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fd1bf6b0f45349d0981f9ddd89269489 already in progress: creating tablet
I20251029 02:52:17.170217 1166 tablet_bootstrap.cc:492] T cd597c319af645909b8f3d1212fcb702 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:17.171618 1166 tablet_bootstrap.cc:654] T cd597c319af645909b8f3d1212fcb702 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:17.172322 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1122) took 3392 ms. Trace:
I20251029 02:52:17.172395 1176 rpcz_store.cc:276] 1029 02:52:13.779547 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:13.780785 (+ 1238us) service_pool.cc:225] Handling call
1029 02:52:17.082212 (+3301427us) ts_tablet_manager.cc:1151] Deleting tablet 59eee86f249e4d6a9d09d30276dcd659
1029 02:52:17.082223 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:17.090000 (+ 7777us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:17.104730 (+ 14730us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:17.172306 (+ 67576us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3280022}
I20251029 02:52:17.172647 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 1006) took 4803 ms. Trace:
I20251029 02:52:17.172741 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1125) took 3356 ms. Trace:
I20251029 02:52:17.172721 1175 rpcz_store.cc:276] 1029 02:52:12.368868 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:12.368924 (+ 56us) service_pool.cc:225] Handling call
1029 02:52:17.172523 (+4803599us) ts_tablet_manager.cc:1151] Deleting tablet 41cfa2d722034618a05de717b28a778d
1029 02:52:17.172533 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:17.172627 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4803393}
I20251029 02:52:17.172797 1176 rpcz_store.cc:276] 1029 02:52:13.815872 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:13.815937 (+ 65us) service_pool.cc:225] Handling call
1029 02:52:17.172627 (+3356690us) ts_tablet_manager.cc:1151] Deleting tablet eaf4018c0d624b1cb8c6528eea7b689e
1029 02:52:17.172634 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:17.172725 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3337078}
I20251029 02:52:17.173079 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1126) took 3343 ms. Trace:
I20251029 02:52:17.173187 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 1007) took 4796 ms. Trace:
I20251029 02:52:17.173144 1176 rpcz_store.cc:276] 1029 02:52:13.829467 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:13.855520 (+ 26053us) service_pool.cc:225] Handling call
1029 02:52:17.172989 (+3317469us) ts_tablet_manager.cc:1151] Deleting tablet e40d4910365f45e9a45d9b3bc5037c95
1029 02:52:17.172997 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:17.173066 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3317213}
I20251029 02:52:17.173249 1175 rpcz_store.cc:276] 1029 02:52:12.376924 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:12.376982 (+ 58us) service_pool.cc:225] Handling call
1029 02:52:17.173074 (+4796092us) ts_tablet_manager.cc:1151] Deleting tablet 99d60ef5325e45059026596a756448de
1029 02:52:17.173082 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:17.173171 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4795905}
W20251029 02:52:17.173599 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 41cfa2d722034618a05de717b28a778d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 41cfa2d722034618a05de717b28a778d already in progress: creating tablet
I20251029 02:52:17.173667 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 1008) took 4740 ms. Trace:
I20251029 02:52:17.173728 1175 rpcz_store.cc:276] 1029 02:52:12.433228 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:12.433284 (+ 56us) service_pool.cc:225] Handling call
1029 02:52:17.173556 (+4740272us) ts_tablet_manager.cc:1151] Deleting tablet 49aaa2d8986a44df979edc1c42612189
1029 02:52:17.173566 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:17.173652 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4740089}
W20251029 02:52:17.173864 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 99d60ef5325e45059026596a756448de because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 99d60ef5325e45059026596a756448de already in progress: creating tablet
I20251029 02:52:17.173954 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1128) took 3254 ms. Trace:
I20251029 02:52:17.174055 1175 tablet_replica.cc:333] T f281191e7c6143f983f11d87f1c9b64e P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
W20251029 02:52:17.174068 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 49aaa2d8986a44df979edc1c42612189 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 49aaa2d8986a44df979edc1c42612189 already in progress: creating tablet
I20251029 02:52:17.174021 1176 rpcz_store.cc:276] 1029 02:52:13.919251 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:13.919297 (+ 46us) service_pool.cc:225] Handling call
1029 02:52:17.173346 (+3254049us) ts_tablet_manager.cc:1151] Deleting tablet 3e9a076e086141ef8ed59d4c8633f961
1029 02:52:17.173834 (+ 488us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:17.173936 (+ 102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3253884}
I20251029 02:52:17.174242 1175 raft_consensus.cc:2243] T f281191e7c6143f983f11d87f1c9b64e P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:17.174401 1175 raft_consensus.cc:2272] T f281191e7c6143f983f11d87f1c9b64e P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:17.175040 1175 ts_tablet_manager.cc:1916] T f281191e7c6143f983f11d87f1c9b64e P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:17.176183 1033 tablet_service.cc:1552] Processing DeleteTablet for tablet 20a762344c5f46f88481c4ac7ad6e43b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:08 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:17.176584 1033 tablet_service.cc:1552] Processing DeleteTablet for tablet 7bb85a12f573494cbb3118d1947eda27 with delete_type TABLET_DATA_DELETED (replaced by tablet a9f78f61ee2841dea71bbbfceacc3ea6) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:17.177484 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 59eee86f249e4d6a9d09d30276dcd659 (table default.Table_8f1e631e578a4010927f77c77cc29cb0 [id=edac8dce6fa747bda675efdf71adb1d6]) successfully deleted
W20251029 02:52:17.177909 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet eaf4018c0d624b1cb8c6528eea7b689e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eaf4018c0d624b1cb8c6528eea7b689e already in progress: creating tablet
W20251029 02:52:17.178300 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet e40d4910365f45e9a45d9b3bc5037c95 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e40d4910365f45e9a45d9b3bc5037c95 already in progress: creating tablet
I20251029 02:52:17.178467 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1129) took 3258 ms. Trace:
I20251029 02:52:17.178550 1176 rpcz_store.cc:276] 1029 02:52:13.919789 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:13.919835 (+ 46us) service_pool.cc:225] Handling call
1029 02:52:17.174200 (+3254365us) ts_tablet_manager.cc:1151] Deleting tablet 2cea2f2fe68545db804a732d0c01a876
1029 02:52:17.178354 (+ 4154us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:17.178450 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3254211}
W20251029 02:52:17.178875 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 3e9a076e086141ef8ed59d4c8633f961 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3e9a076e086141ef8ed59d4c8633f961 already in progress: creating tablet
I20251029 02:52:17.179284 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1131) took 3227 ms. Trace:
I20251029 02:52:17.179366 1176 rpcz_store.cc:276] 1029 02:52:13.952037 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:13.952091 (+ 54us) service_pool.cc:225] Handling call
1029 02:52:17.178766 (+3226675us) ts_tablet_manager.cc:1151] Deleting tablet 786d9148fb1d417a8c2c8b8c53665899
1029 02:52:17.179182 (+ 416us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:17.179270 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3226484}
I20251029 02:52:17.179597 1176 tablet_replica.cc:333] T 67651306f3b44e7eb14dcee9a2007a81 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:17.179773 1176 raft_consensus.cc:2243] T 67651306f3b44e7eb14dcee9a2007a81 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:17.179932 1176 raft_consensus.cc:2272] T 67651306f3b44e7eb14dcee9a2007a81 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251029 02:52:17.180218 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 2cea2f2fe68545db804a732d0c01a876 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2cea2f2fe68545db804a732d0c01a876 already in progress: creating tablet
I20251029 02:52:17.180465 1176 ts_tablet_manager.cc:1916] T 67651306f3b44e7eb14dcee9a2007a81 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 02:52:17.180557 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 786d9148fb1d417a8c2c8b8c53665899 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 786d9148fb1d417a8c2c8b8c53665899 already in progress: creating tablet
W20251029 02:52:17.184504 849 leader_election.cc:343] T 23b9b2ea814645eba63e5f6b12cfb966 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:17.184710 2655 raft_consensus.cc:2804] T 23b9b2ea814645eba63e5f6b12cfb966 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:17.184773 2655 raft_consensus.cc:493] T 23b9b2ea814645eba63e5f6b12cfb966 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:17.184827 2655 raft_consensus.cc:3060] T 23b9b2ea814645eba63e5f6b12cfb966 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:17.185894 1166 tablet_bootstrap.cc:492] T cd597c319af645909b8f3d1212fcb702 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:17.186095 1166 ts_tablet_manager.cc:1403] T cd597c319af645909b8f3d1212fcb702 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.000s
I20251029 02:52:17.186908 2833 sys_catalog.cc:263] Verifying existing consensus state
I20251029 02:52:17.186869 1166 raft_consensus.cc:359] T cd597c319af645909b8f3d1212fcb702 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:17.187674 1166 raft_consensus.cc:385] T cd597c319af645909b8f3d1212fcb702 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:17.187785 1166 raft_consensus.cc:740] T cd597c319af645909b8f3d1212fcb702 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:17.188032 1166 consensus_queue.cc:260] T cd597c319af645909b8f3d1212fcb702 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:17.188555 1166 ts_tablet_manager.cc:1434] T cd597c319af645909b8f3d1212fcb702 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251029 02:52:17.188867 1166 tablet_bootstrap.cc:492] T 589a521f5cc24e6289c37b9cf256bce5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:17.190443 1166 tablet_bootstrap.cc:654] T 589a521f5cc24e6289c37b9cf256bce5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
W20251029 02:52:17.190717 712 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:45003, user_credentials={real_user=slave}} blocked reactor thread for 46211.9us
W20251029 02:52:17.191097 712 leader_election.cc:343] T cc802bda2f6a4e439b605f2cafa37967 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:17.191319 712 leader_election.cc:304] T cc802bda2f6a4e439b605f2cafa37967 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
W20251029 02:52:17.192598 712 leader_election.cc:343] T 506f8c112a3b43e1a3d6bbafb8cb81e8 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:17.192869 712 leader_election.cc:304] T 506f8c112a3b43e1a3d6bbafb8cb81e8 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:17.192790 2722 raft_consensus.cc:2749] T cc802bda2f6a4e439b605f2cafa37967 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:17.193538 2722 raft_consensus.cc:2749] T 506f8c112a3b43e1a3d6bbafb8cb81e8 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:17.197870 1172 tablet_bootstrap.cc:492] T abf18e06df9b482a8b25c078b9495272 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:17.198086 1172 ts_tablet_manager.cc:1403] T abf18e06df9b482a8b25c078b9495272 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.077s user 0.004s sys 0.000s
I20251029 02:52:17.199092 1172 raft_consensus.cc:359] T abf18e06df9b482a8b25c078b9495272 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
W20251029 02:52:17.199671 849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:45003, user_credentials={real_user=slave}} blocked reactor thread for 56885.3us
W20251029 02:52:17.204190 849 consensus_peers.cc:597] T d041e87ceb11441983d0e38b7929c3a4 P c756528642a543eeacd93cef3e335a7d -> Peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Couldn't send request to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d041e87ceb11441983d0e38b7929c3a4. This is attempt 1: this message will repeat every 5th retry.
W20251029 02:52:17.205988 849 consensus_peers.cc:597] T 44ae2b07edca44afbe1b1fb3a5b1115d P c756528642a543eeacd93cef3e335a7d -> Peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Couldn't send request to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 44ae2b07edca44afbe1b1fb3a5b1115d. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:17.206025 1167 tablet_bootstrap.cc:492] T f93b0c39d7174dabbe018b1546dbc198 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:17.206548 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_a3ea3760372c4e8f88ef9506f9426e28"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:17.207083 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a3ea3760372c4e8f88ef9506f9426e28 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:17.210762 2655 raft_consensus.cc:515] T 23b9b2ea814645eba63e5f6b12cfb966 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:17.211410 2655 leader_election.cc:290] T 23b9b2ea814645eba63e5f6b12cfb966 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:17.212085 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "23b9b2ea814645eba63e5f6b12cfb966" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:52:17.213292 1167 tablet_bootstrap.cc:654] T f93b0c39d7174dabbe018b1546dbc198 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
W20251029 02:52:17.213605 849 leader_election.cc:343] T 23b9b2ea814645eba63e5f6b12cfb966 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:17.215540 522 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_f88524c2f85e4721931d212083db66ba" } new_table_name: "default.Table_2821b383d30949a48daa35a61188ab02" modify_external_catalogs: true
W20251029 02:52:17.224293 522 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e9564575e0914392954a92666dd88333 consensus state has no leader
I20251029 02:52:17.224773 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_015f00d002fd4196b1f126905dcaec9f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:17.225708 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_015f00d002fd4196b1f126905dcaec9f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:17.229617 1174 ts_tablet_manager.cc:1929] T 30d03e1e42d34242bd354f9e9ec4f127 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:17.235405 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_77922304a486493b8d07d408c24815ef"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:17.235922 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_77922304a486493b8d07d408c24815ef in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:17.236466 1176 ts_tablet_manager.cc:1929] T 67651306f3b44e7eb14dcee9a2007a81 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:17.236970 1176 log.cc:1199] T 67651306f3b44e7eb14dcee9a2007a81 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/67651306f3b44e7eb14dcee9a2007a81
I20251029 02:52:17.239861 1176 ts_tablet_manager.cc:1950] T 67651306f3b44e7eb14dcee9a2007a81 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:17.241204 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1132) took 3288 ms. Trace:
I20251029 02:52:17.242424 1176 rpcz_store.cc:276] 1029 02:52:13.952479 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:13.952523 (+ 44us) service_pool.cc:225] Handling call
1029 02:52:17.179562 (+3227039us) ts_tablet_manager.cc:1151] Deleting tablet 67651306f3b44e7eb14dcee9a2007a81
1029 02:52:17.179569 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:17.191361 (+ 11792us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:17.236355 (+ 44994us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:17.241189 (+ 4834us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3226896}
I20251029 02:52:17.242864 1176 tablet_replica.cc:333] T 58dde9397b434b089ade9968f369006a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:17.243109 1176 raft_consensus.cc:2243] T 58dde9397b434b089ade9968f369006a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:17.243304 1176 raft_consensus.cc:2272] T 58dde9397b434b089ade9968f369006a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:17.243844 1176 ts_tablet_manager.cc:1916] T 58dde9397b434b089ade9968f369006a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:17.244907 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 67651306f3b44e7eb14dcee9a2007a81 (table default.Table_ff796c36eb2c4bb1809f39e1cb68c223 [id=95bbd9adfe994fecab10c9f23c0bedb0]) successfully deleted
I20251029 02:52:17.245400 2717 raft_consensus.cc:493] T 70b008011d5547bbaf3f45685394139f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:17.245512 2717 raft_consensus.cc:515] T 70b008011d5547bbaf3f45685394139f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:17.246093 2717 leader_election.cc:290] T 70b008011d5547bbaf3f45685394139f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:17.244156 2842 tablet.cc:1722] T ae4231daa2594258aa51419c97a2f50a P c756528642a543eeacd93cef3e335a7d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20251029 02:52:17.245255 849 consensus_peers.cc:597] T c32c28b0e74a4af3a0e69bbf24ab1a1f P c756528642a543eeacd93cef3e335a7d -> Peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Couldn't send request to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c32c28b0e74a4af3a0e69bbf24ab1a1f. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:17.246948 919 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "70b008011d5547bbaf3f45685394139f" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:17.251989 990 leader_election.cc:343] T 70b008011d5547bbaf3f45685394139f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:17.252007 522 master_service.cc:954] ReplaceTablet: received request to replace tablet d557a4750fee4c2b92eaa834619a678c from {username='slave'} at 127.0.0.1:35314
I20251029 02:52:17.199354 1172 raft_consensus.cc:385] T abf18e06df9b482a8b25c078b9495272 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:17.252332 1172 raft_consensus.cc:740] T abf18e06df9b482a8b25c078b9495272 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:17.211972 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "23b9b2ea814645eba63e5f6b12cfb966" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:17.252686 772 raft_consensus.cc:3060] T 23b9b2ea814645eba63e5f6b12cfb966 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:17.252619 1172 consensus_queue.cc:260] T abf18e06df9b482a8b25c078b9495272 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:17.251050 1174 log.cc:1199] T 30d03e1e42d34242bd354f9e9ec4f127 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/30d03e1e42d34242bd354f9e9ec4f127
I20251029 02:52:17.254184 1174 ts_tablet_manager.cc:1950] T 30d03e1e42d34242bd354f9e9ec4f127 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:17.212039 2853 raft_consensus.cc:493] T 786d9148fb1d417a8c2c8b8c53665899 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:17.254828 2853 raft_consensus.cc:515] T 786d9148fb1d417a8c2c8b8c53665899 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:17.254949 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 954) took 6614 ms. Trace:
I20251029 02:52:17.255023 1174 rpcz_store.cc:276] 1029 02:52:10.640357 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:10.640961 (+ 604us) service_pool.cc:225] Handling call
1029 02:52:17.024671 (+6383710us) ts_tablet_manager.cc:1151] Deleting tablet 30d03e1e42d34242bd354f9e9ec4f127
1029 02:52:17.024683 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:17.131227 (+106544us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:17.229503 (+ 98276us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:17.254930 (+ 25427us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6370655}
I20251029 02:52:17.255371 1174 tablet_replica.cc:333] T 59d03a190b784ab18a3e63a572675891 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:17.255489 2853 leader_election.cc:290] T 786d9148fb1d417a8c2c8b8c53665899 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:52:17.255568 1174 raft_consensus.cc:2243] T 59d03a190b784ab18a3e63a572675891 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:17.220268 766 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ae4231daa2594258aa51419c97a2f50a" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:17.255880 1174 raft_consensus.cc:2272] T 59d03a190b784ab18a3e63a572675891 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:17.256485 1172 ts_tablet_manager.cc:1434] T abf18e06df9b482a8b25c078b9495272 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.058s user 0.000s sys 0.002s
I20251029 02:52:17.222395 1175 ts_tablet_manager.cc:1929] T f281191e7c6143f983f11d87f1c9b64e P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:17.257071 1175 log.cc:1199] T f281191e7c6143f983f11d87f1c9b64e P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/f281191e7c6143f983f11d87f1c9b64e
I20251029 02:52:17.257427 1175 ts_tablet_manager.cc:1950] T f281191e7c6143f983f11d87f1c9b64e P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:17.245199 2763 tablet.cc:1722] T ae4231daa2594258aa51419c97a2f50a P b495a49349324f7d81908f2da700f0d9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 02:52:17.258653 919 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "786d9148fb1d417a8c2c8b8c53665899" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:17.260257 711 leader_election.cc:343] T 786d9148fb1d417a8c2c8b8c53665899 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:17.264571 989 leader_election.cc:304] T ae4231daa2594258aa51419c97a2f50a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5; no voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:17.265341 1052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "786d9148fb1d417a8c2c8b8c53665899" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:17.265790 2717 raft_consensus.cc:2749] T ae4231daa2594258aa51419c97a2f50a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:17.269390 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 30d03e1e42d34242bd354f9e9ec4f127 (table default.Table_435f2dfe42134760abc9e8880fa19f32 [id=c6e1c357fbfc49be972748da564cc9a9]) successfully deleted
W20251029 02:52:17.269963 712 leader_election.cc:343] T 786d9148fb1d417a8c2c8b8c53665899 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:17.270105 712 leader_election.cc:304] T 786d9148fb1d417a8c2c8b8c53665899 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:17.270610 2853 raft_consensus.cc:2749] T 786d9148fb1d417a8c2c8b8c53665899 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:17.272038 772 raft_consensus.cc:2468] T 23b9b2ea814645eba63e5f6b12cfb966 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:17.272430 847 leader_election.cc:304] T 23b9b2ea814645eba63e5f6b12cfb966 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5
I20251029 02:52:17.272996 2655 raft_consensus.cc:2804] T 23b9b2ea814645eba63e5f6b12cfb966 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:17.273129 2655 raft_consensus.cc:697] T 23b9b2ea814645eba63e5f6b12cfb966 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:52:17.273361 2655 consensus_queue.cc:237] T 23b9b2ea814645eba63e5f6b12cfb966 P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:17.273363 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "70b008011d5547bbaf3f45685394139f" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:17.273610 776 raft_consensus.cc:2468] T 70b008011d5547bbaf3f45685394139f P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 0.
I20251029 02:52:17.273989 989 leader_election.cc:304] T 70b008011d5547bbaf3f45685394139f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters: c756528642a543eeacd93cef3e335a7d
I20251029 02:52:17.274415 2717 raft_consensus.cc:2804] T 70b008011d5547bbaf3f45685394139f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:17.274488 2717 raft_consensus.cc:493] T 70b008011d5547bbaf3f45685394139f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:17.274569 2717 raft_consensus.cc:3060] T 70b008011d5547bbaf3f45685394139f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:17.275358 2201 consensus_queue.cc:1048] T c081d2eacfac4ca8a0048e2a1c4f8f83 P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:17.277486 2717 raft_consensus.cc:515] T 70b008011d5547bbaf3f45685394139f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:17.278784 2717 leader_election.cc:290] T 70b008011d5547bbaf3f45685394139f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Requested vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:17.279359 2717 raft_consensus.cc:493] T 84993b2f63e7421290a161dde4bb0607 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:17.279417 767 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "70b008011d5547bbaf3f45685394139f" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:17.279529 2717 raft_consensus.cc:515] T 84993b2f63e7421290a161dde4bb0607 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:17.279606 767 raft_consensus.cc:3060] T 70b008011d5547bbaf3f45685394139f P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:17.280570 919 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "70b008011d5547bbaf3f45685394139f" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
W20251029 02:52:17.281138 990 leader_election.cc:343] T 70b008011d5547bbaf3f45685394139f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:17.282459 767 raft_consensus.cc:2468] T 70b008011d5547bbaf3f45685394139f P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 1.
I20251029 02:52:17.283077 989 leader_election.cc:304] T 70b008011d5547bbaf3f45685394139f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters: c756528642a543eeacd93cef3e335a7d
I20251029 02:52:17.283385 2822 raft_consensus.cc:2804] T 70b008011d5547bbaf3f45685394139f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:17.283481 2822 raft_consensus.cc:697] T 70b008011d5547bbaf3f45685394139f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Becoming Leader. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Running, Role: LEADER
I20251029 02:52:17.283690 2822 consensus_queue.cc:237] T 70b008011d5547bbaf3f45685394139f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:17.284405 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "84993b2f63e7421290a161dde4bb0607" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:17.284852 2717 leader_election.cc:290] T 84993b2f63e7421290a161dde4bb0607 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:17.285267 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 1009) took 4679 ms. Trace:
I20251029 02:52:17.285372 1175 rpcz_store.cc:276] 1029 02:52:12.605944 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:12.606209 (+ 265us) service_pool.cc:225] Handling call
1029 02:52:17.174015 (+4567806us) ts_tablet_manager.cc:1151] Deleting tablet f281191e7c6143f983f11d87f1c9b64e
1029 02:52:17.174023 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:17.195287 (+ 21264us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:17.222267 (+ 26980us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:17.285229 (+ 62962us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4567597}
I20251029 02:52:17.285730 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 1010) took 4637 ms. Trace:
I20251029 02:52:17.285804 1175 rpcz_store.cc:276] 1029 02:52:12.647898 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:12.647963 (+ 65us) service_pool.cc:225] Handling call
1029 02:52:17.285618 (+4637655us) ts_tablet_manager.cc:1151] Deleting tablet 9280503662894544a1f227a62a9b8a01
1029 02:52:17.285625 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:17.285704 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4637457}
I20251029 02:52:17.286079 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 1011) took 4545 ms. Trace:
I20251029 02:52:17.286144 1175 rpcz_store.cc:276] 1029 02:52:12.740810 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:12.741149 (+ 339us) service_pool.cc:225] Handling call
1029 02:52:17.285989 (+4544840us) ts_tablet_manager.cc:1151] Deleting tablet fe5aff0f61a44b91bfb37b42457a91f0
1029 02:52:17.285996 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:17.286054 (+ 58us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4544658}
I20251029 02:52:17.286396 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 1012) took 4278 ms. Trace:
I20251029 02:52:17.286463 1175 rpcz_store.cc:276] 1029 02:52:13.008184 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:13.008257 (+ 73us) service_pool.cc:225] Handling call
1029 02:52:17.286304 (+4278047us) ts_tablet_manager.cc:1151] Deleting tablet ca71c4638a6b419b876e22af7e6cf00a
1029 02:52:17.286311 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:17.286367 (+ 56us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4277830}
I20251029 02:52:17.286732 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 1013) took 4196 ms. Trace:
I20251029 02:52:17.286763 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "84993b2f63e7421290a161dde4bb0607" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:17.286810 1175 rpcz_store.cc:276] 1029 02:52:13.089902 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:13.090120 (+ 218us) service_pool.cc:225] Handling call
1029 02:52:17.286651 (+4196531us) ts_tablet_manager.cc:1151] Deleting tablet 71b4b91071a54d6aa66160e7955bc915
1029 02:52:17.286659 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:17.286699 (+ 40us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4196349}
I20251029 02:52:17.287031 776 raft_consensus.cc:2393] T 84993b2f63e7421290a161dde4bb0607 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in current term 1: Already voted for candidate c756528642a543eeacd93cef3e335a7d in this term.
I20251029 02:52:17.287056 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 1014) took 4197 ms. Trace:
I20251029 02:52:17.287133 1175 rpcz_store.cc:276] 1029 02:52:13.090072 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:13.090345 (+ 273us) service_pool.cc:225] Handling call
1029 02:52:17.286978 (+4196633us) ts_tablet_manager.cc:1151] Deleting tablet 30d03e1e42d34242bd354f9e9ec4f127
1029 02:52:17.286985 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:17.287027 (+ 42us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4196498}
I20251029 02:52:17.287356 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 1015) took 4196 ms. Trace:
I20251029 02:52:17.287436 989 leader_election.cc:304] T 84993b2f63e7421290a161dde4bb0607 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5; no voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:17.287438 1175 rpcz_store.cc:276] 1029 02:52:13.090774 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:13.090826 (+ 52us) service_pool.cc:225] Handling call
1029 02:52:17.287282 (+4196456us) ts_tablet_manager.cc:1151] Deleting tablet dabb8d760aae4aa09eb83cdb586c9788
1029 02:52:17.287289 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:17.287325 (+ 36us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4196286}
I20251029 02:52:17.287691 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 1029) took 3870 ms. Trace:
I20251029 02:52:17.287765 1175 rpcz_store.cc:276] 1029 02:52:13.417105 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:13.421503 (+ 4398us) service_pool.cc:225] Handling call
1029 02:52:17.287592 (+3866089us) ts_tablet_manager.cc:1151] Deleting tablet 21c73cb2f1084827b581fb09cf7239a1
1029 02:52:17.287599 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:17.287656 (+ 57us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3865885}
I20251029 02:52:17.287912 2717 raft_consensus.cc:3060] T 84993b2f63e7421290a161dde4bb0607 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:17.288067 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 1031) took 3870 ms. Trace:
I20251029 02:52:17.288147 1175 rpcz_store.cc:276] 1029 02:52:13.417276 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:13.421714 (+ 4438us) service_pool.cc:225] Handling call
1029 02:52:17.287945 (+3866231us) ts_tablet_manager.cc:1151] Deleting tablet 416e7c48daa849cd9635656529488922
1029 02:52:17.287952 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:17.288032 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3866061}
I20251029 02:52:17.288403 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 1030) took 3871 ms. Trace:
I20251029 02:52:17.288491 1175 rpcz_store.cc:276] 1029 02:52:13.417196 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:13.421917 (+ 4721us) service_pool.cc:225] Handling call
1029 02:52:17.288310 (+3866393us) ts_tablet_manager.cc:1151] Deleting tablet eb3c20822d684bea89eb358251e21e33
1029 02:52:17.288317 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:17.288371 (+ 54us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3866226}
I20251029 02:52:17.288756 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 1028) took 3871 ms. Trace:
I20251029 02:52:17.288827 1175 rpcz_store.cc:276] 1029 02:52:13.417003 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:13.422572 (+ 5569us) service_pool.cc:225] Handling call
1029 02:52:17.288653 (+3866081us) ts_tablet_manager.cc:1151] Deleting tablet 15cadd681ceb4daba178a9cc2b7aec08
1029 02:52:17.288660 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:17.288719 (+ 59us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3865903}
I20251029 02:52:17.289119 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 1032) took 3871 ms. Trace:
I20251029 02:52:17.289183 1175 rpcz_store.cc:276] 1029 02:52:13.417351 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:13.425360 (+ 8009us) service_pool.cc:225] Handling call
1029 02:52:17.289031 (+3863671us) ts_tablet_manager.cc:1151] Deleting tablet 31ca9012ddf6401ba78888a0c82b4ef2
1029 02:52:17.289038 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:17.289092 (+ 54us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3857331}
I20251029 02:52:17.289386 1175 tablet_replica.cc:333] T 82289e0f823c4d43bfd8c4e878572b2b P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:17.289567 1175 raft_consensus.cc:2243] T 82289e0f823c4d43bfd8c4e878572b2b P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:17.289721 1175 raft_consensus.cc:2272] T 82289e0f823c4d43bfd8c4e878572b2b P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:17.290264 1175 ts_tablet_manager.cc:1916] T 82289e0f823c4d43bfd8c4e878572b2b P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 02:52:17.291456 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 9280503662894544a1f227a62a9b8a01 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9280503662894544a1f227a62a9b8a01 already in progress: creating tablet
W20251029 02:52:17.291627 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet fe5aff0f61a44b91bfb37b42457a91f0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fe5aff0f61a44b91bfb37b42457a91f0 already in progress: creating tablet
W20251029 02:52:17.291749 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet ca71c4638a6b419b876e22af7e6cf00a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ca71c4638a6b419b876e22af7e6cf00a already in progress: creating tablet
W20251029 02:52:17.291859 511 catalog_manager.cc:4952] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 71b4b91071a54d6aa66160e7955bc915 because the tablet was not found. No further retry: Not found: Tablet not found: 71b4b91071a54d6aa66160e7955bc915
W20251029 02:52:17.291977 511 catalog_manager.cc:4952] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 30d03e1e42d34242bd354f9e9ec4f127 because the tablet was not found. No further retry: Not found: Tablet not found: 30d03e1e42d34242bd354f9e9ec4f127
W20251029 02:52:17.292086 511 catalog_manager.cc:4952] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet dabb8d760aae4aa09eb83cdb586c9788 because the tablet was not found. No further retry: Not found: Tablet not found: dabb8d760aae4aa09eb83cdb586c9788
W20251029 02:52:17.292196 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 21c73cb2f1084827b581fb09cf7239a1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 21c73cb2f1084827b581fb09cf7239a1 already in progress: creating tablet
W20251029 02:52:17.292327 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 416e7c48daa849cd9635656529488922 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 416e7c48daa849cd9635656529488922 already in progress: creating tablet
W20251029 02:52:17.292462 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet eb3c20822d684bea89eb358251e21e33 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eb3c20822d684bea89eb358251e21e33 already in progress: creating tablet
W20251029 02:52:17.292593 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 15cadd681ceb4daba178a9cc2b7aec08 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 15cadd681ceb4daba178a9cc2b7aec08 already in progress: creating tablet
W20251029 02:52:17.292738 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 31ca9012ddf6401ba78888a0c82b4ef2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 31ca9012ddf6401ba78888a0c82b4ef2 already in progress: creating tablet
I20251029 02:52:17.293671 1172 tablet_bootstrap.cc:492] T 2b278db5b2b04344b864797055a5f665 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:17.294664 2717 raft_consensus.cc:2749] T 84993b2f63e7421290a161dde4bb0607 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:17.297837 1167 tablet_bootstrap.cc:492] T f93b0c39d7174dabbe018b1546dbc198 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:17.297981 1167 ts_tablet_manager.cc:1403] T f93b0c39d7174dabbe018b1546dbc198 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.092s user 0.002s sys 0.006s
I20251029 02:52:17.298758 1167 raft_consensus.cc:359] T f93b0c39d7174dabbe018b1546dbc198 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:17.305290 1167 raft_consensus.cc:385] T f93b0c39d7174dabbe018b1546dbc198 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:17.305424 1167 raft_consensus.cc:740] T f93b0c39d7174dabbe018b1546dbc198 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:17.305699 1167 consensus_queue.cc:260] T f93b0c39d7174dabbe018b1546dbc198 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:17.306389 1167 ts_tablet_manager.cc:1434] T f93b0c39d7174dabbe018b1546dbc198 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20251029 02:52:17.306744 1167 tablet_bootstrap.cc:492] T 628a7d0419cb434f8171faf500d585d8 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:17.309053 2747 consensus_queue.cc:1048] T 84993b2f63e7421290a161dde4bb0607 P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:17.313000 2747 consensus_queue.cc:1048] T 84993b2f63e7421290a161dde4bb0607 P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:17.319015 1166 tablet_bootstrap.cc:492] T 589a521f5cc24e6289c37b9cf256bce5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:17.319267 1166 ts_tablet_manager.cc:1403] T 589a521f5cc24e6289c37b9cf256bce5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.130s user 0.003s sys 0.004s
I20251029 02:52:17.311095 2853 raft_consensus.cc:493] T 7816556521144c928d7a02694e22e79f P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:17.320200 1166 raft_consensus.cc:359] T 589a521f5cc24e6289c37b9cf256bce5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:17.320343 1166 raft_consensus.cc:385] T 589a521f5cc24e6289c37b9cf256bce5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:17.320242 2853 raft_consensus.cc:515] T 7816556521144c928d7a02694e22e79f P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:17.320381 1166 raft_consensus.cc:740] T 589a521f5cc24e6289c37b9cf256bce5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:17.320868 2853 leader_election.cc:290] T 7816556521144c928d7a02694e22e79f P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:52:17.323002 1059 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7816556521144c928d7a02694e22e79f" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:17.323119 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7816556521144c928d7a02694e22e79f" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:17.323479 2853 raft_consensus.cc:493] T 688af6580c4d4637a39f817b4e324618 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20251029 02:52:17.323592 712 leader_election.cc:343] T 7816556521144c928d7a02694e22e79f P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:17.323632 2853 raft_consensus.cc:515] T 688af6580c4d4637a39f817b4e324618 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
W20251029 02:52:17.323773 711 leader_election.cc:343] T 7816556521144c928d7a02694e22e79f P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:17.323870 711 leader_election.cc:304] T 7816556521144c928d7a02694e22e79f P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:17.324193 2853 leader_election.cc:290] T 688af6580c4d4637a39f817b4e324618 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:52:17.324226 2857 raft_consensus.cc:2749] T 7816556521144c928d7a02694e22e79f P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:17.324800 1059 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "688af6580c4d4637a39f817b4e324618" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:17.325147 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "688af6580c4d4637a39f817b4e324618" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:17.325316 712 leader_election.cc:343] T 688af6580c4d4637a39f817b4e324618 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
W20251029 02:52:17.325670 711 leader_election.cc:343] T 688af6580c4d4637a39f817b4e324618 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:17.325774 711 leader_election.cc:304] T 688af6580c4d4637a39f817b4e324618 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:17.326099 2857 raft_consensus.cc:2749] T 688af6580c4d4637a39f817b4e324618 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:17.328739 2747 raft_consensus.cc:493] T fcb3db8399aa4cf188b07dda220f7f5b P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:17.328866 2747 raft_consensus.cc:515] T fcb3db8399aa4cf188b07dda220f7f5b P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:17.329427 2747 leader_election.cc:290] T fcb3db8399aa4cf188b07dda220f7f5b P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:17.329972 1059 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fcb3db8399aa4cf188b07dda220f7f5b" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:17.330417 767 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fcb3db8399aa4cf188b07dda220f7f5b" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
W20251029 02:52:17.330622 849 leader_election.cc:343] T fcb3db8399aa4cf188b07dda220f7f5b P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:17.330844 767 raft_consensus.cc:2468] T fcb3db8399aa4cf188b07dda220f7f5b P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:17.331043 1174 ts_tablet_manager.cc:1916] T 59d03a190b784ab18a3e63a572675891 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:17.332209 847 leader_election.cc:304] T fcb3db8399aa4cf188b07dda220f7f5b P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5
I20251029 02:52:17.332563 2426 raft_consensus.cc:2804] T fcb3db8399aa4cf188b07dda220f7f5b P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:17.332640 2426 raft_consensus.cc:493] T fcb3db8399aa4cf188b07dda220f7f5b P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:17.332701 2426 raft_consensus.cc:3060] T fcb3db8399aa4cf188b07dda220f7f5b P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:17.333984 1167 tablet_bootstrap.cc:654] T 628a7d0419cb434f8171faf500d585d8 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:17.335417 2426 raft_consensus.cc:515] T fcb3db8399aa4cf188b07dda220f7f5b P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:17.335963 2426 leader_election.cc:290] T fcb3db8399aa4cf188b07dda220f7f5b P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
W20251029 02:52:17.337186 849 consensus_peers.cc:597] T 58dde9397b434b089ade9968f369006a P c756528642a543eeacd93cef3e335a7d -> Peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Couldn't send request to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:17.337393 1059 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fcb3db8399aa4cf188b07dda220f7f5b" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
W20251029 02:52:17.337822 849 leader_election.cc:343] T fcb3db8399aa4cf188b07dda220f7f5b P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:17.342645 776 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "fcb3db8399aa4cf188b07dda220f7f5b" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:17.342876 776 raft_consensus.cc:3060] T fcb3db8399aa4cf188b07dda220f7f5b P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:17.343463 1166 consensus_queue.cc:260] T 589a521f5cc24e6289c37b9cf256bce5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:17.345727 1166 ts_tablet_manager.cc:1434] T 589a521f5cc24e6289c37b9cf256bce5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.026s user 0.001s sys 0.001s
I20251029 02:52:17.346042 1166 tablet_bootstrap.cc:492] T 748777c6cd6e4f49ab49912b719e48bd P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:17.353322 1167 tablet_bootstrap.cc:492] T 628a7d0419cb434f8171faf500d585d8 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:17.353531 1167 ts_tablet_manager.cc:1403] T 628a7d0419cb434f8171faf500d585d8 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.047s user 0.003s sys 0.000s
I20251029 02:52:17.354353 1167 raft_consensus.cc:359] T 628a7d0419cb434f8171faf500d585d8 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:17.355446 1172 tablet_bootstrap.cc:654] T 2b278db5b2b04344b864797055a5f665 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
W20251029 02:52:17.367247 849 consensus_peers.cc:597] T 67651306f3b44e7eb14dcee9a2007a81 P c756528642a543eeacd93cef3e335a7d -> Peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Couldn't send request to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 67651306f3b44e7eb14dcee9a2007a81. This is attempt 1: this message will repeat every 5th retry.
W20251029 02:52:17.368695 849 consensus_peers.cc:597] T c081d2eacfac4ca8a0048e2a1c4f8f83 P c756528642a543eeacd93cef3e335a7d -> Peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Couldn't send request to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:17.369246 1167 raft_consensus.cc:385] T 628a7d0419cb434f8171faf500d585d8 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:17.369376 1167 raft_consensus.cc:740] T 628a7d0419cb434f8171faf500d585d8 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:17.372696 1167 consensus_queue.cc:260] T 628a7d0419cb434f8171faf500d585d8 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:17.373057 1166 tablet_bootstrap.cc:654] T 748777c6cd6e4f49ab49912b719e48bd P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:17.373476 1167 ts_tablet_manager.cc:1434] T 628a7d0419cb434f8171faf500d585d8 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.020s user 0.001s sys 0.000s
I20251029 02:52:17.376395 2857 raft_consensus.cc:493] T 24f07b1efac64b099b0b5ed467463749 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:17.376544 2857 raft_consensus.cc:515] T 24f07b1efac64b099b0b5ed467463749 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:17.377148 2857 leader_election.cc:290] T 24f07b1efac64b099b0b5ed467463749 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:17.378459 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24f07b1efac64b099b0b5ed467463749" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:17.380541 847 consensus_peers.cc:597] T c32c28b0e74a4af3a0e69bbf24ab1a1f P c756528642a543eeacd93cef3e335a7d -> Peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Couldn't send request to peer b495a49349324f7d81908f2da700f0d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c32c28b0e74a4af3a0e69bbf24ab1a1f. This is attempt 1: this message will repeat every 5th retry.
W20251029 02:52:17.382412 711 leader_election.cc:343] T 24f07b1efac64b099b0b5ed467463749 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:17.382707 1167 tablet_bootstrap.cc:492] T 7816556521144c928d7a02694e22e79f P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:17.384439 1052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24f07b1efac64b099b0b5ed467463749" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:17.384581 2842 tablet.cc:1722] T 6f8cf09f23984b879b60ab6b27c7d52e P c756528642a543eeacd93cef3e335a7d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
W20251029 02:52:17.384984 712 leader_election.cc:343] T 24f07b1efac64b099b0b5ed467463749 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:17.385103 712 leader_election.cc:304] T 24f07b1efac64b099b0b5ed467463749 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:17.385542 2857 raft_consensus.cc:2749] T 24f07b1efac64b099b0b5ed467463749 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:17.389026 1176 ts_tablet_manager.cc:1929] T 58dde9397b434b089ade9968f369006a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:17.389786 776 raft_consensus.cc:2468] T fcb3db8399aa4cf188b07dda220f7f5b P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:17.391878 847 leader_election.cc:304] T fcb3db8399aa4cf188b07dda220f7f5b P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5
I20251029 02:52:17.392271 2747 raft_consensus.cc:2804] T fcb3db8399aa4cf188b07dda220f7f5b P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:17.392398 2747 raft_consensus.cc:697] T fcb3db8399aa4cf188b07dda220f7f5b P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:52:17.392643 2747 consensus_queue.cc:237] T fcb3db8399aa4cf188b07dda220f7f5b P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:17.394171 1175 ts_tablet_manager.cc:1929] T 82289e0f823c4d43bfd8c4e878572b2b P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:17.394644 1175 log.cc:1199] T 82289e0f823c4d43bfd8c4e878572b2b P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/82289e0f823c4d43bfd8c4e878572b2b
I20251029 02:52:17.395337 1175 ts_tablet_manager.cc:1950] T 82289e0f823c4d43bfd8c4e878572b2b P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:17.398981 2763 tablet.cc:1722] T 6f8cf09f23984b879b60ab6b27c7d52e P b495a49349324f7d81908f2da700f0d9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251029 02:52:17.407398 1167 tablet_bootstrap.cc:654] T 7816556521144c928d7a02694e22e79f P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:17.302481 2833 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9672114a6a742e2864c16ff60ba246e: Bootstrap starting.
I20251029 02:52:17.389334 1176 log.cc:1199] T 58dde9397b434b089ade9968f369006a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/58dde9397b434b089ade9968f369006a
I20251029 02:52:17.396226 1052 raft_consensus.cc:3060] T ae4231daa2594258aa51419c97a2f50a P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:17.411466 1176 ts_tablet_manager.cc:1950] T 58dde9397b434b089ade9968f369006a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:17.411768 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1135) took 3377 ms. Trace:
I20251029 02:52:17.411852 1176 rpcz_store.cc:276] 1029 02:52:14.034614 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:14.034678 (+ 64us) service_pool.cc:225] Handling call
1029 02:52:17.242808 (+3208130us) ts_tablet_manager.cc:1151] Deleting tablet 58dde9397b434b089ade9968f369006a
1029 02:52:17.242821 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:17.340988 (+ 98167us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:17.388922 (+ 47934us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:17.411751 (+ 22829us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3207895}
I20251029 02:52:17.412338 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 58dde9397b434b089ade9968f369006a (table default.Table_b9f2a2e3d9f8446fbd46ca6c48155ccf [id=233cd457a662483a84d93d4243a978b1]) successfully deleted
I20251029 02:52:17.412753 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1136) took 3365 ms. Trace:
I20251029 02:52:17.412835 1176 rpcz_store.cc:276] 1029 02:52:14.047073 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:14.047680 (+ 607us) service_pool.cc:225] Handling call
1029 02:52:17.412638 (+3364958us) ts_tablet_manager.cc:1151] Deleting tablet 96427e60edeb45bdab9666a5055b5a99
1029 02:52:17.412647 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:17.412738 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3339077}
W20251029 02:52:17.413240 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 96427e60edeb45bdab9666a5055b5a99 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 96427e60edeb45bdab9666a5055b5a99 already in progress: creating tablet
I20251029 02:52:17.413620 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1139) took 3227 ms. Trace:
I20251029 02:52:17.413761 1176 rpcz_store.cc:276] 1029 02:52:14.186050 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:14.186131 (+ 81us) service_pool.cc:225] Handling call
1029 02:52:17.413526 (+3227395us) ts_tablet_manager.cc:1151] Deleting tablet 7751599f5dfc43baa20dc3487c6eddd9
1029 02:52:17.413535 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:17.413605 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3227196}
I20251029 02:52:17.414125 521 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet 32519fbffcef44a890161e5f3043806c (table default.Table_a78f5837df7b44e791631e421aa7de46 [id=b95fe1c7d6ff4519b90f993f1fa961fd]). Expected version 3 got 0
W20251029 02:52:17.414247 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 7751599f5dfc43baa20dc3487c6eddd9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7751599f5dfc43baa20dc3487c6eddd9 already in progress: creating tablet
I20251029 02:52:17.414381 521 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet 27b08f88b99643d28ac609e45bf8104e (table default.Table_a9e50760813844af85a85c5f8d22f8e0 [id=74a2a41c908146ec8e16c6a3227b94f2]). Expected version 1 got 0
I20251029 02:52:17.414572 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1144) took 3189 ms. Trace:
I20251029 02:52:17.414597 521 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet 6f8cf09f23984b879b60ab6b27c7d52e (table default.Table_96e017f445bc4f1fbebddf65c722b8de [id=98f28a15eced4860ac13a60abdc8e0e2]). Expected version 3 got 0
I20251029 02:52:17.414636 1176 rpcz_store.cc:276] 1029 02:52:14.225257 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:14.226136 (+ 879us) service_pool.cc:225] Handling call
1029 02:52:17.414451 (+3188315us) ts_tablet_manager.cc:1151] Deleting tablet 595b4073cc0446cea041838097af7471
1029 02:52:17.414460 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:17.414556 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3188140}
W20251029 02:52:17.414996 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 595b4073cc0446cea041838097af7471 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 595b4073cc0446cea041838097af7471 already in progress: creating tablet
I20251029 02:52:17.415371 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1145) took 3190 ms. Trace:
I20251029 02:52:17.415449 1176 rpcz_store.cc:276] 1029 02:52:14.225345 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:14.226348 (+ 1003us) service_pool.cc:225] Handling call
1029 02:52:17.415252 (+3188904us) ts_tablet_manager.cc:1151] Deleting tablet e2c8983cded84a1c8b4205c48a4eac33
1029 02:52:17.415264 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:17.415350 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3188785}
W20251029 02:52:17.415860 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet e2c8983cded84a1c8b4205c48a4eac33 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e2c8983cded84a1c8b4205c48a4eac33 already in progress: creating tablet
I20251029 02:52:17.416874 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_eda0a7e4e3a940b0b6e8169f8f36b0b9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:17.417217 1167 tablet_bootstrap.cc:492] T 7816556521144c928d7a02694e22e79f P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:17.417339 1167 ts_tablet_manager.cc:1403] T 7816556521144c928d7a02694e22e79f P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.000s
W20251029 02:52:17.417369 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eda0a7e4e3a940b0b6e8169f8f36b0b9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:17.409896 1166 tablet_bootstrap.cc:492] T 748777c6cd6e4f49ab49912b719e48bd P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:17.417513 1166 ts_tablet_manager.cc:1403] T 748777c6cd6e4f49ab49912b719e48bd P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.072s user 0.003s sys 0.000s
I20251029 02:52:17.418190 1166 raft_consensus.cc:359] T 748777c6cd6e4f49ab49912b719e48bd P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:17.418188 1167 raft_consensus.cc:359] T 7816556521144c928d7a02694e22e79f P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:17.418335 1166 raft_consensus.cc:385] T 748777c6cd6e4f49ab49912b719e48bd P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:17.418366 1167 raft_consensus.cc:385] T 7816556521144c928d7a02694e22e79f P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:17.418383 1166 raft_consensus.cc:740] T 748777c6cd6e4f49ab49912b719e48bd P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:17.418426 1167 raft_consensus.cc:740] T 7816556521144c928d7a02694e22e79f P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:17.418632 1166 consensus_queue.cc:260] T 748777c6cd6e4f49ab49912b719e48bd P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:17.418783 1167 consensus_queue.cc:260] T 7816556521144c928d7a02694e22e79f P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:17.419312 1166 ts_tablet_manager.cc:1434] T 748777c6cd6e4f49ab49912b719e48bd P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251029 02:52:17.419414 1167 ts_tablet_manager.cc:1434] T 7816556521144c928d7a02694e22e79f P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251029 02:52:17.419634 1166 tablet_bootstrap.cc:492] T ad214c870ee649279ed7321c5b69cbee P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:17.419709 1167 tablet_bootstrap.cc:492] T f4729ecd182042c2a10f67ca25224101 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:17.420454 525 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet ae4231daa2594258aa51419c97a2f50a (table default.Table_a9c2f9e099c14953bbce19d79154eb31 [id=f4e028bdd23f4eea975f940d0b2c73f6]). Expected version 1 got 0
I20251029 02:52:17.421736 1166 tablet_bootstrap.cc:654] T ad214c870ee649279ed7321c5b69cbee P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:17.421922 1167 tablet_bootstrap.cc:654] T f4729ecd182042c2a10f67ca25224101 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:17.423334 891 tablet_service.cc:1552] Processing DeleteTablet for tablet eca016aed05e4ce09e1a2bf96e2286f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:58 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:17.423727 2201 raft_consensus.cc:493] T 628a7d0419cb434f8171faf500d585d8 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:17.423840 2201 raft_consensus.cc:515] T 628a7d0419cb434f8171faf500d585d8 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:17.424362 2201 leader_election.cc:290] T 628a7d0419cb434f8171faf500d585d8 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:17.425320 520 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 3fc2ba404f6b4df6a6eea4facaf358dc
I20251029 02:52:17.426049 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_af9e4be8b41a49ec84d8f5d5cd347b4a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:17.427704 526 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 4eec207d19f540518bb567d7ab21d048
I20251029 02:52:17.428290 526 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_76410db5d9f74f4d9c4e6adb8f3a7f40" } new_table_name: "default.Table_1d57de91ade24f33abd0740342749a08" modify_external_catalogs: true
I20251029 02:52:17.430110 519 master_service.cc:954] ReplaceTablet: received request to replace tablet b3dc6e55bb2e47bab7df5df62a1bcb51 from {username='slave'} at 127.0.0.1:35314
I20251029 02:52:17.436956 524 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 4e33a29c6ef34104b73a4f1329a8387a
I20251029 02:52:17.438045 1167 tablet_bootstrap.cc:492] T f4729ecd182042c2a10f67ca25224101 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:17.438181 1167 ts_tablet_manager.cc:1403] T f4729ecd182042c2a10f67ca25224101 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.006s
I20251029 02:52:17.438876 1167 raft_consensus.cc:359] T f4729ecd182042c2a10f67ca25224101 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:17.439030 1167 raft_consensus.cc:385] T f4729ecd182042c2a10f67ca25224101 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:17.439070 1167 raft_consensus.cc:740] T f4729ecd182042c2a10f67ca25224101 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:17.439287 1167 consensus_queue.cc:260] T f4729ecd182042c2a10f67ca25224101 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:17.439738 1167 ts_tablet_manager.cc:1434] T f4729ecd182042c2a10f67ca25224101 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251029 02:52:17.439977 1167 tablet_bootstrap.cc:492] T 70b008011d5547bbaf3f45685394139f P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:17.442291 1167 tablet_bootstrap.cc:654] T 70b008011d5547bbaf3f45685394139f P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
W20251029 02:52:17.447115 849 consensus_peers.cc:597] T f281191e7c6143f983f11d87f1c9b64e P c756528642a543eeacd93cef3e335a7d -> Peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Couldn't send request to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f281191e7c6143f983f11d87f1c9b64e. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:17.447785 888 tablet_service.cc:1552] Processing DeleteTablet for tablet 4e33a29c6ef34104b73a4f1329a8387a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:16 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:17.448316 895 tablet_service.cc:1552] Processing DeleteTablet for tablet 3fc2ba404f6b4df6a6eea4facaf358dc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:16 UTC) from {username='slave'} at 127.0.0.1:35702
W20251029 02:52:17.448530 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_af9e4be8b41a49ec84d8f5d5cd347b4a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:17.381896 1174 ts_tablet_manager.cc:1929] T 59d03a190b784ab18a3e63a572675891 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:17.450098 1174 log.cc:1199] T 59d03a190b784ab18a3e63a572675891 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/59d03a190b784ab18a3e63a572675891
I20251029 02:52:17.450616 891 tablet_service.cc:1552] Processing DeleteTablet for tablet 4eec207d19f540518bb567d7ab21d048 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:16 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:17.450911 1174 ts_tablet_manager.cc:1950] T 59d03a190b784ab18a3e63a572675891 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:17.457425 524 catalog_manager.cc:5649] T defb795026a44e12afb2defb9c9c6db2 P c756528642a543eeacd93cef3e335a7d reported cstate change: term changed from 0 to 1, leader changed from <none> to c756528642a543eeacd93cef3e335a7d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "c756528642a543eeacd93cef3e335a7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } health_report { overall_health: UNKNOWN } } }
I20251029 02:52:17.459712 1167 tablet_bootstrap.cc:492] T 70b008011d5547bbaf3f45685394139f P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:17.461470 1167 ts_tablet_manager.cc:1403] T 70b008011d5547bbaf3f45685394139f P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.006s
I20251029 02:52:17.462617 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 958) took 6807 ms. Trace:
I20251029 02:52:17.462805 1167 raft_consensus.cc:359] T 70b008011d5547bbaf3f45685394139f P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:17.462987 1167 raft_consensus.cc:385] T 70b008011d5547bbaf3f45685394139f P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:17.463056 1167 raft_consensus.cc:740] T 70b008011d5547bbaf3f45685394139f P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:17.463253 1167 consensus_queue.cc:260] T 70b008011d5547bbaf3f45685394139f P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:17.463685 1167 ts_tablet_manager.cc:1434] T 70b008011d5547bbaf3f45685394139f P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251029 02:52:17.463974 1167 tablet_bootstrap.cc:492] T 24f07b1efac64b099b0b5ed467463749 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:17.465440 1167 tablet_bootstrap.cc:654] T 24f07b1efac64b099b0b5ed467463749 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:17.422884 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1153) took 2932 ms. Trace:
I20251029 02:52:17.483825 1176 rpcz_store.cc:276] 1029 02:52:14.490453 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:14.490582 (+ 129us) service_pool.cc:225] Handling call
1029 02:52:17.416064 (+2925482us) ts_tablet_manager.cc:1151] Deleting tablet c0936e54f1d246388a2d82dbc52aac42
1029 02:52:17.422775 (+ 6711us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:17.422866 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2925264}
I20251029 02:52:17.484249 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1152) took 2993 ms. Trace:
I20251029 02:52:17.484339 1176 rpcz_store.cc:276] 1029 02:52:14.490343 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:14.490600 (+ 257us) service_pool.cc:225] Handling call
1029 02:52:17.484130 (+2993530us) ts_tablet_manager.cc:1151] Deleting tablet f93b0c39d7174dabbe018b1546dbc198
1029 02:52:17.484140 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:17.484233 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2927073}
W20251029 02:52:17.484378 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet c0936e54f1d246388a2d82dbc52aac42 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c0936e54f1d246388a2d82dbc52aac42 already in progress: creating tablet
I20251029 02:52:17.484666 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1155) took 2926 ms. Trace:
I20251029 02:52:17.484730 1176 rpcz_store.cc:276] 1029 02:52:14.558035 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:14.558107 (+ 72us) service_pool.cc:225] Handling call
1029 02:52:17.484553 (+2926446us) ts_tablet_manager.cc:1151] Deleting tablet b81f26fb2739487dbc9af7b6e9848a4a
1029 02:52:17.484562 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:17.484655 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2926269}
W20251029 02:52:17.484803 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet f93b0c39d7174dabbe018b1546dbc198 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f93b0c39d7174dabbe018b1546dbc198 already in progress: creating tablet
I20251029 02:52:17.485005 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1154) took 2927 ms. Trace:
I20251029 02:52:17.462759 1174 rpcz_store.cc:276] 1029 02:52:10.655041 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:10.655092 (+ 51us) service_pool.cc:225] Handling call
1029 02:52:17.255324 (+6600232us) ts_tablet_manager.cc:1151] Deleting tablet 59d03a190b784ab18a3e63a572675891
1029 02:52:17.255335 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:17.374771 (+119436us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:17.381792 (+ 7021us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:17.462568 (+ 80776us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6600057}
I20251029 02:52:17.485069 1176 rpcz_store.cc:276] 1029 02:52:14.557891 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:14.563342 (+ 5451us) service_pool.cc:225] Handling call
1029 02:52:17.484900 (+2921558us) ts_tablet_manager.cc:1151] Deleting tablet 3c3ef62323d643dda28f0ca269274600
1029 02:52:17.484907 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:17.484992 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2921312}
W20251029 02:52:17.485226 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet b81f26fb2739487dbc9af7b6e9848a4a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b81f26fb2739487dbc9af7b6e9848a4a already in progress: creating tablet
I20251029 02:52:17.485311 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1167) took 2880 ms. Trace:
I20251029 02:52:17.485363 1176 rpcz_store.cc:276] 1029 02:52:14.605124 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:14.605174 (+ 50us) service_pool.cc:225] Handling call
1029 02:52:17.485233 (+2880059us) ts_tablet_manager.cc:1151] Deleting tablet 0e9a9dd2f6e4424cb1500682d5a49cc4
1029 02:52:17.485239 (+ 6us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:17.485302 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2879883}
I20251029 02:52:17.485432 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 59d03a190b784ab18a3e63a572675891 (table default.Table_2ec9c2215f7147adab6bbfa2fd06d968 [id=3ede90ae95fb46b2ab910914a49a35a5]) successfully deleted
I20251029 02:52:17.485538 1176 tablet_replica.cc:333] T 946f65014d97425aab3851009b742807 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:17.485607 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 959) took 6830 ms. Trace:
I20251029 02:52:17.485682 1176 raft_consensus.cc:2243] T 946f65014d97425aab3851009b742807 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:17.485684 1174 rpcz_store.cc:276] 1029 02:52:10.655138 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:10.655296 (+ 158us) service_pool.cc:225] Handling call
1029 02:52:17.485475 (+6830179us) ts_tablet_manager.cc:1151] Deleting tablet 34d141bee474452693dba1b9d7f2b023
1029 02:52:17.485487 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:17.485587 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6830041}
I20251029 02:52:17.485813 1176 raft_consensus.cc:2272] T 946f65014d97425aab3851009b742807 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:17.486050 1174 tablet_replica.cc:333] T 7a1866afc7564217ad777e7ce53786c5 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
W20251029 02:52:17.486058 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 34d141bee474452693dba1b9d7f2b023 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 34d141bee474452693dba1b9d7f2b023 already in progress: creating tablet
I20251029 02:52:17.486277 1174 raft_consensus.cc:2243] T 7a1866afc7564217ad777e7ce53786c5 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:17.486358 1176 ts_tablet_manager.cc:1916] T 946f65014d97425aab3851009b742807 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 02:52:17.486358 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 3c3ef62323d643dda28f0ca269274600 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3c3ef62323d643dda28f0ca269274600 already in progress: creating tablet
W20251029 02:52:17.486604 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 0e9a9dd2f6e4424cb1500682d5a49cc4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0e9a9dd2f6e4424cb1500682d5a49cc4 already in progress: creating tablet
I20251029 02:52:17.486743 1174 raft_consensus.cc:2272] T 7a1866afc7564217ad777e7ce53786c5 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:17.487593 1174 ts_tablet_manager.cc:1916] T 7a1866afc7564217ad777e7ce53786c5 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:17.424719 1059 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "628a7d0419cb434f8171faf500d585d8" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:17.427199 1033 tablet_service.cc:1552] Processing DeleteTablet for tablet 3fc2ba404f6b4df6a6eea4facaf358dc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:16 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:17.437021 1058 raft_consensus.cc:3060] T 748777c6cd6e4f49ab49912b719e48bd P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
W20251029 02:52:17.489059 849 leader_election.cc:343] T 628a7d0419cb434f8171faf500d585d8 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:17.490473 1167 tablet_bootstrap.cc:492] T 24f07b1efac64b099b0b5ed467463749 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:17.450209 1024 tablet_service.cc:1552] Processing DeleteTablet for tablet 4eec207d19f540518bb567d7ab21d048 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:16 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:17.490687 1167 ts_tablet_manager.cc:1403] T 24f07b1efac64b099b0b5ed467463749 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.004s
I20251029 02:52:17.450331 1031 tablet_service.cc:1552] Processing DeleteTablet for tablet 4e33a29c6ef34104b73a4f1329a8387a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:16 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:17.470474 2847 tablet.cc:1722] T ae4231daa2594258aa51419c97a2f50a P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 02:52:17.475446 2822 raft_consensus.cc:493] T 589a521f5cc24e6289c37b9cf256bce5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:17.491513 2822 raft_consensus.cc:515] T 589a521f5cc24e6289c37b9cf256bce5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:17.492122 2822 leader_election.cc:290] T 589a521f5cc24e6289c37b9cf256bce5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:17.492045 1167 raft_consensus.cc:359] T 24f07b1efac64b099b0b5ed467463749 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:17.492208 1167 raft_consensus.cc:385] T 24f07b1efac64b099b0b5ed467463749 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:17.492256 1167 raft_consensus.cc:740] T 24f07b1efac64b099b0b5ed467463749 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:17.492430 1167 consensus_queue.cc:260] T 24f07b1efac64b099b0b5ed467463749 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:17.492942 1167 ts_tablet_manager.cc:1434] T 24f07b1efac64b099b0b5ed467463749 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251029 02:52:17.493132 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "589a521f5cc24e6289c37b9cf256bce5" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:17.493253 1167 tablet_bootstrap.cc:492] T 589a521f5cc24e6289c37b9cf256bce5 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
W20251029 02:52:17.493582 990 leader_election.cc:343] T 589a521f5cc24e6289c37b9cf256bce5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:17.494837 1167 tablet_bootstrap.cc:654] T 589a521f5cc24e6289c37b9cf256bce5 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:17.495669 1176 ts_tablet_manager.cc:1929] T 946f65014d97425aab3851009b742807 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:17.495774 1176 log.cc:1199] T 946f65014d97425aab3851009b742807 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/946f65014d97425aab3851009b742807
I20251029 02:52:17.496197 1176 ts_tablet_manager.cc:1950] T 946f65014d97425aab3851009b742807 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:17.496796 2747 raft_consensus.cc:493] T f4729ecd182042c2a10f67ca25224101 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:17.496944 2747 raft_consensus.cc:515] T f4729ecd182042c2a10f67ca25224101 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:17.497674 2747 leader_election.cc:290] T f4729ecd182042c2a10f67ca25224101 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:17.498358 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4729ecd182042c2a10f67ca25224101" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:17.498750 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1172) took 2724 ms. Trace:
I20251029 02:52:17.498857 1176 rpcz_store.cc:276] 1029 02:52:14.774336 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:14.774967 (+ 631us) service_pool.cc:225] Handling call
1029 02:52:17.485505 (+2710538us) ts_tablet_manager.cc:1151] Deleting tablet 946f65014d97425aab3851009b742807
1029 02:52:17.485512 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:17.492415 (+ 6903us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:17.495586 (+ 3171us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:17.498729 (+ 3143us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2710308}
W20251029 02:52:17.499281 849 leader_election.cc:343] T f4729ecd182042c2a10f67ca25224101 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:17.499413 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 946f65014d97425aab3851009b742807 (table default.Table_7dd1cd855bb74080b6e242847e5faac6 [id=8a3311a64d244193985716c908a42f27]) successfully deleted
I20251029 02:52:17.501784 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1173) took 2727 ms. Trace:
I20251029 02:52:17.501924 1176 rpcz_store.cc:276] 1029 02:52:14.774423 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:14.867007 (+ 92584us) service_pool.cc:225] Handling call
1029 02:52:17.501651 (+2634644us) ts_tablet_manager.cc:1151] Deleting tablet df669cf8da7f4c98a3f16423c55c15eb
1029 02:52:17.501662 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:17.501770 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2634398}
W20251029 02:52:17.502532 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet df669cf8da7f4c98a3f16423c55c15eb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet df669cf8da7f4c98a3f16423c55c15eb already in progress: creating tablet
I20251029 02:52:17.503253 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1174) took 2635 ms. Trace:
I20251029 02:52:17.503400 1176 rpcz_store.cc:276] 1029 02:52:14.867537 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:14.867811 (+ 274us) service_pool.cc:225] Handling call
1029 02:52:17.503098 (+2635287us) ts_tablet_manager.cc:1151] Deleting tablet fe9e660de57a46c7ba1d82f25e12b4b0
1029 02:52:17.503111 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:17.503233 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2635082}
W20251029 02:52:17.503806 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet fe9e660de57a46c7ba1d82f25e12b4b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fe9e660de57a46c7ba1d82f25e12b4b0 already in progress: creating tablet
I20251029 02:52:17.504187 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1175) took 2636 ms. Trace:
I20251029 02:52:17.504971 1176 rpcz_store.cc:276] 1029 02:52:14.868226 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:14.868563 (+ 337us) service_pool.cc:225] Handling call
1029 02:52:17.504074 (+2635511us) ts_tablet_manager.cc:1151] Deleting tablet e1d00ea3dd3744ffa5258bbe0095851c
1029 02:52:17.504083 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:17.504172 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2635354}
W20251029 02:52:17.505467 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet e1d00ea3dd3744ffa5258bbe0095851c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e1d00ea3dd3744ffa5258bbe0095851c already in progress: creating tablet
I20251029 02:52:17.505848 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1177) took 2636 ms. Trace:
I20251029 02:52:17.506848 1176 rpcz_store.cc:276] 1029 02:52:14.869638 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:14.878184 (+ 8546us) service_pool.cc:225] Handling call
1029 02:52:17.505722 (+2627538us) ts_tablet_manager.cc:1151] Deleting tablet d48d9178108b4eb59062d29b40ba0c11
1029 02:52:17.505730 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:17.505828 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2627298}
W20251029 02:52:17.507444 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet d48d9178108b4eb59062d29b40ba0c11 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d48d9178108b4eb59062d29b40ba0c11 already in progress: creating tablet
I20251029 02:52:17.507762 1176 tablet_replica.cc:333] T 078ace5eada044c88d2462efb2b57aa8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:17.507979 1176 raft_consensus.cc:2243] T 078ace5eada044c88d2462efb2b57aa8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:17.508191 1176 raft_consensus.cc:2272] T 078ace5eada044c88d2462efb2b57aa8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:17.509387 750 tablet_service.cc:1552] Processing DeleteTablet for tablet 3fc2ba404f6b4df6a6eea4facaf358dc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:16 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:17.509933 1176 ts_tablet_manager.cc:1916] T 078ace5eada044c88d2462efb2b57aa8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 02:52:17.510834 989 consensus_peers.cc:597] T 20a762344c5f46f88481c4ac7ad6e43b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 -> Peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Couldn't send request to peer b495a49349324f7d81908f2da700f0d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 20a762344c5f46f88481c4ac7ad6e43b. This is attempt 1: this message will repeat every 5th retry.
W20251029 02:52:17.513453 849 consensus_peers.cc:597] T af4860c6b72f4d988b206b5ef2f22906 P c756528642a543eeacd93cef3e335a7d -> Peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Couldn't send request to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: af4860c6b72f4d988b206b5ef2f22906. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:17.530004 2426 consensus_queue.cc:1048] T d29cc2a0c8b84e3ebadc7f571d666847 P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:17.530130 1176 ts_tablet_manager.cc:1929] T 078ace5eada044c88d2462efb2b57aa8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:17.530261 1176 log.cc:1199] T 078ace5eada044c88d2462efb2b57aa8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/078ace5eada044c88d2462efb2b57aa8
I20251029 02:52:17.530692 1176 ts_tablet_manager.cc:1950] T 078ace5eada044c88d2462efb2b57aa8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:17.531111 767 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "628a7d0419cb434f8171faf500d585d8" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:17.531301 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1195) took 2135 ms. Trace:
I20251029 02:52:17.531417 1176 rpcz_store.cc:276] 1029 02:52:15.395512 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:15.834754 (+439242us) service_pool.cc:225] Handling call
1029 02:52:17.507719 (+1672965us) ts_tablet_manager.cc:1151] Deleting tablet 078ace5eada044c88d2462efb2b57aa8
1029 02:52:17.507728 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:17.516020 (+ 8292us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:17.519351 (+ 3331us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:17.531274 (+ 11923us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1672768}
I20251029 02:52:17.531975 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 078ace5eada044c88d2462efb2b57aa8 (table default.Table_3fae9f42aaba465bb13918661650e73a [id=9173815e55cc4e7a95b8b663e30c622c]) successfully deleted
I20251029 02:52:17.532970 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1196) took 1697 ms. Trace:
I20251029 02:52:17.533147 1176 rpcz_store.cc:276] 1029 02:52:15.835101 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:15.835143 (+ 42us) service_pool.cc:225] Handling call
1029 02:52:17.532793 (+1697650us) ts_tablet_manager.cc:1151] Deleting tablet 7ff2399833e546448b85b9caeb5200fb
1029 02:52:17.532807 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:17.532945 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1697497}
I20251029 02:52:17.533226 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 1043) took 3840 ms. Trace:
I20251029 02:52:17.535674 766 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "589a521f5cc24e6289c37b9cf256bce5" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
W20251029 02:52:17.536406 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 7ff2399833e546448b85b9caeb5200fb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7ff2399833e546448b85b9caeb5200fb already in progress: creating tablet
I20251029 02:52:17.536988 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1197) took 1701 ms. Trace:
I20251029 02:52:17.537606 1176 rpcz_store.cc:276] 1029 02:52:15.835444 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:15.835485 (+ 41us) service_pool.cc:225] Handling call
1029 02:52:17.536812 (+1701327us) ts_tablet_manager.cc:1151] Deleting tablet f66a7af499b446c7a33965c9e37ce460
1029 02:52:17.536827 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:17.536964 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1701196}
I20251029 02:52:17.537711 738 tablet_service.cc:1552] Processing DeleteTablet for tablet 4e33a29c6ef34104b73a4f1329a8387a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:16 UTC) from {username='slave'} at 127.0.0.1:44580
W20251029 02:52:17.538405 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet f66a7af499b446c7a33965c9e37ce460 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f66a7af499b446c7a33965c9e37ce460 already in progress: creating tablet
I20251029 02:52:17.538934 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1198) took 1703 ms. Trace:
W20251029 02:52:17.540612 849 consensus_peers.cc:597] T ad214c870ee649279ed7321c5b69cbee P c756528642a543eeacd93cef3e335a7d -> Peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Couldn't send request to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:17.540788 1176 rpcz_store.cc:276] 1029 02:52:15.835716 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:15.835761 (+ 45us) service_pool.cc:225] Handling call
1029 02:52:17.538789 (+1703028us) ts_tablet_manager.cc:1151] Deleting tablet 2c4e78fbd7e645eb8b3be8ac4bfbb425
1029 02:52:17.538804 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:17.538912 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1702905}
W20251029 02:52:17.541252 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 2c4e78fbd7e645eb8b3be8ac4bfbb425 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2c4e78fbd7e645eb8b3be8ac4bfbb425 already in progress: creating tablet
I20251029 02:52:17.541766 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1200) took 1702 ms. Trace:
I20251029 02:52:17.541882 1176 rpcz_store.cc:276] 1029 02:52:15.838841 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:15.838885 (+ 44us) service_pool.cc:225] Handling call
1029 02:52:17.541526 (+1702641us) ts_tablet_manager.cc:1151] Deleting tablet 59fd5608e4c04fa293ca04595a20d858
1029 02:52:17.541661 (+ 135us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:17.541751 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1702493}
W20251029 02:52:17.543255 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 59fd5608e4c04fa293ca04595a20d858 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 59fd5608e4c04fa293ca04595a20d858 already in progress: creating tablet
I20251029 02:52:17.543682 1176 tablet_replica.cc:333] T 40e844fd7ad74680bcbe7bef936c5b9d P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:17.543970 1176 raft_consensus.cc:2243] T 40e844fd7ad74680bcbe7bef936c5b9d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:17.544247 1176 raft_consensus.cc:2272] T 40e844fd7ad74680bcbe7bef936c5b9d P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:17.545022 1176 ts_tablet_manager.cc:1916] T 40e844fd7ad74680bcbe7bef936c5b9d P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:17.548471 767 raft_consensus.cc:2468] T 628a7d0419cb434f8171faf500d585d8 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:17.549202 847 leader_election.cc:304] T 628a7d0419cb434f8171faf500d585d8 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5
I20251029 02:52:17.549551 2426 raft_consensus.cc:2804] T 628a7d0419cb434f8171faf500d585d8 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:17.549631 2426 raft_consensus.cc:493] T 628a7d0419cb434f8171faf500d585d8 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:17.549708 2426 raft_consensus.cc:3060] T 628a7d0419cb434f8171faf500d585d8 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:17.542248 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4729ecd182042c2a10f67ca25224101" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:17.550004 766 raft_consensus.cc:2468] T 589a521f5cc24e6289c37b9cf256bce5 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 0.
I20251029 02:52:17.550139 772 raft_consensus.cc:2468] T f4729ecd182042c2a10f67ca25224101 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:17.550334 750 tablet_service.cc:1552] Processing DeleteTablet for tablet 4eec207d19f540518bb567d7ab21d048 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:16 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:17.550578 989 leader_election.cc:304] T 589a521f5cc24e6289c37b9cf256bce5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters: c756528642a543eeacd93cef3e335a7d
I20251029 02:52:17.551118 847 leader_election.cc:304] T f4729ecd182042c2a10f67ca25224101 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5
I20251029 02:52:17.551573 2822 raft_consensus.cc:2804] T 589a521f5cc24e6289c37b9cf256bce5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:17.551664 2822 raft_consensus.cc:493] T 589a521f5cc24e6289c37b9cf256bce5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:17.551728 2822 raft_consensus.cc:3060] T 589a521f5cc24e6289c37b9cf256bce5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:17.552858 2649 raft_consensus.cc:2804] T f4729ecd182042c2a10f67ca25224101 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:17.552973 2649 raft_consensus.cc:493] T f4729ecd182042c2a10f67ca25224101 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:17.553040 2649 raft_consensus.cc:3060] T f4729ecd182042c2a10f67ca25224101 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:17.554363 2822 raft_consensus.cc:515] T 589a521f5cc24e6289c37b9cf256bce5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:17.555456 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "589a521f5cc24e6289c37b9cf256bce5" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
I20251029 02:52:17.555542 2649 raft_consensus.cc:515] T f4729ecd182042c2a10f67ca25224101 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
W20251029 02:52:17.555958 990 leader_election.cc:343] T 589a521f5cc24e6289c37b9cf256bce5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:17.556562 2822 leader_election.cc:290] T 589a521f5cc24e6289c37b9cf256bce5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Requested vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:17.556629 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4729ecd182042c2a10f67ca25224101" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:52:17.556891 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "589a521f5cc24e6289c37b9cf256bce5" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
W20251029 02:52:17.557090 849 leader_election.cc:343] T f4729ecd182042c2a10f67ca25224101 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:17.557153 772 raft_consensus.cc:3060] T 589a521f5cc24e6289c37b9cf256bce5 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:17.552428 2426 raft_consensus.cc:515] T 628a7d0419cb434f8171faf500d585d8 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:17.558176 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "628a7d0419cb434f8171faf500d585d8" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
W20251029 02:52:17.558537 849 leader_election.cc:343] T 628a7d0419cb434f8171faf500d585d8 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:17.559284 2649 leader_election.cc:290] T f4729ecd182042c2a10f67ca25224101 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:17.559537 2426 leader_election.cc:290] T 628a7d0419cb434f8171faf500d585d8 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:17.559754 766 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4729ecd182042c2a10f67ca25224101" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:17.559962 766 raft_consensus.cc:3060] T f4729ecd182042c2a10f67ca25224101 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:17.560324 772 raft_consensus.cc:2468] T 589a521f5cc24e6289c37b9cf256bce5 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 1.
I20251029 02:52:17.563273 989 leader_election.cc:304] T 589a521f5cc24e6289c37b9cf256bce5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters: c756528642a543eeacd93cef3e335a7d
I20251029 02:52:17.565042 766 raft_consensus.cc:2468] T f4729ecd182042c2a10f67ca25224101 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:17.565937 847 leader_election.cc:304] T f4729ecd182042c2a10f67ca25224101 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5
I20251029 02:52:17.566319 2426 raft_consensus.cc:2804] T f4729ecd182042c2a10f67ca25224101 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:17.566443 2426 raft_consensus.cc:697] T f4729ecd182042c2a10f67ca25224101 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:52:17.566749 2426 consensus_queue.cc:237] T f4729ecd182042c2a10f67ca25224101 P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:17.568598 766 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "628a7d0419cb434f8171faf500d585d8" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:17.568873 766 raft_consensus.cc:3060] T 628a7d0419cb434f8171faf500d585d8 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:17.571390 1176 ts_tablet_manager.cc:1929] T 40e844fd7ad74680bcbe7bef936c5b9d P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:17.571494 1176 log.cc:1199] T 40e844fd7ad74680bcbe7bef936c5b9d P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/40e844fd7ad74680bcbe7bef936c5b9d
I20251029 02:52:17.571810 1176 ts_tablet_manager.cc:1950] T 40e844fd7ad74680bcbe7bef936c5b9d P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:17.572100 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1201) took 1732 ms. Trace:
I20251029 02:52:17.572178 1176 rpcz_store.cc:276] 1029 02:52:15.839159 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:15.839209 (+ 50us) service_pool.cc:225] Handling call
1029 02:52:17.543623 (+1704414us) ts_tablet_manager.cc:1151] Deleting tablet 40e844fd7ad74680bcbe7bef936c5b9d
1029 02:52:17.543635 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:17.556014 (+ 12379us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:17.571287 (+ 15273us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:17.572086 (+ 799us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1704260}
I20251029 02:52:17.572573 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1222) took 1669 ms. Trace:
I20251029 02:52:17.572638 1176 rpcz_store.cc:276] 1029 02:52:15.903228 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:15.903278 (+ 50us) service_pool.cc:225] Handling call
1029 02:52:17.572446 (+1669168us) ts_tablet_manager.cc:1151] Deleting tablet 18632de3bbbd491a86460497544580c1
1029 02:52:17.572456 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:17.572561 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1668979}
I20251029 02:52:17.572827 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 40e844fd7ad74680bcbe7bef936c5b9d (table default.Table_a66eb4754b854a998e458627d738338b [id=c47f0d4579d74e31be083ac638480e38]) successfully deleted
I20251029 02:52:17.572918 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1223) took 1669 ms. Trace:
I20251029 02:52:17.572978 1176 rpcz_store.cc:276] 1029 02:52:15.903450 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:15.903495 (+ 45us) service_pool.cc:225] Handling call
1029 02:52:17.572818 (+1669323us) ts_tablet_manager.cc:1151] Deleting tablet a3c8b25e6f4849d6aadca322d5365af1
1029 02:52:17.572825 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:17.572906 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1669175}
I20251029 02:52:17.573220 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1224) took 1669 ms. Trace:
I20251029 02:52:17.573269 1176 rpcz_store.cc:276] 1029 02:52:15.903776 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:15.918617 (+ 14841us) service_pool.cc:225] Handling call
1029 02:52:17.573145 (+1654528us) ts_tablet_manager.cc:1151] Deleting tablet 6dc30c614be64b8ba629d9884d450290
1029 02:52:17.573151 (+ 6us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:17.573210 (+ 59us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1654345}
I20251029 02:52:17.573534 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1225) took 1654 ms. Trace:
W20251029 02:52:17.578339 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 18632de3bbbd491a86460497544580c1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 18632de3bbbd491a86460497544580c1 already in progress: creating tablet
I20251029 02:52:17.579798 766 raft_consensus.cc:2468] T 628a7d0419cb434f8171faf500d585d8 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:17.573590 1176 rpcz_store.cc:276] 1029 02:52:15.919006 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:15.919049 (+ 43us) service_pool.cc:225] Handling call
1029 02:52:17.573426 (+1654377us) ts_tablet_manager.cc:1151] Deleting tablet e60b029eb3ec466cbdced20d536f4f47
1029 02:52:17.573436 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:17.573521 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1654258}
I20251029 02:52:17.580502 847 leader_election.cc:304] T 628a7d0419cb434f8171faf500d585d8 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5
I20251029 02:52:17.580675 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1226) took 1661 ms. Trace:
I20251029 02:52:17.580760 1176 rpcz_store.cc:276] 1029 02:52:15.919355 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:15.919391 (+ 36us) service_pool.cc:225] Handling call
1029 02:52:17.580546 (+1661155us) ts_tablet_manager.cc:1151] Deleting tablet f1e975b267464e64a5bbe24ef4e308a3
1029 02:52:17.580557 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:17.580660 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1661040}
I20251029 02:52:17.581640 2822 raft_consensus.cc:2804] T 589a521f5cc24e6289c37b9cf256bce5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:17.581894 2822 raft_consensus.cc:697] T 589a521f5cc24e6289c37b9cf256bce5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Becoming Leader. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Running, Role: LEADER
I20251029 02:52:17.581678 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1227) took 1662 ms. Trace:
I20251029 02:52:17.582118 1176 rpcz_store.cc:276] 1029 02:52:15.919691 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:15.919736 (+ 45us) service_pool.cc:225] Handling call
1029 02:52:17.581563 (+1661827us) ts_tablet_manager.cc:1151] Deleting tablet 688af6580c4d4637a39f817b4e324618
1029 02:52:17.581572 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:17.581664 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1661713}
I20251029 02:52:17.582433 2822 consensus_queue.cc:237] T 589a521f5cc24e6289c37b9cf256bce5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:17.582738 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1221) took 1679 ms. Trace:
I20251029 02:52:17.583043 1176 rpcz_store.cc:276] 1029 02:52:15.903103 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:15.916213 (+ 13110us) service_pool.cc:225] Handling call
1029 02:52:17.582631 (+1666418us) ts_tablet_manager.cc:1151] Deleting tablet e91d6bdca3ad40c48d93e6196644462c
1029 02:52:17.582639 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:17.582725 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1660039}
I20251029 02:52:17.586709 2426 raft_consensus.cc:2804] T 628a7d0419cb434f8171faf500d585d8 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:17.586927 2426 raft_consensus.cc:697] T 628a7d0419cb434f8171faf500d585d8 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:52:17.587275 2426 consensus_queue.cc:237] T 628a7d0419cb434f8171faf500d585d8 P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:17.556890 1175 rpcz_store.cc:276] 1029 02:52:13.624869 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:13.632236 (+ 7367us) service_pool.cc:225] Handling call
1029 02:52:17.289338 (+3657102us) ts_tablet_manager.cc:1151] Deleting tablet 82289e0f823c4d43bfd8c4e878572b2b
1029 02:52:17.289344 (+ 6us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:17.296894 (+ 7550us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:17.393988 (+ 97094us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:17.465363 (+ 71375us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3656820}
I20251029 02:52:17.594961 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 82289e0f823c4d43bfd8c4e878572b2b (table default.Table_e7bb3e262a894c2f84b6edfc3917b30d [id=2cca3b7c50b243b5883b8ac1f353f253]) successfully deleted
I20251029 02:52:17.595075 1175 tablet_replica.cc:333] T 72f9d4408f384e7093316d85f092e1b2 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:17.595310 1175 raft_consensus.cc:2243] T 72f9d4408f384e7093316d85f092e1b2 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:17.595561 1175 raft_consensus.cc:2272] T 72f9d4408f384e7093316d85f092e1b2 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:17.596351 1175 ts_tablet_manager.cc:1916] T 72f9d4408f384e7093316d85f092e1b2 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:17.602598 2822 raft_consensus.cc:493] T cd597c319af645909b8f3d1212fcb702 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:17.602813 2822 raft_consensus.cc:515] T cd597c319af645909b8f3d1212fcb702 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:17.603853 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cd597c319af645909b8f3d1212fcb702" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:17.604349 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cd597c319af645909b8f3d1212fcb702" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:17.605166 989 leader_election.cc:304] T cd597c319af645909b8f3d1212fcb702 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5; no voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:17.605584 2179 raft_consensus.cc:2749] T cd597c319af645909b8f3d1212fcb702 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:17.605758 2822 leader_election.cc:290] T cd597c319af645909b8f3d1212fcb702 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:17.607682 1172 tablet_bootstrap.cc:492] T 2b278db5b2b04344b864797055a5f665 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:17.607890 1172 ts_tablet_manager.cc:1403] T 2b278db5b2b04344b864797055a5f665 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.314s user 0.007s sys 0.000s
I20251029 02:52:17.608868 1172 raft_consensus.cc:359] T 2b278db5b2b04344b864797055a5f665 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:17.616446 2822 consensus_queue.cc:1048] T 372d93914abc41b38a702742b7db0bc2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251029 02:52:17.627033 989 consensus_peers.cc:597] T 372d93914abc41b38a702742b7db0bc2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 -> Peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Couldn't send request to peer b495a49349324f7d81908f2da700f0d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 372d93914abc41b38a702742b7db0bc2. This is attempt 1: this message will repeat every 5th retry.
W20251029 02:52:17.578680 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet a3c8b25e6f4849d6aadca322d5365af1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a3c8b25e6f4849d6aadca322d5365af1 already in progress: creating tablet
W20251029 02:52:17.630733 513 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.3:45003, user_credentials={real_user=slave}} blocked reactor thread for 52049.9us
W20251029 02:52:17.631071 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 6dc30c614be64b8ba629d9884d450290 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6dc30c614be64b8ba629d9884d450290 already in progress: creating tablet
W20251029 02:52:17.631337 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet e60b029eb3ec466cbdced20d536f4f47 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e60b029eb3ec466cbdced20d536f4f47 already in progress: creating tablet
W20251029 02:52:17.631536 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet f1e975b267464e64a5bbe24ef4e308a3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f1e975b267464e64a5bbe24ef4e308a3 already in progress: creating tablet
W20251029 02:52:17.631726 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 688af6580c4d4637a39f817b4e324618 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 688af6580c4d4637a39f817b4e324618 already in progress: creating tablet
W20251029 02:52:17.631919 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet e91d6bdca3ad40c48d93e6196644462c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e91d6bdca3ad40c48d93e6196644462c already in progress: creating tablet
W20251029 02:52:17.632128 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 2849e856dd714da88fca5a9fafe74c22 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2849e856dd714da88fca5a9fafe74c22 already in progress: creating tablet
W20251029 02:52:17.632345 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 19e15a6df77f4e0c9b40ae4f49fdd06f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 19e15a6df77f4e0c9b40ae4f49fdd06f already in progress: creating tablet
W20251029 02:52:17.632539 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 46e054ff563f448d86d8b4f84810291b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 46e054ff563f448d86d8b4f84810291b already in progress: creating tablet
W20251029 02:52:17.632758 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 5bcadc0b68e84ae290690edef392a45b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5bcadc0b68e84ae290690edef392a45b already in progress: creating tablet
W20251029 02:52:17.633212 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 60f0eadc074a4872b6638314fed98e0e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 60f0eadc074a4872b6638314fed98e0e already in progress: creating tablet
W20251029 02:52:17.633543 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 62f4a04568804c3da03393c08676c5da because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 62f4a04568804c3da03393c08676c5da already in progress: creating tablet
I20251029 02:52:17.634544 1166 tablet_bootstrap.cc:492] T ad214c870ee649279ed7321c5b69cbee P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:17.635787 1166 ts_tablet_manager.cc:1403] T ad214c870ee649279ed7321c5b69cbee P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.216s user 0.002s sys 0.002s
I20251029 02:52:17.636706 1166 raft_consensus.cc:359] T ad214c870ee649279ed7321c5b69cbee P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:17.637709 1166 raft_consensus.cc:385] T ad214c870ee649279ed7321c5b69cbee P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:17.637873 1166 raft_consensus.cc:740] T ad214c870ee649279ed7321c5b69cbee P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:17.638948 1166 consensus_queue.cc:260] T ad214c870ee649279ed7321c5b69cbee P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:17.639652 1166 ts_tablet_manager.cc:1434] T ad214c870ee649279ed7321c5b69cbee P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251029 02:52:17.639798 2426 consensus_queue.cc:1048] T ad214c870ee649279ed7321c5b69cbee P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:17.641522 1166 tablet_bootstrap.cc:492] T c081d2eacfac4ca8a0048e2a1c4f8f83 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
W20251029 02:52:17.641829 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 6408901d1e5f4ac8a7bff0de708660eb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6408901d1e5f4ac8a7bff0de708660eb already in progress: creating tablet
W20251029 02:52:17.642153 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 818262aee43c4343b864e01f836e76b8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 818262aee43c4343b864e01f836e76b8 already in progress: creating tablet
W20251029 02:52:17.642454 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 90c2ed692a954a25bdf79b9884f6c5ce because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 90c2ed692a954a25bdf79b9884f6c5ce already in progress: creating tablet
W20251029 02:52:17.642910 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 7fba607c566c4fa1a803cba8d86ca646 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7fba607c566c4fa1a803cba8d86ca646 already in progress: creating tablet
I20251029 02:52:17.643080 1166 tablet_bootstrap.cc:654] T c081d2eacfac4ca8a0048e2a1c4f8f83 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:17.643388 523 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet bf2c37f5eff740a6ae594ffc6c0e5101
W20251029 02:52:17.643419 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet c73059366f4b45528f8b907263c4d085 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c73059366f4b45528f8b907263c4d085 already in progress: creating tablet
I20251029 02:52:17.641232 1176 tablet_replica.cc:333] T 20a762344c5f46f88481c4ac7ad6e43b P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:17.643853 523 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:35314 (request call id 1482) took 1175 ms. Trace:
I20251029 02:52:17.644039 1176 raft_consensus.cc:2243] T 20a762344c5f46f88481c4ac7ad6e43b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:17.644268 1031 tablet_service.cc:1552] Processing DeleteTablet for tablet bf2c37f5eff740a6ae594ffc6c0e5101 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:16 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:17.644335 1176 pending_rounds.cc:70] T 20a762344c5f46f88481c4ac7ad6e43b P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Trying to abort 1 pending ops.
I20251029 02:52:17.643946 523 rpcz_store.cc:276] 1029 02:52:16.468322 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:16.767702 (+299380us) service_pool.cc:225] Handling call
1029 02:52:16.834708 (+ 67006us) spinlock_profiling.cc:243] Waited 66.8 ms on lock 0x616000041254. stack: 00007f4b1e4899f6 00007f4b1e4897b0 00007f4b1d4653ad 0000000000983f04 0000000000983ecd 00007f4b2d7906bd 00007f4b2d7907b3 00007f4b2a357441 00007f4b320119dd 00007f4b3219f256 00007f4b28cb0486 00007f4b28cb01b5 00007f4b2542827e 00007f4b2542715e 00007f4b2542d395 00007f4b2542dcbf
1029 02:52:16.834940 (+ 232us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1029 02:52:16.835099 (+ 159us) catalog_manager.cc:2815] Modifying in-memory table state
1029 02:52:16.835150 (+ 51us) catalog_manager.cc:2824] Locking tablets
1029 02:52:17.408149 (+572999us) catalog_manager.cc:2838] Removing table and tablets from system table
1029 02:52:17.643260 (+235111us) catalog_manager.cc:2856] Removing table from by-name map
1029 02:52:17.643330 (+ 70us) catalog_manager.cc:2872] Committing in-memory state
1029 02:52:17.643371 (+ 41us) catalog_manager.cc:2876] Aborting table tasks
1029 02:52:17.643833 (+ 462us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1029 02:52:17.411298 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1029 02:52:17.411392 (+ 94us) write_op.cc:432] Acquiring schema lock in shared mode
1029 02:52:17.411399 (+ 7us) write_op.cc:435] Acquired schema lock
1029 02:52:17.411403 (+ 4us) tablet.cc:660] Decoding operations
1029 02:52:17.411492 (+ 89us) write_op.cc:620] Acquiring the partition lock for write op
1029 02:52:17.411499 (+ 7us) write_op.cc:641] Partition lock acquired for write op
1029 02:52:17.411503 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
1029 02:52:17.411583 (+ 80us) tablet.cc:699] Row locks acquired
1029 02:52:17.411585 (+ 2us) write_op.cc:260] PREPARE: finished
1029 02:52:17.411610 (+ 25us) write_op.cc:270] Start()
1029 02:52:17.411643 (+ 33us) write_op.cc:276] Timestamp: P: 1761706337411586 usec, L: 0
1029 02:52:17.411646 (+ 3us) op_driver.cc:348] REPLICATION: starting
1029 02:52:17.411787 (+ 141us) log.cc:844] Serialized 725 byte log entry
1029 02:52:17.636503 (+224716us) op_driver.cc:464] REPLICATION: finished
1029 02:52:17.637654 (+ 1151us) write_op.cc:301] APPLY: starting
1029 02:52:17.637663 (+ 9us) tablet.cc:1365] starting BulkCheckPresence
1029 02:52:17.637703 (+ 40us) tablet.cc:1368] finished BulkCheckPresence
1029 02:52:17.637725 (+ 22us) tablet.cc:1370] starting ApplyRowOperation cycle
1029 02:52:17.637775 (+ 50us) tablet.cc:1381] finished ApplyRowOperation cycle
1029 02:52:17.637802 (+ 27us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1029 02:52:17.637809 (+ 7us) write_op.cc:312] APPLY: finished
1029 02:52:17.637860 (+ 51us) log.cc:844] Serialized 29 byte log entry
1029 02:52:17.637899 (+ 39us) write_op.cc:489] Releasing partition, row and schema locks
1029 02:52:17.637912 (+ 13us) write_op.cc:454] Released schema lock
1029 02:52:17.637917 (+ 5us) write_op.cc:341] FINISH: Updating metrics
1029 02:52:17.637988 (+ 71us) write_op.cc:489] Releasing partition, row and schema locks
1029 02:52:17.637991 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":153584384,"child_traces":[["op",{"apply.queue_time_us":1117,"apply.run_cpu_time_us":328,"apply.run_wall_time_us":328,"num_ops":2,"prepare.queue_time_us":2629,"prepare.run_cpu_time_us":552,"prepare.run_wall_time_us":804,"replication_time_us":224838}]]}
I20251029 02:52:17.644483 1176 pending_rounds.cc:77] T 20a762344c5f46f88481c4ac7ad6e43b P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215949155248713728 op_type: NO_OP noop_request { }
I20251029 02:52:17.644583 1176 raft_consensus.cc:2889] T 20a762344c5f46f88481c4ac7ad6e43b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 02:52:17.644665 1176 raft_consensus.cc:2272] T 20a762344c5f46f88481c4ac7ad6e43b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251029 02:52:17.644923 847 consensus_peers.cc:597] T af4860c6b72f4d988b206b5ef2f22906 P c756528642a543eeacd93cef3e335a7d -> Peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Couldn't send request to peer b495a49349324f7d81908f2da700f0d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: af4860c6b72f4d988b206b5ef2f22906. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:17.645406 1176 ts_tablet_manager.cc:1916] T 20a762344c5f46f88481c4ac7ad6e43b P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:17.646359 750 tablet_service.cc:1552] Processing DeleteTablet for tablet bf2c37f5eff740a6ae594ffc6c0e5101 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:16 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:17.647322 891 tablet_service.cc:1552] Processing DeleteTablet for tablet bf2c37f5eff740a6ae594ffc6c0e5101 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:16 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:17.643151 517 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 749f380eafae44ddbe2a244757f37909
I20251029 02:52:17.647967 517 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:35314 (request call id 1487) took 1030 ms. Trace:
I20251029 02:52:17.648063 517 rpcz_store.cc:276] 1029 02:52:16.617258 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:16.899559 (+282301us) service_pool.cc:225] Handling call
1029 02:52:16.899990 (+ 431us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
1029 02:52:16.900189 (+ 199us) catalog_manager.cc:2815] Modifying in-memory table state
1029 02:52:16.900280 (+ 91us) catalog_manager.cc:2824] Locking tablets
1029 02:52:17.406867 (+506587us) catalog_manager.cc:2838] Removing table and tablets from system table
1029 02:52:17.642991 (+236124us) catalog_manager.cc:2856] Removing table from by-name map
1029 02:52:17.643086 (+ 95us) catalog_manager.cc:2872] Committing in-memory state
1029 02:52:17.643123 (+ 37us) catalog_manager.cc:2876] Aborting table tasks
1029 02:52:17.647949 (+ 4826us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1029 02:52:17.410401 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1029 02:52:17.410551 (+ 150us) write_op.cc:432] Acquiring schema lock in shared mode
1029 02:52:17.410557 (+ 6us) write_op.cc:435] Acquired schema lock
1029 02:52:17.410559 (+ 2us) tablet.cc:660] Decoding operations
1029 02:52:17.410647 (+ 88us) write_op.cc:620] Acquiring the partition lock for write op
1029 02:52:17.410656 (+ 9us) write_op.cc:641] Partition lock acquired for write op
1029 02:52:17.410661 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
1029 02:52:17.410735 (+ 74us) tablet.cc:699] Row locks acquired
1029 02:52:17.410738 (+ 3us) write_op.cc:260] PREPARE: finished
1029 02:52:17.410764 (+ 26us) write_op.cc:270] Start()
1029 02:52:17.410788 (+ 24us) write_op.cc:276] Timestamp: P: 1761706337410741 usec, L: 0
1029 02:52:17.410791 (+ 3us) op_driver.cc:348] REPLICATION: starting
1029 02:52:17.410911 (+ 120us) log.cc:844] Serialized 725 byte log entry
1029 02:52:17.636423 (+225512us) op_driver.cc:464] REPLICATION: finished
1029 02:52:17.637092 (+ 669us) write_op.cc:301] APPLY: starting
1029 02:52:17.637112 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
1029 02:52:17.637160 (+ 48us) tablet.cc:1368] finished BulkCheckPresence
1029 02:52:17.637184 (+ 24us) tablet.cc:1370] starting ApplyRowOperation cycle
1029 02:52:17.637263 (+ 79us) tablet.cc:1381] finished ApplyRowOperation cycle
1029 02:52:17.637299 (+ 36us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
1029 02:52:17.637307 (+ 8us) write_op.cc:312] APPLY: finished
1029 02:52:17.637395 (+ 88us) log.cc:844] Serialized 29 byte log entry
1029 02:52:17.637468 (+ 73us) write_op.cc:489] Releasing partition, row and schema locks
1029 02:52:17.637490 (+ 22us) write_op.cc:454] Released schema lock
1029 02:52:17.637498 (+ 8us) write_op.cc:341] FINISH: Updating metrics
1029 02:52:17.637578 (+ 80us) write_op.cc:489] Releasing partition, row and schema locks
1029 02:52:17.637582 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":595,"apply.run_cpu_time_us":482,"apply.run_wall_time_us":483,"num_ops":2,"prepare.queue_time_us":2965,"prepare.run_cpu_time_us":551,"prepare.run_wall_time_us":866,"replication_time_us":225597}]]}
I20251029 02:52:17.649322 891 tablet_service.cc:1552] Processing DeleteTablet for tablet 749f380eafae44ddbe2a244757f37909 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:16 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:17.649335 750 tablet_service.cc:1552] Processing DeleteTablet for tablet 749f380eafae44ddbe2a244757f37909 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:16 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:17.649966 522 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet d557a4750fee4c2b92eaa834619a678c
I20251029 02:52:17.650707 891 tablet_service.cc:1552] Processing DeleteTablet for tablet d557a4750fee4c2b92eaa834619a678c with delete_type TABLET_DATA_DELETED (replaced by tablet 1c24ed8c046c4119ba84dcae5d2154bf) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:17.652582 522 catalog_manager.cc:6719] ReplaceTablet: tablet d557a4750fee4c2b92eaa834619a678c deleted and replaced by tablet 1c24ed8c046c4119ba84dcae5d2154bf
I20251029 02:52:17.652902 2426 raft_consensus.cc:493] T 32519fbffcef44a890161e5f3043806c P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:17.653084 2426 raft_consensus.cc:515] T 32519fbffcef44a890161e5f3043806c P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:17.653290 750 tablet_service.cc:1552] Processing DeleteTablet for tablet d557a4750fee4c2b92eaa834619a678c with delete_type TABLET_DATA_DELETED (replaced by tablet 1c24ed8c046c4119ba84dcae5d2154bf) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:17.653766 2426 leader_election.cc:290] T 32519fbffcef44a890161e5f3043806c P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:17.654347 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "32519fbffcef44a890161e5f3043806c" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:17.654584 1060 raft_consensus.cc:2393] T 32519fbffcef44a890161e5f3043806c P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c756528642a543eeacd93cef3e335a7d in current term 1: Already voted for candidate b495a49349324f7d81908f2da700f0d9 in this term.
I20251029 02:52:17.655315 766 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "32519fbffcef44a890161e5f3043806c" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:17.655689 1031 tablet_service.cc:1552] Processing DeleteTablet for tablet 749f380eafae44ddbe2a244757f37909 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:16 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:17.656121 847 leader_election.cc:304] T 32519fbffcef44a890161e5f3043806c P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9
I20251029 02:52:17.656482 1031 tablet_service.cc:1552] Processing DeleteTablet for tablet d557a4750fee4c2b92eaa834619a678c with delete_type TABLET_DATA_DELETED (replaced by tablet 1c24ed8c046c4119ba84dcae5d2154bf) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:17.656836 2426 raft_consensus.cc:3060] T 32519fbffcef44a890161e5f3043806c P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:17.657841 1176 ts_tablet_manager.cc:1929] T 20a762344c5f46f88481c4ac7ad6e43b P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:17.657953 1176 log.cc:1199] T 20a762344c5f46f88481c4ac7ad6e43b P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/20a762344c5f46f88481c4ac7ad6e43b
I20251029 02:52:17.658305 1176 ts_tablet_manager.cc:1950] T 20a762344c5f46f88481c4ac7ad6e43b P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:17.658849 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 20a762344c5f46f88481c4ac7ad6e43b (table default.Table_3ca4a6512b174cb897b142dfe1259549 [id=a862192912a04942a8f682fc9825b014]) successfully deleted
I20251029 02:52:17.660080 2885 raft_consensus.cc:493] T d31e548feae24b739087c1e211d514f9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:17.660235 2426 raft_consensus.cc:2749] T 32519fbffcef44a890161e5f3043806c P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:17.660236 2885 raft_consensus.cc:515] T d31e548feae24b739087c1e211d514f9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:17.661216 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d31e548feae24b739087c1e211d514f9" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:17.661727 990 leader_election.cc:343] T d31e548feae24b739087c1e211d514f9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:17.661760 1031 tablet_service.cc:1505] Processing CreateTablet for tablet 1256976d606e409b875d608d5c15528d (DEFAULT_TABLE table=default.Table_b5a3de99a1d84f62b07ded51499e5b82 [id=870e40e80dad4ddc94360e49759aa5c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:17.662094 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1256976d606e409b875d608d5c15528d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:17.662252 2885 leader_election.cc:290] T d31e548feae24b739087c1e211d514f9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:17.663523 1176 tablet_replica.cc:333] T 7bb85a12f573494cbb3118d1947eda27 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:17.663712 1176 raft_consensus.cc:2243] T 7bb85a12f573494cbb3118d1947eda27 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:17.663862 1176 raft_consensus.cc:2272] T 7bb85a12f573494cbb3118d1947eda27 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:17.664409 1176 ts_tablet_manager.cc:1916] T 7bb85a12f573494cbb3118d1947eda27 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:17.669821 521 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_f65782ffe391498286f93d5e804bf177" } new_table_name: "default.Table_b9d950092d13427ca8001c487387436a" modify_external_catalogs: true
I20251029 02:52:17.671129 2426 raft_consensus.cc:493] T eca016aed05e4ce09e1a2bf96e2286f5 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:17.671291 2426 raft_consensus.cc:515] T eca016aed05e4ce09e1a2bf96e2286f5 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:17.672139 1176 ts_tablet_manager.cc:1929] T 7bb85a12f573494cbb3118d1947eda27 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:17.672240 1176 log.cc:1199] T 7bb85a12f573494cbb3118d1947eda27 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/7bb85a12f573494cbb3118d1947eda27
I20251029 02:52:17.672274 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "eca016aed05e4ce09e1a2bf96e2286f5" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:17.672580 1176 ts_tablet_manager.cc:1950] T 7bb85a12f573494cbb3118d1947eda27 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
W20251029 02:52:17.672798 849 leader_election.cc:343] T eca016aed05e4ce09e1a2bf96e2286f5 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:17.673086 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 7bb85a12f573494cbb3118d1947eda27 (table default.Table_0100f32eabf24d88a739a3cdac3597e1 [id=f22450cafadd404d8d224cdee646b5a2]) successfully deleted
I20251029 02:52:17.673282 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_3bb798717ff0482d97c9c76ab35a0214"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:17.673810 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3bb798717ff0482d97c9c76ab35a0214 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:17.673794 767 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "eca016aed05e4ce09e1a2bf96e2286f5" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:17.674022 767 raft_consensus.cc:2468] T eca016aed05e4ce09e1a2bf96e2286f5 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:17.674607 847 leader_election.cc:304] T eca016aed05e4ce09e1a2bf96e2286f5 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5
I20251029 02:52:17.674751 523 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_ad45e164be6a481c871f036851e127f2" } new_table_name: "default.Table_aef83d52090548f7aeca8fa0765f1691" modify_external_catalogs: true
I20251029 02:52:17.675272 766 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d31e548feae24b739087c1e211d514f9" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:17.675650 766 raft_consensus.cc:2468] T d31e548feae24b739087c1e211d514f9 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 0.
I20251029 02:52:17.675858 2649 raft_consensus.cc:2804] T eca016aed05e4ce09e1a2bf96e2286f5 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:17.675974 2649 raft_consensus.cc:493] T eca016aed05e4ce09e1a2bf96e2286f5 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:17.676064 2649 raft_consensus.cc:3060] T eca016aed05e4ce09e1a2bf96e2286f5 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:17.676102 989 leader_election.cc:304] T d31e548feae24b739087c1e211d514f9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters: c756528642a543eeacd93cef3e335a7d
I20251029 02:52:17.678483 2426 leader_election.cc:290] T eca016aed05e4ce09e1a2bf96e2286f5 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
W20251029 02:52:17.679775 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 3fc2ba404f6b4df6a6eea4facaf358dc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3fc2ba404f6b4df6a6eea4facaf358dc already in progress: creating tablet
I20251029 02:52:17.680123 2649 raft_consensus.cc:515] T eca016aed05e4ce09e1a2bf96e2286f5 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:17.680792 2649 leader_election.cc:290] T eca016aed05e4ce09e1a2bf96e2286f5 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
W20251029 02:52:17.680848 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 4eec207d19f540518bb567d7ab21d048 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4eec207d19f540518bb567d7ab21d048 already in progress: creating tablet
I20251029 02:52:17.681104 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "eca016aed05e4ce09e1a2bf96e2286f5" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
W20251029 02:52:17.681605 849 leader_election.cc:343] T eca016aed05e4ce09e1a2bf96e2286f5 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
W20251029 02:52:17.681669 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 4e33a29c6ef34104b73a4f1329a8387a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4e33a29c6ef34104b73a4f1329a8387a already in progress: creating tablet
W20251029 02:52:17.682343 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet bf2c37f5eff740a6ae594ffc6c0e5101 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bf2c37f5eff740a6ae594ffc6c0e5101 already in progress: creating tablet
I20251029 02:52:17.682708 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "eca016aed05e4ce09e1a2bf96e2286f5" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:17.682902 772 raft_consensus.cc:3060] T eca016aed05e4ce09e1a2bf96e2286f5 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:17.682466 522 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_447e2f76846e454885124e9615bc17c4" } new_table_name: "default.Table_86cdba66817545ca8252ac0a8a2e28d1" modify_external_catalogs: true
I20251029 02:52:17.683424 2885 raft_consensus.cc:2804] T d31e548feae24b739087c1e211d514f9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:17.683599 2885 raft_consensus.cc:493] T d31e548feae24b739087c1e211d514f9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:17.683715 2885 raft_consensus.cc:3060] T d31e548feae24b739087c1e211d514f9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
W20251029 02:52:17.683990 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 749f380eafae44ddbe2a244757f37909 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 749f380eafae44ddbe2a244757f37909 already in progress: creating tablet
I20251029 02:52:17.684695 2717 raft_consensus.cc:493] T 748777c6cd6e4f49ab49912b719e48bd P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c756528642a543eeacd93cef3e335a7d)
I20251029 02:52:17.684823 2717 raft_consensus.cc:515] T 748777c6cd6e4f49ab49912b719e48bd P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
W20251029 02:52:17.685348 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet d557a4750fee4c2b92eaa834619a678c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d557a4750fee4c2b92eaa834619a678c already in progress: creating tablet
I20251029 02:52:17.685604 2717 leader_election.cc:290] T 748777c6cd6e4f49ab49912b719e48bd P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:17.685629 772 raft_consensus.cc:2468] T eca016aed05e4ce09e1a2bf96e2286f5 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:17.686211 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "748777c6cd6e4f49ab49912b719e48bd" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:17.686417 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "748777c6cd6e4f49ab49912b719e48bd" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:17.687127 990 leader_election.cc:304] T 748777c6cd6e4f49ab49912b719e48bd P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5; no voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:17.687252 2885 raft_consensus.cc:515] T d31e548feae24b739087c1e211d514f9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:17.687500 2179 raft_consensus.cc:2749] T 748777c6cd6e4f49ab49912b719e48bd P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251029 02:52:17.688047 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d31e548feae24b739087c1e211d514f9" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
W20251029 02:52:17.688529 990 leader_election.cc:343] T d31e548feae24b739087c1e211d514f9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:17.689320 847 leader_election.cc:304] T eca016aed05e4ce09e1a2bf96e2286f5 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5
I20251029 02:52:17.689622 2649 raft_consensus.cc:2804] T eca016aed05e4ce09e1a2bf96e2286f5 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:17.689867 2649 raft_consensus.cc:697] T eca016aed05e4ce09e1a2bf96e2286f5 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:52:17.690173 2649 consensus_queue.cc:237] T eca016aed05e4ce09e1a2bf96e2286f5 P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:17.691663 2885 leader_election.cc:290] T d31e548feae24b739087c1e211d514f9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Requested vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:17.692067 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d31e548feae24b739087c1e211d514f9" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:17.692111 2747 raft_consensus.cc:493] T 7816556521144c928d7a02694e22e79f P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:17.692230 772 raft_consensus.cc:3060] T d31e548feae24b739087c1e211d514f9 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:17.692237 2747 raft_consensus.cc:515] T 7816556521144c928d7a02694e22e79f P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:17.692788 2747 leader_election.cc:290] T 7816556521144c928d7a02694e22e79f P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:17.693467 518 catalog_manager.cc:6194] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: default.Table_a9e50760813844af85a85c5f8d22f8e0 [id=74a2a41c908146ec8e16c6a3227b94f2] alter complete (version 1)
I20251029 02:52:17.693812 767 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7816556521144c928d7a02694e22e79f" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:17.694026 767 raft_consensus.cc:2468] T 7816556521144c928d7a02694e22e79f P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:17.694557 847 leader_election.cc:304] T 7816556521144c928d7a02694e22e79f P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:17.694825 2747 raft_consensus.cc:2804] T 7816556521144c928d7a02694e22e79f P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:17.694914 2747 raft_consensus.cc:493] T 7816556521144c928d7a02694e22e79f P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:17.694991 2747 raft_consensus.cc:3060] T 7816556521144c928d7a02694e22e79f P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:17.697664 772 raft_consensus.cc:2468] T d31e548feae24b739087c1e211d514f9 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 1.
I20251029 02:52:17.697741 2747 raft_consensus.cc:515] T 7816556521144c928d7a02694e22e79f P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:17.700165 2747 leader_election.cc:290] T 7816556521144c928d7a02694e22e79f P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:17.701004 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7816556521144c928d7a02694e22e79f" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:17.701223 772 raft_consensus.cc:3060] T 7816556521144c928d7a02694e22e79f P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:17.701633 519 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet b3dc6e55bb2e47bab7df5df62a1bcb51
I20251029 02:52:17.702064 519 catalog_manager.cc:6719] ReplaceTablet: tablet b3dc6e55bb2e47bab7df5df62a1bcb51 deleted and replaced by tablet 2d0a63df94364a19963005071b7395b2
I20251029 02:52:17.703526 891 tablet_service.cc:1552] Processing DeleteTablet for tablet b3dc6e55bb2e47bab7df5df62a1bcb51 with delete_type TABLET_DATA_DELETED (replaced by tablet 2d0a63df94364a19963005071b7395b2) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:17.693292 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7816556521144c928d7a02694e22e79f" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:17.703974 772 raft_consensus.cc:2468] T 7816556521144c928d7a02694e22e79f P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:17.697125 1040 tablet_service.cc:1552] Processing DeleteTablet for tablet d31e548feae24b739087c1e211d514f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:00 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:17.697264 1042 tablet_service.cc:1552] Processing DeleteTablet for tablet 84993b2f63e7421290a161dde4bb0607 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:57 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:17.697379 1027 tablet_service.cc:1552] Processing DeleteTablet for tablet 72f9d4408f384e7093316d85f092e1b2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:05 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:17.697440 1033 tablet_service.cc:1552] Processing DeleteTablet for tablet 20a762344c5f46f88481c4ac7ad6e43b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:08 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:17.697502 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet 372d93914abc41b38a702742b7db0bc2 with delete_type TABLET_DATA_DELETED (replaced by tablet dab6ed2266ed42508738399d388c4da3) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:17.697458 1039 tablet_service.cc:1552] Processing DeleteTablet for tablet 70b008011d5547bbaf3f45685394139f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:03 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:17.698028 989 leader_election.cc:304] T d31e548feae24b739087c1e211d514f9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters: c756528642a543eeacd93cef3e335a7d
I20251029 02:52:17.699409 1024 tablet_service.cc:1552] Processing DeleteTablet for tablet 59eee86f249e4d6a9d09d30276dcd659 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:56 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:17.699908 1059 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7816556521144c928d7a02694e22e79f" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:52:17.704712 1024 tablet_service.cc:1552] Processing DeleteTablet for tablet b3dc6e55bb2e47bab7df5df62a1bcb51 with delete_type TABLET_DATA_DELETED (replaced by tablet 2d0a63df94364a19963005071b7395b2) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:17.704759 2885 raft_consensus.cc:2804] T d31e548feae24b739087c1e211d514f9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:17.705351 750 tablet_service.cc:1552] Processing DeleteTablet for tablet b3dc6e55bb2e47bab7df5df62a1bcb51 with delete_type TABLET_DATA_DELETED (replaced by tablet 2d0a63df94364a19963005071b7395b2) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:17.705778 520 master_service.cc:954] ReplaceTablet: received request to replace tablet c108828dc9ca4c329dcb8a0240bf675f from {username='slave'} at 127.0.0.1:35314
W20251029 02:52:17.705775 849 leader_election.cc:343] T 7816556521144c928d7a02694e22e79f P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
W20251029 02:52:17.706089 849 leader_election.cc:343] T 7816556521144c928d7a02694e22e79f P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:17.707698 1176 tablet_replica.cc:333] T d31e548feae24b739087c1e211d514f9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:17.709831 847 leader_election.cc:304] T 7816556521144c928d7a02694e22e79f P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5
I20251029 02:52:17.710958 2747 raft_consensus.cc:2804] T 7816556521144c928d7a02694e22e79f P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:17.711249 2747 raft_consensus.cc:697] T 7816556521144c928d7a02694e22e79f P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:52:17.711516 2747 consensus_queue.cc:237] T 7816556521144c928d7a02694e22e79f P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:17.719074 2885 raft_consensus.cc:697] T d31e548feae24b739087c1e211d514f9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Becoming Leader. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Running, Role: LEADER
I20251029 02:52:17.722926 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_0847f9152d7041d7a2e814ad64f9d06b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:17.723752 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0847f9152d7041d7a2e814ad64f9d06b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:17.725255 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_d164b5f77ca746b28adb0849b5fb65fb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:17.725857 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d164b5f77ca746b28adb0849b5fb65fb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:17.727104 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_2c1c156e87f540629b084beda3c5768d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:17.727566 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2c1c156e87f540629b084beda3c5768d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:17.734607 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_57d5826b7f1a4a71923282eae27c2e14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:17.735203 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_57d5826b7f1a4a71923282eae27c2e14 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:17.737802 2885 consensus_queue.cc:237] T d31e548feae24b739087c1e211d514f9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:17.742765 2857 raft_consensus.cc:493] T c7b78fd458e4414b9fc3aa064a80b206 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:17.742964 2857 raft_consensus.cc:515] T c7b78fd458e4414b9fc3aa064a80b206 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:17.743652 2857 leader_election.cc:290] T c7b78fd458e4414b9fc3aa064a80b206 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:52:17.743506 521 catalog_manager.cc:5649] T 32519fbffcef44a890161e5f3043806c P b495a49349324f7d81908f2da700f0d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to b495a49349324f7d81908f2da700f0d9 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "b495a49349324f7d81908f2da700f0d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } health_report { overall_health: HEALTHY } } }
I20251029 02:52:17.743889 521 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet 32519fbffcef44a890161e5f3043806c (table default.Table_a78f5837df7b44e791631e421aa7de46 [id=b95fe1c7d6ff4519b90f993f1fa961fd]). Expected version 3 got 0
I20251029 02:52:17.744961 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c7b78fd458e4414b9fc3aa064a80b206" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:17.745556 711 leader_election.cc:343] T c7b78fd458e4414b9fc3aa064a80b206 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:17.744237 1059 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c7b78fd458e4414b9fc3aa064a80b206" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:52:17.747532 712 leader_election.cc:343] T c7b78fd458e4414b9fc3aa064a80b206 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:17.747653 712 leader_election.cc:304] T c7b78fd458e4414b9fc3aa064a80b206 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
W20251029 02:52:17.747786 523 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8471db993eab4fcb9e6f317d48ad8f80 consensus state has no leader
I20251029 02:52:17.748018 2857 raft_consensus.cc:2749] T c7b78fd458e4414b9fc3aa064a80b206 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 02:52:17.752213 522 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9099fb43dbec46cd83f2cb3f4796ecc6 consensus state has no leader
I20251029 02:52:17.754547 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_16362cebff0c4d19a3f59ae8529be683"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:17.755314 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_16362cebff0c4d19a3f59ae8529be683 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:17.758044 522 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_6870fd296d904fdcaeb6e01ed44d4f3d" } new_table_name: "default.Table_d1924caa29b84076b79a327fac47378e" modify_external_catalogs: true
I20251029 02:52:17.761330 524 catalog_manager.cc:6194] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: default.Table_96e017f445bc4f1fbebddf65c722b8de [id=98f28a15eced4860ac13a60abdc8e0e2] alter complete (version 3)
I20251029 02:52:17.765769 2857 raft_consensus.cc:493] T abf18e06df9b482a8b25c078b9495272 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:17.765951 2857 raft_consensus.cc:515] T abf18e06df9b482a8b25c078b9495272 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:17.766597 2857 leader_election.cc:290] T abf18e06df9b482a8b25c078b9495272 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:17.767225 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "abf18e06df9b482a8b25c078b9495272" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:17.767700 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "abf18e06df9b482a8b25c078b9495272" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:17.767822 712 leader_election.cc:343] T abf18e06df9b482a8b25c078b9495272 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
W20251029 02:52:17.768282 711 leader_election.cc:343] T abf18e06df9b482a8b25c078b9495272 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:17.768416 711 leader_election.cc:304] T abf18e06df9b482a8b25c078b9495272 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:17.768831 2857 raft_consensus.cc:2749] T abf18e06df9b482a8b25c078b9495272 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:17.775162 520 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet c108828dc9ca4c329dcb8a0240bf675f
I20251029 02:52:17.775635 520 catalog_manager.cc:6719] ReplaceTablet: tablet c108828dc9ca4c329dcb8a0240bf675f deleted and replaced by tablet daf4973adcce44d7a1d3d3489926a868
I20251029 02:52:17.775807 520 master_service.cc:954] ReplaceTablet: received request to replace tablet 0415b52fadf948238c4e83dc09000f96 from {username='slave'} at 127.0.0.1:35314
I20251029 02:52:17.777019 1031 tablet_service.cc:1552] Processing DeleteTablet for tablet c108828dc9ca4c329dcb8a0240bf675f with delete_type TABLET_DATA_DELETED (replaced by tablet daf4973adcce44d7a1d3d3489926a868) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:17.777443 750 tablet_service.cc:1552] Processing DeleteTablet for tablet c108828dc9ca4c329dcb8a0240bf675f with delete_type TABLET_DATA_DELETED (replaced by tablet daf4973adcce44d7a1d3d3489926a868) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:17.778111 891 tablet_service.cc:1552] Processing DeleteTablet for tablet c108828dc9ca4c329dcb8a0240bf675f with delete_type TABLET_DATA_DELETED (replaced by tablet daf4973adcce44d7a1d3d3489926a868) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:17.778290 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_bdff4e69029249389def3ceac417dfbc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:17.779006 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bdff4e69029249389def3ceac417dfbc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:17.780706 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_570d77b4786e4f80a120744a8e94abf0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:17.781373 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_570d77b4786e4f80a120744a8e94abf0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:17.785465 750 tablet_service.cc:1505] Processing CreateTablet for tablet 1256976d606e409b875d608d5c15528d (DEFAULT_TABLE table=default.Table_b5a3de99a1d84f62b07ded51499e5b82 [id=870e40e80dad4ddc94360e49759aa5c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:17.786569 738 tablet_service.cc:1505] Processing CreateTablet for tablet 81405b2f57d548d98c39317ee053176f (DEFAULT_TABLE table=default.Table_1d64090271db4acc8f23e7b4c0d4947c [id=391db7383c2941cdacbe573658e8eaa5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:17.787029 891 tablet_service.cc:1505] Processing CreateTablet for tablet 1256976d606e409b875d608d5c15528d (DEFAULT_TABLE table=default.Table_b5a3de99a1d84f62b07ded51499e5b82 [id=870e40e80dad4ddc94360e49759aa5c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:17.787070 1031 tablet_service.cc:1505] Processing CreateTablet for tablet 81405b2f57d548d98c39317ee053176f (DEFAULT_TABLE table=default.Table_1d64090271db4acc8f23e7b4c0d4947c [id=391db7383c2941cdacbe573658e8eaa5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:17.787434 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81405b2f57d548d98c39317ee053176f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:17.787451 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1256976d606e409b875d608d5c15528d. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:17.788323 849 consensus_peers.cc:597] T d29cc2a0c8b84e3ebadc7f571d666847 P c756528642a543eeacd93cef3e335a7d -> Peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Couldn't send request to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:17.812628 1176 raft_consensus.cc:2243] T d31e548feae24b739087c1e211d514f9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:17.813148 1176 pending_rounds.cc:70] T d31e548feae24b739087c1e211d514f9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Trying to abort 1 pending ops.
I20251029 02:52:17.813218 1176 pending_rounds.cc:77] T d31e548feae24b739087c1e211d514f9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215949159481425920 op_type: NO_OP noop_request { }
I20251029 02:52:17.813328 1176 raft_consensus.cc:2889] T d31e548feae24b739087c1e211d514f9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 02:52:17.813416 1176 raft_consensus.cc:2272] T d31e548feae24b739087c1e211d514f9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:17.822666 518 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_ff774195536549b3b3a651c34438f014" } modify_external_catalogs: true
I20251029 02:52:17.822901 518 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_ff774195536549b3b3a651c34438f014" } modify_external_catalogs: true
I20251029 02:52:17.824928 525 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_47178701cd044f8fa5357cd607154988" } modify_external_catalogs: true
I20251029 02:52:17.825743 525 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_47178701cd044f8fa5357cd607154988" } modify_external_catalogs: true
I20251029 02:52:17.827060 2747 raft_consensus.cc:493] T f93b0c39d7174dabbe018b1546dbc198 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:17.827278 2747 raft_consensus.cc:515] T f93b0c39d7174dabbe018b1546dbc198 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:17.828641 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f93b0c39d7174dabbe018b1546dbc198" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:52:17.829232 849 leader_election.cc:343] T f93b0c39d7174dabbe018b1546dbc198 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:17.829602 2747 leader_election.cc:290] T f93b0c39d7174dabbe018b1546dbc198 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:17.830647 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f93b0c39d7174dabbe018b1546dbc198" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:17.830868 772 raft_consensus.cc:2468] T f93b0c39d7174dabbe018b1546dbc198 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:17.831133 2842 tablet.cc:1722] T 6f35412a66b74051a475dce5dafa7792 P c756528642a543eeacd93cef3e335a7d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 02:52:17.831615 2763 tablet.cc:1722] T 6f35412a66b74051a475dce5dafa7792 P b495a49349324f7d81908f2da700f0d9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 02:52:17.833002 847 leader_election.cc:304] T f93b0c39d7174dabbe018b1546dbc198 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5
I20251029 02:52:17.833521 2649 raft_consensus.cc:2804] T f93b0c39d7174dabbe018b1546dbc198 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:17.833624 2649 raft_consensus.cc:493] T f93b0c39d7174dabbe018b1546dbc198 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:17.833709 2649 raft_consensus.cc:3060] T f93b0c39d7174dabbe018b1546dbc198 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:17.833775 1175 ts_tablet_manager.cc:1929] T 72f9d4408f384e7093316d85f092e1b2 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:17.833882 1175 log.cc:1199] T 72f9d4408f384e7093316d85f092e1b2 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/72f9d4408f384e7093316d85f092e1b2
I20251029 02:52:17.834256 1175 ts_tablet_manager.cc:1950] T 72f9d4408f384e7093316d85f092e1b2 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:17.836395 2649 raft_consensus.cc:515] T f93b0c39d7174dabbe018b1546dbc198 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:17.837517 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f93b0c39d7174dabbe018b1546dbc198" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:52:17.838048 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1256976d606e409b875d608d5c15528d. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:17.838160 849 leader_election.cc:343] T f93b0c39d7174dabbe018b1546dbc198 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:17.838387 1176 ts_tablet_manager.cc:1916] T d31e548feae24b739087c1e211d514f9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:17.838479 2649 leader_election.cc:290] T f93b0c39d7174dabbe018b1546dbc198 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:17.839227 1172 raft_consensus.cc:385] T 2b278db5b2b04344b864797055a5f665 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:17.839367 1172 raft_consensus.cc:740] T 2b278db5b2b04344b864797055a5f665 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:17.839632 1172 consensus_queue.cc:260] T 2b278db5b2b04344b864797055a5f665 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:17.840260 1172 ts_tablet_manager.cc:1434] T 2b278db5b2b04344b864797055a5f665 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.232s user 0.001s sys 0.001s
I20251029 02:52:17.840559 1172 tablet_bootstrap.cc:492] T 023a4094b424464c8f49d6eb9cc15da4 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:17.842749 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81405b2f57d548d98c39317ee053176f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:17.842844 1172 tablet_bootstrap.cc:654] T 023a4094b424464c8f49d6eb9cc15da4 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:17.842959 523 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet 748777c6cd6e4f49ab49912b719e48bd (table default.Table_1e69e259edba466f99ed7a41dcf6367f [id=a2f3b36dc78a4e94bd054a6a5712b384]). Expected version 1 got 0
I20251029 02:52:17.844713 1176 ts_tablet_manager.cc:1929] T d31e548feae24b739087c1e211d514f9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:17.844810 1176 log.cc:1199] T d31e548feae24b739087c1e211d514f9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/d31e548feae24b739087c1e211d514f9
I20251029 02:52:17.845136 1176 ts_tablet_manager.cc:1950] T d31e548feae24b739087c1e211d514f9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:17.845561 1176 tablet_replica.cc:333] T 84993b2f63e7421290a161dde4bb0607 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:17.845697 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet d31e548feae24b739087c1e211d514f9 (table default.Table_5a915593503d490fa12b10fccd76ac3e [id=6fd26d78f4904f30a2cb76e9a4bec2db]) successfully deleted
I20251029 02:52:17.846063 1176 raft_consensus.cc:2243] T 84993b2f63e7421290a161dde4bb0607 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:17.846240 1176 raft_consensus.cc:2272] T 84993b2f63e7421290a161dde4bb0607 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:17.847955 517 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_a5ee816bf96c4eb891cb8ba5ed50f3ed" } modify_external_catalogs: true
I20251029 02:52:17.848117 517 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_a5ee816bf96c4eb891cb8ba5ed50f3ed" } modify_external_catalogs: true
I20251029 02:52:17.849232 1176 ts_tablet_manager.cc:1916] T 84993b2f63e7421290a161dde4bb0607 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 02:52:17.850354 522 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8616fa24a54147b684f50d373a383c2b consensus state has no leader
I20251029 02:52:17.851248 522 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_f7fa4c862f364b6eb8510a09d693001b" } modify_external_catalogs: true
I20251029 02:52:17.851357 891 tablet_service.cc:1505] Processing CreateTablet for tablet 81405b2f57d548d98c39317ee053176f (DEFAULT_TABLE table=default.Table_1d64090271db4acc8f23e7b4c0d4947c [id=391db7383c2941cdacbe573658e8eaa5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:17.851496 522 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_f7fa4c862f364b6eb8510a09d693001b" } modify_external_catalogs: true
I20251029 02:52:17.851804 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 81405b2f57d548d98c39317ee053176f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:17.852037 1024 tablet_service.cc:1505] Processing CreateTablet for tablet a342154638a041c2aecf43b783e0a796 (DEFAULT_TABLE table=default.Table_6206526535a84403a4b5e8c8ad05887b [id=d3ab64c81f254ce08ebf03c286d43694]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:17.852406 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a342154638a041c2aecf43b783e0a796. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:17.853188 766 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f93b0c39d7174dabbe018b1546dbc198" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:17.853370 766 raft_consensus.cc:3060] T f93b0c39d7174dabbe018b1546dbc198 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
W20251029 02:52:17.854012 847 consensus_peers.cc:597] T f281191e7c6143f983f11d87f1c9b64e P c756528642a543eeacd93cef3e335a7d -> Peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Couldn't send request to peer b495a49349324f7d81908f2da700f0d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f281191e7c6143f983f11d87f1c9b64e. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:17.861330 766 raft_consensus.cc:2468] T f93b0c39d7174dabbe018b1546dbc198 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:17.865079 847 leader_election.cc:304] T f93b0c39d7174dabbe018b1546dbc198 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5
I20251029 02:52:17.865594 2649 raft_consensus.cc:2804] T f93b0c39d7174dabbe018b1546dbc198 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:17.865864 2649 raft_consensus.cc:697] T f93b0c39d7174dabbe018b1546dbc198 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:52:17.866181 2649 consensus_queue.cc:237] T f93b0c39d7174dabbe018b1546dbc198 P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:17.866945 2847 tablet.cc:1722] T 6f35412a66b74051a475dce5dafa7792 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 02:52:17.873929 750 tablet_service.cc:1505] Processing CreateTablet for tablet a342154638a041c2aecf43b783e0a796 (DEFAULT_TABLE table=default.Table_6206526535a84403a4b5e8c8ad05887b [id=d3ab64c81f254ce08ebf03c286d43694]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:17.874770 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a342154638a041c2aecf43b783e0a796. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:17.874807 1024 tablet_service.cc:1505] Processing CreateTablet for tablet d31ac6638b424e1786755ed2dd9e4b64 (DEFAULT_TABLE table=default.Table_8c6b8330348d403c92b6e2da82211db8 [id=65f83da5d3d6430884136239baeddd9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:17.875141 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d31ac6638b424e1786755ed2dd9e4b64. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:17.881322 1176 ts_tablet_manager.cc:1929] T 84993b2f63e7421290a161dde4bb0607 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:17.881434 1176 log.cc:1199] T 84993b2f63e7421290a161dde4bb0607 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/84993b2f63e7421290a161dde4bb0607
I20251029 02:52:17.882339 891 tablet_service.cc:1505] Processing CreateTablet for tablet a342154638a041c2aecf43b783e0a796 (DEFAULT_TABLE table=default.Table_6206526535a84403a4b5e8c8ad05887b [id=d3ab64c81f254ce08ebf03c286d43694]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:17.882557 524 catalog_manager.cc:6194] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: default.Table_a9c2f9e099c14953bbce19d79154eb31 [id=f4e028bdd23f4eea975f940d0b2c73f6] alter complete (version 1)
I20251029 02:52:17.882803 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a342154638a041c2aecf43b783e0a796. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:17.886296 879 tablet_service.cc:1552] Processing DeleteTablet for tablet 59d03a190b784ab18a3e63a572675891 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:57 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:17.886349 895 tablet_service.cc:1552] Processing DeleteTablet for tablet 628a7d0419cb434f8171faf500d585d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:02 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:17.886476 887 tablet_service.cc:1552] Processing DeleteTablet for tablet fcb3db8399aa4cf188b07dda220f7f5b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:59 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:17.886569 888 tablet_service.cc:1552] Processing DeleteTablet for tablet 23b9b2ea814645eba63e5f6b12cfb966 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:57 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:17.886739 900 tablet_service.cc:1552] Processing DeleteTablet for tablet 7816556521144c928d7a02694e22e79f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:58 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:17.887938 1176 ts_tablet_manager.cc:1950] T 84993b2f63e7421290a161dde4bb0607 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:17.888646 1039 tablet_service.cc:1552] Processing DeleteTablet for tablet 589a521f5cc24e6289c37b9cf256bce5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:57 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:17.888736 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 84993b2f63e7421290a161dde4bb0607 (table default.Table_93b405a4f257481289518c5ff28832a0 [id=da230f99a9894ab8aa2fdb4b167169ad]) successfully deleted
I20251029 02:52:17.889148 1039 tablet_service.cc:1552] Processing DeleteTablet for tablet ad214c870ee649279ed7321c5b69cbee with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:05 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:17.891424 738 tablet_service.cc:1552] Processing DeleteTablet for tablet c7b78fd458e4414b9fc3aa064a80b206 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:01 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:17.895972 1176 tablet_replica.cc:333] T 72f9d4408f384e7093316d85f092e1b2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:17.896170 1176 raft_consensus.cc:2243] T 72f9d4408f384e7093316d85f092e1b2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:17.896329 1176 raft_consensus.cc:2272] T 72f9d4408f384e7093316d85f092e1b2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:17.900105 1176 ts_tablet_manager.cc:1916] T 72f9d4408f384e7093316d85f092e1b2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:17.907342 1166 tablet_bootstrap.cc:492] T c081d2eacfac4ca8a0048e2a1c4f8f83 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:17.907570 1166 ts_tablet_manager.cc:1403] T c081d2eacfac4ca8a0048e2a1c4f8f83 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.266s user 0.067s sys 0.070s
I20251029 02:52:17.907582 1039 tablet_service.cc:1505] Processing CreateTablet for tablet c5b3edb898544f9eb6b501e20adbdd4b (DEFAULT_TABLE table=default.Table_00585488392141a58ecfc968a6f3ca61 [id=cd4b3685ce704f098c62b57b104877eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:17.908183 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5b3edb898544f9eb6b501e20adbdd4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:17.908552 1166 raft_consensus.cc:359] T c081d2eacfac4ca8a0048e2a1c4f8f83 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:17.908818 1166 raft_consensus.cc:385] T c081d2eacfac4ca8a0048e2a1c4f8f83 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:17.908928 1166 raft_consensus.cc:740] T c081d2eacfac4ca8a0048e2a1c4f8f83 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:17.909317 1166 consensus_queue.cc:260] T c081d2eacfac4ca8a0048e2a1c4f8f83 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:17.909878 1166 ts_tablet_manager.cc:1434] T c081d2eacfac4ca8a0048e2a1c4f8f83 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 02:52:17.910586 1166 tablet_bootstrap.cc:492] T 6f8cf09f23984b879b60ab6b27c7d52e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:17.911208 900 tablet_service.cc:1505] Processing CreateTablet for tablet d31ac6638b424e1786755ed2dd9e4b64 (DEFAULT_TABLE table=default.Table_8c6b8330348d403c92b6e2da82211db8 [id=65f83da5d3d6430884136239baeddd9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:17.911592 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d31ac6638b424e1786755ed2dd9e4b64. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:17.911610 2847 tablet.cc:1722] T aacb6f4b9284457bb6607e3a85d5c809 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20251029 02:52:17.913658 738 tablet_service.cc:1505] Processing CreateTablet for tablet d31ac6638b424e1786755ed2dd9e4b64 (DEFAULT_TABLE table=default.Table_8c6b8330348d403c92b6e2da82211db8 [id=65f83da5d3d6430884136239baeddd9a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:17.914036 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d31ac6638b424e1786755ed2dd9e4b64. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:17.917775 2763 tablet.cc:1722] T aacb6f4b9284457bb6607e3a85d5c809 P b495a49349324f7d81908f2da700f0d9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20251029 02:52:17.918340 2655 raft_consensus.cc:493] T 70b008011d5547bbaf3f45685394139f P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:17.919761 1166 tablet_bootstrap.cc:654] T 6f8cf09f23984b879b60ab6b27c7d52e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:17.920594 2655 raft_consensus.cc:515] T 70b008011d5547bbaf3f45685394139f P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:17.921244 2655 leader_election.cc:290] T 70b008011d5547bbaf3f45685394139f P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:17.922574 900 tablet_service.cc:1505] Processing CreateTablet for tablet c5b3edb898544f9eb6b501e20adbdd4b (DEFAULT_TABLE table=default.Table_00585488392141a58ecfc968a6f3ca61 [id=cd4b3685ce704f098c62b57b104877eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:17.922928 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5b3edb898544f9eb6b501e20adbdd4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:17.924414 888 tablet_service.cc:1505] Processing CreateTablet for tablet 9e0d410678594cbaa78bd7f56c3cb40e (DEFAULT_TABLE table=default.Table_1aa03bfd183b40188824e6368f5746ee [id=b75b72fea87e458f8a5a943a65d3b01f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:17.924762 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e0d410678594cbaa78bd7f56c3cb40e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:17.926318 766 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "70b008011d5547bbaf3f45685394139f" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:17.926945 766 raft_consensus.cc:2393] T 70b008011d5547bbaf3f45685394139f P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c756528642a543eeacd93cef3e335a7d in current term 1: Already voted for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in this term.
I20251029 02:52:17.927881 748 tablet_service.cc:1505] Processing CreateTablet for tablet c5b3edb898544f9eb6b501e20adbdd4b (DEFAULT_TABLE table=default.Table_00585488392141a58ecfc968a6f3ca61 [id=cd4b3685ce704f098c62b57b104877eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:17.928222 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5b3edb898544f9eb6b501e20adbdd4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:17.930781 742 tablet_service.cc:1505] Processing CreateTablet for tablet 9e0d410678594cbaa78bd7f56c3cb40e (DEFAULT_TABLE table=default.Table_1aa03bfd183b40188824e6368f5746ee [id=b75b72fea87e458f8a5a943a65d3b01f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:17.930846 1039 tablet_service.cc:1505] Processing CreateTablet for tablet 9e0d410678594cbaa78bd7f56c3cb40e (DEFAULT_TABLE table=default.Table_1aa03bfd183b40188824e6368f5746ee [id=b75b72fea87e458f8a5a943a65d3b01f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:17.931166 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e0d410678594cbaa78bd7f56c3cb40e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:17.934947 2842 tablet.cc:1722] T 748777c6cd6e4f49ab49912b719e48bd P c756528642a543eeacd93cef3e335a7d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 02:52:17.941251 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_cd0e071b233b4d45b36009940c6a0084"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:17.941881 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cd0e071b233b4d45b36009940c6a0084 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:17.943810 523 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_7eb1d904f38342dc85bb06681099d741" } modify_external_catalogs: true
I20251029 02:52:17.944092 523 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_7eb1d904f38342dc85bb06681099d741" } modify_external_catalogs: true
I20251029 02:52:17.944687 2847 tablet.cc:1722] T 748777c6cd6e4f49ab49912b719e48bd P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 02:52:17.945363 520 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 0415b52fadf948238c4e83dc09000f96
I20251029 02:52:17.945808 520 catalog_manager.cc:6719] ReplaceTablet: tablet 0415b52fadf948238c4e83dc09000f96 deleted and replaced by tablet 3a1c62fe831b4887a479d0794efbe81d
I20251029 02:52:17.946121 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet 0415b52fadf948238c4e83dc09000f96 with delete_type TABLET_DATA_DELETED (replaced by tablet 3a1c62fe831b4887a479d0794efbe81d) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:17.951014 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e0d410678594cbaa78bd7f56c3cb40e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:17.952801 518 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet e2a739ebb4aa446fba435a9aecd28ed5
I20251029 02:52:17.954476 519 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_0b08a2dc1da148a9973a790b325f302a" } new_table_name: "default.Table_da5d504aece34c94801d0c2926774c6a" modify_external_catalogs: true
I20251029 02:52:17.954856 742 tablet_service.cc:1552] Processing DeleteTablet for tablet 0415b52fadf948238c4e83dc09000f96 with delete_type TABLET_DATA_DELETED (replaced by tablet 3a1c62fe831b4887a479d0794efbe81d) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:17.956775 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet e2a739ebb4aa446fba435a9aecd28ed5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:17 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:17.957159 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_cc7a37c88eb54521b208967338567b54"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:17.957641 1176 ts_tablet_manager.cc:1929] T 72f9d4408f384e7093316d85f092e1b2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:17.957743 1176 log.cc:1199] T 72f9d4408f384e7093316d85f092e1b2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/72f9d4408f384e7093316d85f092e1b2
I20251029 02:52:17.957739 888 tablet_service.cc:1552] Processing DeleteTablet for tablet e2a739ebb4aa446fba435a9aecd28ed5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:17 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:17.958104 1176 ts_tablet_manager.cc:1950] T 72f9d4408f384e7093316d85f092e1b2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:17.958691 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 72f9d4408f384e7093316d85f092e1b2 (table default.Table_8a277d5e05fb437096e36f29a6307d74 [id=e5c07dcc74d1493fa9e1ef153b10d87b]) successfully deleted
I20251029 02:52:17.958989 900 tablet_service.cc:1552] Processing DeleteTablet for tablet 0415b52fadf948238c4e83dc09000f96 with delete_type TABLET_DATA_DELETED (replaced by tablet 3a1c62fe831b4887a479d0794efbe81d) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:17.963677 1058 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "70b008011d5547bbaf3f45685394139f" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:17.966552 849 leader_election.cc:304] T 70b008011d5547bbaf3f45685394139f P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9
W20251029 02:52:17.967067 513 catalog_manager.cc:4952] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 20a762344c5f46f88481c4ac7ad6e43b because the tablet was not found. No further retry: Not found: Tablet not found: 20a762344c5f46f88481c4ac7ad6e43b
I20251029 02:52:17.967125 2426 raft_consensus.cc:3060] T 70b008011d5547bbaf3f45685394139f P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:17.967753 1176 tablet_replica.cc:333] T 372d93914abc41b38a702742b7db0bc2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:17.968173 1176 raft_consensus.cc:2243] T 372d93914abc41b38a702742b7db0bc2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:17.968730 1176 raft_consensus.cc:2272] T 372d93914abc41b38a702742b7db0bc2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:17.970021 1176 ts_tablet_manager.cc:1916] T 372d93914abc41b38a702742b7db0bc2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:17.975548 2717 raft_consensus.cc:493] T c081d2eacfac4ca8a0048e2a1c4f8f83 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:17.975677 2717 raft_consensus.cc:515] T c081d2eacfac4ca8a0048e2a1c4f8f83 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:17.976272 2717 leader_election.cc:290] T c081d2eacfac4ca8a0048e2a1c4f8f83 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:17.976886 919 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c081d2eacfac4ca8a0048e2a1c4f8f83" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:17.978121 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c081d2eacfac4ca8a0048e2a1c4f8f83" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:17.978981 1031 tablet_service.cc:1505] Processing CreateTablet for tablet afd3fb2157c2445bbe8dbf0f28089c2e (DEFAULT_TABLE table=default.Table_9dbf9a53f1e34611976c67bff0ac7f9f [id=1398a570e3c9485db660ecc3e1c4d887]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:17.979398 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet afd3fb2157c2445bbe8dbf0f28089c2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:17.982720 990 leader_election.cc:304] T c081d2eacfac4ca8a0048e2a1c4f8f83 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5; no voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:17.983245 2717 raft_consensus.cc:2749] T c081d2eacfac4ca8a0048e2a1c4f8f83 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:17.986362 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 1047) took 4335 ms. Trace:
I20251029 02:52:17.986581 1175 rpcz_store.cc:276] 1029 02:52:13.650725 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:13.650780 (+ 55us) service_pool.cc:225] Handling call
1029 02:52:17.594981 (+3944201us) ts_tablet_manager.cc:1151] Deleting tablet 72f9d4408f384e7093316d85f092e1b2
1029 02:52:17.594998 (+ 17us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:17.830662 (+235664us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:17.833649 (+ 2987us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:17.986322 (+152673us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3944031}
I20251029 02:52:17.987854 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 1049) took 4336 ms. Trace:
I20251029 02:52:17.988186 1175 rpcz_store.cc:276] 1029 02:52:13.650940 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:13.651002 (+ 62us) service_pool.cc:225] Handling call
1029 02:52:17.987574 (+4336572us) ts_tablet_manager.cc:1151] Deleting tablet 8048506ced4540a5aee9f79bfc4599bf
1029 02:52:17.987586 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:17.987780 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4336355}
I20251029 02:52:17.989459 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 1046) took 4338 ms. Trace:
I20251029 02:52:17.989661 1175 rpcz_store.cc:276] 1029 02:52:13.650627 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:13.651115 (+ 488us) service_pool.cc:225] Handling call
1029 02:52:17.989190 (+4338075us) ts_tablet_manager.cc:1151] Deleting tablet 44ae2b07edca44afbe1b1fb3a5b1115d
1029 02:52:17.989209 (+ 19us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:17.989352 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4337851}
I20251029 02:52:17.992856 525 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 82f83631253a4b5785c814c993678d68
I20251029 02:52:17.994184 2857 consensus_queue.cc:1048] T 32519fbffcef44a890161e5f3043806c P b495a49349324f7d81908f2da700f0d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:17.995399 900 tablet_service.cc:1552] Processing DeleteTablet for tablet 82f83631253a4b5785c814c993678d68 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:17 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:18.001905 1031 tablet_service.cc:1552] Processing DeleteTablet for tablet 82f83631253a4b5785c814c993678d68 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:17 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:18.002445 517 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 48e74fe2c98f4df5ac686a4405c4521b
I20251029 02:52:18.003495 1031 tablet_service.cc:1552] Processing DeleteTablet for tablet 48e74fe2c98f4df5ac686a4405c4521b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:17 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:18.004415 1059 raft_consensus.cc:3060] T defb795026a44e12afb2defb9c9c6db2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
W20251029 02:52:18.006072 847 consensus_peers.cc:597] T 46cd75ff18b14212b556b9bcb3012dd9 P c756528642a543eeacd93cef3e335a7d -> Peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Couldn't send request to peer b495a49349324f7d81908f2da700f0d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 46cd75ff18b14212b556b9bcb3012dd9. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:18.006745 2426 raft_consensus.cc:2749] T 70b008011d5547bbaf3f45685394139f P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:18.009282 2842 tablet.cc:1722] T aacb6f4b9284457bb6607e3a85d5c809 P c756528642a543eeacd93cef3e335a7d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20251029 02:52:18.009581 2655 consensus_queue.cc:1048] T defb795026a44e12afb2defb9c9c6db2 P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:18.020680 1175 tablet_replica.cc:333] T 6142018e44af4d32818adefa2ee43d02 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:18.025189 2763 tablet.cc:1722] T 748777c6cd6e4f49ab49912b719e48bd P b495a49349324f7d81908f2da700f0d9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20251029 02:52:18.025595 1175 raft_consensus.cc:2243] T 6142018e44af4d32818adefa2ee43d02 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:18.029556 1176 ts_tablet_manager.cc:1929] T 372d93914abc41b38a702742b7db0bc2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:18.029690 1176 log.cc:1199] T 372d93914abc41b38a702742b7db0bc2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/372d93914abc41b38a702742b7db0bc2
I20251029 02:52:18.030059 1176 ts_tablet_manager.cc:1950] T 372d93914abc41b38a702742b7db0bc2 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:18.031023 522 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 0d788aeac52047c6bd9edcaf1369c35c
I20251029 02:52:18.031809 1031 tablet_service.cc:1552] Processing DeleteTablet for tablet 0d788aeac52047c6bd9edcaf1369c35c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:17 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:18.032019 900 tablet_service.cc:1552] Processing DeleteTablet for tablet 0d788aeac52047c6bd9edcaf1369c35c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:17 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:18.034477 524 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_79052b7cce26442aa1454f10280e7e7a" } new_table_name: "default.Table_98d6dc24d373463dab3125992ea24306" modify_external_catalogs: true
W20251029 02:52:18.036482 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.3:45003, user_credentials={real_user=slave}} blocked reactor thread for 115316us
I20251029 02:52:18.038456 1175 raft_consensus.cc:2272] T 6142018e44af4d32818adefa2ee43d02 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251029 02:52:18.043289 1167 thread.cc:641] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.548s user 0.048s sys 0.074s
W20251029 02:52:18.043419 1167 thread.cc:608] log-alloc [worker] (thread pool) Time spent starting thread: real 0.548s user 0.048s sys 0.074s
I20251029 02:52:18.045979 1167 tablet_bootstrap.cc:492] T 589a521f5cc24e6289c37b9cf256bce5 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:18.046144 1167 ts_tablet_manager.cc:1403] T 589a521f5cc24e6289c37b9cf256bce5 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.553s user 0.050s sys 0.074s
I20251029 02:52:18.046993 1167 raft_consensus.cc:359] T 589a521f5cc24e6289c37b9cf256bce5 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:18.047201 1167 raft_consensus.cc:385] T 589a521f5cc24e6289c37b9cf256bce5 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:18.047303 1167 raft_consensus.cc:740] T 589a521f5cc24e6289c37b9cf256bce5 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:18.047572 1167 consensus_queue.cc:260] T 589a521f5cc24e6289c37b9cf256bce5 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:18.048126 1167 ts_tablet_manager.cc:1434] T 589a521f5cc24e6289c37b9cf256bce5 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 02:52:18.048588 1167 tablet_bootstrap.cc:492] T abf18e06df9b482a8b25c078b9495272 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:18.051555 1167 tablet_bootstrap.cc:654] T abf18e06df9b482a8b25c078b9495272 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:18.052227 1175 ts_tablet_manager.cc:1916] T 6142018e44af4d32818adefa2ee43d02 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:18.061293 1174 ts_tablet_manager.cc:1929] T 7a1866afc7564217ad777e7ce53786c5 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:18.061430 1174 log.cc:1199] T 7a1866afc7564217ad777e7ce53786c5 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/7a1866afc7564217ad777e7ce53786c5
I20251029 02:52:18.061774 1174 ts_tablet_manager.cc:1950] T 7a1866afc7564217ad777e7ce53786c5 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:18.062577 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 957) took 7407 ms. Trace:
I20251029 02:52:18.062713 1174 rpcz_store.cc:276] 1029 02:52:10.654890 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:10.655682 (+ 792us) service_pool.cc:225] Handling call
1029 02:52:17.486006 (+6830324us) ts_tablet_manager.cc:1151] Deleting tablet 7a1866afc7564217ad777e7ce53786c5
1029 02:52:17.486016 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:17.819895 (+333879us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:18.061191 (+241296us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:18.062490 (+ 1299us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6830154}
I20251029 02:52:18.063150 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 953) took 7431 ms. Trace:
I20251029 02:52:18.063235 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 7a1866afc7564217ad777e7ce53786c5 (table default.Table_46e2f96551d3497c919e1cab3521d63b [id=09ee821707fa4082889bfccb8667b0b7]) successfully deleted
I20251029 02:52:18.063246 1174 rpcz_store.cc:276] 1029 02:52:10.631398 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:10.658567 (+ 27169us) service_pool.cc:225] Handling call
1029 02:52:18.063002 (+7404435us) ts_tablet_manager.cc:1151] Deleting tablet 862d268a526545ba8f25bb91c2992321
1029 02:52:18.063015 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:18.063115 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7400042}
I20251029 02:52:18.063536 1174 tablet_replica.cc:333] T 6d830d30b26b4ba799785fec8a6f0769 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:18.063724 1174 raft_consensus.cc:2243] T 6d830d30b26b4ba799785fec8a6f0769 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus shutting down.
W20251029 02:52:18.063771 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 862d268a526545ba8f25bb91c2992321 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 862d268a526545ba8f25bb91c2992321 already in progress: creating tablet
I20251029 02:52:18.063902 1174 raft_consensus.cc:2272] T 6d830d30b26b4ba799785fec8a6f0769 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:18.064450 1174 ts_tablet_manager.cc:1916] T 6d830d30b26b4ba799785fec8a6f0769 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:18.075992 821 heartbeater.cc:344] Connected to a master server at 127.0.104.62:42343
I20251029 02:52:18.085620 2908 consensus_queue.cc:1048] T 32519fbffcef44a890161e5f3043806c P b495a49349324f7d81908f2da700f0d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:18.086397 523 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 9542585fd9824a55b3ad2f120065a02d
I20251029 02:52:18.087697 1024 tablet_service.cc:1552] Processing DeleteTablet for tablet 9542585fd9824a55b3ad2f120065a02d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:17 UTC) from {username='slave'} at 127.0.0.1:37626
W20251029 02:52:18.088891 849 consensus_peers.cc:597] T 84993b2f63e7421290a161dde4bb0607 P c756528642a543eeacd93cef3e335a7d -> Peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Couldn't send request to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 84993b2f63e7421290a161dde4bb0607. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:18.080544 1172 tablet_bootstrap.cc:492] T 023a4094b424464c8f49d6eb9cc15da4 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:18.093856 2201 consensus_queue.cc:1048] T defb795026a44e12afb2defb9c9c6db2 P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:18.096225 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 372d93914abc41b38a702742b7db0bc2 (table default.Table_4e9d94609b4a4d1f94615dcc9830d717 [id=761ebdd0095c414488aa221cbc99d3bc]) successfully deleted
I20251029 02:52:18.096643 1176 tablet_replica.cc:333] T 70b008011d5547bbaf3f45685394139f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:18.096683 1172 ts_tablet_manager.cc:1403] T 023a4094b424464c8f49d6eb9cc15da4 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.256s user 0.004s sys 0.000s
I20251029 02:52:18.096846 1176 raft_consensus.cc:2243] T 70b008011d5547bbaf3f45685394139f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:18.097074 1176 pending_rounds.cc:70] T 70b008011d5547bbaf3f45685394139f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Trying to abort 1 pending ops.
I20251029 02:52:18.097119 1176 pending_rounds.cc:77] T 70b008011d5547bbaf3f45685394139f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215949157516087296 op_type: NO_OP noop_request { }
I20251029 02:52:18.097206 1176 raft_consensus.cc:2889] T 70b008011d5547bbaf3f45685394139f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 02:52:18.097296 1176 raft_consensus.cc:2272] T 70b008011d5547bbaf3f45685394139f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:18.097945 1172 raft_consensus.cc:359] T 023a4094b424464c8f49d6eb9cc15da4 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:18.098348 1172 raft_consensus.cc:385] T 023a4094b424464c8f49d6eb9cc15da4 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:18.098661 1172 raft_consensus.cc:740] T 023a4094b424464c8f49d6eb9cc15da4 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:18.098621 1176 ts_tablet_manager.cc:1916] T 70b008011d5547bbaf3f45685394139f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 02:52:18.101796 519 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cb9856e5fe244d3e91f30f5e1d276d18 consensus state has no leader
I20251029 02:52:18.109268 1176 ts_tablet_manager.cc:1929] T 70b008011d5547bbaf3f45685394139f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:18.109393 1176 log.cc:1199] T 70b008011d5547bbaf3f45685394139f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/70b008011d5547bbaf3f45685394139f
I20251029 02:52:18.109732 1176 ts_tablet_manager.cc:1950] T 70b008011d5547bbaf3f45685394139f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:18.110309 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 70b008011d5547bbaf3f45685394139f (table default.Table_0bb20faeb14644ca8e2d5acd4af09f85 [id=d39d7d159c7d46b496caa09226366e1f]) successfully deleted
I20251029 02:52:18.098629 1166 tablet_bootstrap.cc:492] T 6f8cf09f23984b879b60ab6b27c7d52e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:18.118706 1166 ts_tablet_manager.cc:1403] T 6f8cf09f23984b879b60ab6b27c7d52e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.208s user 0.004s sys 0.013s
I20251029 02:52:18.099573 1172 consensus_queue.cc:260] T 023a4094b424464c8f49d6eb9cc15da4 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:18.119508 1166 raft_consensus.cc:359] T 6f8cf09f23984b879b60ab6b27c7d52e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:18.119666 1166 raft_consensus.cc:385] T 6f8cf09f23984b879b60ab6b27c7d52e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:18.119719 1166 raft_consensus.cc:740] T 6f8cf09f23984b879b60ab6b27c7d52e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
W20251029 02:52:18.119937 524 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2965344ff7174295b7759d1b2c95ff87 consensus state has no leader
I20251029 02:52:18.119927 1166 consensus_queue.cc:260] T 6f8cf09f23984b879b60ab6b27c7d52e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:18.122931 1166 ts_tablet_manager.cc:1434] T 6f8cf09f23984b879b60ab6b27c7d52e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20251029 02:52:18.123234 1166 tablet_bootstrap.cc:492] T 23b9b2ea814645eba63e5f6b12cfb966 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:18.123208 524 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_b96dfc0b5a574a6ab3f16335d08a7f50" } new_table_name: "default.Table_1922b14d4cfb4671a6224fcd6ace2d4e" modify_external_catalogs: true
I20251029 02:52:18.123513 1172 ts_tablet_manager.cc:1434] T 023a4094b424464c8f49d6eb9cc15da4 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.026s user 0.001s sys 0.001s
I20251029 02:52:18.118049 521 catalog_manager.cc:5649] T f4729ecd182042c2a10f67ca25224101 P c756528642a543eeacd93cef3e335a7d reported cstate change: term changed from 0 to 1, leader changed from <none> to c756528642a543eeacd93cef3e335a7d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "c756528642a543eeacd93cef3e335a7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } health_report { overall_health: UNKNOWN } } }
I20251029 02:52:18.124482 1172 tablet_bootstrap.cc:492] T f18b0ca5a6a44175a912642e9eea1778 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:18.124814 1166 tablet_bootstrap.cc:654] T 23b9b2ea814645eba63e5f6b12cfb966 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:18.127221 521 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet f4729ecd182042c2a10f67ca25224101 (table default.Table_973ef856041949598a23da3dab8b40db [id=1dd2d5488bc647c8b82c53886583b28d]). Expected version 4 got 0
W20251029 02:52:18.130286 513 catalog_manager.cc:4952] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 59eee86f249e4d6a9d09d30276dcd659 because the tablet was not found. No further retry: Not found: Tablet not found: 59eee86f249e4d6a9d09d30276dcd659
W20251029 02:52:18.130905 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet b3dc6e55bb2e47bab7df5df62a1bcb51 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b3dc6e55bb2e47bab7df5df62a1bcb51 already in progress: creating tablet
W20251029 02:52:18.131433 524 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 055184c641ba45a588bca611fb24d130 consensus state has no leader
W20251029 02:52:18.131583 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet c108828dc9ca4c329dcb8a0240bf675f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c108828dc9ca4c329dcb8a0240bf675f already in progress: creating tablet
I20251029 02:52:18.133900 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_fd903b7b9822450fa41766f9f091d82c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:18.134652 1176 tablet_replica.cc:333] T 589a521f5cc24e6289c37b9cf256bce5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:18.136200 1176 raft_consensus.cc:2243] T 589a521f5cc24e6289c37b9cf256bce5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:18.136497 1176 pending_rounds.cc:70] T 589a521f5cc24e6289c37b9cf256bce5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Trying to abort 1 pending ops.
W20251029 02:52:18.143674 510 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 2965344ff7174295b7759d1b2c95ff87 (table default.Table_98d6dc24d373463dab3125992ea24306 [id=a877951ca0224abbbc5a9112f0a598a8], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 2965344ff7174295b7759d1b2c95ff87 consensus state has no leader
W20251029 02:52:18.149300 849 consensus_peers.cc:597] T 7bb85a12f573494cbb3118d1947eda27 P c756528642a543eeacd93cef3e335a7d -> Peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Couldn't send request to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7bb85a12f573494cbb3118d1947eda27. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:18.151438 2426 consensus_queue.cc:1048] T 23b9b2ea814645eba63e5f6b12cfb966 P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:18.149780 521 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet 748777c6cd6e4f49ab49912b719e48bd (table default.Table_1e69e259edba466f99ed7a41dcf6367f [id=a2f3b36dc78a4e94bd054a6a5712b384]). Expected version 1 got 0
I20251029 02:52:18.136584 1176 pending_rounds.cc:77] T 589a521f5cc24e6289c37b9cf256bce5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215949158743601152 op_type: NO_OP noop_request { }
I20251029 02:52:18.152148 1176 raft_consensus.cc:2889] T 589a521f5cc24e6289c37b9cf256bce5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 02:52:18.152648 1176 raft_consensus.cc:2272] T 589a521f5cc24e6289c37b9cf256bce5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:18.154827 1176 ts_tablet_manager.cc:1916] T 589a521f5cc24e6289c37b9cf256bce5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:17.987360 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 72f9d4408f384e7093316d85f092e1b2 (table default.Table_8a277d5e05fb437096e36f29a6307d74 [id=e5c07dcc74d1493fa9e1ef153b10d87b]) successfully deleted
W20251029 02:52:18.164832 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cc7a37c88eb54521b208967338567b54 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:52:18.164896 511 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.1:38481, user_credentials={real_user=slave}} blocked reactor thread for 177532us
I20251029 02:52:18.165304 523 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_a20cf185a2d94a80a1877ecaf2d276f3" } new_table_name: "default.Table_32ea1aa21f0d40369a449e550c317e48" modify_external_catalogs: true
W20251029 02:52:18.166182 511 catalog_manager.cc:4952] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 8048506ced4540a5aee9f79bfc4599bf because the tablet was not found. No further retry: Not found: Tablet not found: 8048506ced4540a5aee9f79bfc4599bf
W20251029 02:52:18.166396 511 catalog_manager.cc:4952] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 44ae2b07edca44afbe1b1fb3a5b1115d because the tablet was not found. No further retry: Not found: Tablet not found: 44ae2b07edca44afbe1b1fb3a5b1115d
W20251029 02:52:18.166616 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fd903b7b9822450fa41766f9f091d82c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:18.167223 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_46e4796809474bca91dc44012fa5f9f6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:18.167785 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_46e4796809474bca91dc44012fa5f9f6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:18.167982 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_c938c99be4f0482b94fc22d82972bb4a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:18.168416 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c938c99be4f0482b94fc22d82972bb4a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:18.170547 1039 tablet_service.cc:1505] Processing CreateTablet for tablet 9ee2b088bcd8430cbe032f1588f7c35b (DEFAULT_TABLE table=default.Table_f3927a91da1e440ab76a73168740c06d [id=6c5b015146c84d238994ff828ab73240]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:18.170562 1031 tablet_service.cc:1505] Processing CreateTablet for tablet 5564fb3eb9e442b5b99a5634ae119891 (DEFAULT_TABLE table=default.Table_5e4e40ac95f149f6b0116c0cb7c5fa72 [id=55a377204e12427fbe760cf9d4e1dac4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:18.170936 1028 tablet_service.cc:1505] Processing CreateTablet for tablet d1bbe01d6ffa4074aad62f30ed3685a0 (DEFAULT_TABLE table=default.Table_6c87fb0ffa1c4d818b953a2b0dace9a3 [id=6c504f217a604e449c49230de9297c28]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:18.171051 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5564fb3eb9e442b5b99a5634ae119891. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:18.171270 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1bbe01d6ffa4074aad62f30ed3685a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:18.171677 750 tablet_service.cc:1552] Processing DeleteTablet for tablet 9542585fd9824a55b3ad2f120065a02d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:17 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:18.171972 888 tablet_service.cc:1552] Processing DeleteTablet for tablet 48e74fe2c98f4df5ac686a4405c4521b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:17 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:18.172047 900 tablet_service.cc:1505] Processing CreateTablet for tablet afd3fb2157c2445bbe8dbf0f28089c2e (DEFAULT_TABLE table=default.Table_9dbf9a53f1e34611976c67bff0ac7f9f [id=1398a570e3c9485db660ecc3e1c4d887]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:18.172367 888 tablet_service.cc:1505] Processing CreateTablet for tablet 9ee2b088bcd8430cbe032f1588f7c35b (DEFAULT_TABLE table=default.Table_f3927a91da1e440ab76a73168740c06d [id=6c5b015146c84d238994ff828ab73240]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:18.172474 887 tablet_service.cc:1505] Processing CreateTablet for tablet 61fc4f8bed574debbdcfc18905858da8 (DEFAULT_TABLE table=default.Table_397408db7cb64d52830da673241064ac [id=c7bbe54e8d254f6c81eec9fffea4e9ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:18.172688 879 tablet_service.cc:1505] Processing CreateTablet for tablet d1bbe01d6ffa4074aad62f30ed3685a0 (DEFAULT_TABLE table=default.Table_6c87fb0ffa1c4d818b953a2b0dace9a3 [id=6c504f217a604e449c49230de9297c28]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:18.172477 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet afd3fb2157c2445bbe8dbf0f28089c2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:18.173213 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ee2b088bcd8430cbe032f1588f7c35b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:18.173476 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1bbe01d6ffa4074aad62f30ed3685a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:18.174777 895 tablet_service.cc:1505] Processing CreateTablet for tablet 5564fb3eb9e442b5b99a5634ae119891 (DEFAULT_TABLE table=default.Table_5e4e40ac95f149f6b0116c0cb7c5fa72 [id=55a377204e12427fbe760cf9d4e1dac4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:18.175104 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5564fb3eb9e442b5b99a5634ae119891. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:18.178643 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61fc4f8bed574debbdcfc18905858da8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:18.178771 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ee2b088bcd8430cbe032f1588f7c35b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:18.181226 750 tablet_service.cc:1505] Processing CreateTablet for tablet afd3fb2157c2445bbe8dbf0f28089c2e (DEFAULT_TABLE table=default.Table_9dbf9a53f1e34611976c67bff0ac7f9f [id=1398a570e3c9485db660ecc3e1c4d887]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:18.181651 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet afd3fb2157c2445bbe8dbf0f28089c2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:18.183372 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_1400dc2050d64e82a07e3fcd8018c0d2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:18.183909 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1400dc2050d64e82a07e3fcd8018c0d2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:18.183925 742 tablet_service.cc:1552] Processing DeleteTablet for tablet 48e74fe2c98f4df5ac686a4405c4521b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:17 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:18.184355 742 tablet_service.cc:1505] Processing CreateTablet for tablet 61fc4f8bed574debbdcfc18905858da8 (DEFAULT_TABLE table=default.Table_397408db7cb64d52830da673241064ac [id=c7bbe54e8d254f6c81eec9fffea4e9ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:18.184717 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61fc4f8bed574debbdcfc18905858da8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:18.185089 738 tablet_service.cc:1552] Processing DeleteTablet for tablet 0d788aeac52047c6bd9edcaf1369c35c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:17 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:18.185640 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_e464d8b2defc45f1aefefab339650ac0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:18.186115 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e464d8b2defc45f1aefefab339650ac0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:18.186836 748 tablet_service.cc:1505] Processing CreateTablet for tablet 5564fb3eb9e442b5b99a5634ae119891 (DEFAULT_TABLE table=default.Table_5e4e40ac95f149f6b0116c0cb7c5fa72 [id=55a377204e12427fbe760cf9d4e1dac4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:18.169786 1024 tablet_service.cc:1505] Processing CreateTablet for tablet 61fc4f8bed574debbdcfc18905858da8 (DEFAULT_TABLE table=default.Table_397408db7cb64d52830da673241064ac [id=c7bbe54e8d254f6c81eec9fffea4e9ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:18.187186 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5564fb3eb9e442b5b99a5634ae119891. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:18.187258 740 tablet_service.cc:1552] Processing DeleteTablet for tablet e2a739ebb4aa446fba435a9aecd28ed5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:17 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:18.187350 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61fc4f8bed574debbdcfc18905858da8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:18.187445 738 tablet_service.cc:1552] Processing DeleteTablet for tablet 82f83631253a4b5785c814c993678d68 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:17 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:18.188613 891 tablet_service.cc:1552] Processing DeleteTablet for tablet 9542585fd9824a55b3ad2f120065a02d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:17 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:18.191115 744 tablet_service.cc:1505] Processing CreateTablet for tablet d1bbe01d6ffa4074aad62f30ed3685a0 (DEFAULT_TABLE table=default.Table_6c87fb0ffa1c4d818b953a2b0dace9a3 [id=6c504f217a604e449c49230de9297c28]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:18.191475 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1bbe01d6ffa4074aad62f30ed3685a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:18.192646 1031 tablet_service.cc:1505] Processing CreateTablet for tablet 8aff7a8dd02046838b662ee3eed8ec1a (DEFAULT_TABLE table=default.Table_255cf3364f1e4be9a543378e563a730f [id=c8a5cd5dabbc49dd9b979ef9c779e3a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:18.193001 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8aff7a8dd02046838b662ee3eed8ec1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:18.194692 739 tablet_service.cc:1505] Processing CreateTablet for tablet 9ee2b088bcd8430cbe032f1588f7c35b (DEFAULT_TABLE table=default.Table_f3927a91da1e440ab76a73168740c06d [id=6c5b015146c84d238994ff828ab73240]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 02:52:18.196435 849 consensus_peers.cc:597] T fcb3db8399aa4cf188b07dda220f7f5b P c756528642a543eeacd93cef3e335a7d -> Peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Couldn't send request to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:18.196935 2717 raft_consensus.cc:493] T ad214c870ee649279ed7321c5b69cbee P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:18.197057 2717 raft_consensus.cc:515] T ad214c870ee649279ed7321c5b69cbee P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:18.197649 2717 leader_election.cc:290] T ad214c870ee649279ed7321c5b69cbee P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:18.197633 738 tablet_service.cc:1505] Processing CreateTablet for tablet 8aff7a8dd02046838b662ee3eed8ec1a (DEFAULT_TABLE table=default.Table_255cf3364f1e4be9a543378e563a730f [id=c8a5cd5dabbc49dd9b979ef9c779e3a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:18.198387 2908 raft_consensus.cc:493] T 2b278db5b2b04344b864797055a5f665 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:18.198601 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ee2b088bcd8430cbe032f1588f7c35b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:18.198555 2908 raft_consensus.cc:515] T 2b278db5b2b04344b864797055a5f665 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:18.199182 2908 leader_election.cc:290] T 2b278db5b2b04344b864797055a5f665 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:52:18.199977 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b278db5b2b04344b864797055a5f665" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:18.200645 711 leader_election.cc:343] T 2b278db5b2b04344b864797055a5f665 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:18.201877 1172 tablet_bootstrap.cc:654] T f18b0ca5a6a44175a912642e9eea1778 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:18.202800 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8aff7a8dd02046838b662ee3eed8ec1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:18.207095 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ad214c870ee649279ed7321c5b69cbee" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:18.208173 1058 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b278db5b2b04344b864797055a5f665" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:52:18.208693 712 leader_election.cc:343] T 2b278db5b2b04344b864797055a5f665 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:18.208808 712 leader_election.cc:304] T 2b278db5b2b04344b864797055a5f665 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:18.209230 2908 raft_consensus.cc:2749] T 2b278db5b2b04344b864797055a5f665 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:18.208513 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "ad214c870ee649279ed7321c5b69cbee" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:18.209993 989 leader_election.cc:304] T ad214c870ee649279ed7321c5b69cbee P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5; no voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:18.210449 2717 raft_consensus.cc:2749] T ad214c870ee649279ed7321c5b69cbee P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:18.210392 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_ad533f1a574e4b30bc8875869b5102a4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:18.211006 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ad533f1a574e4b30bc8875869b5102a4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:52:18.217139 849 consensus_peers.cc:597] T 23b9b2ea814645eba63e5f6b12cfb966 P c756528642a543eeacd93cef3e335a7d -> Peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Couldn't send request to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:18.224792 900 tablet_service.cc:1505] Processing CreateTablet for tablet 8aff7a8dd02046838b662ee3eed8ec1a (DEFAULT_TABLE table=default.Table_255cf3364f1e4be9a543378e563a730f [id=c8a5cd5dabbc49dd9b979ef9c779e3a9]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 02:52:18.225176 523 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 479fd93eb02540868931faee38830223 consensus state has no leader
I20251029 02:52:18.225338 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8aff7a8dd02046838b662ee3eed8ec1a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:18.226869 887 tablet_service.cc:1505] Processing CreateTablet for tablet b5c129532a134bf2ab6a01ece53ac78a (DEFAULT_TABLE table=default.Table_bb58177f826141a1bfdc278c16bcb22a [id=230da540d12342928d3da2f135f281dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:18.227231 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5c129532a134bf2ab6a01ece53ac78a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:18.227407 1028 tablet_service.cc:1505] Processing CreateTablet for tablet 12f909dfd08b4f16a46ed5939f4574d4 (DEFAULT_TABLE table=default.Table_d7867a1cdf554a9c9f4578435d47cf14 [id=fe77ffbff54b42bea36b85812b9f488c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:18.227946 1024 tablet_service.cc:1505] Processing CreateTablet for tablet b5c129532a134bf2ab6a01ece53ac78a (DEFAULT_TABLE table=default.Table_bb58177f826141a1bfdc278c16bcb22a [id=230da540d12342928d3da2f135f281dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:18.228628 523 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_aae6e3dc36e548d6a9116e0535b7a32d" } new_table_name: "default.Table_de4edfacfac14c1f9bb5e313dbe74d65" modify_external_catalogs: true
I20251029 02:52:18.229349 1033 tablet_service.cc:1505] Processing CreateTablet for tablet b22929678f734288829c53fffadfa8a9 (DEFAULT_TABLE table=default.Table_07fdb04dcb96444cb939ea7d0843af6b [id=ac5c9281c407477fb3562f4c08da83ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:18.236006 891 tablet_service.cc:1505] Processing CreateTablet for tablet 12f909dfd08b4f16a46ed5939f4574d4 (DEFAULT_TABLE table=default.Table_d7867a1cdf554a9c9f4578435d47cf14 [id=fe77ffbff54b42bea36b85812b9f488c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:18.236409 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12f909dfd08b4f16a46ed5939f4574d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:18.238227 1167 tablet_bootstrap.cc:492] T abf18e06df9b482a8b25c078b9495272 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:18.238394 1167 ts_tablet_manager.cc:1403] T abf18e06df9b482a8b25c078b9495272 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.190s user 0.004s sys 0.000s
I20251029 02:52:18.239264 1166 tablet_bootstrap.cc:492] T 23b9b2ea814645eba63e5f6b12cfb966 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:18.239225 1167 raft_consensus.cc:359] T abf18e06df9b482a8b25c078b9495272 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:18.239485 1166 ts_tablet_manager.cc:1403] T 23b9b2ea814645eba63e5f6b12cfb966 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.116s user 0.002s sys 0.003s
I20251029 02:52:18.239535 1167 raft_consensus.cc:385] T abf18e06df9b482a8b25c078b9495272 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:18.239674 1167 raft_consensus.cc:740] T abf18e06df9b482a8b25c078b9495272 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:18.239995 1167 consensus_queue.cc:260] T abf18e06df9b482a8b25c078b9495272 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:18.240152 1166 raft_consensus.cc:359] T 23b9b2ea814645eba63e5f6b12cfb966 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:18.240391 1166 raft_consensus.cc:385] T 23b9b2ea814645eba63e5f6b12cfb966 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:18.240487 1166 raft_consensus.cc:740] T 23b9b2ea814645eba63e5f6b12cfb966 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:18.240767 1167 ts_tablet_manager.cc:1434] T abf18e06df9b482a8b25c078b9495272 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20251029 02:52:18.240739 1166 consensus_queue.cc:260] T 23b9b2ea814645eba63e5f6b12cfb966 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:18.241154 1167 tablet_bootstrap.cc:492] T b125cc9aaf4248b8bfa48fb152b832a6 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:18.241286 1166 ts_tablet_manager.cc:1434] T 23b9b2ea814645eba63e5f6b12cfb966 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 02:52:18.244058 1167 tablet_bootstrap.cc:654] T b125cc9aaf4248b8bfa48fb152b832a6 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:18.247970 900 tablet_service.cc:1505] Processing CreateTablet for tablet b22929678f734288829c53fffadfa8a9 (DEFAULT_TABLE table=default.Table_07fdb04dcb96444cb939ea7d0843af6b [id=ac5c9281c407477fb3562f4c08da83ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:18.248330 1027 tablet_service.cc:1505] Processing CreateTablet for tablet d52a7c2d0cb94a3bbedcb413af06d90f (DEFAULT_TABLE table=default.Table_ff6546625fd1420abd9ab22239938a2d [id=98c94a9a65364233993464f0919f77b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:18.248420 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b22929678f734288829c53fffadfa8a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:18.255045 524 master_service.cc:954] ReplaceTablet: received request to replace tablet 3cb36eaa1ffe4db2928e87764eb65a35 from {username='slave'} at 127.0.0.1:35314
I20251029 02:52:18.256728 887 tablet_service.cc:1505] Processing CreateTablet for tablet d52a7c2d0cb94a3bbedcb413af06d90f (DEFAULT_TABLE table=default.Table_ff6546625fd1420abd9ab22239938a2d [id=98c94a9a65364233993464f0919f77b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:18.257153 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d52a7c2d0cb94a3bbedcb413af06d90f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:18.262568 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_553a74353c2a4583ac706c63b70e8c94"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:18.263130 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_553a74353c2a4583ac706c63b70e8c94 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:18.263500 1176 ts_tablet_manager.cc:1929] T 589a521f5cc24e6289c37b9cf256bce5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:18.263612 1176 log.cc:1199] T 589a521f5cc24e6289c37b9cf256bce5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/589a521f5cc24e6289c37b9cf256bce5
I20251029 02:52:18.264410 1176 ts_tablet_manager.cc:1950] T 589a521f5cc24e6289c37b9cf256bce5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:18.268031 965 heartbeater.cc:344] Connected to a master server at 127.0.104.62:42343
I20251029 02:52:18.268450 740 tablet_service.cc:1505] Processing CreateTablet for tablet 12f909dfd08b4f16a46ed5939f4574d4 (DEFAULT_TABLE table=default.Table_d7867a1cdf554a9c9f4578435d47cf14 [id=fe77ffbff54b42bea36b85812b9f488c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:18.270888 1175 ts_tablet_manager.cc:1929] T 6142018e44af4d32818adefa2ee43d02 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:18.271947 1175 log.cc:1199] T 6142018e44af4d32818adefa2ee43d02 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/6142018e44af4d32818adefa2ee43d02
I20251029 02:52:18.272382 1175 ts_tablet_manager.cc:1950] T 6142018e44af4d32818adefa2ee43d02 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:18.272342 754 tablet_service.cc:1505] Processing CreateTablet for tablet b22929678f734288829c53fffadfa8a9 (DEFAULT_TABLE table=default.Table_07fdb04dcb96444cb939ea7d0843af6b [id=ac5c9281c407477fb3562f4c08da83ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:18.275799 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12f909dfd08b4f16a46ed5939f4574d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:18.276829 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5c129532a134bf2ab6a01ece53ac78a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:18.277055 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b22929678f734288829c53fffadfa8a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:18.282217 900 tablet_service.cc:1505] Processing CreateTablet for tablet b7de8eec705c4551a42958fb5a9b4d60 (DEFAULT_TABLE table=default.Table_01a3b212de724677af552f1859bf6481 [id=8ab8e42261534155a7f02a84c97355d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:18.282737 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7de8eec705c4551a42958fb5a9b4d60. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:18.283942 1166 tablet_bootstrap.cc:492] T d29cc2a0c8b84e3ebadc7f571d666847 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:18.286674 1174 ts_tablet_manager.cc:1929] T 6d830d30b26b4ba799785fec8a6f0769 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:18.290235 1058 raft_consensus.cc:3060] T cd597c319af645909b8f3d1212fcb702 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:18.295248 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d52a7c2d0cb94a3bbedcb413af06d90f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:18.295302 1176 tablet_replica.cc:333] T ad214c870ee649279ed7321c5b69cbee P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:18.295506 1176 raft_consensus.cc:2243] T ad214c870ee649279ed7321c5b69cbee P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:18.295651 1176 raft_consensus.cc:2272] T ad214c870ee649279ed7321c5b69cbee P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:18.296154 1176 ts_tablet_manager.cc:1916] T ad214c870ee649279ed7321c5b69cbee P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:18.297672 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_c873cfe841d04574a044e4796fcb3ea7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:18.297783 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_1caf7466260e4f13ab3fcd2d52e80744"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:18.298220 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c873cfe841d04574a044e4796fcb3ea7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:52:18.298403 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1caf7466260e4f13ab3fcd2d52e80744 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:18.300483 1039 tablet_service.cc:1505] Processing CreateTablet for tablet b7de8eec705c4551a42958fb5a9b4d60 (DEFAULT_TABLE table=default.Table_01a3b212de724677af552f1859bf6481 [id=8ab8e42261534155a7f02a84c97355d1]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 02:52:18.303442 849 consensus_peers.cc:597] T 40e844fd7ad74680bcbe7bef936c5b9d P c756528642a543eeacd93cef3e335a7d -> Peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Couldn't send request to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 40e844fd7ad74680bcbe7bef936c5b9d. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:18.303444 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 589a521f5cc24e6289c37b9cf256bce5 (table default.Table_b8095fca27a447f2a9ac0475373352c6 [id=56b57a7a90624bee92fa43661728c0f8]) successfully deleted
I20251029 02:52:18.303997 1042 tablet_service.cc:1505] Processing CreateTablet for tablet a4e0554ae01f47b58127468f8ec3768e (DEFAULT_TABLE table=default.Table_2d6bca94a5a14e4ea414e1863b335737 [id=4f062ac6f69648a0a9d2e3e2daea64f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:18.304975 887 tablet_service.cc:1505] Processing CreateTablet for tablet a4e0554ae01f47b58127468f8ec3768e (DEFAULT_TABLE table=default.Table_2d6bca94a5a14e4ea414e1863b335737 [id=4f062ac6f69648a0a9d2e3e2daea64f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:18.305444 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4e0554ae01f47b58127468f8ec3768e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:18.309505 1031 tablet_service.cc:1505] Processing CreateTablet for tablet 9c8894de7eef4e43957ee7863ebbd087 (DEFAULT_TABLE table=default.Table_07b6fa17dfc04d919f5a344a8b225333 [id=e8cbc6f4a8ab41f1a9302def9062eea3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:18.310478 1176 ts_tablet_manager.cc:1929] T ad214c870ee649279ed7321c5b69cbee P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:18.311002 900 tablet_service.cc:1505] Processing CreateTablet for tablet 9c8894de7eef4e43957ee7863ebbd087 (DEFAULT_TABLE table=default.Table_07b6fa17dfc04d919f5a344a8b225333 [id=e8cbc6f4a8ab41f1a9302def9062eea3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:18.315893 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c8894de7eef4e43957ee7863ebbd087. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:18.310573 1166 tablet_bootstrap.cc:654] T d29cc2a0c8b84e3ebadc7f571d666847 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:18.316107 1176 log.cc:1199] T ad214c870ee649279ed7321c5b69cbee P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/ad214c870ee649279ed7321c5b69cbee
I20251029 02:52:18.316730 1167 tablet_bootstrap.cc:492] T b125cc9aaf4248b8bfa48fb152b832a6 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:18.316960 1167 ts_tablet_manager.cc:1403] T b125cc9aaf4248b8bfa48fb152b832a6 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.076s user 0.001s sys 0.002s
I20251029 02:52:18.317919 1167 raft_consensus.cc:359] T b125cc9aaf4248b8bfa48fb152b832a6 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:18.318204 1167 raft_consensus.cc:385] T b125cc9aaf4248b8bfa48fb152b832a6 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:18.318326 1167 raft_consensus.cc:740] T b125cc9aaf4248b8bfa48fb152b832a6 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:18.318653 1167 consensus_queue.cc:260] T b125cc9aaf4248b8bfa48fb152b832a6 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:18.319442 1167 ts_tablet_manager.cc:1434] T b125cc9aaf4248b8bfa48fb152b832a6 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 02:52:18.312327 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c8894de7eef4e43957ee7863ebbd087. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:18.286893 1174 log.cc:1199] T 6d830d30b26b4ba799785fec8a6f0769 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/6d830d30b26b4ba799785fec8a6f0769
I20251029 02:52:18.320708 1028 tablet_service.cc:1505] Processing CreateTablet for tablet 94407d37393442e295b8f8747fa71ad2 (DEFAULT_TABLE table=default.Table_357785b823ab474dbb17df63a96f7554 [id=1a951277f89a4d9981537bc65ebe155c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:18.322907 1176 ts_tablet_manager.cc:1950] T ad214c870ee649279ed7321c5b69cbee P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
W20251029 02:52:18.324913 847 consensus_peers.cc:597] T 6142018e44af4d32818adefa2ee43d02 P c756528642a543eeacd93cef3e335a7d -> Peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Couldn't send request to peer b495a49349324f7d81908f2da700f0d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6142018e44af4d32818adefa2ee43d02. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:18.323988 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4e0554ae01f47b58127468f8ec3768e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:18.327234 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7de8eec705c4551a42958fb5a9b4d60. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:18.327468 1167 tablet_bootstrap.cc:492] T b0fec06ad37d4c12a7045d424b88cc5e P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:18.327967 744 tablet_service.cc:1505] Processing CreateTablet for tablet b7de8eec705c4551a42958fb5a9b4d60 (DEFAULT_TABLE table=default.Table_01a3b212de724677af552f1859bf6481 [id=8ab8e42261534155a7f02a84c97355d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:18.328275 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b22929678f734288829c53fffadfa8a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:18.328454 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 1050) took 4672 ms. Trace:
I20251029 02:52:18.330449 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 12f909dfd08b4f16a46ed5939f4574d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:18.336272 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet ad214c870ee649279ed7321c5b69cbee (table default.Table_982d01dccabe47bfaac7a582282c5d54 [id=033c1db6a2b44a2e99361183fd18c8f0]) successfully deleted
W20251029 02:52:18.336525 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 0415b52fadf948238c4e83dc09000f96 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0415b52fadf948238c4e83dc09000f96 already in progress: creating tablet
W20251029 02:52:18.336675 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet e2a739ebb4aa446fba435a9aecd28ed5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e2a739ebb4aa446fba435a9aecd28ed5 already in progress: creating tablet
W20251029 02:52:18.336818 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 82f83631253a4b5785c814c993678d68 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 82f83631253a4b5785c814c993678d68 already in progress: creating tablet
W20251029 02:52:18.336946 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 48e74fe2c98f4df5ac686a4405c4521b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 48e74fe2c98f4df5ac686a4405c4521b already in progress: creating tablet
W20251029 02:52:18.337086 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 0d788aeac52047c6bd9edcaf1369c35c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0d788aeac52047c6bd9edcaf1369c35c already in progress: creating tablet
W20251029 02:52:18.337251 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 9542585fd9824a55b3ad2f120065a02d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9542585fd9824a55b3ad2f120065a02d already in progress: creating tablet
I20251029 02:52:18.338807 750 tablet_service.cc:1505] Processing CreateTablet for tablet b5c129532a134bf2ab6a01ece53ac78a (DEFAULT_TABLE table=default.Table_bb58177f826141a1bfdc278c16bcb22a [id=230da540d12342928d3da2f135f281dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:18.341111 1175 rpcz_store.cc:276] 1029 02:52:13.651057 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:13.651337 (+ 280us) service_pool.cc:225] Handling call
1029 02:52:17.990380 (+4339043us) ts_tablet_manager.cc:1151] Deleting tablet 6142018e44af4d32818adefa2ee43d02
1029 02:52:17.990390 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:18.083765 (+ 93375us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:18.230599 (+146834us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:18.323917 (+ 93318us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2249,"tablet-delete.queue_time_us":4338876}
I20251029 02:52:18.341881 739 tablet_service.cc:1505] Processing CreateTablet for tablet 9c8894de7eef4e43957ee7863ebbd087 (DEFAULT_TABLE table=default.Table_07b6fa17dfc04d919f5a344a8b225333 [id=e8cbc6f4a8ab41f1a9302def9062eea3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:18.330061 2655 raft_consensus.cc:493] T 24f07b1efac64b099b0b5ed467463749 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:18.342957 2655 raft_consensus.cc:515] T 24f07b1efac64b099b0b5ed467463749 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:18.343380 748 tablet_service.cc:1505] Processing CreateTablet for tablet a4e0554ae01f47b58127468f8ec3768e (DEFAULT_TABLE table=default.Table_2d6bca94a5a14e4ea414e1863b335737 [id=4f062ac6f69648a0a9d2e3e2daea64f6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:18.343652 2655 leader_election.cc:290] T 24f07b1efac64b099b0b5ed467463749 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:18.344056 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b7de8eec705c4551a42958fb5a9b4d60. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:18.344291 1059 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24f07b1efac64b099b0b5ed467463749" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:52:18.344811 849 leader_election.cc:343] T 24f07b1efac64b099b0b5ed467463749 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:18.344951 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 1048) took 4690 ms. Trace:
I20251029 02:52:18.345046 1175 rpcz_store.cc:276] 1029 02:52:13.650825 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:13.652183 (+ 1358us) service_pool.cc:225] Handling call
1029 02:52:18.341472 (+4689289us) ts_tablet_manager.cc:1151] Deleting tablet af4860c6b72f4d988b206b5ef2f22906
1029 02:52:18.341482 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:18.341544 (+ 62us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4689168}
I20251029 02:52:18.345415 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 1045) took 4694 ms. Trace:
I20251029 02:52:18.345499 1175 rpcz_store.cc:276] 1029 02:52:13.650493 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:13.656778 (+ 6285us) service_pool.cc:225] Handling call
1029 02:52:18.345307 (+4688529us) ts_tablet_manager.cc:1151] Deleting tablet 28f7ceb91a34450cad4b5b45f5285b41
1029 02:52:18.345316 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:18.345380 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4688319}
I20251029 02:52:18.346318 742 tablet_service.cc:1505] Processing CreateTablet for tablet d52a7c2d0cb94a3bbedcb413af06d90f (DEFAULT_TABLE table=default.Table_ff6546625fd1420abd9ab22239938a2d [id=98c94a9a65364233993464f0919f77b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:18.347395 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 1044) took 4697 ms. Trace:
I20251029 02:52:18.347551 766 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24f07b1efac64b099b0b5ed467463749" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:18.347494 1175 rpcz_store.cc:276] 1029 02:52:13.650369 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:13.681607 (+ 31238us) service_pool.cc:225] Handling call
1029 02:52:18.345698 (+4664091us) ts_tablet_manager.cc:1151] Deleting tablet c32c28b0e74a4af3a0e69bbf24ab1a1f
1029 02:52:18.345707 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:18.347356 (+ 1649us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4663944}
I20251029 02:52:18.347744 766 raft_consensus.cc:2468] T 24f07b1efac64b099b0b5ed467463749 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:18.347919 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 1051) took 4613 ms. Trace:
I20251029 02:52:18.348018 1175 rpcz_store.cc:276] 1029 02:52:13.734208 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:13.740451 (+ 6243us) service_pool.cc:225] Handling call
1029 02:52:18.347795 (+4607344us) ts_tablet_manager.cc:1151] Deleting tablet dab6ed2266ed42508738399d388c4da3
1029 02:52:18.347804 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:18.347879 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4607137}
I20251029 02:52:18.348255 847 leader_election.cc:304] T 24f07b1efac64b099b0b5ed467463749 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5
I20251029 02:52:18.348353 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 1052) took 4550 ms. Trace:
I20251029 02:52:18.348433 1175 rpcz_store.cc:276] 1029 02:52:13.798014 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:13.798065 (+ 51us) service_pool.cc:225] Handling call
1029 02:52:18.348203 (+4550138us) ts_tablet_manager.cc:1151] Deleting tablet b4621bf9e6324eaeaa64a6ca51cd5ca9
1029 02:52:18.348211 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:18.348317 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4549968}
I20251029 02:52:18.348664 2655 raft_consensus.cc:2804] T 24f07b1efac64b099b0b5ed467463749 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:18.348734 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 1053) took 4534 ms. Trace:
I20251029 02:52:18.348755 2655 raft_consensus.cc:493] T 24f07b1efac64b099b0b5ed467463749 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:18.348843 2655 raft_consensus.cc:3060] T 24f07b1efac64b099b0b5ed467463749 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:18.348819 1175 rpcz_store.cc:276] 1029 02:52:13.814691 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:13.815555 (+ 864us) service_pool.cc:225] Handling call
1029 02:52:18.348606 (+4533051us) ts_tablet_manager.cc:1151] Deleting tablet eaf4018c0d624b1cb8c6528eea7b689e
1029 02:52:18.348614 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:18.348694 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4532874}
I20251029 02:52:18.349176 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 1054) took 4522 ms. Trace:
I20251029 02:52:18.349259 1175 rpcz_store.cc:276] 1029 02:52:13.826755 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:13.830693 (+ 3938us) service_pool.cc:225] Handling call
1029 02:52:18.349039 (+4518346us) ts_tablet_manager.cc:1151] Deleting tablet e40d4910365f45e9a45d9b3bc5037c95
1029 02:52:18.349047 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:18.349140 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4518142}
I20251029 02:52:18.349545 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 1055) took 4448 ms. Trace:
I20251029 02:52:18.349619 1175 rpcz_store.cc:276] 1029 02:52:13.900934 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:13.901084 (+ 150us) service_pool.cc:225] Handling call
1029 02:52:18.349432 (+4448348us) ts_tablet_manager.cc:1151] Deleting tablet 3e9a076e086141ef8ed59d4c8633f961
1029 02:52:18.349440 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:18.349502 (+ 62us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4448043}
I20251029 02:52:18.349845 1175 tablet_replica.cc:333] T d31e548feae24b739087c1e211d514f9 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:18.350024 1175 raft_consensus.cc:2243] T d31e548feae24b739087c1e211d514f9 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:18.350559 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c8894de7eef4e43957ee7863ebbd087. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:18.350646 1175 raft_consensus.cc:2272] T d31e548feae24b739087c1e211d514f9 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:18.351652 2655 raft_consensus.cc:515] T 24f07b1efac64b099b0b5ed467463749 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:18.352063 1175 ts_tablet_manager.cc:1916] T d31e548feae24b739087c1e211d514f9 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:18.352775 1059 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24f07b1efac64b099b0b5ed467463749" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
W20251029 02:52:18.353535 849 leader_election.cc:343] T 24f07b1efac64b099b0b5ed467463749 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:18.353826 2655 leader_election.cc:290] T 24f07b1efac64b099b0b5ed467463749 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:18.354286 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 6142018e44af4d32818adefa2ee43d02 (table default.Table_a45ccb3fef5d4947ab7a8709b7c32119 [id=b3e01f7d5a5f4675a998eaad985347ca]) successfully deleted
I20251029 02:52:18.354624 1059 raft_consensus.cc:3060] T 6f8cf09f23984b879b60ab6b27c7d52e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
W20251029 02:52:18.354625 511 catalog_manager.cc:4952] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet af4860c6b72f4d988b206b5ef2f22906 because the tablet was not found. No further retry: Not found: Tablet not found: af4860c6b72f4d988b206b5ef2f22906
W20251029 02:52:18.354820 511 catalog_manager.cc:4952] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 28f7ceb91a34450cad4b5b45f5285b41 because the tablet was not found. No further retry: Not found: Tablet not found: 28f7ceb91a34450cad4b5b45f5285b41
W20251029 02:52:18.355002 511 catalog_manager.cc:4952] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet c32c28b0e74a4af3a0e69bbf24ab1a1f because the tablet was not found. No further retry: Not found: Tablet not found: c32c28b0e74a4af3a0e69bbf24ab1a1f
W20251029 02:52:18.355149 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet dab6ed2266ed42508738399d388c4da3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dab6ed2266ed42508738399d388c4da3 already in progress: creating tablet
W20251029 02:52:18.355278 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet b4621bf9e6324eaeaa64a6ca51cd5ca9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b4621bf9e6324eaeaa64a6ca51cd5ca9 already in progress: creating tablet
W20251029 02:52:18.355412 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet eaf4018c0d624b1cb8c6528eea7b689e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eaf4018c0d624b1cb8c6528eea7b689e already in progress: creating tablet
W20251029 02:52:18.355548 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet e40d4910365f45e9a45d9b3bc5037c95 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e40d4910365f45e9a45d9b3bc5037c95 already in progress: creating tablet
W20251029 02:52:18.355683 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 3e9a076e086141ef8ed59d4c8633f961 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3e9a076e086141ef8ed59d4c8633f961 already in progress: creating tablet
I20251029 02:52:18.356191 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94407d37393442e295b8f8747fa71ad2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:18.360229 2908 raft_consensus.cc:493] T 023a4094b424464c8f49d6eb9cc15da4 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:18.360406 2908 raft_consensus.cc:515] T 023a4094b424464c8f49d6eb9cc15da4 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:18.362998 766 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "24f07b1efac64b099b0b5ed467463749" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:18.363186 766 raft_consensus.cc:3060] T 24f07b1efac64b099b0b5ed467463749 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:18.363407 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4e0554ae01f47b58127468f8ec3768e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:18.363435 1052 raft_consensus.cc:3060] T c081d2eacfac4ca8a0048e2a1c4f8f83 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:18.366983 766 raft_consensus.cc:2468] T 24f07b1efac64b099b0b5ed467463749 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:18.367437 847 leader_election.cc:304] T 24f07b1efac64b099b0b5ed467463749 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5
I20251029 02:52:18.367841 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d52a7c2d0cb94a3bbedcb413af06d90f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:18.367874 2649 raft_consensus.cc:2804] T 24f07b1efac64b099b0b5ed467463749 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:18.368018 2649 raft_consensus.cc:697] T 24f07b1efac64b099b0b5ed467463749 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:52:18.368288 2649 consensus_queue.cc:237] T 24f07b1efac64b099b0b5ed467463749 P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:18.370904 1172 tablet_bootstrap.cc:492] T f18b0ca5a6a44175a912642e9eea1778 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:18.374634 887 tablet_service.cc:1505] Processing CreateTablet for tablet 94407d37393442e295b8f8747fa71ad2 (DEFAULT_TABLE table=default.Table_357785b823ab474dbb17df63a96f7554 [id=1a951277f89a4d9981537bc65ebe155c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:18.380177 1172 ts_tablet_manager.cc:1403] T f18b0ca5a6a44175a912642e9eea1778 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.247s user 0.004s sys 0.008s
I20251029 02:52:18.383168 2908 leader_election.cc:290] T 023a4094b424464c8f49d6eb9cc15da4 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:52:18.383802 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "023a4094b424464c8f49d6eb9cc15da4" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:18.384320 754 tablet_service.cc:1505] Processing CreateTablet for tablet 94407d37393442e295b8f8747fa71ad2 (DEFAULT_TABLE table=default.Table_357785b823ab474dbb17df63a96f7554 [id=1a951277f89a4d9981537bc65ebe155c]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 02:52:18.384405 711 leader_election.cc:343] T 023a4094b424464c8f49d6eb9cc15da4 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:18.384959 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94407d37393442e295b8f8747fa71ad2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:18.388402 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5c129532a134bf2ab6a01ece53ac78a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:18.401857 518 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_dc31c9a0d89947c1aeeedd96b7678134" } new_table_name: "default.Table_0824caee7ed74f48a0cb0f98ca1ff51e" modify_external_catalogs: true
I20251029 02:52:18.403255 1172 raft_consensus.cc:359] T f18b0ca5a6a44175a912642e9eea1778 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:18.403417 1172 raft_consensus.cc:385] T f18b0ca5a6a44175a912642e9eea1778 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:18.403470 1172 raft_consensus.cc:740] T f18b0ca5a6a44175a912642e9eea1778 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:18.403687 1172 consensus_queue.cc:260] T f18b0ca5a6a44175a912642e9eea1778 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:18.404284 1172 ts_tablet_manager.cc:1434] T f18b0ca5a6a44175a912642e9eea1778 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.023s user 0.001s sys 0.001s
I20251029 02:52:18.408905 1167 tablet_bootstrap.cc:654] T b0fec06ad37d4c12a7045d424b88cc5e P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:18.410197 519 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet 32519fbffcef44a890161e5f3043806c (table default.Table_a78f5837df7b44e791631e421aa7de46 [id=b95fe1c7d6ff4519b90f993f1fa961fd]). Expected version 3 got 0
I20251029 02:52:18.421837 1039 tablet_service.cc:1505] Processing CreateTablet for tablet 01eb63da62ff4f43bd8a23631553e5b8 (DEFAULT_TABLE table=default.Table_68c016f54d7f45818aa4eb2f91c27539 [id=7cff46ea159f42b189d74d067fa5cdc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:18.422395 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01eb63da62ff4f43bd8a23631553e5b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:18.335279 1174 ts_tablet_manager.cc:1950] T 6d830d30b26b4ba799785fec8a6f0769 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:18.430272 2656 consensus_queue.cc:1048] T fcb3db8399aa4cf188b07dda220f7f5b P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:18.430830 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 94407d37393442e295b8f8747fa71ad2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:18.437325 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 962) took 7649 ms. Trace:
I20251029 02:52:18.437582 1174 rpcz_store.cc:276] 1029 02:52:10.788270 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:10.788322 (+ 52us) service_pool.cc:225] Handling call
1029 02:52:18.063461 (+7275139us) ts_tablet_manager.cc:1151] Deleting tablet 6d830d30b26b4ba799785fec8a6f0769
1029 02:52:18.063472 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:18.200601 (+137129us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:18.286489 (+ 85888us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:18.437279 (+150790us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7263716}
I20251029 02:52:18.439653 1166 tablet_bootstrap.cc:492] T d29cc2a0c8b84e3ebadc7f571d666847 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:18.440083 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 961) took 7651 ms. Trace:
I20251029 02:52:18.440129 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 6d830d30b26b4ba799785fec8a6f0769 (table default.Table_19165ff9f7f24a97bf835e72701455d3 [id=ae6032816e6c4b80aa294478cbd49460]) successfully deleted
I20251029 02:52:18.443462 1166 ts_tablet_manager.cc:1403] T d29cc2a0c8b84e3ebadc7f571d666847 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.160s user 0.000s sys 0.005s
I20251029 02:52:18.447451 1166 raft_consensus.cc:359] T d29cc2a0c8b84e3ebadc7f571d666847 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
W20251029 02:52:18.452920 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.3:45003, user_credentials={real_user=slave}} blocked reactor thread for 111705us
W20251029 02:52:18.455713 523 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c92a3c77be8943fb929d321559e3a46c consensus state has no leader
I20251029 02:52:18.456560 524 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 3cb36eaa1ffe4db2928e87764eb65a35
I20251029 02:52:18.456936 524 catalog_manager.cc:6719] ReplaceTablet: tablet 3cb36eaa1ffe4db2928e87764eb65a35 deleted and replaced by tablet c7ff396a667a4a10a34c111806bf5aa7
I20251029 02:52:18.457872 900 tablet_service.cc:1552] Processing DeleteTablet for tablet 3cb36eaa1ffe4db2928e87764eb65a35 with delete_type TABLET_DATA_DELETED (replaced by tablet c7ff396a667a4a10a34c111806bf5aa7) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:18.459022 900 tablet_service.cc:1505] Processing CreateTablet for tablet 01eb63da62ff4f43bd8a23631553e5b8 (DEFAULT_TABLE table=default.Table_68c016f54d7f45818aa4eb2f91c27539 [id=7cff46ea159f42b189d74d067fa5cdc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:18.455438 1174 rpcz_store.cc:276] 1029 02:52:10.788154 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:10.788452 (+ 298us) service_pool.cc:225] Handling call
1029 02:52:18.439933 (+7651481us) ts_tablet_manager.cc:1151] Deleting tablet 5a1aa131cf4e4edca5bb2515edd7556a
1029 02:52:18.439944 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:18.440043 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7640089}
I20251029 02:52:18.459484 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01eb63da62ff4f43bd8a23631553e5b8. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:18.459573 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 5a1aa131cf4e4edca5bb2515edd7556a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5a1aa131cf4e4edca5bb2515edd7556a already in progress: creating tablet
I20251029 02:52:18.466897 1166 raft_consensus.cc:385] T d29cc2a0c8b84e3ebadc7f571d666847 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:18.467079 1166 raft_consensus.cc:740] T d29cc2a0c8b84e3ebadc7f571d666847 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:18.467511 1166 consensus_queue.cc:260] T d29cc2a0c8b84e3ebadc7f571d666847 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
W20251029 02:52:18.461658 849 consensus_peers.cc:597] T 946f65014d97425aab3851009b742807 P c756528642a543eeacd93cef3e335a7d -> Peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Couldn't send request to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 946f65014d97425aab3851009b742807. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:18.468286 1166 ts_tablet_manager.cc:1434] T d29cc2a0c8b84e3ebadc7f571d666847 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.021s user 0.001s sys 0.001s
I20251029 02:52:18.468686 1166 tablet_bootstrap.cc:492] T 46cd75ff18b14212b556b9bcb3012dd9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:18.461874 1039 tablet_service.cc:1552] Processing DeleteTablet for tablet 3cb36eaa1ffe4db2928e87764eb65a35 with delete_type TABLET_DATA_DELETED (replaced by tablet c7ff396a667a4a10a34c111806bf5aa7) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:18.460301 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 964) took 7659 ms. Trace:
W20251029 02:52:18.470328 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 3cb36eaa1ffe4db2928e87764eb65a35 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3cb36eaa1ffe4db2928e87764eb65a35 already in progress: creating tablet
I20251029 02:52:18.432441 1172 tablet_bootstrap.cc:492] T 7fb5b908db77453da5cfb32e2c955f41 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:18.441282 744 tablet_service.cc:1505] Processing CreateTablet for tablet 01eb63da62ff4f43bd8a23631553e5b8 (DEFAULT_TABLE table=default.Table_68c016f54d7f45818aa4eb2f91c27539 [id=7cff46ea159f42b189d74d067fa5cdc2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:18.472162 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01eb63da62ff4f43bd8a23631553e5b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:18.473155 1172 tablet_bootstrap.cc:654] T 7fb5b908db77453da5cfb32e2c955f41 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:18.460467 739 tablet_service.cc:1552] Processing DeleteTablet for tablet 3cb36eaa1ffe4db2928e87764eb65a35 with delete_type TABLET_DATA_DELETED (replaced by tablet c7ff396a667a4a10a34c111806bf5aa7) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:18.477102 2932 tablet.cc:1722] T 6f8cf09f23984b879b60ab6b27c7d52e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251029 02:52:18.483249 1172 tablet_bootstrap.cc:492] T 7fb5b908db77453da5cfb32e2c955f41 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:18.483489 1172 ts_tablet_manager.cc:1403] T 7fb5b908db77453da5cfb32e2c955f41 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.051s user 0.001s sys 0.001s
I20251029 02:52:18.484202 1172 raft_consensus.cc:359] T 7fb5b908db77453da5cfb32e2c955f41 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:18.484426 1172 raft_consensus.cc:385] T 7fb5b908db77453da5cfb32e2c955f41 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:18.484511 1172 raft_consensus.cc:740] T 7fb5b908db77453da5cfb32e2c955f41 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:18.484757 1172 consensus_queue.cc:260] T 7fb5b908db77453da5cfb32e2c955f41 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:18.488543 1172 ts_tablet_manager.cc:1434] T 7fb5b908db77453da5cfb32e2c955f41 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251029 02:52:18.488848 1172 tablet_bootstrap.cc:492] T e3ac6597d52441569a7aeef051087ad2 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:18.490336 1172 tablet_bootstrap.cc:654] T e3ac6597d52441569a7aeef051087ad2 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:18.492089 1166 tablet_bootstrap.cc:654] T 46cd75ff18b14212b556b9bcb3012dd9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
W20251029 02:52:18.492596 518 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8af82b9846464e739d0c052b9a31ff7b consensus state has no leader
I20251029 02:52:18.494141 2649 raft_consensus.cc:493] T abf18e06df9b482a8b25c078b9495272 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:18.494357 2649 raft_consensus.cc:515] T abf18e06df9b482a8b25c078b9495272 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:18.495705 2649 leader_election.cc:290] T abf18e06df9b482a8b25c078b9495272 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:18.496012 766 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "abf18e06df9b482a8b25c078b9495272" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:18.496359 766 raft_consensus.cc:2468] T abf18e06df9b482a8b25c078b9495272 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:18.496780 847 leader_election.cc:304] T abf18e06df9b482a8b25c078b9495272 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:18.497373 2426 raft_consensus.cc:2804] T abf18e06df9b482a8b25c078b9495272 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:18.497457 2426 raft_consensus.cc:493] T abf18e06df9b482a8b25c078b9495272 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:18.497519 2426 raft_consensus.cc:3060] T abf18e06df9b482a8b25c078b9495272 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:18.500109 2426 raft_consensus.cc:515] T abf18e06df9b482a8b25c078b9495272 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:18.500222 1167 tablet_bootstrap.cc:492] T b0fec06ad37d4c12a7045d424b88cc5e P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:18.500348 1167 ts_tablet_manager.cc:1403] T b0fec06ad37d4c12a7045d424b88cc5e P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.176s user 0.002s sys 0.001s
I20251029 02:52:18.500741 2426 leader_election.cc:290] T abf18e06df9b482a8b25c078b9495272 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:18.502318 1167 raft_consensus.cc:359] T b0fec06ad37d4c12a7045d424b88cc5e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:18.503772 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "abf18e06df9b482a8b25c078b9495272" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:18.503950 773 raft_consensus.cc:3060] T abf18e06df9b482a8b25c078b9495272 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:18.470384 1174 rpcz_store.cc:276] 1029 02:52:10.800506 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:10.802914 (+ 2408us) service_pool.cc:225] Handling call
1029 02:52:18.459957 (+7657043us) ts_tablet_manager.cc:1151] Deleting tablet 8378d67f1da743cca91803e37ea54281
1029 02:52:18.459997 (+ 40us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:18.460222 (+ 225us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7656861}
W20251029 02:52:18.504812 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 8378d67f1da743cca91803e37ea54281 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8378d67f1da743cca91803e37ea54281 already in progress: creating tablet
I20251029 02:52:18.505133 1174 tablet_replica.cc:333] T 2e97d1bc06a24e8d8ba11acb6fd280d8 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:18.505321 1174 raft_consensus.cc:2243] T 2e97d1bc06a24e8d8ba11acb6fd280d8 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:18.505625 1174 raft_consensus.cc:2272] T 2e97d1bc06a24e8d8ba11acb6fd280d8 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:18.506122 1174 ts_tablet_manager.cc:1916] T 2e97d1bc06a24e8d8ba11acb6fd280d8 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:18.507799 773 raft_consensus.cc:2468] T abf18e06df9b482a8b25c078b9495272 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:18.508280 847 leader_election.cc:304] T abf18e06df9b482a8b25c078b9495272 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:18.508788 2655 raft_consensus.cc:2804] T abf18e06df9b482a8b25c078b9495272 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:18.509263 2655 raft_consensus.cc:697] T abf18e06df9b482a8b25c078b9495272 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:52:18.509536 2655 consensus_queue.cc:237] T abf18e06df9b482a8b25c078b9495272 P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:18.513206 2426 raft_consensus.cc:493] T b125cc9aaf4248b8bfa48fb152b832a6 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:18.513374 2426 raft_consensus.cc:515] T b125cc9aaf4248b8bfa48fb152b832a6 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:18.513383 524 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_7fd7b56a9c09475a80cce932d6e15cc1" } modify_external_catalogs: true
I20251029 02:52:18.513551 524 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_7fd7b56a9c09475a80cce932d6e15cc1" } modify_external_catalogs: true
I20251029 02:52:18.514140 2426 leader_election.cc:290] T b125cc9aaf4248b8bfa48fb152b832a6 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:18.514751 900 tablet_service.cc:1505] Processing CreateTablet for tablet 3f3aee27041a49ceb3218d3f43bc6273 (DEFAULT_TABLE table=default.Table_2a4d6457e4db4e9280cd96080b7223b0 [id=3c32e21c17dc4625873c30c019cb2a85]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:18.514871 766 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b125cc9aaf4248b8bfa48fb152b832a6" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:18.514921 526 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_a3ee7878447a4c47970b01ff8b52955f" } new_table_name: "default.Table_65fabe5555af4c2bb042425a54bf6215" modify_external_catalogs: true
I20251029 02:52:18.515154 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f3aee27041a49ceb3218d3f43bc6273. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:18.515359 847 leader_election.cc:343] T b125cc9aaf4248b8bfa48fb152b832a6 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:18.516698 523 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_9d8abd5ee0364a16bd9c5156a01dd6da" } new_table_name: "default.Table_c77977d40ceb4a8c8a059e695f80869b" modify_external_catalogs: true
I20251029 02:52:18.517037 522 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_7c1cb64e05be4f6e8ac28286c982a3e7" } modify_external_catalogs: true
I20251029 02:52:18.517172 522 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_7c1cb64e05be4f6e8ac28286c982a3e7" } modify_external_catalogs: true
I20251029 02:52:18.517841 517 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_0f4514215d3042c5a46ac59072d2bbb2" } modify_external_catalogs: true
I20251029 02:52:18.517992 517 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_0f4514215d3042c5a46ac59072d2bbb2" } modify_external_catalogs: true
I20251029 02:52:18.518316 525 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_37dad1121b2144ee9eeb3ab7c38a0609" } modify_external_catalogs: true
I20251029 02:52:18.518440 525 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_37dad1121b2144ee9eeb3ab7c38a0609" } modify_external_catalogs: true
I20251029 02:52:18.519631 1039 tablet_service.cc:1505] Processing CreateTablet for tablet 923fa94e0b404efcb93ac5b1f38609de (DEFAULT_TABLE table=default.Table_26c8d3fab4e146319800a03dc27ced50 [id=a0f26d8f0ede4038822403024188177b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:18.520017 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 923fa94e0b404efcb93ac5b1f38609de. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:18.520038 518 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_9cd59d4e874c4239b50b475aa8def07c" } modify_external_catalogs: true
I20251029 02:52:18.520151 518 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_9cd59d4e874c4239b50b475aa8def07c" } modify_external_catalogs: true
I20251029 02:52:18.425808 1175 ts_tablet_manager.cc:1929] T d31e548feae24b739087c1e211d514f9 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:18.520377 1175 log.cc:1199] T d31e548feae24b739087c1e211d514f9 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/d31e548feae24b739087c1e211d514f9
I20251029 02:52:18.520718 1175 ts_tablet_manager.cc:1950] T d31e548feae24b739087c1e211d514f9 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:18.521394 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 1056) took 4619 ms. Trace:
I20251029 02:52:18.521487 1175 rpcz_store.cc:276] 1029 02:52:13.901496 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:13.901586 (+ 90us) service_pool.cc:225] Handling call
1029 02:52:18.349784 (+4448198us) ts_tablet_manager.cc:1151] Deleting tablet d31e548feae24b739087c1e211d514f9
1029 02:52:18.349791 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:18.422768 (+ 72977us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:18.425721 (+ 2953us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:18.521375 (+ 95654us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4448006}
I20251029 02:52:18.521862 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 1057) took 4619 ms. Trace:
I20251029 02:52:18.521965 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet d31e548feae24b739087c1e211d514f9 (table default.Table_5a915593503d490fa12b10fccd76ac3e [id=6fd26d78f4904f30a2cb76e9a4bec2db]) successfully deleted
I20251029 02:52:18.521920 1175 rpcz_store.cc:276] 1029 02:52:13.902333 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:13.902388 (+ 55us) service_pool.cc:225] Handling call
1029 02:52:18.521742 (+4619354us) ts_tablet_manager.cc:1151] Deleting tablet 2cea2f2fe68545db804a732d0c01a876
1029 02:52:18.521752 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:18.521847 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4619185}
I20251029 02:52:18.522179 1175 tablet_replica.cc:333] T 23b9b2ea814645eba63e5f6b12cfb966 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
W20251029 02:52:18.522339 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 2cea2f2fe68545db804a732d0c01a876 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2cea2f2fe68545db804a732d0c01a876 already in progress: creating tablet
I20251029 02:52:18.522358 1175 raft_consensus.cc:2243] T 23b9b2ea814645eba63e5f6b12cfb966 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:18.522821 891 tablet_service.cc:1505] Processing CreateTablet for tablet c3dabe9d8e534953838ce5f0e3259e9d (DEFAULT_TABLE table=default.Table_37dad1121b2144ee9eeb3ab7c38a0609 [id=be64f276351541b3b926db57c2af7378]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:18.522835 742 tablet_service.cc:1505] Processing CreateTablet for tablet 923fa94e0b404efcb93ac5b1f38609de (DEFAULT_TABLE table=default.Table_26c8d3fab4e146319800a03dc27ced50 [id=a0f26d8f0ede4038822403024188177b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:18.523166 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 923fa94e0b404efcb93ac5b1f38609de. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:18.523173 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3dabe9d8e534953838ce5f0e3259e9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:18.523414 2649 consensus_queue.cc:1048] T f4729ecd182042c2a10f67ca25224101 P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:18.526875 739 tablet_service.cc:1505] Processing CreateTablet for tablet c3dabe9d8e534953838ce5f0e3259e9d (DEFAULT_TABLE table=default.Table_37dad1121b2144ee9eeb3ab7c38a0609 [id=be64f276351541b3b926db57c2af7378]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:18.527259 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3dabe9d8e534953838ce5f0e3259e9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:18.534926 521 catalog_manager.cc:6194] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: default.Table_b9d950092d13427ca8001c487387436a [id=d37290c8b4e8435a8b54bac83428a9ce] alter complete (version 1)
W20251029 02:52:18.535526 526 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a8cef3074aef43b2981eaffdc5a5b5bb consensus state has no leader
I20251029 02:52:18.536827 879 tablet_service.cc:1552] Processing DeleteTablet for tablet 70b008011d5547bbaf3f45685394139f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:03 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:18.537436 879 tablet_service.cc:1552] Processing DeleteTablet for tablet 24f07b1efac64b099b0b5ed467463749 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:59 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:18.535218 1031 tablet_service.cc:1505] Processing CreateTablet for tablet c3dabe9d8e534953838ce5f0e3259e9d (DEFAULT_TABLE table=default.Table_37dad1121b2144ee9eeb3ab7c38a0609 [id=be64f276351541b3b926db57c2af7378]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:18.540439 1028 tablet_service.cc:1505] Processing CreateTablet for tablet 3f3aee27041a49ceb3218d3f43bc6273 (DEFAULT_TABLE table=default.Table_2a4d6457e4db4e9280cd96080b7223b0 [id=3c32e21c17dc4625873c30c019cb2a85]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:18.540911 750 tablet_service.cc:1505] Processing CreateTablet for tablet 3f3aee27041a49ceb3218d3f43bc6273 (DEFAULT_TABLE table=default.Table_2a4d6457e4db4e9280cd96080b7223b0 [id=3c32e21c17dc4625873c30c019cb2a85]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:18.541363 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f3aee27041a49ceb3218d3f43bc6273. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:18.546713 521 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_07b6fa17dfc04d919f5a344a8b225333" } new_table_name: "default.Table_6538a6d5d3984da0ace555f80399827b" modify_external_catalogs: true
I20251029 02:52:18.546713 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_e0250e6a474d4907b9bd8f39596af69c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:18.547277 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e0250e6a474d4907b9bd8f39596af69c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:18.550714 1059 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "abf18e06df9b482a8b25c078b9495272" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:18.554194 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_7ac30f4a6b164b459121be1019009bbc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:18.554800 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7ac30f4a6b164b459121be1019009bbc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:52:18.556699 847 consensus_peers.cc:597] T 72f9d4408f384e7093316d85f092e1b2 P c756528642a543eeacd93cef3e335a7d -> Peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Couldn't send request to peer b495a49349324f7d81908f2da700f0d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 72f9d4408f384e7093316d85f092e1b2. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:18.557665 1166 tablet_bootstrap.cc:492] T 46cd75ff18b14212b556b9bcb3012dd9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:18.557849 1166 ts_tablet_manager.cc:1403] T 46cd75ff18b14212b556b9bcb3012dd9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.089s user 0.002s sys 0.002s
I20251029 02:52:18.558630 1166 raft_consensus.cc:359] T 46cd75ff18b14212b556b9bcb3012dd9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:18.559638 1166 raft_consensus.cc:385] T 46cd75ff18b14212b556b9bcb3012dd9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:18.559739 1166 raft_consensus.cc:740] T 46cd75ff18b14212b556b9bcb3012dd9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:18.559738 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3dabe9d8e534953838ce5f0e3259e9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:18.560012 1166 consensus_queue.cc:260] T 46cd75ff18b14212b556b9bcb3012dd9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
W20251029 02:52:18.562201 523 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c0672a948e5e41c5bbfe7e8199025986 consensus state has no leader
I20251029 02:52:18.562289 1175 raft_consensus.cc:2272] T 23b9b2ea814645eba63e5f6b12cfb966 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:18.563898 1175 ts_tablet_manager.cc:1916] T 23b9b2ea814645eba63e5f6b12cfb966 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:18.565601 1166 ts_tablet_manager.cc:1434] T 46cd75ff18b14212b556b9bcb3012dd9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20251029 02:52:18.565641 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f3aee27041a49ceb3218d3f43bc6273. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:18.567986 523 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet b125cc9aaf4248b8bfa48fb152b832a6 (table default.Table_110e1b64737c4040ada22360d5a373cb [id=3513c69f6b3941849ac5b8ae0a5846d3]). Expected version 5 got 0
I20251029 02:52:18.568217 523 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet 32519fbffcef44a890161e5f3043806c (table default.Table_a78f5837df7b44e791631e421aa7de46 [id=b95fe1c7d6ff4519b90f993f1fa961fd]). Expected version 3 got 0
I20251029 02:52:18.568535 523 catalog_manager.cc:5649] T abf18e06df9b482a8b25c078b9495272 P c756528642a543eeacd93cef3e335a7d reported cstate change: term changed from 0 to 1, leader changed from <none> to c756528642a543eeacd93cef3e335a7d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "c756528642a543eeacd93cef3e335a7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } health_report { overall_health: UNKNOWN } } }
I20251029 02:52:18.570039 1166 tablet_bootstrap.cc:492] T 6142018e44af4d32818adefa2ee43d02 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:18.571708 1166 tablet_bootstrap.cc:654] T 6142018e44af4d32818adefa2ee43d02 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:18.572566 1172 tablet_bootstrap.cc:492] T e3ac6597d52441569a7aeef051087ad2 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:18.572779 1172 ts_tablet_manager.cc:1403] T e3ac6597d52441569a7aeef051087ad2 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.084s user 0.002s sys 0.002s
I20251029 02:52:18.573462 1172 raft_consensus.cc:359] T e3ac6597d52441569a7aeef051087ad2 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:18.573678 1172 raft_consensus.cc:385] T e3ac6597d52441569a7aeef051087ad2 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:18.573769 1172 raft_consensus.cc:740] T e3ac6597d52441569a7aeef051087ad2 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:18.574035 1172 consensus_queue.cc:260] T e3ac6597d52441569a7aeef051087ad2 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:18.574667 1172 ts_tablet_manager.cc:1434] T e3ac6597d52441569a7aeef051087ad2 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 02:52:18.576313 748 tablet_service.cc:1505] Processing CreateTablet for tablet 5d363855f4964e9f922a89d4195e1ba3 (DEFAULT_TABLE table=default.Table_acc016af001143dc8e6dce38c782721f [id=8ad94ae8abf3427db5986427305ed1cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:18.576659 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d363855f4964e9f922a89d4195e1ba3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:18.576669 891 tablet_service.cc:1505] Processing CreateTablet for tablet 5d363855f4964e9f922a89d4195e1ba3 (DEFAULT_TABLE table=default.Table_acc016af001143dc8e6dce38c782721f [id=8ad94ae8abf3427db5986427305ed1cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:18.578995 900 tablet_service.cc:1505] Processing CreateTablet for tablet c4f73fbc5d77464a84bcc53d6193377b (DEFAULT_TABLE table=default.Table_a92bbb1e05604a78a67943dc495cc225 [id=c247356c3a204adb8024d5b1eafc54c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:18.579463 887 tablet_service.cc:1505] Processing CreateTablet for tablet 923fa94e0b404efcb93ac5b1f38609de (DEFAULT_TABLE table=default.Table_26c8d3fab4e146319800a03dc27ced50 [id=a0f26d8f0ede4038822403024188177b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:18.581347 740 tablet_service.cc:1505] Processing CreateTablet for tablet c4f73fbc5d77464a84bcc53d6193377b (DEFAULT_TABLE table=default.Table_a92bbb1e05604a78a67943dc495cc225 [id=c247356c3a204adb8024d5b1eafc54c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:18.581701 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4f73fbc5d77464a84bcc53d6193377b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:18.588052 738 tablet_service.cc:1505] Processing CreateTablet for tablet e4d17459491f4f76b990b64b68fc06a5 (DEFAULT_TABLE table=default.Table_2af94979b80a40e3804eed706e1d4cbe [id=7146697b15e8464280030f42924e7dbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:18.588482 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4d17459491f4f76b990b64b68fc06a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:18.590773 522 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 643e20d9974447daa78885b3401f3b77
W20251029 02:52:18.591766 849 leader_election.cc:343] T abf18e06df9b482a8b25c078b9495272 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
W20251029 02:52:18.593544 849 consensus_peers.cc:597] T 7816556521144c928d7a02694e22e79f P c756528642a543eeacd93cef3e335a7d -> Peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Couldn't send request to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:18.594079 749 tablet_service.cc:1552] Processing DeleteTablet for tablet 643e20d9974447daa78885b3401f3b77 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:18 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:18.597589 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_a75246042ab0422cbfc4c83f2ab5b81c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:18.598186 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a75246042ab0422cbfc4c83f2ab5b81c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:18.599253 879 tablet_service.cc:1552] Processing DeleteTablet for tablet 643e20d9974447daa78885b3401f3b77 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:18 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:18.601375 2649 consensus_queue.cc:1048] T 628a7d0419cb434f8171faf500d585d8 P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:18.601722 744 tablet_service.cc:1552] Processing DeleteTablet for tablet 2b278db5b2b04344b864797055a5f665 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:00 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:18.624044 879 tablet_service.cc:1505] Processing CreateTablet for tablet e4d17459491f4f76b990b64b68fc06a5 (DEFAULT_TABLE table=default.Table_2af94979b80a40e3804eed706e1d4cbe [id=7146697b15e8464280030f42924e7dbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:18.624646 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d363855f4964e9f922a89d4195e1ba3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:18.624910 517 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet af74d277abb149e09900c46baf2ea1d3
I20251029 02:52:18.626689 744 tablet_service.cc:1552] Processing DeleteTablet for tablet af74d277abb149e09900c46baf2ea1d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:18 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:18.627622 744 tablet_service.cc:1505] Processing CreateTablet for tablet 7e97eebe1c134ad8b40f66d26871dc50 (DEFAULT_TABLE table=default.Table_2b0686bf14ba41edafd8823ec10bcae6 [id=ca5ca6e78fff4723a29689855326efad]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:18.627991 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e97eebe1c134ad8b40f66d26871dc50. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:18.630422 1172 tablet_bootstrap.cc:492] T 9492c1027aba4455ad8f539addaed5a7 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:18.630743 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4f73fbc5d77464a84bcc53d6193377b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:18.630940 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 923fa94e0b404efcb93ac5b1f38609de. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:18.631906 1172 tablet_bootstrap.cc:654] T 9492c1027aba4455ad8f539addaed5a7 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:18.638646 525 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet c3dabe9d8e534953838ce5f0e3259e9d
I20251029 02:52:18.639219 525 master_service.cc:954] ReplaceTablet: received request to replace tablet b64880e732f54510a3954248cc940fbc from {username='slave'} at 127.0.0.1:35314
I20251029 02:52:18.639441 895 tablet_service.cc:1552] Processing DeleteTablet for tablet af74d277abb149e09900c46baf2ea1d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:18 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:18.639617 518 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 2e96a398bfd74cf68cf0377a813dc1dd
I20251029 02:52:18.640420 524 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 854bb36d1d3340a9bab2ad2eb4ee8cb5
I20251029 02:52:18.642635 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4d17459491f4f76b990b64b68fc06a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:18.647956 888 tablet_service.cc:1552] Processing DeleteTablet for tablet 854bb36d1d3340a9bab2ad2eb4ee8cb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:18 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:18.575556 1028 tablet_service.cc:1505] Processing CreateTablet for tablet 5d363855f4964e9f922a89d4195e1ba3 (DEFAULT_TABLE table=default.Table_acc016af001143dc8e6dce38c782721f [id=8ad94ae8abf3427db5986427305ed1cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:18.648138 895 tablet_service.cc:1552] Processing DeleteTablet for tablet 2e96a398bfd74cf68cf0377a813dc1dd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:18 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:18.648507 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d363855f4964e9f922a89d4195e1ba3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:18.577556 1031 tablet_service.cc:1505] Processing CreateTablet for tablet c4f73fbc5d77464a84bcc53d6193377b (DEFAULT_TABLE table=default.Table_a92bbb1e05604a78a67943dc495cc225 [id=c247356c3a204adb8024d5b1eafc54c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:18.650822 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4f73fbc5d77464a84bcc53d6193377b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:18.652944 748 tablet_service.cc:1552] Processing DeleteTablet for tablet 2e96a398bfd74cf68cf0377a813dc1dd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:18 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:18.654130 891 tablet_service.cc:1552] Processing DeleteTablet for tablet c3dabe9d8e534953838ce5f0e3259e9d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:18 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:18.654278 740 tablet_service.cc:1552] Processing DeleteTablet for tablet c3dabe9d8e534953838ce5f0e3259e9d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:18 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:18.580166 1039 tablet_service.cc:1505] Processing CreateTablet for tablet e4d17459491f4f76b990b64b68fc06a5 (DEFAULT_TABLE table=default.Table_2af94979b80a40e3804eed706e1d4cbe [id=7146697b15e8464280030f42924e7dbd]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:18.654966 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4d17459491f4f76b990b64b68fc06a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:18.657040 742 tablet_service.cc:1552] Processing DeleteTablet for tablet 854bb36d1d3340a9bab2ad2eb4ee8cb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:18 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:18.590968 1059 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "abf18e06df9b482a8b25c078b9495272" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:52:18.591773 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b125cc9aaf4248b8bfa48fb152b832a6" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:18.592727 1024 tablet_service.cc:1552] Processing DeleteTablet for tablet 643e20d9974447daa78885b3401f3b77 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:18 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:18.610574 1107 heartbeater.cc:344] Connected to a master server at 127.0.104.62:42343
I20251029 02:52:18.625164 1042 tablet_service.cc:1505] Processing CreateTablet for tablet 7e97eebe1c134ad8b40f66d26871dc50 (DEFAULT_TABLE table=default.Table_2b0686bf14ba41edafd8823ec10bcae6 [id=ca5ca6e78fff4723a29689855326efad]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:18.625763 1033 tablet_service.cc:1552] Processing DeleteTablet for tablet af74d277abb149e09900c46baf2ea1d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:18 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:18.658851 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e97eebe1c134ad8b40f66d26871dc50. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:18.659029 1176 tablet_replica.cc:333] T 643e20d9974447daa78885b3401f3b77 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:18.659327 1176 raft_consensus.cc:2243] T 643e20d9974447daa78885b3401f3b77 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:18.659533 1176 raft_consensus.cc:2272] T 643e20d9974447daa78885b3401f3b77 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:18.660035 1176 ts_tablet_manager.cc:1916] T 643e20d9974447daa78885b3401f3b77 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:18.641837 1027 tablet_service.cc:1552] Processing DeleteTablet for tablet c3dabe9d8e534953838ce5f0e3259e9d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:18 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:18.642000 1034 tablet_service.cc:1552] Processing DeleteTablet for tablet 854bb36d1d3340a9bab2ad2eb4ee8cb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:18 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:18.642000 1040 tablet_service.cc:1552] Processing DeleteTablet for tablet 2e96a398bfd74cf68cf0377a813dc1dd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:18 UTC) from {username='slave'} at 127.0.0.1:37626
W20251029 02:52:18.668790 849 leader_election.cc:343] T abf18e06df9b482a8b25c078b9495272 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
W20251029 02:52:18.669461 849 leader_election.cc:343] T b125cc9aaf4248b8bfa48fb152b832a6 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:18.669554 849 leader_election.cc:304] T b125cc9aaf4248b8bfa48fb152b832a6 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9
I20251029 02:52:18.669883 2747 raft_consensus.cc:2749] T b125cc9aaf4248b8bfa48fb152b832a6 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 02:52:18.670529 849 consensus_peers.cc:597] T 078ace5eada044c88d2462efb2b57aa8 P c756528642a543eeacd93cef3e335a7d -> Peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Couldn't send request to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 078ace5eada044c88d2462efb2b57aa8. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:18.670745 1034 tablet_service.cc:1505] Processing CreateTablet for tablet b00b7696d0dc46a2aec79ebf3c66f39f (DEFAULT_TABLE table=default.Table_abf33e38626c4d3e9bd944fecb1d1490 [id=e93a75bd377f4e6f9eec37f7d49d7373]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:18.671157 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b00b7696d0dc46a2aec79ebf3c66f39f. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:18.671713 849 consensus_peers.cc:597] T 72f9d4408f384e7093316d85f092e1b2 P c756528642a543eeacd93cef3e335a7d -> Peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Couldn't send request to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 72f9d4408f384e7093316d85f092e1b2. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:18.672044 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "023a4094b424464c8f49d6eb9cc15da4" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:18.658456 1174 ts_tablet_manager.cc:1929] T 2e97d1bc06a24e8d8ba11acb6fd280d8 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:18.673032 1174 log.cc:1199] T 2e97d1bc06a24e8d8ba11acb6fd280d8 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/2e97d1bc06a24e8d8ba11acb6fd280d8
I20251029 02:52:18.674719 1040 tablet_service.cc:1505] Processing CreateTablet for tablet 44c13b35b573487fa5114f7c856a6b8e (DEFAULT_TABLE table=default.Table_b997b92037ef41199e6a64c11b0bfb81 [id=a878dffbe954402d98caf97c4a5451fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:18.674785 740 tablet_service.cc:1505] Processing CreateTablet for tablet b00b7696d0dc46a2aec79ebf3c66f39f (DEFAULT_TABLE table=default.Table_abf33e38626c4d3e9bd944fecb1d1490 [id=e93a75bd377f4e6f9eec37f7d49d7373]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:18.675093 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44c13b35b573487fa5114f7c856a6b8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:18.675141 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b00b7696d0dc46a2aec79ebf3c66f39f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:18.676601 1176 ts_tablet_manager.cc:1929] T 643e20d9974447daa78885b3401f3b77 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251029 02:52:18.676739 1176 log.cc:1199] T 643e20d9974447daa78885b3401f3b77 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/643e20d9974447daa78885b3401f3b77
I20251029 02:52:18.677114 1176 ts_tablet_manager.cc:1950] T 643e20d9974447daa78885b3401f3b77 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:18.678766 742 tablet_service.cc:1505] Processing CreateTablet for tablet 44c13b35b573487fa5114f7c856a6b8e (DEFAULT_TABLE table=default.Table_b997b92037ef41199e6a64c11b0bfb81 [id=a878dffbe954402d98caf97c4a5451fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:18.679107 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44c13b35b573487fa5114f7c856a6b8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:18.685678 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 643e20d9974447daa78885b3401f3b77 (table default.Table_7c1cb64e05be4f6e8ac28286c982a3e7 [id=b4fc453bf54d45c4b8dddc3c49bf48f8]) successfully deleted
W20251029 02:52:18.685921 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet af74d277abb149e09900c46baf2ea1d3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet af74d277abb149e09900c46baf2ea1d3 already in progress: creating tablet
W20251029 02:52:18.686106 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet c3dabe9d8e534953838ce5f0e3259e9d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c3dabe9d8e534953838ce5f0e3259e9d already in progress: creating tablet
W20251029 02:52:18.690052 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 854bb36d1d3340a9bab2ad2eb4ee8cb5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 854bb36d1d3340a9bab2ad2eb4ee8cb5 already in progress: creating tablet
W20251029 02:52:18.691051 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 2e96a398bfd74cf68cf0377a813dc1dd because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2e96a398bfd74cf68cf0377a813dc1dd already in progress: creating tablet
W20251029 02:52:18.694352 521 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9c8894de7eef4e43957ee7863ebbd087 consensus state has no leader
I20251029 02:52:18.694536 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_82605703457f430e9edcf160c498ad0c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:18.694784 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_810b941251184116ac99d736431a5978"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:18.695046 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_82605703457f430e9edcf160c498ad0c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:52:18.695125 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_810b941251184116ac99d736431a5978 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:18.696419 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_ea69b281d8ca4cf5a9536a17ce80e7d2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:18.696861 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ea69b281d8ca4cf5a9536a17ce80e7d2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:18.699050 1166 tablet_bootstrap.cc:492] T 6142018e44af4d32818adefa2ee43d02 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:18.699231 1166 ts_tablet_manager.cc:1403] T 6142018e44af4d32818adefa2ee43d02 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.129s user 0.003s sys 0.000s
I20251029 02:52:18.700008 1042 tablet_service.cc:1505] Processing CreateTablet for tablet 0997a490132e44138908aaaa91184ec7 (DEFAULT_TABLE table=default.Table_9ad0c74be72449c7b3868451dd9ac8d2 [id=5a96fb76b10c48b585e251ba7cd80e88]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:18.700013 1166 raft_consensus.cc:359] T 6142018e44af4d32818adefa2ee43d02 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:18.700356 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0997a490132e44138908aaaa91184ec7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:18.700368 1166 raft_consensus.cc:385] T 6142018e44af4d32818adefa2ee43d02 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:18.700551 1166 raft_consensus.cc:740] T 6142018e44af4d32818adefa2ee43d02 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:18.700903 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_5a9988516aa349aeb264f073696be956"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:18.701064 712 leader_election.cc:343] T 023a4094b424464c8f49d6eb9cc15da4 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:18.700786 1166 consensus_queue.cc:260] T 6142018e44af4d32818adefa2ee43d02 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:18.701180 712 leader_election.cc:304] T 023a4094b424464c8f49d6eb9cc15da4 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
W20251029 02:52:18.701346 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5a9988516aa349aeb264f073696be956 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:18.701642 1166 ts_tablet_manager.cc:1434] T 6142018e44af4d32818adefa2ee43d02 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 02:52:18.701768 1175 ts_tablet_manager.cc:1929] T 23b9b2ea814645eba63e5f6b12cfb966 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:18.701898 1175 log.cc:1199] T 23b9b2ea814645eba63e5f6b12cfb966 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/23b9b2ea814645eba63e5f6b12cfb966
I20251029 02:52:18.702767 2908 raft_consensus.cc:2749] T 023a4094b424464c8f49d6eb9cc15da4 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:18.707648 879 tablet_service.cc:1505] Processing CreateTablet for tablet 7e97eebe1c134ad8b40f66d26871dc50 (DEFAULT_TABLE table=default.Table_2b0686bf14ba41edafd8823ec10bcae6 [id=ca5ca6e78fff4723a29689855326efad]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:18.708272 895 tablet_service.cc:1505] Processing CreateTablet for tablet 0997a490132e44138908aaaa91184ec7 (DEFAULT_TABLE table=default.Table_9ad0c74be72449c7b3868451dd9ac8d2 [id=5a96fb76b10c48b585e251ba7cd80e88]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:18.708585 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e97eebe1c134ad8b40f66d26871dc50. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:18.709108 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0997a490132e44138908aaaa91184ec7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:18.709554 891 tablet_service.cc:1505] Processing CreateTablet for tablet 44c13b35b573487fa5114f7c856a6b8e (DEFAULT_TABLE table=default.Table_b997b92037ef41199e6a64c11b0bfb81 [id=a878dffbe954402d98caf97c4a5451fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:18.709872 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44c13b35b573487fa5114f7c856a6b8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:18.711230 900 tablet_service.cc:1505] Processing CreateTablet for tablet b00b7696d0dc46a2aec79ebf3c66f39f (DEFAULT_TABLE table=default.Table_abf33e38626c4d3e9bd944fecb1d1490 [id=e93a75bd377f4e6f9eec37f7d49d7373]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:18.711580 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b00b7696d0dc46a2aec79ebf3c66f39f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:18.711889 738 tablet_service.cc:1505] Processing CreateTablet for tablet 0997a490132e44138908aaaa91184ec7 (DEFAULT_TABLE table=default.Table_9ad0c74be72449c7b3868451dd9ac8d2 [id=5a96fb76b10c48b585e251ba7cd80e88]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:18.712420 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0997a490132e44138908aaaa91184ec7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:18.716302 888 tablet_service.cc:1505] Processing CreateTablet for tablet e9387edc3c434f5881a3a43e79a3af58 (DEFAULT_TABLE table=default.Table_a75c04341d7244f3ba161a938bb401b3 [id=0dc6f8e6205d4e3997946747bbc86e2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:18.716621 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9387edc3c434f5881a3a43e79a3af58. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:18.717648 1166 tablet_bootstrap.cc:492] T 6dff036b32904ddb9598be4b79504bb6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:18.720824 520 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_2d6bca94a5a14e4ea414e1863b335737" } new_table_name: "default.Table_b67322fbe55d4bf4a6baba1868cac544" modify_external_catalogs: true
I20251029 02:52:18.722077 2932 tablet.cc:1722] T 32519fbffcef44a890161e5f3043806c P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251029 02:52:18.724639 1040 tablet_service.cc:1505] Processing CreateTablet for tablet e9387edc3c434f5881a3a43e79a3af58 (DEFAULT_TABLE table=default.Table_a75c04341d7244f3ba161a938bb401b3 [id=0dc6f8e6205d4e3997946747bbc86e2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:18.726361 1174 ts_tablet_manager.cc:1950] T 2e97d1bc06a24e8d8ba11acb6fd280d8 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:18.728480 1166 tablet_bootstrap.cc:654] T 6dff036b32904ddb9598be4b79504bb6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
W20251029 02:52:18.728725 849 consensus_peers.cc:597] T 628a7d0419cb434f8171faf500d585d8 P c756528642a543eeacd93cef3e335a7d -> Peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Couldn't send request to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:18.740011 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9387edc3c434f5881a3a43e79a3af58. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:18.758347 849 consensus_peers.cc:597] T eca016aed05e4ce09e1a2bf96e2286f5 P c756528642a543eeacd93cef3e335a7d -> Peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Couldn't send request to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:18.758622 1175 ts_tablet_manager.cc:1950] T 23b9b2ea814645eba63e5f6b12cfb966 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
W20251029 02:52:18.759508 849 consensus_peers.cc:597] T f4729ecd182042c2a10f67ca25224101 P c756528642a543eeacd93cef3e335a7d -> Peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Couldn't send request to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:18.759852 1166 tablet_bootstrap.cc:492] T 6dff036b32904ddb9598be4b79504bb6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:18.760030 1166 ts_tablet_manager.cc:1403] T 6dff036b32904ddb9598be4b79504bb6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.042s user 0.002s sys 0.002s
I20251029 02:52:18.760037 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 1058) took 4857 ms. Trace:
I20251029 02:52:18.760249 1175 rpcz_store.cc:276] 1029 02:52:13.902970 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:13.903016 (+ 46us) service_pool.cc:225] Handling call
1029 02:52:18.522141 (+4619125us) ts_tablet_manager.cc:1151] Deleting tablet 23b9b2ea814645eba63e5f6b12cfb966
1029 02:52:18.522149 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:18.619641 (+ 97492us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:18.694556 (+ 74915us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:18.760017 (+ 65461us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4618979}
I20251029 02:52:18.760730 1175 tablet_replica.cc:333] T d29cc2a0c8b84e3ebadc7f571d666847 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:18.760954 1175 raft_consensus.cc:2243] T d29cc2a0c8b84e3ebadc7f571d666847 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:18.761009 1166 raft_consensus.cc:359] T 6dff036b32904ddb9598be4b79504bb6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:18.761266 1166 raft_consensus.cc:385] T 6dff036b32904ddb9598be4b79504bb6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:18.761199 1175 raft_consensus.cc:2272] T d29cc2a0c8b84e3ebadc7f571d666847 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:18.761341 1166 raft_consensus.cc:740] T 6dff036b32904ddb9598be4b79504bb6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:18.766052 1175 ts_tablet_manager.cc:1916] T d29cc2a0c8b84e3ebadc7f571d666847 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:18.766945 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 966) took 7957 ms. Trace:
I20251029 02:52:18.767076 1174 rpcz_store.cc:276] 1029 02:52:10.809234 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:10.809286 (+ 52us) service_pool.cc:225] Handling call
1029 02:52:18.505066 (+7695780us) ts_tablet_manager.cc:1151] Deleting tablet 2e97d1bc06a24e8d8ba11acb6fd280d8
1029 02:52:18.505077 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:18.553712 (+ 48635us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:18.658352 (+104640us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:18.749998 (+ 91646us) spinlock_profiling.cc:243] Waited 1.25 ms on lock 0x61800000d518. stack: 00007f1e09cad9f6 00007f1e09cad7b0 00007f1e08c893ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f1e09c962fd 00007f1e09c9736e 00007f1e1d8d2c9d 00007f1e1d851203 00007f1e19a866f3 00007f1e19ab92d4 00007f1e19a99d70 00007f1e19a99afd 0000000000a8867c 00007f1e09cf3d7f
1029 02:52:18.766919 (+ 16921us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2867584,"tablet-delete.queue_time_us":7695598}
I20251029 02:52:18.767587 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 2e97d1bc06a24e8d8ba11acb6fd280d8 (table default.Table_24be2b894899463a810d2fe7511d4d6b [id=43e922b1efe44ad9882b5084b22d1560]) successfully deleted
I20251029 02:52:18.769133 1174 tablet_replica.cc:333] T 061f3d38c4604fa4b57e74ff6f6fc7da P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:18.769381 1174 raft_consensus.cc:2243] T 061f3d38c4604fa4b57e74ff6f6fc7da P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:18.769573 1174 raft_consensus.cc:2272] T 061f3d38c4604fa4b57e74ff6f6fc7da P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:18.770153 1174 ts_tablet_manager.cc:1916] T 061f3d38c4604fa4b57e74ff6f6fc7da P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:18.773942 1175 ts_tablet_manager.cc:1929] T d29cc2a0c8b84e3ebadc7f571d666847 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:18.774744 1175 log.cc:1199] T d29cc2a0c8b84e3ebadc7f571d666847 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/d29cc2a0c8b84e3ebadc7f571d666847
I20251029 02:52:18.776896 1172 tablet_bootstrap.cc:492] T 9492c1027aba4455ad8f539addaed5a7 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:18.777065 1172 ts_tablet_manager.cc:1403] T 9492c1027aba4455ad8f539addaed5a7 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.147s user 0.004s sys 0.000s
W20251029 02:52:18.777684 849 consensus_peers.cc:597] T f93b0c39d7174dabbe018b1546dbc198 P c756528642a543eeacd93cef3e335a7d -> Peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Couldn't send request to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:18.779119 1167 raft_consensus.cc:385] T b0fec06ad37d4c12a7045d424b88cc5e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:18.779269 1167 raft_consensus.cc:740] T b0fec06ad37d4c12a7045d424b88cc5e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:18.779654 1167 consensus_queue.cc:260] T b0fec06ad37d4c12a7045d424b88cc5e P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:18.780694 1167 ts_tablet_manager.cc:1434] T b0fec06ad37d4c12a7045d424b88cc5e P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.280s user 0.018s sys 0.000s
I20251029 02:52:18.781236 1175 ts_tablet_manager.cc:1950] T d29cc2a0c8b84e3ebadc7f571d666847 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:18.783875 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 1059) took 4878 ms. Trace:
I20251029 02:52:18.784041 1175 rpcz_store.cc:276] 1029 02:52:13.905516 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:13.905561 (+ 45us) service_pool.cc:225] Handling call
1029 02:52:18.760683 (+4855122us) ts_tablet_manager.cc:1151] Deleting tablet d29cc2a0c8b84e3ebadc7f571d666847
1029 02:52:18.760693 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:18.770860 (+ 10167us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:18.773835 (+ 2975us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:18.783697 (+ 9862us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4854958}
I20251029 02:52:18.785634 2747 consensus_queue.cc:1048] T eca016aed05e4ce09e1a2bf96e2286f5 P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:18.787137 1175 tablet_replica.cc:333] T eca016aed05e4ce09e1a2bf96e2286f5 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:18.787379 1175 raft_consensus.cc:2243] T eca016aed05e4ce09e1a2bf96e2286f5 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:18.787575 1175 raft_consensus.cc:2272] T eca016aed05e4ce09e1a2bf96e2286f5 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:18.788158 1175 ts_tablet_manager.cc:1916] T eca016aed05e4ce09e1a2bf96e2286f5 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:18.789983 1172 raft_consensus.cc:359] T 9492c1027aba4455ad8f539addaed5a7 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:18.790158 1172 raft_consensus.cc:385] T 9492c1027aba4455ad8f539addaed5a7 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:18.790217 1172 raft_consensus.cc:740] T 9492c1027aba4455ad8f539addaed5a7 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:18.790392 1172 consensus_queue.cc:260] T 9492c1027aba4455ad8f539addaed5a7 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:18.795308 1166 consensus_queue.cc:260] T 6dff036b32904ddb9598be4b79504bb6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:18.796111 1166 ts_tablet_manager.cc:1434] T 6dff036b32904ddb9598be4b79504bb6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.036s user 0.001s sys 0.001s
I20251029 02:52:18.802788 2747 consensus_queue.cc:1048] T 7816556521144c928d7a02694e22e79f P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251029 02:52:18.805148 847 consensus_peers.cc:597] T eca016aed05e4ce09e1a2bf96e2286f5 P c756528642a543eeacd93cef3e335a7d -> Peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Couldn't send request to peer b495a49349324f7d81908f2da700f0d9. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:18.806932 1175 ts_tablet_manager.cc:1929] T eca016aed05e4ce09e1a2bf96e2286f5 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:18.807054 1175 log.cc:1199] T eca016aed05e4ce09e1a2bf96e2286f5 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/eca016aed05e4ce09e1a2bf96e2286f5
I20251029 02:52:18.814733 2946 tablet.cc:1722] T 32519fbffcef44a890161e5f3043806c P b495a49349324f7d81908f2da700f0d9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251029 02:52:18.815054 2747 raft_consensus.cc:493] T 589a521f5cc24e6289c37b9cf256bce5 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:18.815203 2747 raft_consensus.cc:515] T 589a521f5cc24e6289c37b9cf256bce5 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:18.815771 2747 leader_election.cc:290] T 589a521f5cc24e6289c37b9cf256bce5 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
W20251029 02:52:18.816784 849 leader_election.cc:343] T 589a521f5cc24e6289c37b9cf256bce5 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Not found: Tablet not found: 589a521f5cc24e6289c37b9cf256bce5
I20251029 02:52:18.817426 1167 tablet_bootstrap.cc:492] T 055184c641ba45a588bca611fb24d130 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:18.817778 766 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "589a521f5cc24e6289c37b9cf256bce5" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:18.817984 766 raft_consensus.cc:2393] T 589a521f5cc24e6289c37b9cf256bce5 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c756528642a543eeacd93cef3e335a7d in current term 1: Already voted for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in this term.
I20251029 02:52:18.818864 847 leader_election.cc:304] T 589a521f5cc24e6289c37b9cf256bce5 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9
I20251029 02:52:18.819290 2747 raft_consensus.cc:3060] T 589a521f5cc24e6289c37b9cf256bce5 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:18.820544 1166 tablet_bootstrap.cc:492] T eca016aed05e4ce09e1a2bf96e2286f5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:18.822135 1166 tablet_bootstrap.cc:654] T eca016aed05e4ce09e1a2bf96e2286f5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:18.823030 1175 ts_tablet_manager.cc:1950] T eca016aed05e4ce09e1a2bf96e2286f5 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:18.823393 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 1060) took 4917 ms. Trace:
I20251029 02:52:18.823505 1175 rpcz_store.cc:276] 1029 02:52:13.906233 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:13.906280 (+ 47us) service_pool.cc:225] Handling call
1029 02:52:18.787089 (+4880809us) ts_tablet_manager.cc:1151] Deleting tablet eca016aed05e4ce09e1a2bf96e2286f5
1029 02:52:18.787102 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:18.792576 (+ 5474us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:18.806825 (+ 14249us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:18.823371 (+ 16546us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4880675}
I20251029 02:52:18.823882 1175 tablet_replica.cc:333] T ad214c870ee649279ed7321c5b69cbee P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:18.824121 1175 raft_consensus.cc:2243] T ad214c870ee649279ed7321c5b69cbee P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:18.824311 1175 raft_consensus.cc:2272] T ad214c870ee649279ed7321c5b69cbee P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:18.824936 1175 ts_tablet_manager.cc:1916] T ad214c870ee649279ed7321c5b69cbee P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 02:52:18.829640 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.3:45003, user_credentials={real_user=slave}} blocked reactor thread for 185416us
W20251029 02:52:18.834618 711 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:41701, user_credentials={real_user=slave}} blocked reactor thread for 97207.2us
I20251029 02:52:18.838009 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 23b9b2ea814645eba63e5f6b12cfb966 (table default.Table_ab94a2a8b29d498a9f201f39228d7bfe [id=4876e3df916248d2a64ba3cad736b5ab]) successfully deleted
I20251029 02:52:18.838495 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet d29cc2a0c8b84e3ebadc7f571d666847 (table default.Table_8fee569eca38413dbac56707725c6a46 [id=50de770b3b9b4d8c85546014a959da24]) successfully deleted
I20251029 02:52:18.839891 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet eca016aed05e4ce09e1a2bf96e2286f5 (table default.Table_ee6dec2fc7834c78914ddc9f13890c55 [id=2faf0479a2a6477fa1d70d30094ef1db]) successfully deleted
I20251029 02:52:18.841125 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_104c94e8fd7d4479a81d52d44bbcea78"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:18.841727 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_104c94e8fd7d4479a81d52d44bbcea78 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:18.842036 738 tablet_service.cc:1505] Processing CreateTablet for tablet e9387edc3c434f5881a3a43e79a3af58 (DEFAULT_TABLE table=default.Table_a75c04341d7244f3ba161a938bb401b3 [id=0dc6f8e6205d4e3997946747bbc86e2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:18.842036 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_da61417a3a7c4cf6802ec3f656cceea1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:18.842553 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e9387edc3c434f5881a3a43e79a3af58. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:18.842550 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_da61417a3a7c4cf6802ec3f656cceea1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:18.842484 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_83e15301d9354c94994239617d931177"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:18.842950 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_83e15301d9354c94994239617d931177 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:18.843590 525 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet b64880e732f54510a3954248cc940fbc
I20251029 02:52:18.844067 525 catalog_manager.cc:6719] ReplaceTablet: tablet b64880e732f54510a3954248cc940fbc deleted and replaced by tablet e095f00f34a145b1bbd22deecb899ecc
I20251029 02:52:18.844357 525 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_9dbf9a53f1e34611976c67bff0ac7f9f" } new_table_name: "default.Table_5cfd989a6edc401ab42490cc500c8948" modify_external_catalogs: true
I20251029 02:52:18.844400 517 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet f4729ecd182042c2a10f67ca25224101 (table default.Table_973ef856041949598a23da3dab8b40db [id=1dd2d5488bc647c8b82c53886583b28d]). Expected version 4 got 0
I20251029 02:52:18.845607 517 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet 32519fbffcef44a890161e5f3043806c (table default.Table_a78f5837df7b44e791631e421aa7de46 [id=b95fe1c7d6ff4519b90f993f1fa961fd]). Expected version 3 got 0
I20251029 02:52:18.845980 519 catalog_manager.cc:6194] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: default.Table_1d57de91ade24f33abd0740342749a08 [id=9f95160639ad40e995a6c89e297a0fad] alter complete (version 5)
I20251029 02:52:18.848217 888 tablet_service.cc:1552] Processing DeleteTablet for tablet b64880e732f54510a3954248cc940fbc with delete_type TABLET_DATA_DELETED (replaced by tablet e095f00f34a145b1bbd22deecb899ecc) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:18.850380 1042 tablet_service.cc:1552] Processing DeleteTablet for tablet b64880e732f54510a3954248cc940fbc with delete_type TABLET_DATA_DELETED (replaced by tablet e095f00f34a145b1bbd22deecb899ecc) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:18.850922 740 tablet_service.cc:1552] Processing DeleteTablet for tablet b64880e732f54510a3954248cc940fbc with delete_type TABLET_DATA_DELETED (replaced by tablet e095f00f34a145b1bbd22deecb899ecc) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:18.852650 518 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_69fcb07379dc4054b90c1c6d3e9c7985" } new_table_name: "default.Table_d5738f8e6a034761b854ca361a4428ab" modify_external_catalogs: true
I20251029 02:52:18.860304 523 catalog_manager.cc:6194] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: default.Table_1e69e259edba466f99ed7a41dcf6367f [id=a2f3b36dc78a4e94bd054a6a5712b384] alter complete (version 1)
I20251029 02:52:18.862016 526 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_6442d40b6bad4ac9a2dbd42a8a3cb096" } new_table_name: "default.Table_c7372ed8954c4607896e1db7a4330b04" modify_external_catalogs: true
W20251029 02:52:18.863327 519 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.3:48627 (request call id 118) took 797 ms (client timeout 1000 ms). Trace:
I20251029 02:52:18.863879 521 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_09c621ba9fce4850ae1f0019e3c2abf8" } modify_external_catalogs: true
W20251029 02:52:18.863426 519 rpcz_store.cc:269] 1029 02:52:18.066224 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:18.102019 (+ 35795us) service_pool.cc:225] Handling call
1029 02:52:18.104069 (+ 2050us) spinlock_profiling.cc:243] Waited 1.43 ms on lock 0x616000041254. stack: 00007f4b1e4899f6 00007f4b1e4897b0 00007f4b1d4653ad 0000000000983f04 0000000000983ecd 00007f4b2d7906bd 00007f4b2d7907b3 00007f4b2a357441 00007f4b320119dd 00007f4b32198aa8 00007f4b28cabca6 00007f4b28cab9d5 00007f4b2542827e 00007f4b2542715e 00007f4b2542d395 00007f4b2542dcbf
1029 02:52:18.863304 (+759235us) inbound_call.cc:173] Queueing success response
Related trace 'op':
1029 02:52:18.609255 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
1029 02:52:18.609358 (+ 103us) write_op.cc:432] Acquiring schema lock in shared mode
1029 02:52:18.609363 (+ 5us) write_op.cc:435] Acquired schema lock
1029 02:52:18.609365 (+ 2us) tablet.cc:660] Decoding operations
1029 02:52:18.609438 (+ 73us) write_op.cc:620] Acquiring the partition lock for write op
1029 02:52:18.609445 (+ 7us) write_op.cc:641] Partition lock acquired for write op
1029 02:52:18.609450 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
1029 02:52:18.609494 (+ 44us) tablet.cc:699] Row locks acquired
1029 02:52:18.609497 (+ 3us) write_op.cc:260] PREPARE: finished
1029 02:52:18.609521 (+ 24us) write_op.cc:270] Start()
1029 02:52:18.609545 (+ 24us) write_op.cc:276] Timestamp: P: 1761706338609448 usec, L: 0
1029 02:52:18.609548 (+ 3us) op_driver.cc:348] REPLICATION: starting
1029 02:52:18.609674 (+ 126us) log.cc:844] Serialized 356 byte log entry
1029 02:52:18.636411 (+ 26737us) op_driver.cc:464] REPLICATION: finished
1029 02:52:18.637798 (+ 1387us) write_op.cc:301] APPLY: starting
1029 02:52:18.637807 (+ 9us) tablet.cc:1365] starting BulkCheckPresence
1029 02:52:18.637830 (+ 23us) tablet.cc:1368] finished BulkCheckPresence
1029 02:52:18.637865 (+ 35us) tablet.cc:1370] starting ApplyRowOperation cycle
1029 02:52:18.637891 (+ 26us) tablet.cc:1381] finished ApplyRowOperation cycle
1029 02:52:18.637912 (+ 21us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
1029 02:52:18.637919 (+ 7us) write_op.cc:312] APPLY: finished
1029 02:52:18.637959 (+ 40us) log.cc:844] Serialized 23 byte log entry
1029 02:52:18.637992 (+ 33us) write_op.cc:489] Releasing partition, row and schema locks
1029 02:52:18.638001 (+ 9us) write_op.cc:454] Released schema lock
1029 02:52:18.638005 (+ 4us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"reported_tablets":6,"spinlock_wait_cycles":3282688,"child_traces":[["op",{"apply.queue_time_us":1361,"apply.run_cpu_time_us":262,"apply.run_wall_time_us":262,"num_ops":1,"prepare.queue_time_us":71637,"prepare.run_cpu_time_us":494,"prepare.run_wall_time_us":747,"replication_time_us":26848,"thread_start_us":54632,"threads_started":1}]]}
I20251029 02:52:18.864030 521 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_09c621ba9fce4850ae1f0019e3c2abf8" } modify_external_catalogs: true
I20251029 02:52:18.864277 519 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_f90425a19e964b75b94c68952d26ae1e" } modify_external_catalogs: true
I20251029 02:52:18.864414 519 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_f90425a19e964b75b94c68952d26ae1e" } modify_external_catalogs: true
I20251029 02:52:18.865759 524 master_service.cc:954] ReplaceTablet: received request to replace tablet 2849e856dd714da88fca5a9fafe74c22 from {username='slave'} at 127.0.0.1:35314
I20251029 02:52:18.866214 524 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_f2291437414b4849a6e2443780dd1c78" } modify_external_catalogs: true
I20251029 02:52:18.866334 524 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_f2291437414b4849a6e2443780dd1c78" } modify_external_catalogs: true
W20251029 02:52:18.867921 525 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet afd3fb2157c2445bbe8dbf0f28089c2e consensus state has no leader
I20251029 02:52:18.867987 517 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_ad533f1a574e4b30bc8875869b5102a4" } modify_external_catalogs: true
I20251029 02:52:18.868111 517 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_ad533f1a574e4b30bc8875869b5102a4" } modify_external_catalogs: true
I20251029 02:52:18.868191 525 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_896799e8c5bd4815973929a8ba006a36" } modify_external_catalogs: true
I20251029 02:52:18.868299 525 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_896799e8c5bd4815973929a8ba006a36" } modify_external_catalogs: true
W20251029 02:52:18.863361 520 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a4e0554ae01f47b58127468f8ec3768e consensus state has no leader
I20251029 02:52:18.871274 888 tablet_service.cc:1552] Processing DeleteTablet for tablet 2e97d1bc06a24e8d8ba11acb6fd280d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:55 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:18.871472 891 tablet_service.cc:1552] Processing DeleteTablet for tablet 589a521f5cc24e6289c37b9cf256bce5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:57 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:18.873225 740 tablet_service.cc:1552] Processing DeleteTablet for tablet 23b9b2ea814645eba63e5f6b12cfb966 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:57 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:18.873445 744 tablet_service.cc:1552] Processing DeleteTablet for tablet f18b0ca5a6a44175a912642e9eea1778 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:58 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:18.873721 1042 tablet_service.cc:1552] Processing DeleteTablet for tablet c081d2eacfac4ca8a0048e2a1c4f8f83 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:04 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:18.874214 738 tablet_service.cc:1552] Processing DeleteTablet for tablet 7fb5b908db77453da5cfb32e2c955f41 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:03 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:18.874380 742 tablet_service.cc:1552] Processing DeleteTablet for tablet 023a4094b424464c8f49d6eb9cc15da4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:58 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:18.877741 1175 ts_tablet_manager.cc:1929] T ad214c870ee649279ed7321c5b69cbee P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:18.878753 1176 tablet_replica.cc:333] T c081d2eacfac4ca8a0048e2a1c4f8f83 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:18.879027 1176 raft_consensus.cc:2243] T c081d2eacfac4ca8a0048e2a1c4f8f83 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:18.879222 1176 raft_consensus.cc:2272] T c081d2eacfac4ca8a0048e2a1c4f8f83 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251029 02:52:18.879680 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet b64880e732f54510a3954248cc940fbc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b64880e732f54510a3954248cc940fbc already in progress: creating tablet
I20251029 02:52:18.880282 1176 ts_tablet_manager.cc:1916] T c081d2eacfac4ca8a0048e2a1c4f8f83 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:18.887513 1176 ts_tablet_manager.cc:1929] T c081d2eacfac4ca8a0048e2a1c4f8f83 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:18.888378 1176 log.cc:1199] T c081d2eacfac4ca8a0048e2a1c4f8f83 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/c081d2eacfac4ca8a0048e2a1c4f8f83
I20251029 02:52:18.888804 1176 ts_tablet_manager.cc:1950] T c081d2eacfac4ca8a0048e2a1c4f8f83 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:18.889415 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet c081d2eacfac4ca8a0048e2a1c4f8f83 (table default.Table_dbf2e434d7964eac80709c205f61191e [id=b6afde5f860040be9ede53221978f44d]) successfully deleted
I20251029 02:52:18.890266 1166 tablet_bootstrap.cc:492] T eca016aed05e4ce09e1a2bf96e2286f5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:18.890465 1166 ts_tablet_manager.cc:1403] T eca016aed05e4ce09e1a2bf96e2286f5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.070s user 0.001s sys 0.010s
I20251029 02:52:18.891582 1166 raft_consensus.cc:359] T eca016aed05e4ce09e1a2bf96e2286f5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:18.891813 1166 raft_consensus.cc:385] T eca016aed05e4ce09e1a2bf96e2286f5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:18.891921 1166 raft_consensus.cc:740] T eca016aed05e4ce09e1a2bf96e2286f5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:18.892184 1166 consensus_queue.cc:260] T eca016aed05e4ce09e1a2bf96e2286f5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:18.893074 523 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet 6dff036b32904ddb9598be4b79504bb6 (table default.Table_0ae102f09fdc41bcbead0148653df9e2 [id=65ec61bf400a4628b98e64e7d43c025c]). Expected version 2 got 0
I20251029 02:52:18.898417 1175 log.cc:1199] T ad214c870ee649279ed7321c5b69cbee P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/ad214c870ee649279ed7321c5b69cbee
I20251029 02:52:18.899277 1175 ts_tablet_manager.cc:1950] T ad214c870ee649279ed7321c5b69cbee P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:18.890996 2950 raft_consensus.cc:493] T f18b0ca5a6a44175a912642e9eea1778 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:18.900828 2950 raft_consensus.cc:515] T f18b0ca5a6a44175a912642e9eea1778 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:18.901021 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 1061) took 4994 ms. Trace:
I20251029 02:52:18.901098 1175 rpcz_store.cc:276] 1029 02:52:13.906892 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:13.906936 (+ 44us) service_pool.cc:225] Handling call
1029 02:52:18.823810 (+4916874us) ts_tablet_manager.cc:1151] Deleting tablet ad214c870ee649279ed7321c5b69cbee
1029 02:52:18.823822 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:18.858301 (+ 34479us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:18.877630 (+ 19329us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:18.901005 (+ 23375us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4878645}
I20251029 02:52:18.901523 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet ad214c870ee649279ed7321c5b69cbee (table default.Table_982d01dccabe47bfaac7a582282c5d54 [id=033c1db6a2b44a2e99361183fd18c8f0]) successfully deleted
I20251029 02:52:18.901681 2950 leader_election.cc:290] T f18b0ca5a6a44175a912642e9eea1778 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:18.902858 1166 ts_tablet_manager.cc:1434] T eca016aed05e4ce09e1a2bf96e2286f5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20251029 02:52:18.902853 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_c4d539ac548a4514befd48f0460d1061"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:18.903400 1175 tablet_replica.cc:333] T 786d9148fb1d417a8c2c8b8c53665899 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
W20251029 02:52:18.903424 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c4d539ac548a4514befd48f0460d1061 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:18.904896 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f18b0ca5a6a44175a912642e9eea1778" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:18.905471 1172 ts_tablet_manager.cc:1434] T 9492c1027aba4455ad8f539addaed5a7 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.128s user 0.012s sys 0.005s
W20251029 02:52:18.906596 712 leader_election.cc:343] T f18b0ca5a6a44175a912642e9eea1778 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:18.908133 1172 tablet_bootstrap.cc:492] T 1df0d944717d487a98ea83a8ec04bd46 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:18.909607 1172 tablet_bootstrap.cc:654] T 1df0d944717d487a98ea83a8ec04bd46 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:18.910476 1175 raft_consensus.cc:2243] T 786d9148fb1d417a8c2c8b8c53665899 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:18.910732 1175 raft_consensus.cc:2272] T 786d9148fb1d417a8c2c8b8c53665899 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:18.911314 1175 ts_tablet_manager.cc:1916] T 786d9148fb1d417a8c2c8b8c53665899 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:18.915525 2747 raft_consensus.cc:2749] T 589a521f5cc24e6289c37b9cf256bce5 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:18.918568 522 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_b2d77ffcda024d1e80fd8c298ddfd3e6" } new_table_name: "default.Table_05327aa5d63642fbb6db4ae3cbcc591b" modify_external_catalogs: true
I20251029 02:52:18.918648 1174 ts_tablet_manager.cc:1929] T 061f3d38c4604fa4b57e74ff6f6fc7da P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:18.918840 1174 log.cc:1199] T 061f3d38c4604fa4b57e74ff6f6fc7da P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/061f3d38c4604fa4b57e74ff6f6fc7da
W20251029 02:52:18.920238 526 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 41a171c7ba55422b8700c4f898dc7d9e consensus state has no leader
I20251029 02:52:18.920607 521 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 8fa6a27df46b4cf786470792431cefbc
I20251029 02:52:18.921201 524 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet f3ede523203545d18b09d535fdcabb6f
I20251029 02:52:18.922252 919 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f18b0ca5a6a44175a912642e9eea1778" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:18.923357 1040 tablet_service.cc:1552] Processing DeleteTablet for tablet 8fa6a27df46b4cf786470792431cefbc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:18 UTC) from {username='slave'} at 127.0.0.1:37626
W20251029 02:52:18.920246 518 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 926b02920aef457e84253c9bd0497be6 consensus state has no leader
I20251029 02:52:18.924330 1042 tablet_service.cc:1552] Processing DeleteTablet for tablet f3ede523203545d18b09d535fdcabb6f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:18 UTC) from {username='slave'} at 127.0.0.1:37626
W20251029 02:52:18.924822 711 leader_election.cc:343] T f18b0ca5a6a44175a912642e9eea1778 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:18.924944 711 leader_election.cc:304] T f18b0ca5a6a44175a912642e9eea1778 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:18.925405 1060 raft_consensus.cc:3060] T d29cc2a0c8b84e3ebadc7f571d666847 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:18.925482 742 tablet_service.cc:1552] Processing DeleteTablet for tablet 8fa6a27df46b4cf786470792431cefbc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:18 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:18.922465 1174 ts_tablet_manager.cc:1950] T 061f3d38c4604fa4b57e74ff6f6fc7da P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
W20251029 02:52:18.926709 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 8fa6a27df46b4cf786470792431cefbc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8fa6a27df46b4cf786470792431cefbc already in progress: creating tablet
W20251029 02:52:18.926951 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet f3ede523203545d18b09d535fdcabb6f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f3ede523203545d18b09d535fdcabb6f already in progress: creating tablet
I20251029 02:52:18.927309 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_531e36fbab1c4f06bcdbd7013ddd3e2f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:18.927786 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_531e36fbab1c4f06bcdbd7013ddd3e2f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:18.928674 738 tablet_service.cc:1552] Processing DeleteTablet for tablet f3ede523203545d18b09d535fdcabb6f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:18 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:18.929147 887 tablet_service.cc:1552] Processing DeleteTablet for tablet 8fa6a27df46b4cf786470792431cefbc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:18 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:18.932264 519 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 0573fe69797f48539df5caf4b3777b4d
I20251029 02:52:18.933336 738 tablet_service.cc:1552] Processing DeleteTablet for tablet 0573fe69797f48539df5caf4b3777b4d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:18 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:18.933617 1042 tablet_service.cc:1552] Processing DeleteTablet for tablet 0573fe69797f48539df5caf4b3777b4d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:18 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:18.934214 887 tablet_service.cc:1552] Processing DeleteTablet for tablet f3ede523203545d18b09d535fdcabb6f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:18 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:18.934762 887 tablet_service.cc:1552] Processing DeleteTablet for tablet 0573fe69797f48539df5caf4b3777b4d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:18 UTC) from {username='slave'} at 127.0.0.1:35702
W20251029 02:52:18.935361 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 0573fe69797f48539df5caf4b3777b4d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0573fe69797f48539df5caf4b3777b4d already in progress: creating tablet
I20251029 02:52:18.935931 1167 tablet_bootstrap.cc:654] T 055184c641ba45a588bca611fb24d130 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:18.945995 1166 tablet_bootstrap.cc:492] T f4729ecd182042c2a10f67ca25224101 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:18.947729 2949 raft_consensus.cc:2749] T f18b0ca5a6a44175a912642e9eea1778 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:18.953472 524 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet f4729ecd182042c2a10f67ca25224101 (table default.Table_973ef856041949598a23da3dab8b40db [id=1dd2d5488bc647c8b82c53886583b28d]). Expected version 4 got 0
I20251029 02:52:18.955636 1166 tablet_bootstrap.cc:654] T f4729ecd182042c2a10f67ca25224101 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:18.956506 525 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 62d8e7f41345410b9914bbf4343349b6
I20251029 02:52:18.957681 523 catalog_manager.cc:6194] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: default.Table_a78f5837df7b44e791631e421aa7de46 [id=b95fe1c7d6ff4519b90f993f1fa961fd] alter complete (version 3)
I20251029 02:52:18.959285 519 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet f4729ecd182042c2a10f67ca25224101 (table default.Table_973ef856041949598a23da3dab8b40db [id=1dd2d5488bc647c8b82c53886583b28d]). Expected version 4 got 0
I20251029 02:52:18.959756 1039 tablet_service.cc:1552] Processing DeleteTablet for tablet d29cc2a0c8b84e3ebadc7f571d666847 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:00 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:18.959919 1040 tablet_service.cc:1552] Processing DeleteTablet for tablet 6142018e44af4d32818adefa2ee43d02 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:01 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:18.960328 1031 tablet_service.cc:1552] Processing DeleteTablet for tablet 23b9b2ea814645eba63e5f6b12cfb966 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:57 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:18.960158 1034 tablet_service.cc:1552] Processing DeleteTablet for tablet 46cd75ff18b14212b556b9bcb3012dd9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:05 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:18.960457 1042 tablet_service.cc:1552] Processing DeleteTablet for tablet c081d2eacfac4ca8a0048e2a1c4f8f83 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:04 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:18.960507 1176 tablet_replica.cc:333] T d29cc2a0c8b84e3ebadc7f571d666847 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:18.960985 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 967) took 8151 ms. Trace:
I20251029 02:52:18.961874 1172 tablet_bootstrap.cc:492] T 1df0d944717d487a98ea83a8ec04bd46 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:18.962038 1172 ts_tablet_manager.cc:1403] T 1df0d944717d487a98ea83a8ec04bd46 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.054s user 0.001s sys 0.002s
I20251029 02:52:18.963001 1176 raft_consensus.cc:2243] T d29cc2a0c8b84e3ebadc7f571d666847 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:18.963294 1176 raft_consensus.cc:2272] T d29cc2a0c8b84e3ebadc7f571d666847 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:18.960151 517 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet c6202b39a22f4afab755038b3eddb81a
I20251029 02:52:18.963478 1172 raft_consensus.cc:359] T 1df0d944717d487a98ea83a8ec04bd46 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:18.963646 1172 raft_consensus.cc:385] T 1df0d944717d487a98ea83a8ec04bd46 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:18.963711 1172 raft_consensus.cc:740] T 1df0d944717d487a98ea83a8ec04bd46 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:18.963900 1172 consensus_queue.cc:260] T 1df0d944717d487a98ea83a8ec04bd46 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:18.963900 519 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet e3ac6597d52441569a7aeef051087ad2 (table default.Table_2c15df0a33124ce69282c48d6455d1b2 [id=242159edd3bc4771ac206d4e450b9475]). Expected version 2 got 0
I20251029 02:52:18.961709 887 tablet_service.cc:1552] Processing DeleteTablet for tablet b0fec06ad37d4c12a7045d424b88cc5e with delete_type TABLET_DATA_DELETED (replaced by tablet 4eec207d19f540518bb567d7ab21d048) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:18.964464 1174 rpcz_store.cc:276] 1029 02:52:10.809371 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:10.809420 (+ 49us) service_pool.cc:225] Handling call
1029 02:52:18.769068 (+7959648us) ts_tablet_manager.cc:1151] Deleting tablet 061f3d38c4604fa4b57e74ff6f6fc7da
1029 02:52:18.769086 (+ 18us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:18.804253 (+ 35167us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:18.918436 (+114183us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:18.960960 (+ 42524us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7959488}
I20251029 02:52:18.964879 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 975) took 8065 ms. Trace:
I20251029 02:52:18.964954 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 061f3d38c4604fa4b57e74ff6f6fc7da (table default.Table_0f40dc35be6d46b3a991e93384fa1e1f [id=7f5f847a87ab464b932a46f3c04e43d6]) successfully deleted
I20251029 02:52:18.964955 1174 rpcz_store.cc:276] 1029 02:52:10.899328 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:10.899379 (+ 51us) service_pool.cc:225] Handling call
1029 02:52:18.964756 (+8065377us) ts_tablet_manager.cc:1151] Deleting tablet 12432c84ca104b0e99926a247318906a
1029 02:52:18.964767 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:18.964863 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8065202}
I20251029 02:52:18.965245 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 977) took 8064 ms. Trace:
I20251029 02:52:18.965309 1174 rpcz_store.cc:276] 1029 02:52:10.900395 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:10.900441 (+ 46us) service_pool.cc:225] Handling call
1029 02:52:18.965148 (+8064707us) ts_tablet_manager.cc:1151] Deleting tablet e74eda48dca647efacabe525ef316944
1029 02:52:18.965157 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:18.965231 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8064531}
I20251029 02:52:18.965587 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 985) took 7996 ms. Trace:
I20251029 02:52:18.965652 1174 rpcz_store.cc:276] 1029 02:52:10.969461 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:10.969521 (+ 60us) service_pool.cc:225] Handling call
1029 02:52:18.965493 (+7995972us) ts_tablet_manager.cc:1151] Deleting tablet 4b841723726e40bda0187fb84bf36a6a
1029 02:52:18.965501 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:18.965573 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7995774}
W20251029 02:52:18.965788 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 12432c84ca104b0e99926a247318906a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 12432c84ca104b0e99926a247318906a already in progress: creating tablet
W20251029 02:52:18.965983 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet e74eda48dca647efacabe525ef316944 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e74eda48dca647efacabe525ef316944 already in progress: creating tablet
I20251029 02:52:18.966169 1172 ts_tablet_manager.cc:1434] T 1df0d944717d487a98ea83a8ec04bd46 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20251029 02:52:18.966316 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 991) took 7755 ms. Trace:
I20251029 02:52:18.966485 1174 rpcz_store.cc:276] 1029 02:52:11.210731 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:11.210781 (+ 50us) service_pool.cc:225] Handling call
1029 02:52:18.965834 (+7755053us) ts_tablet_manager.cc:1151] Deleting tablet 509288c033d54e4b9da3f26a30dc46a0
1029 02:52:18.965843 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:18.966296 (+ 453us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7754813}
W20251029 02:52:18.966800 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 4b841723726e40bda0187fb84bf36a6a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4b841723726e40bda0187fb84bf36a6a already in progress: creating tablet
W20251029 02:52:18.967236 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 509288c033d54e4b9da3f26a30dc46a0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 509288c033d54e4b9da3f26a30dc46a0 already in progress: creating tablet
I20251029 02:52:18.967252 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 992) took 7756 ms. Trace:
I20251029 02:52:18.967341 1174 rpcz_store.cc:276] 1029 02:52:11.210897 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:11.210952 (+ 55us) service_pool.cc:225] Handling call
1029 02:52:18.966930 (+7755978us) ts_tablet_manager.cc:1151] Deleting tablet c172417d1d444c439e9cda611a2ed551
1029 02:52:18.966945 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:18.967210 (+ 265us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7755785}
I20251029 02:52:18.967729 1174 tablet_replica.cc:333] T 8048506ced4540a5aee9f79bfc4599bf P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:18.968001 1174 raft_consensus.cc:2243] T 8048506ced4540a5aee9f79bfc4599bf P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:18.968214 1174 raft_consensus.cc:2272] T 8048506ced4540a5aee9f79bfc4599bf P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
W20251029 02:52:18.969398 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet c172417d1d444c439e9cda611a2ed551 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c172417d1d444c439e9cda611a2ed551 already in progress: creating tablet
I20251029 02:52:18.973157 1042 tablet_service.cc:1552] Processing DeleteTablet for tablet eca016aed05e4ce09e1a2bf96e2286f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:58 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:18.976589 1176 ts_tablet_manager.cc:1916] T d29cc2a0c8b84e3ebadc7f571d666847 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 02:52:18.985260 522 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f6fdaac654d4434d81963746f53205b2 consensus state has no leader
I20251029 02:52:18.987015 1174 ts_tablet_manager.cc:1916] T 8048506ced4540a5aee9f79bfc4599bf P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:18.994303 1175 ts_tablet_manager.cc:1929] T 786d9148fb1d417a8c2c8b8c53665899 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:18.998796 1176 ts_tablet_manager.cc:1929] T d29cc2a0c8b84e3ebadc7f571d666847 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:18.994494 738 tablet_service.cc:1552] Processing DeleteTablet for tablet 1df0d944717d487a98ea83a8ec04bd46 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:01 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:18.999018 1175 log.cc:1199] T 786d9148fb1d417a8c2c8b8c53665899 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/786d9148fb1d417a8c2c8b8c53665899
I20251029 02:52:18.999270 1176 log.cc:1199] T d29cc2a0c8b84e3ebadc7f571d666847 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/d29cc2a0c8b84e3ebadc7f571d666847
I20251029 02:52:18.999411 1175 ts_tablet_manager.cc:1950] T 786d9148fb1d417a8c2c8b8c53665899 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:18.999697 1176 ts_tablet_manager.cc:1950] T d29cc2a0c8b84e3ebadc7f571d666847 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:19.000284 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet d29cc2a0c8b84e3ebadc7f571d666847 (table default.Table_8fee569eca38413dbac56707725c6a46 [id=50de770b3b9b4d8c85546014a959da24]) successfully deleted
I20251029 02:52:19.000640 1176 tablet_replica.cc:333] T 6142018e44af4d32818adefa2ee43d02 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:19.001040 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 1063) took 5089 ms. Trace:
I20251029 02:52:19.002401 1176 raft_consensus.cc:2243] T 6142018e44af4d32818adefa2ee43d02 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:19.002614 1176 raft_consensus.cc:2272] T 6142018e44af4d32818adefa2ee43d02 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:19.003206 1175 rpcz_store.cc:276] 1029 02:52:13.911963 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:13.913261 (+ 1298us) service_pool.cc:225] Handling call
1029 02:52:18.903351 (+4990090us) ts_tablet_manager.cc:1151] Deleting tablet 786d9148fb1d417a8c2c8b8c53665899
1029 02:52:18.903361 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:18.979456 (+ 76095us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:18.994173 (+ 14717us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:19.001021 (+ 6848us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4936294}
I20251029 02:52:19.003698 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 786d9148fb1d417a8c2c8b8c53665899 (table default.Table_0f200bab93784e9d97c7d001330e303a [id=0809df5782144d1da622b5726db932f4]) successfully deleted
I20251029 02:52:19.005121 1175 tablet_replica.cc:333] T 589a521f5cc24e6289c37b9cf256bce5 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:19.006183 1176 ts_tablet_manager.cc:1916] T 6142018e44af4d32818adefa2ee43d02 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:19.005465 1172 tablet_bootstrap.cc:492] T b125cc9aaf4248b8bfa48fb152b832a6 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:19.008740 1175 raft_consensus.cc:2243] T 589a521f5cc24e6289c37b9cf256bce5 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:19.008965 1175 raft_consensus.cc:2272] T 589a521f5cc24e6289c37b9cf256bce5 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:19.009630 1175 ts_tablet_manager.cc:1916] T 589a521f5cc24e6289c37b9cf256bce5 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:19.012259 1166 tablet_bootstrap.cc:492] T f4729ecd182042c2a10f67ca25224101 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:19.012449 1166 ts_tablet_manager.cc:1403] T f4729ecd182042c2a10f67ca25224101 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.067s user 0.002s sys 0.002s
I20251029 02:52:19.013353 1166 raft_consensus.cc:359] T f4729ecd182042c2a10f67ca25224101 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:19.010272 1172 tablet_bootstrap.cc:654] T b125cc9aaf4248b8bfa48fb152b832a6 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:19.039005 2952 tablet.cc:1722] T 32519fbffcef44a890161e5f3043806c P c756528642a543eeacd93cef3e335a7d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20251029 02:52:19.044802 1167 tablet_bootstrap.cc:492] T 055184c641ba45a588bca611fb24d130 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:19.048492 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_9e9ca87cd8a04a2ea4f1bfa00e1287d9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:19.049136 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9e9ca87cd8a04a2ea4f1bfa00e1287d9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:19.049621 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_3f79cf786ed04f769e3f2f08bd109229"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:19.050063 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3f79cf786ed04f769e3f2f08bd109229 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:19.058382 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_2410d595bde241a084883280aaed23b3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:19.058630 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_a3bef4a3b367428983bb707cacb4f663"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:19.058917 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2410d595bde241a084883280aaed23b3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:52:19.058948 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a3bef4a3b367428983bb707cacb4f663 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:19.077363 1175 ts_tablet_manager.cc:1929] T 589a521f5cc24e6289c37b9cf256bce5 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:19.077564 1175 log.cc:1199] T 589a521f5cc24e6289c37b9cf256bce5 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/589a521f5cc24e6289c37b9cf256bce5
I20251029 02:52:19.077939 1175 ts_tablet_manager.cc:1950] T 589a521f5cc24e6289c37b9cf256bce5 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:19.080783 524 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_06ebf0bbf7f648bca4075740bb6ff66a" } new_table_name: "default.Table_b2b8451524e7419682961023cc62ba95" modify_external_catalogs: true
I20251029 02:52:19.081907 517 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_15d2f76cb37642f5a8f3611fbcdd5259" } new_table_name: "default.Table_1d559709d20746339371e190c66738e4" modify_external_catalogs: true
I20251029 02:52:19.082360 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_e83f9801f1db4c51bddc74af9f67fe30"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:19.085359 492 master_runner.cc:186] Error getting master registration for 127.0.104.62:42343: OK, instance_id { permanent_uuid: "e9672114a6a742e2864c16ff60ba246e" instance_seqno: 1761706336177483 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20251029 02:52:19.085564 492 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251029 02:52:19.082916 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e83f9801f1db4c51bddc74af9f67fe30 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:19.082954 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_1cb57c999d6441a7b0355b2a1b5a9338"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:19.088479 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1cb57c999d6441a7b0355b2a1b5a9338 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:19.083377 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_7943765870a6468682d2b39b11741c17"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:19.090235 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7943765870a6468682d2b39b11741c17 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:19.083122 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_949a1bcd83f24789aba2511ddc5389f5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:19.091840 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_949a1bcd83f24789aba2511ddc5389f5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:19.090832 2949 raft_consensus.cc:493] T 9492c1027aba4455ad8f539addaed5a7 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:19.093976 2949 raft_consensus.cc:515] T 9492c1027aba4455ad8f539addaed5a7 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:19.094718 2949 leader_election.cc:290] T 9492c1027aba4455ad8f539addaed5a7 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:52:19.095662 1172 tablet_bootstrap.cc:492] T b125cc9aaf4248b8bfa48fb152b832a6 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:19.095810 1172 ts_tablet_manager.cc:1403] T b125cc9aaf4248b8bfa48fb152b832a6 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.090s user 0.002s sys 0.003s
I20251029 02:52:19.096460 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 1064) took 5171 ms. Trace:
W20251029 02:52:19.097553 524 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 266aaea633bf4c059b9c02f092a474be consensus state has no leader
I20251029 02:52:19.097555 1175 rpcz_store.cc:276] 1029 02:52:13.925167 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:13.925230 (+ 63us) service_pool.cc:225] Handling call
1029 02:52:19.004019 (+5078789us) ts_tablet_manager.cc:1151] Deleting tablet 589a521f5cc24e6289c37b9cf256bce5
1029 02:52:19.005078 (+ 1059us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:19.065863 (+ 60785us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:19.077251 (+ 11388us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:19.096443 (+ 19192us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5036891}
I20251029 02:52:19.098014 1175 tablet_replica.cc:333] T 70b008011d5547bbaf3f45685394139f P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:19.097988 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 589a521f5cc24e6289c37b9cf256bce5 (table default.Table_b8095fca27a447f2a9ac0475373352c6 [id=56b57a7a90624bee92fa43661728c0f8]) successfully deleted
I20251029 02:52:19.098263 1175 raft_consensus.cc:2243] T 70b008011d5547bbaf3f45685394139f P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:19.098474 1175 raft_consensus.cc:2272] T 70b008011d5547bbaf3f45685394139f P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:19.099156 1175 ts_tablet_manager.cc:1916] T 70b008011d5547bbaf3f45685394139f P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 02:52:19.104878 517 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0bc634974f1f4cbba8092f1ddff4c0ff consensus state has no leader
I20251029 02:52:19.113873 1172 raft_consensus.cc:359] T b125cc9aaf4248b8bfa48fb152b832a6 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:19.114070 1172 raft_consensus.cc:385] T b125cc9aaf4248b8bfa48fb152b832a6 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:19.114127 1172 raft_consensus.cc:740] T b125cc9aaf4248b8bfa48fb152b832a6 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:19.114317 1172 consensus_queue.cc:260] T b125cc9aaf4248b8bfa48fb152b832a6 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:19.114984 1172 ts_tablet_manager.cc:1434] T b125cc9aaf4248b8bfa48fb152b832a6 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.019s user 0.001s sys 0.000s
I20251029 02:52:19.115300 1172 tablet_bootstrap.cc:492] T d1866d6dcb0349bba5db86163375cada P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:19.118361 1175 ts_tablet_manager.cc:1929] T 70b008011d5547bbaf3f45685394139f P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:19.118714 1175 log.cc:1199] T 70b008011d5547bbaf3f45685394139f P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/70b008011d5547bbaf3f45685394139f
I20251029 02:52:19.119230 1175 ts_tablet_manager.cc:1950] T 70b008011d5547bbaf3f45685394139f P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:19.119583 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 1065) took 5193 ms. Trace:
I20251029 02:52:19.119714 1175 rpcz_store.cc:276] 1029 02:52:13.925827 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:13.926317 (+ 490us) service_pool.cc:225] Handling call
1029 02:52:19.097956 (+5171639us) ts_tablet_manager.cc:1151] Deleting tablet 70b008011d5547bbaf3f45685394139f
1029 02:52:19.097966 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:19.114797 (+ 16831us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:19.118267 (+ 3470us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:19.119565 (+ 1298us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5130768}
I20251029 02:52:19.123376 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 70b008011d5547bbaf3f45685394139f (table default.Table_0bb20faeb14644ca8e2d5acd4af09f85 [id=d39d7d159c7d46b496caa09226366e1f]) successfully deleted
I20251029 02:52:19.123848 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 1067) took 5073 ms. Trace:
I20251029 02:52:19.124451 519 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_abf33e38626c4d3e9bd944fecb1d1490" } new_table_name: "default.Table_05607b075b454d49a9a68e71b13c95e5" modify_external_catalogs: true
I20251029 02:52:19.121829 1172 tablet_bootstrap.cc:654] T d1866d6dcb0349bba5db86163375cada P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:19.125787 522 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_a68b9a5d4e444e579e08cc644d3aee8c" } modify_external_catalogs: true
I20251029 02:52:19.125913 522 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_a68b9a5d4e444e579e08cc644d3aee8c" } modify_external_catalogs: true
I20251029 02:52:19.124007 1175 rpcz_store.cc:276] 1029 02:52:14.050779 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:14.050836 (+ 57us) service_pool.cc:225] Handling call
1029 02:52:19.123704 (+5072868us) ts_tablet_manager.cc:1151] Deleting tablet 96427e60edeb45bdab9666a5055b5a99
1029 02:52:19.123714 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:19.123829 (+ 115us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5072647}
I20251029 02:52:19.055673 1174 ts_tablet_manager.cc:1929] T 8048506ced4540a5aee9f79bfc4599bf P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:19.126756 518 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_a75246042ab0422cbfc4c83f2ab5b81c" } modify_external_catalogs: true
I20251029 02:52:19.126878 518 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_a75246042ab0422cbfc4c83f2ab5b81c" } modify_external_catalogs: true
I20251029 02:52:19.126892 1174 log.cc:1199] T 8048506ced4540a5aee9f79bfc4599bf P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/8048506ced4540a5aee9f79bfc4599bf
I20251029 02:52:19.127271 1174 ts_tablet_manager.cc:1950] T 8048506ced4540a5aee9f79bfc4599bf P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
W20251029 02:52:19.127506 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 96427e60edeb45bdab9666a5055b5a99 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 96427e60edeb45bdab9666a5055b5a99 already in progress: creating tablet
I20251029 02:52:19.127902 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 1070) took 5036 ms. Trace:
I20251029 02:52:19.128018 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 993) took 7878 ms. Trace:
I20251029 02:52:19.127976 1175 rpcz_store.cc:276] 1029 02:52:14.091287 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:14.136887 (+ 45600us) service_pool.cc:225] Handling call
1029 02:52:19.127771 (+4990884us) ts_tablet_manager.cc:1151] Deleting tablet 7751599f5dfc43baa20dc3487c6eddd9
1029 02:52:19.127780 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:19.127881 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4990715}
I20251029 02:52:19.128099 1174 rpcz_store.cc:276] 1029 02:52:11.249214 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:11.249511 (+ 297us) service_pool.cc:225] Handling call
1029 02:52:18.967669 (+7718158us) ts_tablet_manager.cc:1151] Deleting tablet 8048506ced4540a5aee9f79bfc4599bf
1029 02:52:18.967678 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:19.026773 (+ 59095us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:19.054886 (+ 28113us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:19.127997 (+ 73111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7717943}
W20251029 02:52:19.128391 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 7751599f5dfc43baa20dc3487c6eddd9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7751599f5dfc43baa20dc3487c6eddd9 already in progress: creating tablet
I20251029 02:52:19.128507 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 995) took 7879 ms. Trace:
I20251029 02:52:19.128582 1174 rpcz_store.cc:276] 1029 02:52:11.249460 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:11.249692 (+ 232us) service_pool.cc:225] Handling call
1029 02:52:19.128390 (+7878698us) ts_tablet_manager.cc:1151] Deleting tablet 7dbc9c8b110a48f0a801ad468738c3a3
1029 02:52:19.128401 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:19.128490 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7878567}
I20251029 02:52:19.128744 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 1075) took 4951 ms. Trace:
I20251029 02:52:19.128813 1175 rpcz_store.cc:276] 1029 02:52:14.177765 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:14.177819 (+ 54us) service_pool.cc:225] Handling call
1029 02:52:19.128626 (+4950807us) ts_tablet_manager.cc:1151] Deleting tablet 595b4073cc0446cea041838097af7471
1029 02:52:19.128634 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:19.128727 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4950621}
I20251029 02:52:19.128882 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 994) took 7879 ms. Trace:
I20251029 02:52:19.128966 1174 rpcz_store.cc:276] 1029 02:52:11.249342 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:11.249446 (+ 104us) service_pool.cc:225] Handling call
1029 02:52:19.128786 (+7879340us) ts_tablet_manager.cc:1151] Deleting tablet c7a3e6d8e3fb45898d8a5e094dbf0fd7
1029 02:52:19.128794 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:19.128868 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":244,"tablet-delete.queue_time_us":7879105}
W20251029 02:52:19.129184 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 595b4073cc0446cea041838097af7471 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 595b4073cc0446cea041838097af7471 already in progress: creating tablet
I20251029 02:52:19.129215 1174 tablet_replica.cc:333] T c32c28b0e74a4af3a0e69bbf24ab1a1f P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:19.129386 1174 raft_consensus.cc:2243] T c32c28b0e74a4af3a0e69bbf24ab1a1f P c756528642a543eeacd93cef3e335a7d [term 2 LEADER]: Raft consensus shutting down.
I20251029 02:52:19.129678 1174 pending_rounds.cc:70] T c32c28b0e74a4af3a0e69bbf24ab1a1f P c756528642a543eeacd93cef3e335a7d: Trying to abort 1 pending ops.
I20251029 02:52:19.129720 1174 pending_rounds.cc:77] T c32c28b0e74a4af3a0e69bbf24ab1a1f P c756528642a543eeacd93cef3e335a7d: Aborting op as it isn't in flight: id { term: 2 index: 2 } timestamp: 7215949152815239168 op_type: NO_OP noop_request { }
I20251029 02:52:19.129797 1174 raft_consensus.cc:2889] T c32c28b0e74a4af3a0e69bbf24ab1a1f P c756528642a543eeacd93cef3e335a7d [term 2 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 02:52:19.129870 1174 raft_consensus.cc:2272] T c32c28b0e74a4af3a0e69bbf24ab1a1f P c756528642a543eeacd93cef3e335a7d [term 2 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:19.130393 1174 ts_tablet_manager.cc:1916] T c32c28b0e74a4af3a0e69bbf24ab1a1f P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:19.064870 2426 consensus_queue.cc:1048] T f93b0c39d7174dabbe018b1546dbc198 P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:19.095381 919 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9492c1027aba4455ad8f539addaed5a7" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:19.131433 519 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b00b7696d0dc46a2aec79ebf3c66f39f consensus state has no leader
W20251029 02:52:19.131932 711 leader_election.cc:343] T 9492c1027aba4455ad8f539addaed5a7 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:19.133096 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 8048506ced4540a5aee9f79bfc4599bf (table default.Table_f2a72498cf4540949baff93eba1f5ef0 [id=a047685192f34bf1a1f3d1361cdaf272]) successfully deleted
W20251029 02:52:19.133306 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 7dbc9c8b110a48f0a801ad468738c3a3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7dbc9c8b110a48f0a801ad468738c3a3 already in progress: creating tablet
W20251029 02:52:19.133489 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet c7a3e6d8e3fb45898d8a5e094dbf0fd7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c7a3e6d8e3fb45898d8a5e094dbf0fd7 already in progress: creating tablet
I20251029 02:52:19.133734 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 1076) took 4951 ms. Trace:
I20251029 02:52:19.133872 1175 rpcz_store.cc:276] 1029 02:52:14.182631 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:14.182688 (+ 57us) service_pool.cc:225] Handling call
1029 02:52:19.133601 (+4950913us) ts_tablet_manager.cc:1151] Deleting tablet e2c8983cded84a1c8b4205c48a4eac33
1029 02:52:19.133610 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:19.133717 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4950732}
I20251029 02:52:19.134316 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 1083) took 4832 ms. Trace:
W20251029 02:52:19.134356 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet e2c8983cded84a1c8b4205c48a4eac33 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e2c8983cded84a1c8b4205c48a4eac33 already in progress: creating tablet
I20251029 02:52:19.134455 1175 rpcz_store.cc:276] 1029 02:52:14.301860 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:14.301907 (+ 47us) service_pool.cc:225] Handling call
1029 02:52:19.134197 (+4832290us) ts_tablet_manager.cc:1151] Deleting tablet c0936e54f1d246388a2d82dbc52aac42
1029 02:52:19.134206 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:19.134300 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4832112}
I20251029 02:52:19.134935 517 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_9e67f2d45e8f44c6bff1b72166c092a2" } modify_external_catalogs: true
I20251029 02:52:19.135108 517 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_9e67f2d45e8f44c6bff1b72166c092a2" } modify_external_catalogs: true
I20251029 02:52:19.138211 1175 tablet_replica.cc:333] T f93b0c39d7174dabbe018b1546dbc198 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:19.138451 1175 raft_consensus.cc:2243] T f93b0c39d7174dabbe018b1546dbc198 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:19.138746 1175 raft_consensus.cc:2272] T f93b0c39d7174dabbe018b1546dbc198 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:19.138880 519 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_0a33656128144bd98100260c9f0df989" } new_table_name: "default.Table_1a72358be85144d5b95be93d90d6bd7e" modify_external_catalogs: true
W20251029 02:52:19.139014 847 consensus_peers.cc:597] T f93b0c39d7174dabbe018b1546dbc198 P c756528642a543eeacd93cef3e335a7d -> Peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Couldn't send request to peer b495a49349324f7d81908f2da700f0d9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:19.142443 1174 ts_tablet_manager.cc:1929] T c32c28b0e74a4af3a0e69bbf24ab1a1f P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.2
I20251029 02:52:19.142658 1174 log.cc:1199] T c32c28b0e74a4af3a0e69bbf24ab1a1f P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/c32c28b0e74a4af3a0e69bbf24ab1a1f
I20251029 02:52:19.143253 1174 ts_tablet_manager.cc:1950] T c32c28b0e74a4af3a0e69bbf24ab1a1f P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:19.143545 1175 ts_tablet_manager.cc:1916] T f93b0c39d7174dabbe018b1546dbc198 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 02:52:19.143853 510 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet f6fdaac654d4434d81963746f53205b2 (table default.Table_05327aa5d63642fbb6db4ae3cbcc591b [id=c309ea497e8b4c8dbafeb7162d58236d], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet f6fdaac654d4434d81963746f53205b2 consensus state has no leader
I20251029 02:52:19.144217 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 996) took 7882 ms. Trace:
I20251029 02:52:19.144368 1174 rpcz_store.cc:276] 1029 02:52:11.261861 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:11.261929 (+ 68us) service_pool.cc:225] Handling call
1029 02:52:19.129176 (+7867247us) ts_tablet_manager.cc:1151] Deleting tablet c32c28b0e74a4af3a0e69bbf24ab1a1f
1029 02:52:19.129183 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:19.137762 (+ 8579us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:19.142351 (+ 4589us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:19.144195 (+ 1844us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7867004}
I20251029 02:52:19.145896 1174 tablet_replica.cc:333] T af4860c6b72f4d988b206b5ef2f22906 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:19.146574 1167 ts_tablet_manager.cc:1403] T 055184c641ba45a588bca611fb24d130 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.329s user 0.006s sys 0.001s
I20251029 02:52:19.147828 1167 raft_consensus.cc:359] T 055184c641ba45a588bca611fb24d130 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:19.148191 1167 raft_consensus.cc:385] T 055184c641ba45a588bca611fb24d130 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:19.148342 1167 raft_consensus.cc:740] T 055184c641ba45a588bca611fb24d130 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:19.148765 1167 consensus_queue.cc:260] T 055184c641ba45a588bca611fb24d130 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:19.149946 1167 ts_tablet_manager.cc:1434] T 055184c641ba45a588bca611fb24d130 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251029 02:52:19.150452 1167 tablet_bootstrap.cc:492] T 5815626b05204b578cc4a110f246bf10 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:19.145094 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet c32c28b0e74a4af3a0e69bbf24ab1a1f (table default.Table_3d94f8b10e884dc0b338dcfbb9c82309 [id=3db1589f811041eaa1cd7412a26391bc]) successfully deleted
I20251029 02:52:19.154731 517 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 44b518c3a0964a6fa863b2784d80c8a4
I20251029 02:52:19.156282 1166 raft_consensus.cc:385] T f4729ecd182042c2a10f67ca25224101 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:19.156365 1166 raft_consensus.cc:740] T f4729ecd182042c2a10f67ca25224101 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:19.156395 887 tablet_service.cc:1552] Processing DeleteTablet for tablet 44b518c3a0964a6fa863b2784d80c8a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:19 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:19.031327 1176 ts_tablet_manager.cc:1929] T 6142018e44af4d32818adefa2ee43d02 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:19.156617 1176 log.cc:1199] T 6142018e44af4d32818adefa2ee43d02 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/6142018e44af4d32818adefa2ee43d02
I20251029 02:52:19.156572 1166 consensus_queue.cc:260] T f4729ecd182042c2a10f67ca25224101 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:19.156948 1176 ts_tablet_manager.cc:1950] T 6142018e44af4d32818adefa2ee43d02 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:19.157063 1166 ts_tablet_manager.cc:1434] T f4729ecd182042c2a10f67ca25224101 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.144s user 0.000s sys 0.000s
I20251029 02:52:19.056568 1060 raft_consensus.cc:3060] T 23b9b2ea814645eba63e5f6b12cfb966 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:19.157475 738 tablet_service.cc:1552] Processing DeleteTablet for tablet 44b518c3a0964a6fa863b2784d80c8a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:19 UTC) from {username='slave'} at 127.0.0.1:44580
W20251029 02:52:19.158579 519 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0de19a308a534cdaa38f46ab43789322 consensus state has no leader
I20251029 02:52:19.056694 2885 raft_consensus.cc:493] T eca016aed05e4ce09e1a2bf96e2286f5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:19.160735 2885 raft_consensus.cc:515] T eca016aed05e4ce09e1a2bf96e2286f5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:19.161324 2885 leader_election.cc:290] T eca016aed05e4ce09e1a2bf96e2286f5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
W20251029 02:52:19.162047 989 leader_election.cc:343] T eca016aed05e4ce09e1a2bf96e2286f5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: eca016aed05e4ce09e1a2bf96e2286f5
I20251029 02:52:19.095383 1059 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9492c1027aba4455ad8f539addaed5a7" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:19.107252 2964 raft_consensus.cc:493] T 46cd75ff18b14212b556b9bcb3012dd9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:19.163026 2964 raft_consensus.cc:515] T 46cd75ff18b14212b556b9bcb3012dd9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
W20251029 02:52:19.163273 712 leader_election.cc:343] T 9492c1027aba4455ad8f539addaed5a7 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:19.163378 712 leader_election.cc:304] T 9492c1027aba4455ad8f539addaed5a7 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:19.163571 2964 leader_election.cc:290] T 46cd75ff18b14212b556b9bcb3012dd9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:19.163743 2949 raft_consensus.cc:2749] T 9492c1027aba4455ad8f539addaed5a7 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20251029 02:52:19.164778 989 leader_election.cc:343] T 46cd75ff18b14212b556b9bcb3012dd9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 46cd75ff18b14212b556b9bcb3012dd9
I20251029 02:52:19.165622 1042 tablet_service.cc:1552] Processing DeleteTablet for tablet 44b518c3a0964a6fa863b2784d80c8a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:19 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:19.166783 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_23333cfa5eaf4bc6813923745cda6c5a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:19.167073 519 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_80b931e916ee46cf8704adef42106b95" } modify_external_catalogs: true
I20251029 02:52:19.167219 519 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_80b931e916ee46cf8704adef42106b95" } modify_external_catalogs: true
W20251029 02:52:19.167341 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_23333cfa5eaf4bc6813923745cda6c5a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:19.168680 524 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_1963d5301e1a4a8ea388fbedfa53c6df" } modify_external_catalogs: true
I20251029 02:52:19.168818 524 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_1963d5301e1a4a8ea388fbedfa53c6df" } modify_external_catalogs: true
I20251029 02:52:19.169888 520 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet f4729ecd182042c2a10f67ca25224101 (table default.Table_973ef856041949598a23da3dab8b40db [id=1dd2d5488bc647c8b82c53886583b28d]). Expected version 4 got 0
I20251029 02:52:19.175825 1166 tablet_bootstrap.cc:492] T b125cc9aaf4248b8bfa48fb152b832a6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:19.175912 1176 tablet_replica.cc:333] T 23b9b2ea814645eba63e5f6b12cfb966 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:19.176175 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 6142018e44af4d32818adefa2ee43d02 (table default.Table_a45ccb3fef5d4947ab7a8709b7c32119 [id=b3e01f7d5a5f4675a998eaad985347ca]) successfully deleted
I20251029 02:52:19.178053 1166 tablet_bootstrap.cc:654] T b125cc9aaf4248b8bfa48fb152b832a6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
W20251029 02:52:19.179886 849 consensus_peers.cc:597] T 643e20d9974447daa78885b3401f3b77 P c756528642a543eeacd93cef3e335a7d -> Peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Couldn't send request to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 643e20d9974447daa78885b3401f3b77. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:19.180956 1172 tablet_bootstrap.cc:492] T d1866d6dcb0349bba5db86163375cada P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:19.181108 1172 ts_tablet_manager.cc:1403] T d1866d6dcb0349bba5db86163375cada P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.066s user 0.003s sys 0.000s
I20251029 02:52:19.181959 1172 raft_consensus.cc:359] T d1866d6dcb0349bba5db86163375cada P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:19.182158 1172 raft_consensus.cc:385] T d1866d6dcb0349bba5db86163375cada P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:19.182220 1172 raft_consensus.cc:740] T d1866d6dcb0349bba5db86163375cada P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:19.182474 1172 consensus_queue.cc:260] T d1866d6dcb0349bba5db86163375cada P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:19.182987 1172 ts_tablet_manager.cc:1434] T d1866d6dcb0349bba5db86163375cada P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 02:52:19.185459 821 heartbeater.cc:344] Connected to a master server at 127.0.104.62:42343
I20251029 02:52:19.187033 517 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_13076d1361cd4ef9ae70f064a6b8679c" } new_table_name: "default.Table_0ad8fce71fdd4d34aab2366cdb321b86" modify_external_catalogs: true
I20251029 02:52:19.191946 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_cba0d764db7147d59d1a3260084ae20c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:19.192512 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cba0d764db7147d59d1a3260084ae20c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:19.146265 1174 raft_consensus.cc:2243] T af4860c6b72f4d988b206b5ef2f22906 P c756528642a543eeacd93cef3e335a7d [term 2 LEADER]: Raft consensus shutting down.
I20251029 02:52:19.193413 1174 pending_rounds.cc:70] T af4860c6b72f4d988b206b5ef2f22906 P c756528642a543eeacd93cef3e335a7d: Trying to abort 1 pending ops.
I20251029 02:52:19.193531 1174 pending_rounds.cc:77] T af4860c6b72f4d988b206b5ef2f22906 P c756528642a543eeacd93cef3e335a7d: Aborting op as it isn't in flight: id { term: 2 index: 2 } timestamp: 7215949154160910336 op_type: NO_OP noop_request { }
I20251029 02:52:19.193799 1174 raft_consensus.cc:2889] T af4860c6b72f4d988b206b5ef2f22906 P c756528642a543eeacd93cef3e335a7d [term 2 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20251029 02:52:19.193943 1174 raft_consensus.cc:2272] T af4860c6b72f4d988b206b5ef2f22906 P c756528642a543eeacd93cef3e335a7d [term 2 FOLLOWER]: Raft consensus is shut down!
W20251029 02:52:19.195102 517 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1194e03172954aadb819cc9a6a5050fe consensus state has no leader
I20251029 02:52:19.197671 1167 tablet_bootstrap.cc:654] T 5815626b05204b578cc4a110f246bf10 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:19.208753 1172 tablet_bootstrap.cc:492] T 294afb9b012c40c0b05b73795d6f0ac8 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:19.212159 1172 tablet_bootstrap.cc:654] T 294afb9b012c40c0b05b73795d6f0ac8 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:19.225505 2949 raft_consensus.cc:493] T e3ac6597d52441569a7aeef051087ad2 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:19.225661 2949 raft_consensus.cc:515] T e3ac6597d52441569a7aeef051087ad2 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:19.226296 2949 leader_election.cc:290] T e3ac6597d52441569a7aeef051087ad2 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:52:19.227121 1059 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e3ac6597d52441569a7aeef051087ad2" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:52:19.227671 712 leader_election.cc:343] T e3ac6597d52441569a7aeef051087ad2 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:19.228823 517 master_service.cc:954] ReplaceTablet: received request to replace tablet 81f936792aae4b699b3122457d7b631d from {username='slave'} at 127.0.0.1:35314
I20251029 02:52:19.234566 1176 raft_consensus.cc:2243] T 23b9b2ea814645eba63e5f6b12cfb966 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:19.234891 1176 raft_consensus.cc:2272] T 23b9b2ea814645eba63e5f6b12cfb966 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:19.237636 517 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet f4729ecd182042c2a10f67ca25224101 (table default.Table_973ef856041949598a23da3dab8b40db [id=1dd2d5488bc647c8b82c53886583b28d]). Expected version 4 got 0
I20251029 02:52:19.238336 1176 ts_tablet_manager.cc:1916] T 23b9b2ea814645eba63e5f6b12cfb966 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:19.205624 919 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "eca016aed05e4ce09e1a2bf96e2286f5" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:19.206372 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "46cd75ff18b14212b556b9bcb3012dd9" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:19.242715 990 leader_election.cc:304] T eca016aed05e4ce09e1a2bf96e2286f5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5; no voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:19.243072 2961 raft_consensus.cc:2749] T eca016aed05e4ce09e1a2bf96e2286f5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:19.243530 990 leader_election.cc:304] T 46cd75ff18b14212b556b9bcb3012dd9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5; no voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:19.243805 2961 raft_consensus.cc:2749] T 46cd75ff18b14212b556b9bcb3012dd9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:19.215266 1174 ts_tablet_manager.cc:1916] T af4860c6b72f4d988b206b5ef2f22906 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:19.235211 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e3ac6597d52441569a7aeef051087ad2" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:19.245079 711 leader_election.cc:343] T e3ac6597d52441569a7aeef051087ad2 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:19.245196 711 leader_election.cc:304] T e3ac6597d52441569a7aeef051087ad2 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:19.245638 2949 raft_consensus.cc:2749] T e3ac6597d52441569a7aeef051087ad2 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:19.249436 520 master_service.cc:954] ReplaceTablet: received request to replace tablet e3000183230e4350b47f5085952f2db6 from {username='slave'} at 127.0.0.1:35314
I20251029 02:52:19.250391 517 master_service.cc:954] ReplaceTablet: received request to replace tablet 48aa20640dfe43288326ed6d6b9bc1cf from {username='slave'} at 127.0.0.1:35314
I20251029 02:52:19.251120 1176 ts_tablet_manager.cc:1929] T 23b9b2ea814645eba63e5f6b12cfb966 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:19.251315 1176 log.cc:1199] T 23b9b2ea814645eba63e5f6b12cfb966 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/23b9b2ea814645eba63e5f6b12cfb966
I20251029 02:52:19.251765 1176 ts_tablet_manager.cc:1950] T 23b9b2ea814645eba63e5f6b12cfb966 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:19.252187 1176 tablet_replica.cc:333] T 46cd75ff18b14212b556b9bcb3012dd9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:19.252408 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 23b9b2ea814645eba63e5f6b12cfb966 (table default.Table_ab94a2a8b29d498a9f201f39228d7bfe [id=4876e3df916248d2a64ba3cad736b5ab]) successfully deleted
I20251029 02:52:19.252843 1176 raft_consensus.cc:2243] T 46cd75ff18b14212b556b9bcb3012dd9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:19.253041 1176 raft_consensus.cc:2272] T 46cd75ff18b14212b556b9bcb3012dd9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:19.253278 1166 tablet_bootstrap.cc:492] T b125cc9aaf4248b8bfa48fb152b832a6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:19.253475 1166 ts_tablet_manager.cc:1403] T b125cc9aaf4248b8bfa48fb152b832a6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.078s user 0.002s sys 0.002s
I20251029 02:52:19.253525 1175 ts_tablet_manager.cc:1929] T f93b0c39d7174dabbe018b1546dbc198 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:19.253588 1176 ts_tablet_manager.cc:1916] T 46cd75ff18b14212b556b9bcb3012dd9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:19.253657 1175 log.cc:1199] T f93b0c39d7174dabbe018b1546dbc198 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/f93b0c39d7174dabbe018b1546dbc198
I20251029 02:52:19.254024 1175 ts_tablet_manager.cc:1950] T f93b0c39d7174dabbe018b1546dbc198 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:19.254354 1166 raft_consensus.cc:359] T b125cc9aaf4248b8bfa48fb152b832a6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:19.254560 1166 raft_consensus.cc:385] T b125cc9aaf4248b8bfa48fb152b832a6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:19.254644 1166 raft_consensus.cc:740] T b125cc9aaf4248b8bfa48fb152b832a6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:19.254823 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 1082) took 4953 ms. Trace:
I20251029 02:52:19.254846 1166 consensus_queue.cc:260] T b125cc9aaf4248b8bfa48fb152b832a6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:19.254926 1175 rpcz_store.cc:276] 1029 02:52:14.301738 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:14.305586 (+ 3848us) service_pool.cc:225] Handling call
1029 02:52:19.138164 (+4832578us) ts_tablet_manager.cc:1151] Deleting tablet f93b0c39d7174dabbe018b1546dbc198
1029 02:52:19.138173 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:19.180344 (+ 42171us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:19.253404 (+ 73060us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:19.254803 (+ 1399us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4832400}
I20251029 02:52:19.255321 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet f93b0c39d7174dabbe018b1546dbc198 (table default.Table_a2eda0beb1fb4fd3aa2a64eded1e17ad [id=61fac9a7d5b049e08485ce8695ba5a48]) successfully deleted
I20251029 02:52:19.255913 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 1084) took 4935 ms. Trace:
I20251029 02:52:19.256146 1166 ts_tablet_manager.cc:1434] T b125cc9aaf4248b8bfa48fb152b832a6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251029 02:52:19.256065 1175 rpcz_store.cc:276] 1029 02:52:14.320115 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:14.320181 (+ 66us) service_pool.cc:225] Handling call
1029 02:52:19.255751 (+4935570us) ts_tablet_manager.cc:1151] Deleting tablet 3c3ef62323d643dda28f0ca269274600
1029 02:52:19.255763 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:19.255895 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4934838}
W20251029 02:52:19.256438 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 3c3ef62323d643dda28f0ca269274600 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3c3ef62323d643dda28f0ca269274600 already in progress: creating tablet
I20251029 02:52:19.256559 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 1085) took 4918 ms. Trace:
I20251029 02:52:19.256646 1175 rpcz_store.cc:276] 1029 02:52:14.338531 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:14.338583 (+ 52us) service_pool.cc:225] Handling call
1029 02:52:19.256457 (+4917874us) ts_tablet_manager.cc:1151] Deleting tablet b81f26fb2739487dbc9af7b6e9848a4a
1029 02:52:19.256468 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:19.256545 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4917646}
I20251029 02:52:19.257141 1175 tablet_replica.cc:333] T c081d2eacfac4ca8a0048e2a1c4f8f83 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:19.257360 1175 raft_consensus.cc:2243] T c081d2eacfac4ca8a0048e2a1c4f8f83 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus shutting down.
W20251029 02:52:19.257565 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet b81f26fb2739487dbc9af7b6e9848a4a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b81f26fb2739487dbc9af7b6e9848a4a already in progress: creating tablet
I20251029 02:52:19.257599 1175 raft_consensus.cc:2272] T c081d2eacfac4ca8a0048e2a1c4f8f83 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:19.258769 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_4b5bd78da8af4f63b6f0e99546b381d4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:19.259493 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4b5bd78da8af4f63b6f0e99546b381d4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:19.260594 2949 raft_consensus.cc:493] T 7fb5b908db77453da5cfb32e2c955f41 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:19.260722 2949 raft_consensus.cc:515] T 7fb5b908db77453da5cfb32e2c955f41 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:19.261330 2949 leader_election.cc:290] T 7fb5b908db77453da5cfb32e2c955f41 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:52:19.263481 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7fb5b908db77453da5cfb32e2c955f41" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:19.264143 711 leader_election.cc:343] T 7fb5b908db77453da5cfb32e2c955f41 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:19.264979 1175 ts_tablet_manager.cc:1916] T c081d2eacfac4ca8a0048e2a1c4f8f83 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:19.270988 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_652b301189b1475293a17d420bba679f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:19.271545 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_652b301189b1475293a17d420bba679f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:19.272956 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_dc1320ec0fe44ca495d927b9297ed5af"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:19.273149 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_987be38a134f4846b4dff6150ef64354"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:19.273444 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dc1320ec0fe44ca495d927b9297ed5af in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:52:19.273615 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_987be38a134f4846b4dff6150ef64354 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:19.274477 1172 tablet_bootstrap.cc:492] T 294afb9b012c40c0b05b73795d6f0ac8 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:19.274672 1172 ts_tablet_manager.cc:1403] T 294afb9b012c40c0b05b73795d6f0ac8 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.066s user 0.002s sys 0.003s
I20251029 02:52:19.275493 1172 raft_consensus.cc:359] T 294afb9b012c40c0b05b73795d6f0ac8 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:19.275753 1172 raft_consensus.cc:385] T 294afb9b012c40c0b05b73795d6f0ac8 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:19.275946 1172 raft_consensus.cc:740] T 294afb9b012c40c0b05b73795d6f0ac8 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:19.276398 1172 consensus_queue.cc:260] T 294afb9b012c40c0b05b73795d6f0ac8 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:19.277781 1172 ts_tablet_manager.cc:1434] T 294afb9b012c40c0b05b73795d6f0ac8 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251029 02:52:19.280536 2426 raft_consensus.cc:493] T 97b2ecae572a4b7ea1b9595074556708 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b495a49349324f7d81908f2da700f0d9)
I20251029 02:52:19.280679 2426 raft_consensus.cc:515] T 97b2ecae572a4b7ea1b9595074556708 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:19.281250 2426 leader_election.cc:290] T 97b2ecae572a4b7ea1b9595074556708 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
W20251029 02:52:19.282017 847 leader_election.cc:343] T 97b2ecae572a4b7ea1b9595074556708 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 97b2ecae572a4b7ea1b9595074556708
I20251029 02:52:19.284391 2949 raft_consensus.cc:493] T 294afb9b012c40c0b05b73795d6f0ac8 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:19.284516 2949 raft_consensus.cc:515] T 294afb9b012c40c0b05b73795d6f0ac8 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:19.285109 2949 leader_election.cc:290] T 294afb9b012c40c0b05b73795d6f0ac8 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:52:19.285737 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "294afb9b012c40c0b05b73795d6f0ac8" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:19.286245 711 leader_election.cc:343] T 294afb9b012c40c0b05b73795d6f0ac8 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:19.290364 1172 tablet_bootstrap.cc:492] T 3a2b95997454452bb985c51fc02d77e2 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:19.292225 1175 ts_tablet_manager.cc:1929] T c081d2eacfac4ca8a0048e2a1c4f8f83 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:19.293293 1167 tablet_bootstrap.cc:492] T 5815626b05204b578cc4a110f246bf10 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:19.294831 1167 ts_tablet_manager.cc:1403] T 5815626b05204b578cc4a110f246bf10 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.144s user 0.002s sys 0.002s
I20251029 02:52:19.298638 1175 log.cc:1199] T c081d2eacfac4ca8a0048e2a1c4f8f83 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/c081d2eacfac4ca8a0048e2a1c4f8f83
I20251029 02:52:19.294199 1172 tablet_bootstrap.cc:654] T 3a2b95997454452bb985c51fc02d77e2 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:19.299587 526 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_6f4e7e4f715747b492f4d5b8e3ab8c5a" } new_table_name: "default.Table_6172c6e3d0fc4b5b9c7a37bfb31a0ac4" modify_external_catalogs: true
I20251029 02:52:19.301898 1167 raft_consensus.cc:359] T 5815626b05204b578cc4a110f246bf10 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:19.302585 1175 ts_tablet_manager.cc:1950] T c081d2eacfac4ca8a0048e2a1c4f8f83 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:19.303050 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "294afb9b012c40c0b05b73795d6f0ac8" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:52:19.303534 712 leader_election.cc:343] T 294afb9b012c40c0b05b73795d6f0ac8 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:19.303666 712 leader_election.cc:304] T 294afb9b012c40c0b05b73795d6f0ac8 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:19.304049 2949 raft_consensus.cc:2749] T 294afb9b012c40c0b05b73795d6f0ac8 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:19.305532 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 1095) took 4874 ms. Trace:
I20251029 02:52:19.305666 1175 rpcz_store.cc:276] 1029 02:52:14.430740 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:14.431066 (+ 326us) service_pool.cc:225] Handling call
1029 02:52:19.257097 (+4826031us) ts_tablet_manager.cc:1151] Deleting tablet c081d2eacfac4ca8a0048e2a1c4f8f83
1029 02:52:19.257106 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:19.268252 (+ 11146us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:19.292134 (+ 23882us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:19.305513 (+ 13379us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4825758}
I20251029 02:52:19.306166 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet c081d2eacfac4ca8a0048e2a1c4f8f83 (table default.Table_dbf2e434d7964eac80709c205f61191e [id=b6afde5f860040be9ede53221978f44d]) successfully deleted
I20251029 02:52:19.306280 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 1098) took 4866 ms. Trace:
I20251029 02:52:19.306340 1175 rpcz_store.cc:276] 1029 02:52:14.439607 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:14.439663 (+ 56us) service_pool.cc:225] Handling call
1029 02:52:19.306129 (+4866466us) ts_tablet_manager.cc:1151] Deleting tablet 0e9a9dd2f6e4424cb1500682d5a49cc4
1029 02:52:19.306139 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:19.306263 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4866272}
I20251029 02:52:19.306753 1175 tablet_replica.cc:333] T 84993b2f63e7421290a161dde4bb0607 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:19.306951 1175 raft_consensus.cc:2243] T 84993b2f63e7421290a161dde4bb0607 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:19.307159 1175 raft_consensus.cc:2272] T 84993b2f63e7421290a161dde4bb0607 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:19.307350 521 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_a9e50760813844af85a85c5f8d22f8e0" } new_table_name: "default.Table_90cf46f2fe2f4537aebcc726d215cff6" modify_external_catalogs: true
I20251029 02:52:19.307744 1175 ts_tablet_manager.cc:1916] T 84993b2f63e7421290a161dde4bb0607 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:19.308812 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_2124f7b7bcbc4277a1ad7282f21d4ae8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:19.309355 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2124f7b7bcbc4277a1ad7282f21d4ae8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:52:19.309551 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 0e9a9dd2f6e4424cb1500682d5a49cc4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0e9a9dd2f6e4424cb1500682d5a49cc4 already in progress: creating tablet
W20251029 02:52:19.312091 526 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bf67a67854d048a49a96cccccf500ab6 consensus state has no leader
I20251029 02:52:19.312489 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_d76695e16db9463f8c2e04ef98d39a0a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:19.313529 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d76695e16db9463f8c2e04ef98d39a0a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:19.317886 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_d80394d599864f1fb36d8008be9d70c2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:19.304829 849 leader_election.cc:343] T 97b2ecae572a4b7ea1b9595074556708 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Not found: Tablet not found: 97b2ecae572a4b7ea1b9595074556708
I20251029 02:52:19.318783 849 leader_election.cc:304] T 97b2ecae572a4b7ea1b9595074556708 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9
W20251029 02:52:19.318902 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d80394d599864f1fb36d8008be9d70c2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:52:19.319345 849 consensus_peers.cc:597] T 24f07b1efac64b099b0b5ed467463749 P c756528642a543eeacd93cef3e335a7d -> Peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Couldn't send request to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:19.319661 2747 raft_consensus.cc:2749] T 97b2ecae572a4b7ea1b9595074556708 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251029 02:52:19.320844 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7fb5b908db77453da5cfb32e2c955f41" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:52:19.321355 712 leader_election.cc:343] T 7fb5b908db77453da5cfb32e2c955f41 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:19.321471 712 leader_election.cc:304] T 7fb5b908db77453da5cfb32e2c955f41 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:19.321820 2949 raft_consensus.cc:2749] T 7fb5b908db77453da5cfb32e2c955f41 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:19.321906 1167 raft_consensus.cc:385] T 5815626b05204b578cc4a110f246bf10 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:19.322057 1167 raft_consensus.cc:740] T 5815626b05204b578cc4a110f246bf10 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:19.322187 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_4c57cfb0a2fb4bc4a10960416fe5fa3a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:19.322649 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4c57cfb0a2fb4bc4a10960416fe5fa3a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:19.324910 1167 consensus_queue.cc:260] T 5815626b05204b578cc4a110f246bf10 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:19.325816 1167 ts_tablet_manager.cc:1434] T 5815626b05204b578cc4a110f246bf10 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.025s user 0.001s sys 0.001s
I20251029 02:52:19.326181 1172 tablet_bootstrap.cc:492] T 3a2b95997454452bb985c51fc02d77e2 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:19.326220 1167 tablet_bootstrap.cc:492] T d31e548feae24b739087c1e211d514f9 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:19.326370 1172 ts_tablet_manager.cc:1403] T 3a2b95997454452bb985c51fc02d77e2 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.036s user 0.002s sys 0.002s
I20251029 02:52:19.335131 1172 raft_consensus.cc:359] T 3a2b95997454452bb985c51fc02d77e2 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:19.335176 526 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_6206526535a84403a4b5e8c8ad05887b" } new_table_name: "default.Table_2b1ea9736c834f9d8b25fa6a8ff7a3d7" modify_external_catalogs: true
I20251029 02:52:19.335439 1172 raft_consensus.cc:385] T 3a2b95997454452bb985c51fc02d77e2 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:19.335565 1172 raft_consensus.cc:740] T 3a2b95997454452bb985c51fc02d77e2 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:19.335803 1172 consensus_queue.cc:260] T 3a2b95997454452bb985c51fc02d77e2 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:19.336359 1172 ts_tablet_manager.cc:1434] T 3a2b95997454452bb985c51fc02d77e2 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 02:52:19.336791 1172 tablet_bootstrap.cc:492] T 08371b3daa184a0e9f4eb3d313b52c56 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:19.338440 1166 tablet_bootstrap.cc:492] T 628a7d0419cb434f8171faf500d585d8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:19.348640 2747 consensus_queue.cc:1048] T abf18e06df9b482a8b25c078b9495272 P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:19.348826 1059 raft_consensus.cc:3060] T 6dff036b32904ddb9598be4b79504bb6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:19.349320 1176 ts_tablet_manager.cc:1929] T 46cd75ff18b14212b556b9bcb3012dd9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20251029 02:52:19.352207 526 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a342154638a041c2aecf43b783e0a796 consensus state has no leader
I20251029 02:52:19.361039 1175 ts_tablet_manager.cc:1929] T 84993b2f63e7421290a161dde4bb0607 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:19.361183 1175 log.cc:1199] T 84993b2f63e7421290a161dde4bb0607 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/84993b2f63e7421290a161dde4bb0607
I20251029 02:52:19.361379 1176 log.cc:1199] T 46cd75ff18b14212b556b9bcb3012dd9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/46cd75ff18b14212b556b9bcb3012dd9
I20251029 02:52:19.361600 1175 ts_tablet_manager.cc:1950] T 84993b2f63e7421290a161dde4bb0607 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:19.361851 1176 ts_tablet_manager.cc:1950] T 46cd75ff18b14212b556b9bcb3012dd9 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:19.362623 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 46cd75ff18b14212b556b9bcb3012dd9 (table default.Table_2ccb14f87f774da3b62823c798b31a7c [id=0829f9f13eea4c2081a9545b98e928bf]) successfully deleted
W20251029 02:52:19.363469 513 catalog_manager.cc:4952] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet c081d2eacfac4ca8a0048e2a1c4f8f83 because the tablet was not found. No further retry: Not found: Tablet not found: c081d2eacfac4ca8a0048e2a1c4f8f83
I20251029 02:52:19.363883 1176 tablet_replica.cc:333] T eca016aed05e4ce09e1a2bf96e2286f5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:19.365478 1176 raft_consensus.cc:2243] T eca016aed05e4ce09e1a2bf96e2286f5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:19.350670 1166 tablet_bootstrap.cc:654] T 628a7d0419cb434f8171faf500d585d8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:19.366627 526 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_ff347dc490d441baa678753aa33b710b" } modify_external_catalogs: true
I20251029 02:52:19.366838 526 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_ff347dc490d441baa678753aa33b710b" } modify_external_catalogs: true
I20251029 02:52:19.368384 517 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_95f035fda47144eb9d79be23614b41b6" } new_table_name: "default.Table_21e4218e82574af6882a525378b46a6e" modify_external_catalogs: true
I20251029 02:52:19.372023 1172 tablet_bootstrap.cc:654] T 08371b3daa184a0e9f4eb3d313b52c56 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:19.373083 1176 raft_consensus.cc:2272] T eca016aed05e4ce09e1a2bf96e2286f5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:19.374652 2952 tablet.cc:1722] T 27b08f88b99643d28ac609e45bf8104e P c756528642a543eeacd93cef3e335a7d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20251029 02:52:19.381157 517 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 03f59f9206024dfbaa4e67a5b6bfb902 consensus state has no leader
I20251029 02:52:19.381507 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_ae547cee980b4988a4f096b110e967fd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:19.382071 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ae547cee980b4988a4f096b110e967fd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:52:19.382201 847 consensus_peers.cc:597] T 23b9b2ea814645eba63e5f6b12cfb966 P c756528642a543eeacd93cef3e335a7d -> Peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Couldn't send request to peer b495a49349324f7d81908f2da700f0d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 23b9b2ea814645eba63e5f6b12cfb966. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:19.384627 2981 tablet.cc:1722] T 27b08f88b99643d28ac609e45bf8104e P b495a49349324f7d81908f2da700f0d9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 02:52:19.389353 2961 raft_consensus.cc:493] T f4729ecd182042c2a10f67ca25224101 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:19.389519 2961 raft_consensus.cc:515] T f4729ecd182042c2a10f67ca25224101 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:19.390144 2961 leader_election.cc:290] T f4729ecd182042c2a10f67ca25224101 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:19.390769 1174 ts_tablet_manager.cc:1929] T af4860c6b72f4d988b206b5ef2f22906 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.2
I20251029 02:52:19.390895 1174 log.cc:1199] T af4860c6b72f4d988b206b5ef2f22906 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/af4860c6b72f4d988b206b5ef2f22906
I20251029 02:52:19.391276 1174 ts_tablet_manager.cc:1950] T af4860c6b72f4d988b206b5ef2f22906 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:19.391336 767 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4729ecd182042c2a10f67ca25224101" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:19.393908 525 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet 055184c641ba45a588bca611fb24d130 (table default.Table_1922b14d4cfb4671a6224fcd6ace2d4e [id=aec3db766dee438c89fb2f1939ee7b7b]). Expected version 2 got 0
I20251029 02:52:19.395071 1167 tablet_bootstrap.cc:654] T d31e548feae24b739087c1e211d514f9 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:19.398524 523 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet b125cc9aaf4248b8bfa48fb152b832a6 (table default.Table_110e1b64737c4040ada22360d5a373cb [id=3513c69f6b3941849ac5b8ae0a5846d3]). Expected version 5 got 0
I20251029 02:52:19.399039 520 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet b125cc9aaf4248b8bfa48fb152b832a6 (table default.Table_110e1b64737c4040ada22360d5a373cb [id=3513c69f6b3941849ac5b8ae0a5846d3]). Expected version 5 got 0
I20251029 02:52:19.400243 1176 ts_tablet_manager.cc:1916] T eca016aed05e4ce09e1a2bf96e2286f5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:19.403548 525 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_6172c6e3d0fc4b5b9c7a37bfb31a0ac4" } new_table_name: "default.Table_6e2a477ce83c4deba033b4b270f6880e" modify_external_catalogs: true
I20251029 02:52:19.405488 1042 tablet_service.cc:1505] Processing CreateTablet for tablet 1e1a3d6e43834bddacd45cc3da301dbb (DEFAULT_TABLE table=default.Table_810b941251184116ac99d736431a5978 [id=3edf556f120f41b39ac2d7ffd9baee4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:19.406006 887 tablet_service.cc:1505] Processing CreateTablet for tablet 1e1a3d6e43834bddacd45cc3da301dbb (DEFAULT_TABLE table=default.Table_810b941251184116ac99d736431a5978 [id=3edf556f120f41b39ac2d7ffd9baee4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:19.406417 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e1a3d6e43834bddacd45cc3da301dbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:19.406708 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f4729ecd182042c2a10f67ca25224101" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:19.406854 738 tablet_service.cc:1505] Processing CreateTablet for tablet 1e1a3d6e43834bddacd45cc3da301dbb (DEFAULT_TABLE table=default.Table_810b941251184116ac99d736431a5978 [id=3edf556f120f41b39ac2d7ffd9baee4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:19.407240 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e1a3d6e43834bddacd45cc3da301dbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:19.407286 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_7ad2035911a64b3fa4965915468d8730"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:19.407747 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7ad2035911a64b3fa4965915468d8730 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:19.409072 990 leader_election.cc:304] T f4729ecd182042c2a10f67ca25224101 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5; no voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:19.409529 2961 raft_consensus.cc:2749] T f4729ecd182042c2a10f67ca25224101 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:19.414875 524 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 2c424a54c9714fe490677e64e3dd2931
I20251029 02:52:19.415783 2949 raft_consensus.cc:493] T 055184c641ba45a588bca611fb24d130 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:19.415941 2949 raft_consensus.cc:515] T 055184c641ba45a588bca611fb24d130 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:19.416580 2949 leader_election.cc:290] T 055184c641ba45a588bca611fb24d130 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:52:19.417196 1034 tablet_service.cc:1552] Processing DeleteTablet for tablet 2c424a54c9714fe490677e64e3dd2931 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:19 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:19.417585 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "055184c641ba45a588bca611fb24d130" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:19.417776 915 raft_consensus.cc:2468] T 055184c641ba45a588bca611fb24d130 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b495a49349324f7d81908f2da700f0d9 in term 0.
I20251029 02:52:19.417884 742 tablet_service.cc:1552] Processing DeleteTablet for tablet 2c424a54c9714fe490677e64e3dd2931 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:19 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:19.418287 711 leader_election.cc:304] T 055184c641ba45a588bca611fb24d130 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:19.418315 887 tablet_service.cc:1552] Processing DeleteTablet for tablet 2c424a54c9714fe490677e64e3dd2931 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:19 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:19.418637 2949 raft_consensus.cc:2804] T 055184c641ba45a588bca611fb24d130 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:19.418730 2949 raft_consensus.cc:493] T 055184c641ba45a588bca611fb24d130 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:19.418800 2949 raft_consensus.cc:3060] T 055184c641ba45a588bca611fb24d130 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:19.419369 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 997) took 8156 ms. Trace:
I20251029 02:52:19.419449 1174 rpcz_store.cc:276] 1029 02:52:11.262663 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:11.262713 (+ 50us) service_pool.cc:225] Handling call
1029 02:52:19.145751 (+7883038us) ts_tablet_manager.cc:1151] Deleting tablet af4860c6b72f4d988b206b5ef2f22906
1029 02:52:19.145826 (+ 75us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:19.296570 (+150744us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:19.390651 (+ 94081us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:19.419275 (+ 28624us) spinlock_profiling.cc:243] Waited 27.6 ms on lock 0x61800000d518. stack: 00007f1e09cad9f6 00007f1e09cad7b0 00007f1e08c893ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f1e09c962fd 00007f1e09c9736e 00007f1e1d8d2c9d 00007f1e1d851203 00007f1e19a866f3 00007f1e19ab92d4 00007f1e19a99d70 00007f1e19a99afd 0000000000a8867c 00007f1e09cf3d7f
1029 02:52:19.419355 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":63500672,"tablet-delete.queue_time_us":7882354}
I20251029 02:52:19.419883 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet af4860c6b72f4d988b206b5ef2f22906 (table default.Table_f5db3e3913ca4ea98b02981cd04a5965 [id=7eb08c679e0e441bb1ad4be13b2494ef]) successfully deleted
I20251029 02:52:19.420280 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 998) took 8129 ms. Trace:
I20251029 02:52:19.420357 1174 rpcz_store.cc:276] 1029 02:52:11.290820 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:11.290946 (+ 126us) service_pool.cc:225] Handling call
1029 02:52:19.420155 (+8129209us) ts_tablet_manager.cc:1151] Deleting tablet 96f7876e5d6d4a0db236df5e7f266fd2
1029 02:52:19.420165 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:19.420264 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8128968}
W20251029 02:52:19.420753 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 96f7876e5d6d4a0db236df5e7f266fd2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 96f7876e5d6d4a0db236df5e7f266fd2 already in progress: creating tablet
I20251029 02:52:19.421104 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 999) took 8093 ms. Trace:
I20251029 02:52:19.421180 1174 rpcz_store.cc:276] 1029 02:52:11.327543 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:11.328080 (+ 537us) service_pool.cc:225] Handling call
1029 02:52:19.420997 (+8092917us) ts_tablet_manager.cc:1151] Deleting tablet ecf3b900206b4c86a7adbc83a769d90e
1029 02:52:19.421010 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:19.421088 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":8092702}
I20251029 02:52:19.421553 2949 raft_consensus.cc:515] T 055184c641ba45a588bca611fb24d130 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:19.421763 1174 tablet_replica.cc:333] T 1bdb67125493430398ff5c80b73de2ac P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:19.421959 1174 raft_consensus.cc:2243] T 1bdb67125493430398ff5c80b73de2ac P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:19.422330 1174 raft_consensus.cc:2272] T 1bdb67125493430398ff5c80b73de2ac P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:19.423058 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "055184c641ba45a588bca611fb24d130" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
W20251029 02:52:19.423576 712 leader_election.cc:343] T 055184c641ba45a588bca611fb24d130 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:19.425076 2949 leader_election.cc:290] T 055184c641ba45a588bca611fb24d130 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 election: Requested vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:52:19.425611 1172 tablet_bootstrap.cc:492] T 08371b3daa184a0e9f4eb3d313b52c56 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:19.425766 1172 ts_tablet_manager.cc:1403] T 08371b3daa184a0e9f4eb3d313b52c56 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.089s user 0.007s sys 0.007s
I20251029 02:52:19.426666 1172 raft_consensus.cc:359] T 08371b3daa184a0e9f4eb3d313b52c56 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:19.426853 1172 raft_consensus.cc:385] T 08371b3daa184a0e9f4eb3d313b52c56 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:19.426934 1172 raft_consensus.cc:740] T 08371b3daa184a0e9f4eb3d313b52c56 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:19.427206 1172 consensus_queue.cc:260] T 08371b3daa184a0e9f4eb3d313b52c56 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:19.428232 1172 ts_tablet_manager.cc:1434] T 08371b3daa184a0e9f4eb3d313b52c56 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251029 02:52:19.428560 1172 tablet_bootstrap.cc:492] T 877e278deb4a4b01ac66b06be0890baf P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:19.429538 2980 tablet.cc:1722] T 27b08f88b99643d28ac609e45bf8104e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 02:52:19.429960 522 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 2d01627906b444dc8386191004e650e2
I20251029 02:52:19.430441 519 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 4c17d21145e54cc78134cab37059ce1e
I20251029 02:52:19.432708 738 tablet_service.cc:1552] Processing DeleteTablet for tablet 2d01627906b444dc8386191004e650e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:19 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:19.433570 742 tablet_service.cc:1552] Processing DeleteTablet for tablet 4c17d21145e54cc78134cab37059ce1e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:19 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:19.434930 1034 tablet_service.cc:1552] Processing DeleteTablet for tablet 2d01627906b444dc8386191004e650e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:19 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:19.438766 1031 tablet_service.cc:1552] Processing DeleteTablet for tablet 4c17d21145e54cc78134cab37059ce1e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:19 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:19.439213 1174 ts_tablet_manager.cc:1916] T 1bdb67125493430398ff5c80b73de2ac P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:19.439368 1031 tablet_service.cc:1505] Processing CreateTablet for tablet 902188c70bbc4d6c9cd4c1ba2c2fe295 (DEFAULT_TABLE table=default.Table_a75246042ab0422cbfc4c83f2ab5b81c [id=38b27ed58e314e0ea476e4ae53705cd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:19.424373 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "055184c641ba45a588bca611fb24d130" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
I20251029 02:52:19.439960 1034 tablet_service.cc:1505] Processing CreateTablet for tablet bec66cf322b24103abfeb41ebb367488 (DEFAULT_TABLE table=default.Table_7ac30f4a6b164b459121be1019009bbc [id=045b10aa70be447dbdecca17b0c8ea12]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:19.440053 915 raft_consensus.cc:3060] T 055184c641ba45a588bca611fb24d130 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:19.440972 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bec66cf322b24103abfeb41ebb367488. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:19.441179 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e1a3d6e43834bddacd45cc3da301dbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:19.427796 2747 consensus_queue.cc:1048] T 24f07b1efac64b099b0b5ed467463749 P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:19.431181 1167 tablet_bootstrap.cc:492] T d31e548feae24b739087c1e211d514f9 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:19.442957 1167 ts_tablet_manager.cc:1403] T d31e548feae24b739087c1e211d514f9 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.117s user 0.004s sys 0.000s
I20251029 02:52:19.443658 1167 raft_consensus.cc:359] T d31e548feae24b739087c1e211d514f9 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:19.443802 1167 raft_consensus.cc:385] T d31e548feae24b739087c1e211d514f9 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:19.443845 1167 raft_consensus.cc:740] T d31e548feae24b739087c1e211d514f9 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:19.444008 1167 consensus_queue.cc:260] T d31e548feae24b739087c1e211d514f9 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:19.444375 1167 ts_tablet_manager.cc:1434] T d31e548feae24b739087c1e211d514f9 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20251029 02:52:19.444638 1167 tablet_bootstrap.cc:492] T 1df0d944717d487a98ea83a8ec04bd46 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:19.446153 1167 tablet_bootstrap.cc:654] T 1df0d944717d487a98ea83a8ec04bd46 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:19.447518 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 902188c70bbc4d6c9cd4c1ba2c2fe295. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:19.451822 744 tablet_service.cc:1505] Processing CreateTablet for tablet 5793a23424c542aa8c5c57d031d2c149 (DEFAULT_TABLE table=default.Table_e0250e6a474d4907b9bd8f39596af69c [id=29aa2b54ea5246a8905118785ef1d4ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:19.452209 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5793a23424c542aa8c5c57d031d2c149. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:19.453262 738 tablet_service.cc:1505] Processing CreateTablet for tablet bec66cf322b24103abfeb41ebb367488 (DEFAULT_TABLE table=default.Table_7ac30f4a6b164b459121be1019009bbc [id=045b10aa70be447dbdecca17b0c8ea12]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:19.453610 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bec66cf322b24103abfeb41ebb367488. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:19.454174 891 tablet_service.cc:1552] Processing DeleteTablet for tablet 4c17d21145e54cc78134cab37059ce1e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:19 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:19.454275 895 tablet_service.cc:1505] Processing CreateTablet for tablet bec66cf322b24103abfeb41ebb367488 (DEFAULT_TABLE table=default.Table_7ac30f4a6b164b459121be1019009bbc [id=045b10aa70be447dbdecca17b0c8ea12]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:19.454555 888 tablet_service.cc:1505] Processing CreateTablet for tablet 902188c70bbc4d6c9cd4c1ba2c2fe295 (DEFAULT_TABLE table=default.Table_a75246042ab0422cbfc4c83f2ab5b81c [id=38b27ed58e314e0ea476e4ae53705cd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:19.454886 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 902188c70bbc4d6c9cd4c1ba2c2fe295. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:19.454806 742 tablet_service.cc:1505] Processing CreateTablet for tablet 902188c70bbc4d6c9cd4c1ba2c2fe295 (DEFAULT_TABLE table=default.Table_a75246042ab0422cbfc4c83f2ab5b81c [id=38b27ed58e314e0ea476e4ae53705cd3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:19.455125 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 902188c70bbc4d6c9cd4c1ba2c2fe295. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:19.459856 2747 raft_consensus.cc:493] T b0fec06ad37d4c12a7045d424b88cc5e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:19.459988 2747 raft_consensus.cc:515] T b0fec06ad37d4c12a7045d424b88cc5e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:19.460570 2747 leader_election.cc:290] T b0fec06ad37d4c12a7045d424b88cc5e P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:19.461143 1052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0fec06ad37d4c12a7045d424b88cc5e" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:19.461428 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0fec06ad37d4c12a7045d424b88cc5e" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:19.461597 773 raft_consensus.cc:2468] T b0fec06ad37d4c12a7045d424b88cc5e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
W20251029 02:52:19.461650 849 leader_election.cc:343] T b0fec06ad37d4c12a7045d424b88cc5e P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:19.461953 847 leader_election.cc:304] T b0fec06ad37d4c12a7045d424b88cc5e P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5
I20251029 02:52:19.462603 2747 raft_consensus.cc:2804] T b0fec06ad37d4c12a7045d424b88cc5e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:19.462680 2747 raft_consensus.cc:493] T b0fec06ad37d4c12a7045d424b88cc5e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:19.462736 2747 raft_consensus.cc:3060] T b0fec06ad37d4c12a7045d424b88cc5e P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:19.463114 900 tablet_service.cc:1505] Processing CreateTablet for tablet 5793a23424c542aa8c5c57d031d2c149 (DEFAULT_TABLE table=default.Table_e0250e6a474d4907b9bd8f39596af69c [id=29aa2b54ea5246a8905118785ef1d4ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:19.463439 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5793a23424c542aa8c5c57d031d2c149. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:19.463798 2949 raft_consensus.cc:493] T b125cc9aaf4248b8bfa48fb152b832a6 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:19.463919 2949 raft_consensus.cc:515] T b125cc9aaf4248b8bfa48fb152b832a6 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:19.464519 2949 leader_election.cc:290] T b125cc9aaf4248b8bfa48fb152b832a6 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:19.465314 2747 raft_consensus.cc:515] T b0fec06ad37d4c12a7045d424b88cc5e P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:19.466296 1052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0fec06ad37d4c12a7045d424b88cc5e" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:52:19.466605 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b0fec06ad37d4c12a7045d424b88cc5e" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:19.466758 773 raft_consensus.cc:3060] T b0fec06ad37d4c12a7045d424b88cc5e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
W20251029 02:52:19.466809 849 leader_election.cc:343] T b0fec06ad37d4c12a7045d424b88cc5e P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:19.468272 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bec66cf322b24103abfeb41ebb367488. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:19.469489 773 raft_consensus.cc:2468] T b0fec06ad37d4c12a7045d424b88cc5e P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:19.469936 847 leader_election.cc:304] T b0fec06ad37d4c12a7045d424b88cc5e P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5
I20251029 02:52:19.470422 2655 raft_consensus.cc:2804] T b0fec06ad37d4c12a7045d424b88cc5e P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:19.470568 2655 raft_consensus.cc:697] T b0fec06ad37d4c12a7045d424b88cc5e P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:52:19.470821 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b125cc9aaf4248b8bfa48fb152b832a6" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:19.470800 2655 consensus_queue.cc:237] T b0fec06ad37d4c12a7045d424b88cc5e P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:19.470950 916 raft_consensus.cc:2468] T b125cc9aaf4248b8bfa48fb152b832a6 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b495a49349324f7d81908f2da700f0d9 in term 0.
I20251029 02:52:19.471225 2747 leader_election.cc:290] T b0fec06ad37d4c12a7045d424b88cc5e P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:19.471971 711 leader_election.cc:304] T b125cc9aaf4248b8bfa48fb152b832a6 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:19.472241 2949 raft_consensus.cc:2804] T b125cc9aaf4248b8bfa48fb152b832a6 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:19.472302 2949 raft_consensus.cc:493] T b125cc9aaf4248b8bfa48fb152b832a6 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:19.472362 2949 raft_consensus.cc:3060] T b125cc9aaf4248b8bfa48fb152b832a6 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:19.475250 2949 raft_consensus.cc:515] T b125cc9aaf4248b8bfa48fb152b832a6 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:19.475821 2949 leader_election.cc:290] T b125cc9aaf4248b8bfa48fb152b832a6 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 election: Requested vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:19.476089 887 tablet_service.cc:1552] Processing DeleteTablet for tablet 2d01627906b444dc8386191004e650e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:19 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:19.476485 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_762e1fd7769c4e09957fba5622c238d5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:19.476891 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b125cc9aaf4248b8bfa48fb152b832a6" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
W20251029 02:52:19.477075 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_762e1fd7769c4e09957fba5622c238d5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:19.477084 916 raft_consensus.cc:3060] T b125cc9aaf4248b8bfa48fb152b832a6 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:19.478107 518 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 902188c70bbc4d6c9cd4c1ba2c2fe295
I20251029 02:52:19.480213 916 raft_consensus.cc:2468] T b125cc9aaf4248b8bfa48fb152b832a6 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b495a49349324f7d81908f2da700f0d9 in term 1.
I20251029 02:52:19.480683 711 leader_election.cc:304] T b125cc9aaf4248b8bfa48fb152b832a6 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:19.481545 887 tablet_service.cc:1552] Processing DeleteTablet for tablet 902188c70bbc4d6c9cd4c1ba2c2fe295 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:19 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:19.481623 738 tablet_service.cc:1552] Processing DeleteTablet for tablet 902188c70bbc4d6c9cd4c1ba2c2fe295 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:19 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:19.482066 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_273e8e890c2a4eeab49a97dcb392b1f3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:19.482370 915 raft_consensus.cc:2468] T 055184c641ba45a588bca611fb24d130 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b495a49349324f7d81908f2da700f0d9 in term 1.
W20251029 02:52:19.482621 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_273e8e890c2a4eeab49a97dcb392b1f3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:19.482800 711 leader_election.cc:304] T 055184c641ba45a588bca611fb24d130 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5
I20251029 02:52:19.485910 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_56159ac1d36948e38c1ca46a13d5a81d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:19.485975 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_153fb459f00845158311f068069c5c25"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:19.486402 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_56159ac1d36948e38c1ca46a13d5a81d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:52:19.486402 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_153fb459f00845158311f068069c5c25 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:19.489990 2949 raft_consensus.cc:2804] T b125cc9aaf4248b8bfa48fb152b832a6 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:19.492130 1055 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b125cc9aaf4248b8bfa48fb152b832a6" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:52:19.492327 1055 raft_consensus.cc:3060] T b125cc9aaf4248b8bfa48fb152b832a6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:19.492712 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b125cc9aaf4248b8bfa48fb152b832a6" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:19.495326 1055 raft_consensus.cc:2468] T b125cc9aaf4248b8bfa48fb152b832a6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b495a49349324f7d81908f2da700f0d9 in term 1.
I20251029 02:52:19.495769 2649 raft_consensus.cc:493] T 44c0488eba0543078ff4f0b3d69366f2 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0cfd4bee16f84b7c8b0ff3ce14db20d5)
I20251029 02:52:19.495903 2649 raft_consensus.cc:515] T 44c0488eba0543078ff4f0b3d69366f2 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:19.496568 2649 leader_election.cc:290] T 44c0488eba0543078ff4f0b3d69366f2 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:19.496980 2649 raft_consensus.cc:493] T e65eaaae43ac435ebd665546de9d68dd P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0cfd4bee16f84b7c8b0ff3ce14db20d5)
I20251029 02:52:19.497174 1042 tablet_service.cc:1552] Processing DeleteTablet for tablet 902188c70bbc4d6c9cd4c1ba2c2fe295 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:19 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:19.497109 2649 raft_consensus.cc:515] T e65eaaae43ac435ebd665546de9d68dd P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:19.497659 1034 tablet_service.cc:1505] Processing CreateTablet for tablet 5793a23424c542aa8c5c57d031d2c149 (DEFAULT_TABLE table=default.Table_e0250e6a474d4907b9bd8f39596af69c [id=29aa2b54ea5246a8905118785ef1d4ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:19.497759 2649 leader_election.cc:290] T e65eaaae43ac435ebd665546de9d68dd P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:19.498040 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5793a23424c542aa8c5c57d031d2c149. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:19.499369 849 consensus_peers.cc:597] T c081d2eacfac4ca8a0048e2a1c4f8f83 P c756528642a543eeacd93cef3e335a7d -> Peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Couldn't send request to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c081d2eacfac4ca8a0048e2a1c4f8f83. This is attempt 1: this message will repeat every 5th retry.
W20251029 02:52:19.499612 849 leader_election.cc:343] T e65eaaae43ac435ebd665546de9d68dd P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Not found: Tablet not found: e65eaaae43ac435ebd665546de9d68dd
W20251029 02:52:19.499774 849 leader_election.cc:343] T 44c0488eba0543078ff4f0b3d69366f2 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Not found: Tablet not found: 44c0488eba0543078ff4f0b3d69366f2
I20251029 02:52:19.499970 2656 raft_consensus.cc:493] T 2f198c3259864f7ab3625cb7bfae20b0 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0cfd4bee16f84b7c8b0ff3ce14db20d5)
I20251029 02:52:19.500094 2656 raft_consensus.cc:515] T 2f198c3259864f7ab3625cb7bfae20b0 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:19.500710 2656 leader_election.cc:290] T 2f198c3259864f7ab3625cb7bfae20b0 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
W20251029 02:52:19.502183 849 leader_election.cc:343] T 2f198c3259864f7ab3625cb7bfae20b0 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Not found: Tablet not found: 2f198c3259864f7ab3625cb7bfae20b0
I20251029 02:52:19.503587 2949 raft_consensus.cc:697] T b125cc9aaf4248b8bfa48fb152b832a6 P b495a49349324f7d81908f2da700f0d9 [term 1 LEADER]: Becoming Leader. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Running, Role: LEADER
I20251029 02:52:19.503758 2987 raft_consensus.cc:2804] T 055184c641ba45a588bca611fb24d130 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:19.505856 1167 tablet_bootstrap.cc:492] T 1df0d944717d487a98ea83a8ec04bd46 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:19.506044 1167 ts_tablet_manager.cc:1403] T 1df0d944717d487a98ea83a8ec04bd46 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.061s user 0.002s sys 0.002s
I20251029 02:52:19.507004 1167 raft_consensus.cc:359] T 1df0d944717d487a98ea83a8ec04bd46 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:19.507217 1167 raft_consensus.cc:385] T 1df0d944717d487a98ea83a8ec04bd46 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:19.507308 1167 raft_consensus.cc:740] T 1df0d944717d487a98ea83a8ec04bd46 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:19.507572 1167 consensus_queue.cc:260] T 1df0d944717d487a98ea83a8ec04bd46 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:19.508270 1167 ts_tablet_manager.cc:1434] T 1df0d944717d487a98ea83a8ec04bd46 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251029 02:52:19.503924 2949 consensus_queue.cc:237] T b125cc9aaf4248b8bfa48fb152b832a6 P b495a49349324f7d81908f2da700f0d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:19.511749 1167 tablet_bootstrap.cc:492] T 506f8c112a3b43e1a3d6bbafb8cb81e8 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:19.513767 1167 tablet_bootstrap.cc:654] T 506f8c112a3b43e1a3d6bbafb8cb81e8 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:19.515676 523 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet b125cc9aaf4248b8bfa48fb152b832a6 (table default.Table_110e1b64737c4040ada22360d5a373cb [id=3513c69f6b3941849ac5b8ae0a5846d3]). Expected version 5 got 0
I20251029 02:52:19.515880 523 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet 6dff036b32904ddb9598be4b79504bb6 (table default.Table_0ae102f09fdc41bcbead0148653df9e2 [id=65ec61bf400a4628b98e64e7d43c025c]). Expected version 2 got 0
I20251029 02:52:19.522229 2980 tablet.cc:1722] T 6dff036b32904ddb9598be4b79504bb6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 02:52:19.529469 1166 tablet_bootstrap.cc:492] T 628a7d0419cb434f8171faf500d585d8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:19.529749 1166 ts_tablet_manager.cc:1403] T 628a7d0419cb434f8171faf500d585d8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.191s user 0.003s sys 0.007s
I20251029 02:52:19.531349 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_bab353e338ed483e9f685de2637cc326"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:19.531901 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bab353e338ed483e9f685de2637cc326 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:19.533109 1166 raft_consensus.cc:359] T 628a7d0419cb434f8171faf500d585d8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:19.533365 1166 raft_consensus.cc:385] T 628a7d0419cb434f8171faf500d585d8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:19.533466 1166 raft_consensus.cc:740] T 628a7d0419cb434f8171faf500d585d8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:19.533741 1166 consensus_queue.cc:260] T 628a7d0419cb434f8171faf500d585d8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:19.534117 1174 ts_tablet_manager.cc:1929] T 1bdb67125493430398ff5c80b73de2ac P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251029 02:52:19.534284 1174 log.cc:1199] T 1bdb67125493430398ff5c80b73de2ac P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/1bdb67125493430398ff5c80b73de2ac
I20251029 02:52:19.534524 1166 ts_tablet_manager.cc:1434] T 628a7d0419cb434f8171faf500d585d8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251029 02:52:19.535512 1176 ts_tablet_manager.cc:1929] T eca016aed05e4ce09e1a2bf96e2286f5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:19.535642 1176 log.cc:1199] T eca016aed05e4ce09e1a2bf96e2286f5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/eca016aed05e4ce09e1a2bf96e2286f5
I20251029 02:52:19.536307 1176 ts_tablet_manager.cc:1950] T eca016aed05e4ce09e1a2bf96e2286f5 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:19.541517 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_6db362a6fbd742eab5cdd05ecbbc8647"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:19.542104 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6db362a6fbd742eab5cdd05ecbbc8647 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:19.542697 1167 tablet_bootstrap.cc:492] T 506f8c112a3b43e1a3d6bbafb8cb81e8 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:19.544726 1167 ts_tablet_manager.cc:1403] T 506f8c112a3b43e1a3d6bbafb8cb81e8 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.033s user 0.004s sys 0.001s
I20251029 02:52:19.545536 1031 tablet_service.cc:1505] Processing CreateTablet for tablet 44c5224ef3444516a675c23662d34bec (DEFAULT_TABLE table=default.Table_c873cfe841d04574a044e4796fcb3ea7 [id=15051d88fe2a4d8994212386945b6056]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:19.548619 1167 raft_consensus.cc:359] T 506f8c112a3b43e1a3d6bbafb8cb81e8 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:19.551879 1167 raft_consensus.cc:385] T 506f8c112a3b43e1a3d6bbafb8cb81e8 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:19.552372 1167 raft_consensus.cc:740] T 506f8c112a3b43e1a3d6bbafb8cb81e8 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:19.553148 1167 consensus_queue.cc:260] T 506f8c112a3b43e1a3d6bbafb8cb81e8 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:19.555295 1167 ts_tablet_manager.cc:1434] T 506f8c112a3b43e1a3d6bbafb8cb81e8 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20251029 02:52:19.555435 1107 heartbeater.cc:344] Connected to a master server at 127.0.104.62:42343
I20251029 02:52:19.557396 1167 tablet_bootstrap.cc:492] T cc802bda2f6a4e439b605f2cafa37967 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:19.562251 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "055184c641ba45a588bca611fb24d130" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:19.563537 1167 tablet_bootstrap.cc:654] T cc802bda2f6a4e439b605f2cafa37967 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:19.567133 1059 raft_consensus.cc:3060] T f4729ecd182042c2a10f67ca25224101 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:19.570917 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44c5224ef3444516a675c23662d34bec. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:19.572852 526 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet b79327b8c1354810a8b1b0d9e62fe3c8
I20251029 02:52:19.556286 1174 ts_tablet_manager.cc:1950] T 1bdb67125493430398ff5c80b73de2ac P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:19.574365 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 1012) took 7887 ms. Trace:
I20251029 02:52:19.574568 1174 rpcz_store.cc:276] 1029 02:52:11.686757 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:11.690738 (+ 3981us) service_pool.cc:225] Handling call
1029 02:52:19.421710 (+7730972us) ts_tablet_manager.cc:1151] Deleting tablet 1bdb67125493430398ff5c80b73de2ac
1029 02:52:19.421720 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:19.530372 (+108652us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:19.533978 (+ 3606us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:19.574343 (+ 40365us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7730800}
I20251029 02:52:19.575294 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 1014) took 7840 ms. Trace:
I20251029 02:52:19.575529 1174 rpcz_store.cc:276] 1029 02:52:11.734805 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:11.735104 (+ 299us) service_pool.cc:225] Handling call
1029 02:52:19.575165 (+7840061us) ts_tablet_manager.cc:1151] Deleting tablet 81ee839d811a4f46b93a57ede397bd93
1029 02:52:19.575177 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:19.575278 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7839871}
I20251029 02:52:19.575971 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 1015) took 7841 ms. Trace:
I20251029 02:52:19.576102 1174 rpcz_store.cc:276] 1029 02:52:11.734922 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:11.735322 (+ 400us) service_pool.cc:225] Handling call
1029 02:52:19.575870 (+7840548us) ts_tablet_manager.cc:1151] Deleting tablet 0f96af1cb0ea4b46bd846ae0739f4947
1029 02:52:19.575879 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:19.575957 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7840407}
I20251029 02:52:19.576494 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 1016) took 7841 ms. Trace:
I20251029 02:52:19.576624 1174 rpcz_store.cc:276] 1029 02:52:11.735053 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:11.735489 (+ 436us) service_pool.cc:225] Handling call
1029 02:52:19.576384 (+7840895us) ts_tablet_manager.cc:1151] Deleting tablet a530fe8de8ae4be5b0fd9a5f7e94ea9e
1029 02:52:19.576395 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:19.576482 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7840756}
I20251029 02:52:19.577075 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 1017) took 7839 ms. Trace:
I20251029 02:52:19.577184 1174 rpcz_store.cc:276] 1029 02:52:11.738028 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:11.738112 (+ 84us) service_pool.cc:225] Handling call
1029 02:52:19.576970 (+7838858us) ts_tablet_manager.cc:1151] Deleting tablet 426f9be3b0074f60989acc43ec1e9bd6
1029 02:52:19.576978 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:19.577059 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7838679}
I20251029 02:52:19.577533 1174 tablet_replica.cc:333] T 97b2ecae572a4b7ea1b9595074556708 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:19.577771 1174 raft_consensus.cc:2243] T 97b2ecae572a4b7ea1b9595074556708 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:19.578161 1174 raft_consensus.cc:2272] T 97b2ecae572a4b7ea1b9595074556708 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:19.578140 1034 tablet_service.cc:1552] Processing DeleteTablet for tablet b79327b8c1354810a8b1b0d9e62fe3c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:19 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:19.578884 888 tablet_service.cc:1552] Processing DeleteTablet for tablet b79327b8c1354810a8b1b0d9e62fe3c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:19 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:19.578998 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 1bdb67125493430398ff5c80b73de2ac (table default.Table_2846af457a4b4e27a739af50c7f8c7f7 [id=8c3d3d88395e4712af0f62c4ad14edea]) successfully deleted
I20251029 02:52:19.579380 1174 ts_tablet_manager.cc:1916] T 97b2ecae572a4b7ea1b9595074556708 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 02:52:19.579478 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 81ee839d811a4f46b93a57ede397bd93 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 81ee839d811a4f46b93a57ede397bd93 already in progress: creating tablet
W20251029 02:52:19.579959 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 0f96af1cb0ea4b46bd846ae0739f4947 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0f96af1cb0ea4b46bd846ae0739f4947 already in progress: creating tablet
W20251029 02:52:19.580595 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet a530fe8de8ae4be5b0fd9a5f7e94ea9e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a530fe8de8ae4be5b0fd9a5f7e94ea9e already in progress: creating tablet
I20251029 02:52:19.581454 1166 tablet_bootstrap.cc:492] T 5815626b05204b578cc4a110f246bf10 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
W20251029 02:52:19.580955 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 426f9be3b0074f60989acc43ec1e9bd6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 426f9be3b0074f60989acc43ec1e9bd6 already in progress: creating tablet
I20251029 02:52:19.581710 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet eca016aed05e4ce09e1a2bf96e2286f5 (table default.Table_ee6dec2fc7834c78914ddc9f13890c55 [id=2faf0479a2a6477fa1d70d30094ef1db]) successfully deleted
I20251029 02:52:19.581491 1176 tablet_replica.cc:333] T 44b518c3a0964a6fa863b2784d80c8a4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:19.584214 1167 tablet_bootstrap.cc:492] T cc802bda2f6a4e439b605f2cafa37967 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:19.584285 1176 raft_consensus.cc:2243] T 44b518c3a0964a6fa863b2784d80c8a4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:19.584354 1167 ts_tablet_manager.cc:1403] T cc802bda2f6a4e439b605f2cafa37967 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.001s
I20251029 02:52:19.584617 1176 raft_consensus.cc:2272] T 44b518c3a0964a6fa863b2784d80c8a4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:19.585173 1176 ts_tablet_manager.cc:1916] T 44b518c3a0964a6fa863b2784d80c8a4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:19.585158 1167 raft_consensus.cc:359] T cc802bda2f6a4e439b605f2cafa37967 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:19.585353 1167 raft_consensus.cc:385] T cc802bda2f6a4e439b605f2cafa37967 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:19.585400 1167 raft_consensus.cc:740] T cc802bda2f6a4e439b605f2cafa37967 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:19.594982 1167 consensus_queue.cc:260] T cc802bda2f6a4e439b605f2cafa37967 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:19.597368 1167 ts_tablet_manager.cc:1434] T cc802bda2f6a4e439b605f2cafa37967 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20251029 02:52:19.598564 1034 tablet_service.cc:1505] Processing CreateTablet for tablet 3abc92dd0c0942d49e52311ff9824383 (DEFAULT_TABLE table=default.Table_1caf7466260e4f13ab3fcd2d52e80744 [id=731357aa50324045a19cd7f543ff0cde]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:19.599082 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3abc92dd0c0942d49e52311ff9824383. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:19.599195 1167 tablet_bootstrap.cc:492] T 7fb5b908db77453da5cfb32e2c955f41 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:19.599603 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 1093) took 5187 ms. Trace:
I20251029 02:52:19.599699 1175 rpcz_store.cc:276] 1029 02:52:14.412155 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:14.467928 (+ 55773us) service_pool.cc:225] Handling call
1029 02:52:19.306711 (+4838783us) ts_tablet_manager.cc:1151] Deleting tablet 84993b2f63e7421290a161dde4bb0607
1029 02:52:19.306720 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:19.311050 (+ 4330us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:19.360917 (+ 49867us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:19.598913 (+237996us) spinlock_profiling.cc:243] Waited 17.2 ms on lock 0x616000117454. stack: 00007f1174e9f9f6 00007f1174e9f7b0 00007f1173e7b3ad 0000000000983f04 0000000000983ecd 000000000098389a 00007f1180d69f8a 00007f1180d28354 00007f11820b9936 00007f1184c78684 00007f1184cab2d4 00007f1184c8bd70 00007f1184c8bafd 0000000000a8867c 00007f1174ee5d7f 00007f1174ee904b
1029 02:52:19.599580 (+ 667us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":39499136,"tablet-delete.queue_time_us":4838549}
I20251029 02:52:19.600071 1175 tablet_replica.cc:333] T 946f65014d97425aab3851009b742807 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:19.600265 1175 raft_consensus.cc:2243] T 946f65014d97425aab3851009b742807 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:19.600425 1175 raft_consensus.cc:2272] T 946f65014d97425aab3851009b742807 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:19.600754 1167 tablet_bootstrap.cc:654] T 7fb5b908db77453da5cfb32e2c955f41 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:19.601020 1175 ts_tablet_manager.cc:1916] T 946f65014d97425aab3851009b742807 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:19.602886 888 tablet_service.cc:1505] Processing CreateTablet for tablet 44c5224ef3444516a675c23662d34bec (DEFAULT_TABLE table=default.Table_c873cfe841d04574a044e4796fcb3ea7 [id=15051d88fe2a4d8994212386945b6056]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:19.603340 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44c5224ef3444516a675c23662d34bec. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:19.603937 1166 tablet_bootstrap.cc:654] T 5815626b05204b578cc4a110f246bf10 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:19.604534 2961 raft_consensus.cc:493] T 628a7d0419cb434f8171faf500d585d8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:19.604650 2961 raft_consensus.cc:515] T 628a7d0419cb434f8171faf500d585d8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:19.605266 2961 leader_election.cc:290] T 628a7d0419cb434f8171faf500d585d8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:19.605998 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "628a7d0419cb434f8171faf500d585d8" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:19.607055 712 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:45003, user_credentials={real_user=slave}} blocked reactor thread for 110720us
W20251029 02:52:19.616672 849 consensus_peers.cc:597] T abf18e06df9b482a8b25c078b9495272 P c756528642a543eeacd93cef3e335a7d -> Peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Couldn't send request to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:19.607183 2989 raft_consensus.cc:493] T 5815626b05204b578cc4a110f246bf10 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:19.619627 2655 raft_consensus.cc:493] T 217144957930495aa3e9dfcfde268e71 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0cfd4bee16f84b7c8b0ff3ce14db20d5)
I20251029 02:52:19.619756 2655 raft_consensus.cc:515] T 217144957930495aa3e9dfcfde268e71 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:19.620380 2655 leader_election.cc:290] T 217144957930495aa3e9dfcfde268e71 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
W20251029 02:52:19.621454 849 leader_election.cc:343] T 217144957930495aa3e9dfcfde268e71 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Not found: Tablet not found: 217144957930495aa3e9dfcfde268e71
I20251029 02:52:19.618136 2989 raft_consensus.cc:515] T 5815626b05204b578cc4a110f246bf10 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:19.634366 773 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "628a7d0419cb434f8171faf500d585d8" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:19.635202 888 tablet_service.cc:1505] Processing CreateTablet for tablet 3abc92dd0c0942d49e52311ff9824383 (DEFAULT_TABLE table=default.Table_1caf7466260e4f13ab3fcd2d52e80744 [id=731357aa50324045a19cd7f543ff0cde]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:19.636812 2989 leader_election.cc:290] T 5815626b05204b578cc4a110f246bf10 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:52:19.636936 1174 ts_tablet_manager.cc:1929] T 97b2ecae572a4b7ea1b9595074556708 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20251029 02:52:19.639931 712 leader_election.cc:343] T 055184c641ba45a588bca611fb24d130 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:19.641184 989 leader_election.cc:304] T 628a7d0419cb434f8171faf500d585d8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5; no voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:19.641490 1174 log.cc:1199] T 97b2ecae572a4b7ea1b9595074556708 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/97b2ecae572a4b7ea1b9595074556708
I20251029 02:52:19.641603 2961 raft_consensus.cc:2749] T 628a7d0419cb434f8171faf500d585d8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:19.642220 1174 ts_tablet_manager.cc:1950] T 97b2ecae572a4b7ea1b9595074556708 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:19.644500 1167 tablet_bootstrap.cc:492] T 7fb5b908db77453da5cfb32e2c955f41 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:19.644713 1167 ts_tablet_manager.cc:1403] T 7fb5b908db77453da5cfb32e2c955f41 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.046s user 0.000s sys 0.008s
I20251029 02:52:19.645645 1167 raft_consensus.cc:359] T 7fb5b908db77453da5cfb32e2c955f41 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:19.645877 1167 raft_consensus.cc:385] T 7fb5b908db77453da5cfb32e2c955f41 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:19.645977 1167 raft_consensus.cc:740] T 7fb5b908db77453da5cfb32e2c955f41 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:19.646288 1167 consensus_queue.cc:260] T 7fb5b908db77453da5cfb32e2c955f41 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:19.646955 1167 ts_tablet_manager.cc:1434] T 7fb5b908db77453da5cfb32e2c955f41 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251029 02:52:19.660257 2989 raft_consensus.cc:493] T 1df0d944717d487a98ea83a8ec04bd46 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:19.660456 2989 raft_consensus.cc:515] T 1df0d944717d487a98ea83a8ec04bd46 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
W20251029 02:52:19.666241 847 leader_election.cc:343] T 44c0488eba0543078ff4f0b3d69366f2 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 44c0488eba0543078ff4f0b3d69366f2
I20251029 02:52:19.666392 847 leader_election.cc:304] T 44c0488eba0543078ff4f0b3d69366f2 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9
I20251029 02:52:19.666637 2989 leader_election.cc:290] T 1df0d944717d487a98ea83a8ec04bd46 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
W20251029 02:52:19.666960 847 leader_election.cc:343] T e65eaaae43ac435ebd665546de9d68dd P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: e65eaaae43ac435ebd665546de9d68dd
I20251029 02:52:19.667057 847 leader_election.cc:304] T e65eaaae43ac435ebd665546de9d68dd P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9
W20251029 02:52:19.667340 847 leader_election.cc:343] T 2f198c3259864f7ab3625cb7bfae20b0 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 2f198c3259864f7ab3625cb7bfae20b0
I20251029 02:52:19.667408 847 leader_election.cc:304] T 2f198c3259864f7ab3625cb7bfae20b0 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9
W20251029 02:52:19.667692 847 consensus_peers.cc:597] T ad214c870ee649279ed7321c5b69cbee P c756528642a543eeacd93cef3e335a7d -> Peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Couldn't send request to peer b495a49349324f7d81908f2da700f0d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ad214c870ee649279ed7321c5b69cbee. This is attempt 1: this message will repeat every 5th retry.
W20251029 02:52:19.667804 847 consensus_peers.cc:597] T 84993b2f63e7421290a161dde4bb0607 P c756528642a543eeacd93cef3e335a7d -> Peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Couldn't send request to peer b495a49349324f7d81908f2da700f0d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 84993b2f63e7421290a161dde4bb0607. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:19.668078 2655 raft_consensus.cc:2749] T 44c0488eba0543078ff4f0b3d69366f2 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251029 02:52:19.668182 525 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet bf67a67854d048a49a96cccccf500ab6 consensus state has no leader
I20251029 02:52:19.668205 2655 raft_consensus.cc:2749] T e65eaaae43ac435ebd665546de9d68dd P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251029 02:52:19.668318 2655 raft_consensus.cc:2749] T 2f198c3259864f7ab3625cb7bfae20b0 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251029 02:52:19.672356 520 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_c243191d6825474288c6942a1efeb0dd" } new_table_name: "default.Table_857b0213ebc144668cd7c5edff52bba4" modify_external_catalogs: true
W20251029 02:52:19.681346 847 leader_election.cc:343] T 217144957930495aa3e9dfcfde268e71 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 217144957930495aa3e9dfcfde268e71
I20251029 02:52:19.681484 847 leader_election.cc:304] T 217144957930495aa3e9dfcfde268e71 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9
I20251029 02:52:19.682067 2655 raft_consensus.cc:2749] T 217144957930495aa3e9dfcfde268e71 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251029 02:52:19.686404 711 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:41701, user_credentials={real_user=slave}} blocked reactor thread for 203624us
I20251029 02:52:19.687047 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 84993b2f63e7421290a161dde4bb0607 (table default.Table_93b405a4f257481289518c5ff28832a0 [id=da230f99a9894ab8aa2fdb4b167169ad]) successfully deleted
I20251029 02:52:19.687906 738 tablet_service.cc:1505] Processing CreateTablet for tablet 44c5224ef3444516a675c23662d34bec (DEFAULT_TABLE table=default.Table_c873cfe841d04574a044e4796fcb3ea7 [id=15051d88fe2a4d8994212386945b6056]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:19.688257 744 tablet_service.cc:1552] Processing DeleteTablet for tablet b79327b8c1354810a8b1b0d9e62fe3c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:19 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:19.688324 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44c5224ef3444516a675c23662d34bec. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:19.688516 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5815626b05204b578cc4a110f246bf10" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:19.688702 915 raft_consensus.cc:2468] T 5815626b05204b578cc4a110f246bf10 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b495a49349324f7d81908f2da700f0d9 in term 0.
I20251029 02:52:19.689656 740 tablet_service.cc:1505] Processing CreateTablet for tablet 3abc92dd0c0942d49e52311ff9824383 (DEFAULT_TABLE table=default.Table_1caf7466260e4f13ab3fcd2d52e80744 [id=731357aa50324045a19cd7f543ff0cde]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:19.690002 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1df0d944717d487a98ea83a8ec04bd46" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:19.690173 916 raft_consensus.cc:2468] T 1df0d944717d487a98ea83a8ec04bd46 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b495a49349324f7d81908f2da700f0d9 in term 0.
I20251029 02:52:19.690531 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3abc92dd0c0942d49e52311ff9824383. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:19.691257 711 leader_election.cc:304] T 5815626b05204b578cc4a110f246bf10 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:19.691644 2985 raft_consensus.cc:2804] T 5815626b05204b578cc4a110f246bf10 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:19.691713 2985 raft_consensus.cc:493] T 5815626b05204b578cc4a110f246bf10 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:19.691769 2985 raft_consensus.cc:3060] T 5815626b05204b578cc4a110f246bf10 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:19.694787 711 leader_election.cc:304] T 1df0d944717d487a98ea83a8ec04bd46 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:19.695348 1172 tablet_bootstrap.cc:654] T 877e278deb4a4b01ac66b06be0890baf P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:19.695482 2992 raft_consensus.cc:2804] T 1df0d944717d487a98ea83a8ec04bd46 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:19.702849 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3abc92dd0c0942d49e52311ff9824383. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:19.703302 895 tablet_service.cc:1505] Processing CreateTablet for tablet e257e5ffc4314f61b2df087d59139129 (DEFAULT_TABLE table=default.Table_553a74353c2a4583ac706c63b70e8c94 [id=21f0bc58ea744818afc7e72331a60ac3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:19.703656 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 1020) took 7809 ms. Trace:
I20251029 02:52:19.703719 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e257e5ffc4314f61b2df087d59139129. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:19.703771 1174 rpcz_store.cc:276] 1029 02:52:11.894323 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:11.898671 (+ 4348us) service_pool.cc:225] Handling call
1029 02:52:19.577490 (+7678819us) ts_tablet_manager.cc:1151] Deleting tablet 97b2ecae572a4b7ea1b9595074556708
1029 02:52:19.577501 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:19.610747 (+ 33246us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:19.636809 (+ 26062us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:19.703635 (+ 66826us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7678305}
I20251029 02:52:19.704093 1174 tablet_replica.cc:333] T 44c0488eba0543078ff4f0b3d69366f2 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:19.704283 1174 raft_consensus.cc:2243] T 44c0488eba0543078ff4f0b3d69366f2 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:19.704442 1174 raft_consensus.cc:2272] T 44c0488eba0543078ff4f0b3d69366f2 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:19.706772 1167 tablet_bootstrap.cc:492] T 08371b3daa184a0e9f4eb3d313b52c56 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:19.707120 2649 raft_consensus.cc:493] T afb19ba39399410bb90b6225d7b5b5b1 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0cfd4bee16f84b7c8b0ff3ce14db20d5)
I20251029 02:52:19.707234 2649 raft_consensus.cc:515] T afb19ba39399410bb90b6225d7b5b5b1 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:19.707860 2649 leader_election.cc:290] T afb19ba39399410bb90b6225d7b5b5b1 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:19.708180 1167 tablet_bootstrap.cc:654] T 08371b3daa184a0e9f4eb3d313b52c56 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:19.714197 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 97b2ecae572a4b7ea1b9595074556708 (table default.Table_3b4bdfaf4ddf43cfa2d2287594c57816 [id=222ef1b283554565b4930328c2c8975b]) successfully deleted
I20251029 02:52:19.701521 2992 raft_consensus.cc:493] T 1df0d944717d487a98ea83a8ec04bd46 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:19.715794 2992 raft_consensus.cc:3060] T 1df0d944717d487a98ea83a8ec04bd46 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:19.718451 521 catalog_manager.cc:6194] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: default.Table_90cf46f2fe2f4537aebcc726d215cff6 [id=74a2a41c908146ec8e16c6a3227b94f2] alter complete (version 2)
I20251029 02:52:19.720043 523 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_2aba26f0828343a6981aad6b2e3ac97f" } new_table_name: "default.Table_bc94c6255afc48ea9f94d0a4533ee3ec" modify_external_catalogs: true
I20251029 02:52:19.720252 521 master_service.cc:954] ReplaceTablet: received request to replace tablet 98bbaec01186435b8bcc7257a1390185 from {username='slave'} at 127.0.0.1:35314
I20251029 02:52:19.720711 521 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_07fdb04dcb96444cb939ea7d0843af6b" } modify_external_catalogs: true
I20251029 02:52:19.720870 521 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_07fdb04dcb96444cb939ea7d0843af6b" } modify_external_catalogs: true
I20251029 02:52:19.720913 522 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_697672643d0d4986badc08e39aa0bb14" } new_table_name: "default.Table_96f5ac69176245fc9a4d1b244fcf28b1" modify_external_catalogs: true
I20251029 02:52:19.720899 2985 raft_consensus.cc:515] T 5815626b05204b578cc4a110f246bf10 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:19.721503 2985 leader_election.cc:290] T 5815626b05204b578cc4a110f246bf10 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 election: Requested vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:52:19.722066 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_3f96167cf5b048b2846521470dfea839"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:19.722285 525 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_61c29c73a4354026ba1dab6dd81536ea" } new_table_name: "default.Table_525b7fdbf9d843c480b0c77c795c0d62" modify_external_catalogs: true
W20251029 02:52:19.722662 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3f96167cf5b048b2846521470dfea839 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:19.723969 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_f996bd8d41254afcb7aca941ebc3aa3b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:19.724424 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f996bd8d41254afcb7aca941ebc3aa3b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:52:19.724927 847 leader_election.cc:343] T afb19ba39399410bb90b6225d7b5b5b1 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: afb19ba39399410bb90b6225d7b5b5b1
I20251029 02:52:19.725538 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_2fa0c67ec1ea41128a9284f86f15a7c1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:19.725936 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2fa0c67ec1ea41128a9284f86f15a7c1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:19.727099 744 tablet_service.cc:1505] Processing CreateTablet for tablet e257e5ffc4314f61b2df087d59139129 (DEFAULT_TABLE table=default.Table_553a74353c2a4583ac706c63b70e8c94 [id=21f0bc58ea744818afc7e72331a60ac3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:19.727468 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e257e5ffc4314f61b2df087d59139129. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:19.733158 2992 raft_consensus.cc:515] T 1df0d944717d487a98ea83a8ec04bd46 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:19.733723 2992 leader_election.cc:290] T 1df0d944717d487a98ea83a8ec04bd46 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 election: Requested vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:52:19.734949 524 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet 3a2b95997454452bb985c51fc02d77e2 (table default.Table_eddccb434b6d4c048d894eefcee2065b [id=0c3d955bdfd34db2ac137dd6f5d84a2b]). Expected version 2 got 0
I20251029 02:52:19.735167 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5815626b05204b578cc4a110f246bf10" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
I20251029 02:52:19.735344 916 raft_consensus.cc:3060] T 5815626b05204b578cc4a110f246bf10 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:19.736423 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1df0d944717d487a98ea83a8ec04bd46" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
I20251029 02:52:19.736594 915 raft_consensus.cc:3060] T 1df0d944717d487a98ea83a8ec04bd46 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:19.736671 524 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_32cbbd63da094e50be86cd48ec2be3bc" } modify_external_catalogs: true
I20251029 02:52:19.736843 524 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_32cbbd63da094e50be86cd48ec2be3bc" } modify_external_catalogs: true
I20251029 02:52:19.738178 916 raft_consensus.cc:2468] T 5815626b05204b578cc4a110f246bf10 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b495a49349324f7d81908f2da700f0d9 in term 1.
I20251029 02:52:19.738627 711 leader_election.cc:304] T 5815626b05204b578cc4a110f246bf10 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:19.738831 526 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_b3877f1183ba42e9b368d74299f13f55" } modify_external_catalogs: true
I20251029 02:52:19.739019 526 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_b3877f1183ba42e9b368d74299f13f55" } modify_external_catalogs: true
I20251029 02:52:19.739114 2986 raft_consensus.cc:493] T d1866d6dcb0349bba5db86163375cada P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:19.739233 2986 raft_consensus.cc:515] T d1866d6dcb0349bba5db86163375cada P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:19.740000 2986 leader_election.cc:290] T d1866d6dcb0349bba5db86163375cada P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:52:19.740162 2986 raft_consensus.cc:2804] T 5815626b05204b578cc4a110f246bf10 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:19.740294 2986 raft_consensus.cc:697] T 5815626b05204b578cc4a110f246bf10 P b495a49349324f7d81908f2da700f0d9 [term 1 LEADER]: Becoming Leader. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Running, Role: LEADER
I20251029 02:52:19.740581 2986 consensus_queue.cc:237] T 5815626b05204b578cc4a110f246bf10 P b495a49349324f7d81908f2da700f0d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:19.742256 1167 tablet_bootstrap.cc:492] T 08371b3daa184a0e9f4eb3d313b52c56 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:19.742731 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d1866d6dcb0349bba5db86163375cada" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:19.743206 1167 ts_tablet_manager.cc:1403] T 08371b3daa184a0e9f4eb3d313b52c56 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.037s user 0.003s sys 0.005s
W20251029 02:52:19.743417 711 leader_election.cc:343] T d1866d6dcb0349bba5db86163375cada P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:19.743777 740 tablet_service.cc:1552] Processing DeleteTablet for tablet d1866d6dcb0349bba5db86163375cada with delete_type TABLET_DATA_DELETED (replaced by tablet 313b963f9d63437ba0dd32385b21f5a6) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:19.743906 738 tablet_service.cc:1552] Processing DeleteTablet for tablet 294afb9b012c40c0b05b73795d6f0ac8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:59 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:19.744807 1167 raft_consensus.cc:359] T 08371b3daa184a0e9f4eb3d313b52c56 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:19.745021 1167 raft_consensus.cc:385] T 08371b3daa184a0e9f4eb3d313b52c56 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:19.745110 1167 raft_consensus.cc:740] T 08371b3daa184a0e9f4eb3d313b52c56 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:19.745404 1167 consensus_queue.cc:260] T 08371b3daa184a0e9f4eb3d313b52c56 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:19.746603 1167 ts_tablet_manager.cc:1434] T 08371b3daa184a0e9f4eb3d313b52c56 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20251029 02:52:19.746707 915 raft_consensus.cc:2468] T 1df0d944717d487a98ea83a8ec04bd46 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b495a49349324f7d81908f2da700f0d9 in term 1.
I20251029 02:52:19.750730 711 leader_election.cc:304] T 1df0d944717d487a98ea83a8ec04bd46 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:19.751051 2992 raft_consensus.cc:2804] T 1df0d944717d487a98ea83a8ec04bd46 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:19.751158 2992 raft_consensus.cc:697] T 1df0d944717d487a98ea83a8ec04bd46 P b495a49349324f7d81908f2da700f0d9 [term 1 LEADER]: Becoming Leader. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Running, Role: LEADER
I20251029 02:52:19.751359 2992 consensus_queue.cc:237] T 1df0d944717d487a98ea83a8ec04bd46 P b495a49349324f7d81908f2da700f0d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:19.754773 1167 tablet_bootstrap.cc:492] T 294afb9b012c40c0b05b73795d6f0ac8 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:19.756918 1167 tablet_bootstrap.cc:654] T 294afb9b012c40c0b05b73795d6f0ac8 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:19.756987 900 tablet_service.cc:1552] Processing DeleteTablet for tablet 1bdb67125493430398ff5c80b73de2ac with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:11 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:19.757508 887 tablet_service.cc:1552] Processing DeleteTablet for tablet 5815626b05204b578cc4a110f246bf10 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:08 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:19.757542 2655 raft_consensus.cc:493] T 08371b3daa184a0e9f4eb3d313b52c56 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:19.759071 2655 raft_consensus.cc:515] T 08371b3daa184a0e9f4eb3d313b52c56 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:19.759704 2655 leader_election.cc:290] T 08371b3daa184a0e9f4eb3d313b52c56 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:19.761605 766 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "08371b3daa184a0e9f4eb3d313b52c56" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:19.761793 766 raft_consensus.cc:2468] T 08371b3daa184a0e9f4eb3d313b52c56 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:19.762212 847 leader_election.cc:304] T 08371b3daa184a0e9f4eb3d313b52c56 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:19.696061 2987 raft_consensus.cc:697] T 055184c641ba45a588bca611fb24d130 P b495a49349324f7d81908f2da700f0d9 [term 1 LEADER]: Becoming Leader. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Running, Role: LEADER
I20251029 02:52:19.762457 2655 raft_consensus.cc:2804] T 08371b3daa184a0e9f4eb3d313b52c56 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:19.762544 2655 raft_consensus.cc:493] T 08371b3daa184a0e9f4eb3d313b52c56 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:19.762600 2655 raft_consensus.cc:3060] T 08371b3daa184a0e9f4eb3d313b52c56 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:19.762691 2987 consensus_queue.cc:237] T 055184c641ba45a588bca611fb24d130 P b495a49349324f7d81908f2da700f0d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:19.765242 2655 raft_consensus.cc:515] T 08371b3daa184a0e9f4eb3d313b52c56 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:19.766022 2655 leader_election.cc:290] T 08371b3daa184a0e9f4eb3d313b52c56 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:19.766615 766 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "08371b3daa184a0e9f4eb3d313b52c56" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:19.766768 766 raft_consensus.cc:3060] T 08371b3daa184a0e9f4eb3d313b52c56 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:19.769856 766 raft_consensus.cc:2468] T 08371b3daa184a0e9f4eb3d313b52c56 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:19.770431 847 leader_election.cc:304] T 08371b3daa184a0e9f4eb3d313b52c56 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:19.770751 2655 raft_consensus.cc:2804] T 08371b3daa184a0e9f4eb3d313b52c56 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:19.770888 2655 raft_consensus.cc:697] T 08371b3daa184a0e9f4eb3d313b52c56 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:52:19.771145 2655 consensus_queue.cc:237] T 08371b3daa184a0e9f4eb3d313b52c56 P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:19.773056 1174 ts_tablet_manager.cc:1916] T 44c0488eba0543078ff4f0b3d69366f2 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 02:52:19.780161 520 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 51a6e45378344f86847a44cf4950307b consensus state has no leader
I20251029 02:52:19.781577 1174 ts_tablet_manager.cc:1929] T 44c0488eba0543078ff4f0b3d69366f2 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:19.781678 1174 log.cc:1199] T 44c0488eba0543078ff4f0b3d69366f2 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/44c0488eba0543078ff4f0b3d69366f2
I20251029 02:52:19.782002 1174 ts_tablet_manager.cc:1950] T 44c0488eba0543078ff4f0b3d69366f2 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
W20251029 02:52:19.785226 847 consensus_peers.cc:597] T d29cc2a0c8b84e3ebadc7f571d666847 P c756528642a543eeacd93cef3e335a7d -> Peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Couldn't send request to peer b495a49349324f7d81908f2da700f0d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d29cc2a0c8b84e3ebadc7f571d666847. This is attempt 1: this message will repeat every 5th retry.
W20251029 02:52:19.788784 522 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f3dfd1d767294ea08ef5e957b349ee12 consensus state has no leader
I20251029 02:52:19.802863 519 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_7ca8e3e58d62408c8010ea1aeb204a94" } modify_external_catalogs: true
I20251029 02:52:19.803082 519 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_7ca8e3e58d62408c8010ea1aeb204a94" } modify_external_catalogs: true
I20251029 02:52:19.804949 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 1025) took 7565 ms. Trace:
I20251029 02:52:19.805066 1174 rpcz_store.cc:276] 1029 02:52:12.239858 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:12.240177 (+ 319us) service_pool.cc:225] Handling call
1029 02:52:19.704052 (+7463875us) ts_tablet_manager.cc:1151] Deleting tablet 44c0488eba0543078ff4f0b3d69366f2
1029 02:52:19.704061 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:19.778266 (+ 74205us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:19.781480 (+ 3214us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:19.804900 (+ 23420us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7463633}
I20251029 02:52:19.806295 522 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_d07c65e3c15d47aa9b724a151431aa52" } modify_external_catalogs: true
I20251029 02:52:19.806504 522 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_d07c65e3c15d47aa9b724a151431aa52" } modify_external_catalogs: true
I20251029 02:52:19.808316 1174 tablet_replica.cc:333] T 217144957930495aa3e9dfcfde268e71 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:19.809262 1174 raft_consensus.cc:2243] T 217144957930495aa3e9dfcfde268e71 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:19.809309 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 44c0488eba0543078ff4f0b3d69366f2 (table default.Table_bc32d7beda4f4957817b20124bf48e9c [id=df809b5c66d64eb1846fa5eb9da11781]) successfully deleted
I20251029 02:52:19.810410 1174 raft_consensus.cc:2272] T 217144957930495aa3e9dfcfde268e71 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:19.811586 1174 ts_tablet_manager.cc:1916] T 217144957930495aa3e9dfcfde268e71 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:19.818311 1167 tablet_bootstrap.cc:492] T 294afb9b012c40c0b05b73795d6f0ac8 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:19.819658 1167 ts_tablet_manager.cc:1403] T 294afb9b012c40c0b05b73795d6f0ac8 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.065s user 0.008s sys 0.008s
I20251029 02:52:19.825709 1166 tablet_bootstrap.cc:492] T 5815626b05204b578cc4a110f246bf10 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:19.820153 887 tablet_service.cc:1505] Processing CreateTablet for tablet fed40a07c28549db90af66acdab11521 (DEFAULT_TABLE table=default.Table_e464d8b2defc45f1aefefab339650ac0 [id=baf82b1773ef426a8ea379cf59987dc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:19.825845 1166 ts_tablet_manager.cc:1403] T 5815626b05204b578cc4a110f246bf10 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.244s user 0.003s sys 0.000s
I20251029 02:52:19.822288 900 tablet_service.cc:1505] Processing CreateTablet for tablet c5fe79779cb14993a02497d746085ac9 (DEFAULT_TABLE table=default.Table_1400dc2050d64e82a07e3fcd8018c0d2 [id=532c91430c034154aeaa767a5a1c7ff6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:19.826040 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fed40a07c28549db90af66acdab11521. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:19.826232 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5fe79779cb14993a02497d746085ac9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:19.827188 1166 raft_consensus.cc:359] T 5815626b05204b578cc4a110f246bf10 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:19.827358 1166 raft_consensus.cc:385] T 5815626b05204b578cc4a110f246bf10 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:19.827404 1166 raft_consensus.cc:740] T 5815626b05204b578cc4a110f246bf10 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:19.827565 1166 consensus_queue.cc:260] T 5815626b05204b578cc4a110f246bf10 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:19.828109 1166 ts_tablet_manager.cc:1434] T 5815626b05204b578cc4a110f246bf10 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 02:52:19.828348 1166 tablet_bootstrap.cc:492] T 055184c641ba45a588bca611fb24d130 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:19.830263 1166 tablet_bootstrap.cc:654] T 055184c641ba45a588bca611fb24d130 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:19.833941 1055 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5815626b05204b578cc4a110f246bf10" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:19.834152 1055 raft_consensus.cc:2468] T 5815626b05204b578cc4a110f246bf10 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b495a49349324f7d81908f2da700f0d9 in term 0.
I20251029 02:52:19.835420 520 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet f4729ecd182042c2a10f67ca25224101 (table default.Table_973ef856041949598a23da3dab8b40db [id=1dd2d5488bc647c8b82c53886583b28d]). Expected version 4 got 0
I20251029 02:52:19.833621 1167 raft_consensus.cc:359] T 294afb9b012c40c0b05b73795d6f0ac8 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:19.843892 1034 tablet_service.cc:1505] Processing CreateTablet for tablet e257e5ffc4314f61b2df087d59139129 (DEFAULT_TABLE table=default.Table_553a74353c2a4583ac706c63b70e8c94 [id=21f0bc58ea744818afc7e72331a60ac3]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:19.843963 1167 raft_consensus.cc:385] T 294afb9b012c40c0b05b73795d6f0ac8 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:19.844360 1167 raft_consensus.cc:740] T 294afb9b012c40c0b05b73795d6f0ac8 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:19.844961 1167 consensus_queue.cc:260] T 294afb9b012c40c0b05b73795d6f0ac8 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:19.846104 1167 ts_tablet_manager.cc:1434] T 294afb9b012c40c0b05b73795d6f0ac8 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20251029 02:52:19.847282 1167 tablet_bootstrap.cc:492] T 023a4094b424464c8f49d6eb9cc15da4 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:19.848414 1042 tablet_service.cc:1505] Processing CreateTablet for tablet fed40a07c28549db90af66acdab11521 (DEFAULT_TABLE table=default.Table_e464d8b2defc45f1aefefab339650ac0 [id=baf82b1773ef426a8ea379cf59987dc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:19.848564 1039 tablet_service.cc:1505] Processing CreateTablet for tablet 8ccca343c67942c7a97e3c19bcfb5993 (DEFAULT_TABLE table=default.Table_fd903b7b9822450fa41766f9f091d82c [id=c5f98f9d0f0141e789d6d1075fc2ee37]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:19.848829 1040 tablet_service.cc:1505] Processing CreateTablet for tablet c5fe79779cb14993a02497d746085ac9 (DEFAULT_TABLE table=default.Table_1400dc2050d64e82a07e3fcd8018c0d2 [id=532c91430c034154aeaa767a5a1c7ff6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:19.851786 1167 tablet_bootstrap.cc:654] T 023a4094b424464c8f49d6eb9cc15da4 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:19.853919 1052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "5815626b05204b578cc4a110f246bf10" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:52:19.854101 1052 raft_consensus.cc:3060] T 5815626b05204b578cc4a110f246bf10 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:19.854748 521 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet b22929678f734288829c53fffadfa8a9
W20251029 02:52:19.854769 525 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5c40f2f9cf2b41a3b129e7ef1bf41eb3 consensus state has no leader
I20251029 02:52:19.854966 1056 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1df0d944717d487a98ea83a8ec04bd46" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:52:19.855273 1062 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "d1866d6dcb0349bba5db86163375cada" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:19.855544 1055 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "1df0d944717d487a98ea83a8ec04bd46" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:52:19.856603 523 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 47348ffa21574e4084f0b725a78e2a70 consensus state has no leader
I20251029 02:52:19.857753 526 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 2d33bc6dafb64d948cd62feb50193b26
I20251029 02:52:19.858501 524 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 09567e75ea1140e5ba506a49fbb1d77e
I20251029 02:52:19.859704 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet b22929678f734288829c53fffadfa8a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:19 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:19.859714 887 tablet_service.cc:1552] Processing DeleteTablet for tablet 2d33bc6dafb64d948cd62feb50193b26 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:19 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:19.859987 887 tablet_service.cc:1552] Processing DeleteTablet for tablet 09567e75ea1140e5ba506a49fbb1d77e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:19 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:19.860755 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet 2d33bc6dafb64d948cd62feb50193b26 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:19 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:19.861317 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet 09567e75ea1140e5ba506a49fbb1d77e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:19 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:19.862372 887 tablet_service.cc:1505] Processing CreateTablet for tablet 8ccca343c67942c7a97e3c19bcfb5993 (DEFAULT_TABLE table=default.Table_fd903b7b9822450fa41766f9f091d82c [id=c5f98f9d0f0141e789d6d1075fc2ee37]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:19.862902 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ccca343c67942c7a97e3c19bcfb5993. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:19.863061 888 tablet_service.cc:1552] Processing DeleteTablet for tablet b22929678f734288829c53fffadfa8a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:19 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:19.866760 744 tablet_service.cc:1505] Processing CreateTablet for tablet fed40a07c28549db90af66acdab11521 (DEFAULT_TABLE table=default.Table_e464d8b2defc45f1aefefab339650ac0 [id=baf82b1773ef426a8ea379cf59987dc8]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 02:52:19.869287 712 leader_election.cc:343] T 1df0d944717d487a98ea83a8ec04bd46 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
W20251029 02:52:19.869560 847 consensus_peers.cc:597] T 946f65014d97425aab3851009b742807 P c756528642a543eeacd93cef3e335a7d -> Peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Couldn't send request to peer b495a49349324f7d81908f2da700f0d9. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20251029 02:52:19.869616 712 leader_election.cc:343] T d1866d6dcb0349bba5db86163375cada P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:19.869709 712 leader_election.cc:304] T d1866d6dcb0349bba5db86163375cada P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
W20251029 02:52:19.870085 712 leader_election.cc:343] T 1df0d944717d487a98ea83a8ec04bd46 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:19.870457 2985 raft_consensus.cc:2749] T d1866d6dcb0349bba5db86163375cada P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:19.872009 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_858074c10a52451095df3d1be46f6b57"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:19.869263 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fed40a07c28549db90af66acdab11521. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:19.872581 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_858074c10a52451095df3d1be46f6b57 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:19.874711 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_07b504e33dfe4756a8c11fa7fc043f25"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:19.875214 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_07b504e33dfe4756a8c11fa7fc043f25 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:19.875342 524 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_724cce6ff005499cae9f0077c069e82a" } new_table_name: "default.Table_b90a97409b234cbd97e863c7235041ed" modify_external_catalogs: true
I20251029 02:52:19.876061 888 tablet_service.cc:1505] Processing CreateTablet for tablet 086f01b797c84717b4c10c84581b7a35 (DEFAULT_TABLE table=default.Table_cc7a37c88eb54521b208967338567b54 [id=4f5a7772423c4332947b1128ff75dfb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:19.876427 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_9149348f34044c90bfbf0255c90bd9cf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:19.876947 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9149348f34044c90bfbf0255c90bd9cf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:19.880816 1174 ts_tablet_manager.cc:1929] T 217144957930495aa3e9dfcfde268e71 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:19.881740 1174 log.cc:1199] T 217144957930495aa3e9dfcfde268e71 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/217144957930495aa3e9dfcfde268e71
I20251029 02:52:19.882398 1174 ts_tablet_manager.cc:1950] T 217144957930495aa3e9dfcfde268e71 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:19.882632 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_ba665c4339d94f1d9d81f5e31b5186b7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:19.883206 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ba665c4339d94f1d9d81f5e31b5186b7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:19.883738 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 1027) took 7643 ms. Trace:
I20251029 02:52:19.881194 1167 tablet_bootstrap.cc:492] T 023a4094b424464c8f49d6eb9cc15da4 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:19.885268 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_e5a909b41e1e4ef19422115834d84b7f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:19.885841 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e5a909b41e1e4ef19422115834d84b7f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:19.885876 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ccca343c67942c7a97e3c19bcfb5993. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:19.886268 1031 tablet_service.cc:1505] Processing CreateTablet for tablet 086f01b797c84717b4c10c84581b7a35 (DEFAULT_TABLE table=default.Table_cc7a37c88eb54521b208967338567b54 [id=4f5a7772423c4332947b1128ff75dfb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:19.887341 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 086f01b797c84717b4c10c84581b7a35. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:19.887614 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_2e5be05f9cb045709f53fb3f77a06964"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:19.888039 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2e5be05f9cb045709f53fb3f77a06964 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:19.889153 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5fe79779cb14993a02497d746085ac9. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:19.884383 2747 raft_consensus.cc:493] T 506f8c112a3b43e1a3d6bbafb8cb81e8 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:19.891268 2747 raft_consensus.cc:515] T 506f8c112a3b43e1a3d6bbafb8cb81e8 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:19.891850 2747 leader_election.cc:290] T 506f8c112a3b43e1a3d6bbafb8cb81e8 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:19.892335 1167 ts_tablet_manager.cc:1403] T 023a4094b424464c8f49d6eb9cc15da4 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.045s user 0.000s sys 0.003s
I20251029 02:52:19.893056 1167 raft_consensus.cc:359] T 023a4094b424464c8f49d6eb9cc15da4 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:19.895044 2989 raft_consensus.cc:493] T c45ffc304e73490387ce800b702b5881 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:19.899703 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e257e5ffc4314f61b2df087d59139129. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:19.901068 1055 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "08371b3daa184a0e9f4eb3d313b52c56" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:19.901460 1062 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "08371b3daa184a0e9f4eb3d313b52c56" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:52:19.902307 1058 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "506f8c112a3b43e1a3d6bbafb8cb81e8" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:19.904983 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fed40a07c28549db90af66acdab11521. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:19.894810 2747 raft_consensus.cc:493] T 82289e0f823c4d43bfd8c4e878572b2b P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0cfd4bee16f84b7c8b0ff3ce14db20d5)
I20251029 02:52:19.905308 2747 raft_consensus.cc:515] T 82289e0f823c4d43bfd8c4e878572b2b P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:19.905900 2747 leader_election.cc:290] T 82289e0f823c4d43bfd8c4e878572b2b P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
W20251029 02:52:19.903116 849 consensus_peers.cc:597] T d29cc2a0c8b84e3ebadc7f571d666847 P c756528642a543eeacd93cef3e335a7d -> Peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Couldn't send request to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d29cc2a0c8b84e3ebadc7f571d666847. This is attempt 1: this message will repeat every 5th retry.
W20251029 02:52:19.906387 849 leader_election.cc:343] T afb19ba39399410bb90b6225d7b5b5b1 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Not found: Tablet not found: afb19ba39399410bb90b6225d7b5b5b1
I20251029 02:52:19.906486 849 leader_election.cc:304] T afb19ba39399410bb90b6225d7b5b5b1 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9
I20251029 02:52:19.906872 2747 raft_consensus.cc:2749] T afb19ba39399410bb90b6225d7b5b5b1 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20251029 02:52:19.907029 849 leader_election.cc:343] T 08371b3daa184a0e9f4eb3d313b52c56 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
W20251029 02:52:19.907305 849 leader_election.cc:343] T 08371b3daa184a0e9f4eb3d313b52c56 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
W20251029 02:52:19.907609 849 leader_election.cc:343] T 506f8c112a3b43e1a3d6bbafb8cb81e8 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
W20251029 02:52:19.907743 849 consensus_peers.cc:597] T 23b9b2ea814645eba63e5f6b12cfb966 P c756528642a543eeacd93cef3e335a7d -> Peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Couldn't send request to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 23b9b2ea814645eba63e5f6b12cfb966. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:19.908807 887 tablet_service.cc:1505] Processing CreateTablet for tablet a3fe29bae27d4147809df3833f9807c5 (DEFAULT_TABLE table=default.Table_cd0e071b233b4d45b36009940c6a0084 [id=126c78342d1746719f4a177e84422465]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 02:52:19.908975 849 leader_election.cc:343] T 82289e0f823c4d43bfd8c4e878572b2b P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Not found: Tablet not found: 82289e0f823c4d43bfd8c4e878572b2b
I20251029 02:52:19.909042 1062 raft_consensus.cc:3060] T 628a7d0419cb434f8171faf500d585d8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:19.909349 1167 raft_consensus.cc:385] T 023a4094b424464c8f49d6eb9cc15da4 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:19.909425 1167 raft_consensus.cc:740] T 023a4094b424464c8f49d6eb9cc15da4 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:19.910531 1028 tablet_service.cc:1505] Processing CreateTablet for tablet a3fe29bae27d4147809df3833f9807c5 (DEFAULT_TABLE table=default.Table_cd0e071b233b4d45b36009940c6a0084 [id=126c78342d1746719f4a177e84422465]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:19.910866 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3fe29bae27d4147809df3833f9807c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:19.910941 900 tablet_service.cc:1505] Processing CreateTablet for tablet 6538fe3658bf45dcb4b0a566989a208c (DEFAULT_TABLE table=default.Table_917b4493547e4d55be883cc4a51d45a1 [id=194da79b21534801b99e6c1fa87d86ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:19.911638 1167 consensus_queue.cc:260] T 023a4094b424464c8f49d6eb9cc15da4 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:19.913638 1167 ts_tablet_manager.cc:1434] T 023a4094b424464c8f49d6eb9cc15da4 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.021s user 0.000s sys 0.001s
I20251029 02:52:19.913893 1167 tablet_bootstrap.cc:492] T 2b278db5b2b04344b864797055a5f665 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:19.915107 1027 tablet_service.cc:1505] Processing CreateTablet for tablet 6538fe3658bf45dcb4b0a566989a208c (DEFAULT_TABLE table=default.Table_917b4493547e4d55be883cc4a51d45a1 [id=194da79b21534801b99e6c1fa87d86ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:19.915459 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6538fe3658bf45dcb4b0a566989a208c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:19.884799 1174 rpcz_store.cc:276] 1029 02:52:12.240109 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:12.240179 (+ 70us) service_pool.cc:225] Handling call
1029 02:52:19.808246 (+7568067us) ts_tablet_manager.cc:1151] Deleting tablet 217144957930495aa3e9dfcfde268e71
1029 02:52:19.808262 (+ 16us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:19.821584 (+ 13322us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:19.880697 (+ 59113us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:19.883718 (+ 3021us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7567684}
I20251029 02:52:19.916451 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 217144957930495aa3e9dfcfde268e71 (table default.Table_15cde6ff49784e5082de7519ce61c81d [id=3241aaa6ea9148b7a101833462ab3eeb]) successfully deleted
I20251029 02:52:19.917754 1174 tablet_replica.cc:333] T 44ae2b07edca44afbe1b1fb3a5b1115d P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:19.919137 1174 raft_consensus.cc:2243] T 44ae2b07edca44afbe1b1fb3a5b1115d P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:19.920792 1174 raft_consensus.cc:2272] T 44ae2b07edca44afbe1b1fb3a5b1115d P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:19.938901 1052 raft_consensus.cc:2468] T 5815626b05204b578cc4a110f246bf10 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b495a49349324f7d81908f2da700f0d9 in term 1.
I20251029 02:52:19.940478 519 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 78c7cbd6bde646c69eae1698d1b0f7a9
I20251029 02:52:19.941190 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_c9cbdca76fe849b6b6afc3080c6f6997"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:19.941787 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c9cbdca76fe849b6b6afc3080c6f6997 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:19.942693 522 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet d22b01dc872b4ebb8f65b586b29eea00
I20251029 02:52:19.946638 895 tablet_service.cc:1552] Processing DeleteTablet for tablet 78c7cbd6bde646c69eae1698d1b0f7a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:19 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:19.947419 891 tablet_service.cc:1552] Processing DeleteTablet for tablet d22b01dc872b4ebb8f65b586b29eea00 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:19 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:19.952505 2655 raft_consensus.cc:493] T d31e548feae24b739087c1e211d514f9 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:19.952647 2655 raft_consensus.cc:515] T d31e548feae24b739087c1e211d514f9 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:19.953267 2655 leader_election.cc:290] T d31e548feae24b739087c1e211d514f9 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
W20251029 02:52:19.955945 849 leader_election.cc:343] T d31e548feae24b739087c1e211d514f9 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Not found: Tablet not found: d31e548feae24b739087c1e211d514f9
W20251029 02:52:19.956729 524 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5171c8aa52ac497d9858bb6d0ea8dfee consensus state has no leader
I20251029 02:52:19.957903 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet 78c7cbd6bde646c69eae1698d1b0f7a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:19 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:19.958393 1174 ts_tablet_manager.cc:1916] T 44ae2b07edca44afbe1b1fb3a5b1115d P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:19.895184 2989 raft_consensus.cc:515] T c45ffc304e73490387ce800b702b5881 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:19.971901 1042 tablet_service.cc:1552] Processing DeleteTablet for tablet d22b01dc872b4ebb8f65b586b29eea00 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:19 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:19.972815 2989 leader_election.cc:290] T c45ffc304e73490387ce800b702b5881 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:52:19.973819 740 tablet_service.cc:1552] Processing DeleteTablet for tablet b22929678f734288829c53fffadfa8a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:19 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:19.974109 2655 raft_consensus.cc:493] T 59eee86f249e4d6a9d09d30276dcd659 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0cfd4bee16f84b7c8b0ff3ce14db20d5)
I20251029 02:52:19.974248 2655 raft_consensus.cc:515] T 59eee86f249e4d6a9d09d30276dcd659 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:19.974306 1052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c45ffc304e73490387ce800b702b5881" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:52:19.974820 712 leader_election.cc:343] T c45ffc304e73490387ce800b702b5881 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:19.974984 2655 leader_election.cc:290] T 59eee86f249e4d6a9d09d30276dcd659 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:19.975764 738 tablet_service.cc:1505] Processing CreateTablet for tablet c5fe79779cb14993a02497d746085ac9 (DEFAULT_TABLE table=default.Table_1400dc2050d64e82a07e3fcd8018c0d2 [id=532c91430c034154aeaa767a5a1c7ff6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:19.976158 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c5fe79779cb14993a02497d746085ac9. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:19.977691 849 leader_election.cc:343] T 59eee86f249e4d6a9d09d30276dcd659 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Not found: Tablet not found: 59eee86f249e4d6a9d09d30276dcd659
I20251029 02:52:19.978500 767 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "506f8c112a3b43e1a3d6bbafb8cb81e8" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:19.978719 767 raft_consensus.cc:2468] T 506f8c112a3b43e1a3d6bbafb8cb81e8 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
W20251029 02:52:19.979791 847 leader_election.cc:343] T 82289e0f823c4d43bfd8c4e878572b2b P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 82289e0f823c4d43bfd8c4e878572b2b
I20251029 02:52:19.979914 847 leader_election.cc:304] T 82289e0f823c4d43bfd8c4e878572b2b P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9
I20251029 02:52:19.980518 847 leader_election.cc:304] T 506f8c112a3b43e1a3d6bbafb8cb81e8 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5
I20251029 02:52:19.980808 2426 raft_consensus.cc:2749] T 82289e0f823c4d43bfd8c4e878572b2b P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251029 02:52:19.981009 2426 raft_consensus.cc:2804] T 506f8c112a3b43e1a3d6bbafb8cb81e8 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:19.981081 2426 raft_consensus.cc:493] T 506f8c112a3b43e1a3d6bbafb8cb81e8 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:19.981160 2426 raft_consensus.cc:3060] T 506f8c112a3b43e1a3d6bbafb8cb81e8 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
W20251029 02:52:19.981642 847 leader_election.cc:343] T d31e548feae24b739087c1e211d514f9 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: d31e548feae24b739087c1e211d514f9
I20251029 02:52:19.981778 847 leader_election.cc:304] T d31e548feae24b739087c1e211d514f9 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9
I20251029 02:52:19.982316 2649 raft_consensus.cc:2749] T d31e548feae24b739087c1e211d514f9 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:19.983222 740 tablet_service.cc:1552] Processing DeleteTablet for tablet 2d33bc6dafb64d948cd62feb50193b26 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:19 UTC) from {username='slave'} at 127.0.0.1:44580
W20251029 02:52:19.983464 847 leader_election.cc:343] T 59eee86f249e4d6a9d09d30276dcd659 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 59eee86f249e4d6a9d09d30276dcd659
I20251029 02:52:19.983568 847 leader_election.cc:304] T 59eee86f249e4d6a9d09d30276dcd659 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9
I20251029 02:52:19.984377 742 tablet_service.cc:1505] Processing CreateTablet for tablet 8ccca343c67942c7a97e3c19bcfb5993 (DEFAULT_TABLE table=default.Table_fd903b7b9822450fa41766f9f091d82c [id=c5f98f9d0f0141e789d6d1075fc2ee37]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:19.984563 749 tablet_service.cc:1552] Processing DeleteTablet for tablet 78c7cbd6bde646c69eae1698d1b0f7a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:19 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:19.984760 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ccca343c67942c7a97e3c19bcfb5993. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:19.985056 2656 raft_consensus.cc:2749] T 59eee86f249e4d6a9d09d30276dcd659 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251029 02:52:19.985085 750 tablet_service.cc:1505] Processing CreateTablet for tablet 086f01b797c84717b4c10c84581b7a35 (DEFAULT_TABLE table=default.Table_cc7a37c88eb54521b208967338567b54 [id=4f5a7772423c4332947b1128ff75dfb7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:19.985308 2426 raft_consensus.cc:515] T 506f8c112a3b43e1a3d6bbafb8cb81e8 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:19.985438 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 086f01b797c84717b4c10c84581b7a35. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:19.985889 2426 leader_election.cc:290] T 506f8c112a3b43e1a3d6bbafb8cb81e8 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:19.986963 1052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "506f8c112a3b43e1a3d6bbafb8cb81e8" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
W20251029 02:52:19.987588 849 leader_election.cc:343] T 506f8c112a3b43e1a3d6bbafb8cb81e8 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:19.990836 748 tablet_service.cc:1505] Processing CreateTablet for tablet a3fe29bae27d4147809df3833f9807c5 (DEFAULT_TABLE table=default.Table_cd0e071b233b4d45b36009940c6a0084 [id=126c78342d1746719f4a177e84422465]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:19.991205 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3fe29bae27d4147809df3833f9807c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:19.991237 1172 tablet_bootstrap.cc:492] T 877e278deb4a4b01ac66b06be0890baf P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:19.991349 1172 ts_tablet_manager.cc:1403] T 877e278deb4a4b01ac66b06be0890baf P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.563s user 0.009s sys 0.007s
I20251029 02:52:19.992038 1172 raft_consensus.cc:359] T 877e278deb4a4b01ac66b06be0890baf P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:19.992184 1172 raft_consensus.cc:385] T 877e278deb4a4b01ac66b06be0890baf P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:19.992228 1172 raft_consensus.cc:740] T 877e278deb4a4b01ac66b06be0890baf P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:19.992448 1172 consensus_queue.cc:260] T 877e278deb4a4b01ac66b06be0890baf P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:19.992973 1172 ts_tablet_manager.cc:1434] T 877e278deb4a4b01ac66b06be0890baf P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 02:52:19.993232 1172 tablet_bootstrap.cc:492] T 35d9b8dfd61e4d8db441427a49fb9dca P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:19.994043 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 086f01b797c84717b4c10c84581b7a35. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:19.994264 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3fe29bae27d4147809df3833f9807c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:19.994266 891 tablet_service.cc:1505] Processing CreateTablet for tablet fb63ec8e38fe4cbb8df2065b760c9112 (DEFAULT_TABLE table=default.Table_daad6fda420f4659bc87d5991d60cf2f [id=9cd9b155c5ad467c8da68c8c03a68206]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:19.994719 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6538fe3658bf45dcb4b0a566989a208c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:19.994755 739 tablet_service.cc:1505] Processing CreateTablet for tablet 6538fe3658bf45dcb4b0a566989a208c (DEFAULT_TABLE table=default.Table_917b4493547e4d55be883cc4a51d45a1 [id=194da79b21534801b99e6c1fa87d86ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:19.994891 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb63ec8e38fe4cbb8df2065b760c9112. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:19.995096 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6538fe3658bf45dcb4b0a566989a208c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:19.997094 754 tablet_service.cc:1505] Processing CreateTablet for tablet fb63ec8e38fe4cbb8df2065b760c9112 (DEFAULT_TABLE table=default.Table_daad6fda420f4659bc87d5991d60cf2f [id=9cd9b155c5ad467c8da68c8c03a68206]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:19.997392 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb63ec8e38fe4cbb8df2065b760c9112. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:19.998796 756 tablet_service.cc:1552] Processing DeleteTablet for tablet d22b01dc872b4ebb8f65b586b29eea00 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:19 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:19.999974 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c45ffc304e73490387ce800b702b5881" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:20.000617 740 tablet_service.cc:1552] Processing DeleteTablet for tablet 09567e75ea1140e5ba506a49fbb1d77e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:19 UTC) from {username='slave'} at 127.0.0.1:44580
W20251029 02:52:20.001463 711 leader_election.cc:343] T c45ffc304e73490387ce800b702b5881 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:20.001569 711 leader_election.cc:304] T c45ffc304e73490387ce800b702b5881 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:20.001912 2989 raft_consensus.cc:2749] T c45ffc304e73490387ce800b702b5881 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:20.005695 1167 tablet_bootstrap.cc:654] T 2b278db5b2b04344b864797055a5f665 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:20.008934 1174 ts_tablet_manager.cc:1929] T 44ae2b07edca44afbe1b1fb3a5b1115d P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:20.009176 1174 log.cc:1199] T 44ae2b07edca44afbe1b1fb3a5b1115d P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/44ae2b07edca44afbe1b1fb3a5b1115d
I20251029 02:52:20.009642 1174 ts_tablet_manager.cc:1950] T 44ae2b07edca44afbe1b1fb3a5b1115d P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:20.009615 1042 tablet_service.cc:1505] Processing CreateTablet for tablet fb63ec8e38fe4cbb8df2065b760c9112 (DEFAULT_TABLE table=default.Table_daad6fda420f4659bc87d5991d60cf2f [id=9cd9b155c5ad467c8da68c8c03a68206]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:20.010083 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb63ec8e38fe4cbb8df2065b760c9112. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:20.010198 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 1026) took 7770 ms. Trace:
I20251029 02:52:20.010293 1174 rpcz_store.cc:276] 1029 02:52:12.239975 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:12.254937 (+ 14962us) service_pool.cc:225] Handling call
1029 02:52:19.917450 (+7662513us) ts_tablet_manager.cc:1151] Deleting tablet 44ae2b07edca44afbe1b1fb3a5b1115d
1029 02:52:19.917515 (+ 65us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:20.003239 (+ 85724us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:20.008839 (+ 5600us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:20.010181 (+ 1342us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7662094}
I20251029 02:52:20.010713 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 44ae2b07edca44afbe1b1fb3a5b1115d (table default.Table_c2719a3251c84739a6681f80f1d8f94d [id=a746921b2aba4cd6af22a293991968e8]) successfully deleted
I20251029 02:52:20.011029 740 tablet_service.cc:1505] Processing CreateTablet for tablet 2d4b8fc3e0d24fd693d5f8b5a2231f79 (DEFAULT_TABLE table=default.Table_16362cebff0c4d19a3f59ae8529be683 [id=a21cca4d156643ed9bfed8be498bce2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:20.011197 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 1028) took 7685 ms. Trace:
I20251029 02:52:20.011277 1174 rpcz_store.cc:276] 1029 02:52:12.325767 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:12.325842 (+ 75us) service_pool.cc:225] Handling call
1029 02:52:20.011052 (+7685210us) ts_tablet_manager.cc:1151] Deleting tablet fd1bf6b0f45349d0981f9ddd89269489
1029 02:52:20.011067 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:20.011177 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7684942}
I20251029 02:52:20.011385 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d4b8fc3e0d24fd693d5f8b5a2231f79. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:20.011684 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet fd1bf6b0f45349d0981f9ddd89269489 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fd1bf6b0f45349d0981f9ddd89269489 already in progress: creating tablet
I20251029 02:52:20.012099 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 1029) took 7683 ms. Trace:
I20251029 02:52:20.012267 1174 rpcz_store.cc:276] 1029 02:52:12.329103 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:12.329154 (+ 51us) service_pool.cc:225] Handling call
1029 02:52:20.011960 (+7682806us) ts_tablet_manager.cc:1151] Deleting tablet 1150d2210a234f1aa554f486cdcabc4c
1029 02:52:20.011971 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:20.012080 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7682586}
W20251029 02:52:20.012707 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 1150d2210a234f1aa554f486cdcabc4c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1150d2210a234f1aa554f486cdcabc4c already in progress: creating tablet
I20251029 02:52:20.013144 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 1030) took 7642 ms. Trace:
I20251029 02:52:20.013254 1174 rpcz_store.cc:276] 1029 02:52:12.370300 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:12.372633 (+ 2333us) service_pool.cc:225] Handling call
1029 02:52:20.013035 (+7640402us) ts_tablet_manager.cc:1151] Deleting tablet 41cfa2d722034618a05de717b28a778d
1029 02:52:20.013049 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:20.013128 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7640211}
I20251029 02:52:20.013865 1034 tablet_service.cc:1505] Processing CreateTablet for tablet 2d4b8fc3e0d24fd693d5f8b5a2231f79 (DEFAULT_TABLE table=default.Table_16362cebff0c4d19a3f59ae8529be683 [id=a21cca4d156643ed9bfed8be498bce2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:20.014261 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d4b8fc3e0d24fd693d5f8b5a2231f79. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:20.014420 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 41cfa2d722034618a05de717b28a778d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 41cfa2d722034618a05de717b28a778d already in progress: creating tablet
I20251029 02:52:20.014900 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 1031) took 7633 ms. Trace:
I20251029 02:52:20.013375 1167 tablet_bootstrap.cc:492] T 2b278db5b2b04344b864797055a5f665 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:20.015494 1167 ts_tablet_manager.cc:1403] T 2b278db5b2b04344b864797055a5f665 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.102s user 0.004s sys 0.008s
I20251029 02:52:20.016324 1167 raft_consensus.cc:359] T 2b278db5b2b04344b864797055a5f665 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:20.017491 1167 raft_consensus.cc:385] T 2b278db5b2b04344b864797055a5f665 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:20.017800 1167 raft_consensus.cc:740] T 2b278db5b2b04344b864797055a5f665 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:20.018179 1167 consensus_queue.cc:260] T 2b278db5b2b04344b864797055a5f665 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:20.019320 1167 ts_tablet_manager.cc:1434] T 2b278db5b2b04344b864797055a5f665 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20251029 02:52:20.020654 1167 tablet_bootstrap.cc:492] T e3000183230e4350b47f5085952f2db6 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:20.025183 1167 tablet_bootstrap.cc:654] T e3000183230e4350b47f5085952f2db6 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:20.015338 1174 rpcz_store.cc:276] 1029 02:52:12.381403 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:12.381451 (+ 48us) service_pool.cc:225] Handling call
1029 02:52:20.014776 (+7633325us) ts_tablet_manager.cc:1151] Deleting tablet 99d60ef5325e45059026596a756448de
1029 02:52:20.014788 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:20.014879 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7633037}
I20251029 02:52:20.029174 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 1032) took 7594 ms. Trace:
I20251029 02:52:20.029490 1174 rpcz_store.cc:276] 1029 02:52:12.434923 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:12.448847 (+ 13924us) service_pool.cc:225] Handling call
1029 02:52:20.029032 (+7580185us) ts_tablet_manager.cc:1151] Deleting tablet 49aaa2d8986a44df979edc1c42612189
1029 02:52:20.029043 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:20.029152 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7579963}
W20251029 02:52:20.029964 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 99d60ef5325e45059026596a756448de because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 99d60ef5325e45059026596a756448de already in progress: creating tablet
W20251029 02:52:20.030292 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 49aaa2d8986a44df979edc1c42612189 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 49aaa2d8986a44df979edc1c42612189 already in progress: creating tablet
I20251029 02:52:20.030436 1174 tablet_replica.cc:333] T f281191e7c6143f983f11d87f1c9b64e P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:20.030948 1174 raft_consensus.cc:2243] T f281191e7c6143f983f11d87f1c9b64e P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:20.031774 1174 raft_consensus.cc:2272] T f281191e7c6143f983f11d87f1c9b64e P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:20.037148 1166 tablet_bootstrap.cc:492] T 055184c641ba45a588bca611fb24d130 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:20.037482 1166 ts_tablet_manager.cc:1403] T 055184c641ba45a588bca611fb24d130 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.209s user 0.017s sys 0.020s
I20251029 02:52:20.033275 887 tablet_service.cc:1505] Processing CreateTablet for tablet 2d4b8fc3e0d24fd693d5f8b5a2231f79 (DEFAULT_TABLE table=default.Table_16362cebff0c4d19a3f59ae8529be683 [id=a21cca4d156643ed9bfed8be498bce2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:20.039839 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d4b8fc3e0d24fd693d5f8b5a2231f79. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:20.036675 891 tablet_service.cc:1552] Processing DeleteTablet for tablet 1df0d944717d487a98ea83a8ec04bd46 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:01 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:20.043533 526 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_6c87fb0ffa1c4d818b953a2b0dace9a3" } new_table_name: "default.Table_c981ab0884004673ad534a4e6bf22cc0" modify_external_catalogs: true
I20251029 02:52:20.045002 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_d9f6f90a7e80471eab4a475a01b712d3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:20.045586 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d9f6f90a7e80471eab4a475a01b712d3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:20.049232 524 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_aef83d52090548f7aeca8fa0765f1691" } new_table_name: "default.Table_c7dbb79386884cc49914ee437d974351" modify_external_catalogs: true
I20251029 02:52:20.058446 517 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet f4729ecd182042c2a10f67ca25224101 (table default.Table_973ef856041949598a23da3dab8b40db [id=1dd2d5488bc647c8b82c53886583b28d]). Expected version 4 got 0
I20251029 02:52:20.061229 521 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_3c3bcfdfd5914e7b892a8af9aaaac175" } modify_external_catalogs: true
I20251029 02:52:20.061177 525 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_e2177489f8c84dfa9aa73bd309e20652" } new_table_name: "default.Table_41bc250962b040ddbbdb3b1fc9bd74f4" modify_external_catalogs: true
I20251029 02:52:20.061373 521 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_3c3bcfdfd5914e7b892a8af9aaaac175" } modify_external_catalogs: true
I20251029 02:52:20.061722 1174 ts_tablet_manager.cc:1916] T f281191e7c6143f983f11d87f1c9b64e P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:20.062682 518 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_6f38863737b34dfa90488cad665a8628" } new_table_name: "default.Table_41530f589e9749fb8d7f21460e63db86" modify_external_catalogs: true
I20251029 02:52:20.064174 522 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_a587f8486e2a4542a9e7cf80c0306f23" } modify_external_catalogs: true
I20251029 02:52:20.064339 522 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_a587f8486e2a4542a9e7cf80c0306f23" } modify_external_catalogs: true
I20251029 02:52:20.064383 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "506f8c112a3b43e1a3d6bbafb8cb81e8" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:20.064553 772 raft_consensus.cc:3060] T 506f8c112a3b43e1a3d6bbafb8cb81e8 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:20.066977 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet 628a7d0419cb434f8171faf500d585d8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:02 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:20.067495 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet 5815626b05204b578cc4a110f246bf10 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:08 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:20.067931 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_a1f3bae2359c48cc8689b2d818943796"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:20.068581 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a1f3bae2359c48cc8689b2d818943796 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:20.069427 1166 raft_consensus.cc:359] T 055184c641ba45a588bca611fb24d130 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:20.069639 1166 raft_consensus.cc:385] T 055184c641ba45a588bca611fb24d130 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:20.069725 1166 raft_consensus.cc:740] T 055184c641ba45a588bca611fb24d130 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:20.069990 1166 consensus_queue.cc:260] T 055184c641ba45a588bca611fb24d130 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:20.070415 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_0f2ddac3ba294c829d245f25d343f3be"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:20.070870 1166 ts_tablet_manager.cc:1434] T 055184c641ba45a588bca611fb24d130 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.033s user 0.001s sys 0.001s
W20251029 02:52:20.071034 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0f2ddac3ba294c829d245f25d343f3be in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:20.071224 1166 tablet_bootstrap.cc:492] T 24f07b1efac64b099b0b5ed467463749 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:20.072108 772 raft_consensus.cc:2468] T 506f8c112a3b43e1a3d6bbafb8cb81e8 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:20.072660 1166 tablet_bootstrap.cc:654] T 24f07b1efac64b099b0b5ed467463749 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:20.073598 847 leader_election.cc:304] T 506f8c112a3b43e1a3d6bbafb8cb81e8 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5
I20251029 02:52:20.074095 2656 raft_consensus.cc:2804] T 506f8c112a3b43e1a3d6bbafb8cb81e8 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:20.074219 2656 raft_consensus.cc:697] T 506f8c112a3b43e1a3d6bbafb8cb81e8 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:52:20.074455 2656 consensus_queue.cc:237] T 506f8c112a3b43e1a3d6bbafb8cb81e8 P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:20.079764 1174 ts_tablet_manager.cc:1929] T f281191e7c6143f983f11d87f1c9b64e P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:20.081158 1174 log.cc:1199] T f281191e7c6143f983f11d87f1c9b64e P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/f281191e7c6143f983f11d87f1c9b64e
I20251029 02:52:20.082406 1174 ts_tablet_manager.cc:1950] T f281191e7c6143f983f11d87f1c9b64e P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:20.083766 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 1033) took 7477 ms. Trace:
I20251029 02:52:20.085178 1174 rpcz_store.cc:276] 1029 02:52:12.606284 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:12.606337 (+ 53us) service_pool.cc:225] Handling call
1029 02:52:20.030386 (+7424049us) ts_tablet_manager.cc:1151] Deleting tablet f281191e7c6143f983f11d87f1c9b64e
1029 02:52:20.030398 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:20.068334 (+ 37936us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:20.079628 (+ 11294us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:20.083741 (+ 4113us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":20352,"tablet-delete.queue_time_us":7423835}
I20251029 02:52:20.087288 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 1035) took 7438 ms. Trace:
I20251029 02:52:20.087633 1174 rpcz_store.cc:276] 1029 02:52:12.649122 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:12.655194 (+ 6072us) service_pool.cc:225] Handling call
1029 02:52:20.086401 (+7431207us) ts_tablet_manager.cc:1151] Deleting tablet 9280503662894544a1f227a62a9b8a01
1029 02:52:20.086416 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:20.087269 (+ 853us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7430969}
W20251029 02:52:20.088384 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 9280503662894544a1f227a62a9b8a01 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9280503662894544a1f227a62a9b8a01 already in progress: creating tablet
W20251029 02:52:20.088819 525 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2c28131d740c411dbb07898d222630ff consensus state has no leader
I20251029 02:52:20.088872 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 1036) took 7352 ms. Trace:
I20251029 02:52:20.089416 1174 rpcz_store.cc:276] 1029 02:52:12.736224 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:12.736330 (+ 106us) service_pool.cc:225] Handling call
1029 02:52:20.088755 (+7352425us) ts_tablet_manager.cc:1151] Deleting tablet fe5aff0f61a44b91bfb37b42457a91f0
1029 02:52:20.088765 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:20.088854 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7352228}
W20251029 02:52:20.090232 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet fe5aff0f61a44b91bfb37b42457a91f0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fe5aff0f61a44b91bfb37b42457a91f0 already in progress: creating tablet
W20251029 02:52:20.090312 526 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d1bbe01d6ffa4074aad62f30ed3685a0 consensus state has no leader
I20251029 02:52:20.090703 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 1038) took 7087 ms. Trace:
W20251029 02:52:20.091077 524 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8471db993eab4fcb9e6f317d48ad8f80 consensus state has no leader
I20251029 02:52:20.091894 1174 rpcz_store.cc:276] 1029 02:52:13.003471 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:13.006589 (+ 3118us) service_pool.cc:225] Handling call
1029 02:52:20.090592 (+7084003us) ts_tablet_manager.cc:1151] Deleting tablet ca71c4638a6b419b876e22af7e6cf00a
1029 02:52:20.090603 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:20.090688 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7083749}
W20251029 02:52:20.092705 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet ca71c4638a6b419b876e22af7e6cf00a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ca71c4638a6b419b876e22af7e6cf00a already in progress: creating tablet
I20251029 02:52:20.093097 1174 tablet_replica.cc:333] T afb19ba39399410bb90b6225d7b5b5b1 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:20.093175 1172 tablet_bootstrap.cc:654] T 35d9b8dfd61e4d8db441427a49fb9dca P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:20.093580 1174 raft_consensus.cc:2243] T afb19ba39399410bb90b6225d7b5b5b1 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:20.094822 1174 raft_consensus.cc:2272] T afb19ba39399410bb90b6225d7b5b5b1 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:20.095796 1174 ts_tablet_manager.cc:1916] T afb19ba39399410bb90b6225d7b5b5b1 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:20.104141 1028 tablet_service.cc:1505] Processing CreateTablet for tablet 0c94bd8cf035428280cc7449dcc036b7 (DEFAULT_TABLE table=default.Table_b3a7b65a549244a6af331747d706ecf3 [id=c21ccfd7b1964426938b032b019df906]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:20.107612 887 tablet_service.cc:1552] Processing DeleteTablet for tablet 506f8c112a3b43e1a3d6bbafb8cb81e8 with delete_type TABLET_DATA_DELETED (replaced by tablet ef2e9dd8f90a47389d9062d1f99a1da1) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:20.112447 887 tablet_service.cc:1552] Processing DeleteTablet for tablet d31e548feae24b739087c1e211d514f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:00 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:20.115221 887 tablet_service.cc:1552] Processing DeleteTablet for tablet 7fb5b908db77453da5cfb32e2c955f41 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:03 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:20.116710 524 master_service.cc:954] ReplaceTablet: received request to replace tablet a27496be3431446da9cb83452a3893e3 from {username='slave'} at 127.0.0.1:35314
I20251029 02:52:20.118412 900 tablet_service.cc:1552] Processing DeleteTablet for tablet cc802bda2f6a4e439b605f2cafa37967 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:57 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:20.120095 1174 ts_tablet_manager.cc:1929] T afb19ba39399410bb90b6225d7b5b5b1 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:20.123435 739 tablet_service.cc:1505] Processing CreateTablet for tablet 0c94bd8cf035428280cc7449dcc036b7 (DEFAULT_TABLE table=default.Table_b3a7b65a549244a6af331747d706ecf3 [id=c21ccfd7b1964426938b032b019df906]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:20.123955 521 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 8c972c88f64e486bb2b75306549bb23f
I20251029 02:52:20.124146 1174 log.cc:1199] T afb19ba39399410bb90b6225d7b5b5b1 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/afb19ba39399410bb90b6225d7b5b5b1
I20251029 02:52:20.124171 522 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet f72dae5a0e604eb384c1f5f06fe680bb
I20251029 02:52:20.124204 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c94bd8cf035428280cc7449dcc036b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:20.124768 1174 ts_tablet_manager.cc:1950] T afb19ba39399410bb90b6225d7b5b5b1 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
W20251029 02:52:20.124784 518 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 15137fc1c70347b381ffdfe5653a5f22 consensus state has no leader
I20251029 02:52:20.125516 740 tablet_service.cc:1552] Processing DeleteTablet for tablet 8c972c88f64e486bb2b75306549bb23f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:20 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:20.125694 748 tablet_service.cc:1552] Processing DeleteTablet for tablet f72dae5a0e604eb384c1f5f06fe680bb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:20 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:20.126119 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 1039) took 7019 ms. Trace:
I20251029 02:52:20.126431 1174 rpcz_store.cc:276] 1029 02:52:13.106399 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:13.106457 (+ 58us) service_pool.cc:225] Handling call
1029 02:52:20.093039 (+6986582us) ts_tablet_manager.cc:1151] Deleting tablet afb19ba39399410bb90b6225d7b5b5b1
1029 02:52:20.093052 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:20.110109 (+ 17057us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:20.119977 (+ 9868us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:20.126103 (+ 6126us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6986383}
I20251029 02:52:20.127216 1166 tablet_bootstrap.cc:492] T 24f07b1efac64b099b0b5ed467463749 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:20.127449 1166 ts_tablet_manager.cc:1403] T 24f07b1efac64b099b0b5ed467463749 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.056s user 0.003s sys 0.000s
I20251029 02:52:20.127354 887 tablet_service.cc:1552] Processing DeleteTablet for tablet 8c972c88f64e486bb2b75306549bb23f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:20 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:20.127741 1039 tablet_service.cc:1552] Processing DeleteTablet for tablet 8c972c88f64e486bb2b75306549bb23f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:20 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:20.127868 1031 tablet_service.cc:1552] Processing DeleteTablet for tablet f72dae5a0e604eb384c1f5f06fe680bb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:20 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:20.128244 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet afb19ba39399410bb90b6225d7b5b5b1 (table default.Table_d4caf12f7662463c824210f3c9cd7e0f [id=7dfb670d18824f5dbdd321940fea965e]) successfully deleted
I20251029 02:52:20.128262 891 tablet_service.cc:1552] Processing DeleteTablet for tablet f72dae5a0e604eb384c1f5f06fe680bb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:20 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:20.128310 1166 raft_consensus.cc:359] T 24f07b1efac64b099b0b5ed467463749 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:20.128563 1166 raft_consensus.cc:385] T 24f07b1efac64b099b0b5ed467463749 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:20.128623 1174 tablet_replica.cc:333] T e65eaaae43ac435ebd665546de9d68dd P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:20.128669 1166 raft_consensus.cc:740] T 24f07b1efac64b099b0b5ed467463749 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:20.129060 1174 raft_consensus.cc:2243] T e65eaaae43ac435ebd665546de9d68dd P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:20.129011 1166 consensus_queue.cc:260] T 24f07b1efac64b099b0b5ed467463749 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:20.129225 1174 raft_consensus.cc:2272] T e65eaaae43ac435ebd665546de9d68dd P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:20.129736 1174 ts_tablet_manager.cc:1916] T e65eaaae43ac435ebd665546de9d68dd P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:20.134111 523 catalog_manager.cc:5649] T b125cc9aaf4248b8bfa48fb152b832a6 P b495a49349324f7d81908f2da700f0d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to b495a49349324f7d81908f2da700f0d9 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "b495a49349324f7d81908f2da700f0d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } health_report { overall_health: UNKNOWN } } }
I20251029 02:52:20.135352 1166 ts_tablet_manager.cc:1434] T 24f07b1efac64b099b0b5ed467463749 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20251029 02:52:20.136293 1166 tablet_bootstrap.cc:492] T b0fec06ad37d4c12a7045d424b88cc5e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:20.134896 526 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_b2b8451524e7419682961023cc62ba95" } modify_external_catalogs: true
I20251029 02:52:20.137316 526 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_b2b8451524e7419682961023cc62ba95" } modify_external_catalogs: true
I20251029 02:52:20.138466 1166 tablet_bootstrap.cc:654] T b0fec06ad37d4c12a7045d424b88cc5e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:20.138818 523 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet b125cc9aaf4248b8bfa48fb152b832a6 (table default.Table_110e1b64737c4040ada22360d5a373cb [id=3513c69f6b3941849ac5b8ae0a5846d3]). Expected version 5 got 0
I20251029 02:52:20.140017 523 catalog_manager.cc:5649] T 055184c641ba45a588bca611fb24d130 P b495a49349324f7d81908f2da700f0d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to b495a49349324f7d81908f2da700f0d9 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "b495a49349324f7d81908f2da700f0d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } health_report { overall_health: HEALTHY } } }
I20251029 02:52:20.140487 523 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet 055184c641ba45a588bca611fb24d130 (table default.Table_1922b14d4cfb4671a6224fcd6ace2d4e [id=aec3db766dee438c89fb2f1939ee7b7b]). Expected version 2 got 0
I20251029 02:52:20.140750 523 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet 08371b3daa184a0e9f4eb3d313b52c56 (table default.Table_1453b5168e64406688903f1d6927f549 [id=40f1beecaad24a64b4348663c9ae9afd]). Expected version 1 got 0
I20251029 02:52:20.137030 525 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet f4729ecd182042c2a10f67ca25224101 (table default.Table_973ef856041949598a23da3dab8b40db [id=1dd2d5488bc647c8b82c53886583b28d]). Expected version 4 got 0
W20251029 02:52:20.146718 510 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 8471db993eab4fcb9e6f317d48ad8f80 (table default.Table_c7dbb79386884cc49914ee437d974351 [id=592b3a043426469193575e824213d36e], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 8471db993eab4fcb9e6f317d48ad8f80 consensus state has no leader
I20251029 02:52:20.147881 524 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_3ddda212ceb444adae605530b0985d6a" } modify_external_catalogs: true
I20251029 02:52:20.148088 524 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_3ddda212ceb444adae605530b0985d6a" } modify_external_catalogs: true
I20251029 02:52:20.149780 517 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_24deffef51354bcda12021bdf6bab67b" } modify_external_catalogs: true
I20251029 02:52:20.149974 517 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_24deffef51354bcda12021bdf6bab67b" } modify_external_catalogs: true
I20251029 02:52:20.150655 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c94bd8cf035428280cc7449dcc036b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:20.160351 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_77dc63248995469d9ff43beb7dbe0bab"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:20.160962 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_77dc63248995469d9ff43beb7dbe0bab in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:20.162549 521 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_f47c6012c3a343d3baa413d0cd44571c" } new_table_name: "default.Table_61073ea06da94bd7b1d89dd71c8a2806" modify_external_catalogs: true
I20251029 02:52:20.163879 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_149cbe11c7c9451da5dfb9cff7001762"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:20.164405 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_149cbe11c7c9451da5dfb9cff7001762 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:20.167198 1167 tablet_bootstrap.cc:492] T e3000183230e4350b47f5085952f2db6 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:20.167407 1167 ts_tablet_manager.cc:1403] T e3000183230e4350b47f5085952f2db6 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.147s user 0.011s sys 0.015s
I20251029 02:52:20.168174 1167 raft_consensus.cc:359] T e3000183230e4350b47f5085952f2db6 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:20.168416 1167 raft_consensus.cc:385] T e3000183230e4350b47f5085952f2db6 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:20.168536 1167 raft_consensus.cc:740] T e3000183230e4350b47f5085952f2db6 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:20.168828 1167 consensus_queue.cc:260] T e3000183230e4350b47f5085952f2db6 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:20.169512 1167 ts_tablet_manager.cc:1434] T e3000183230e4350b47f5085952f2db6 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 02:52:20.169831 1167 tablet_bootstrap.cc:492] T c45ffc304e73490387ce800b702b5881 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:20.171339 1167 tablet_bootstrap.cc:654] T c45ffc304e73490387ce800b702b5881 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:20.171373 1166 tablet_bootstrap.cc:492] T b0fec06ad37d4c12a7045d424b88cc5e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:20.171582 1166 ts_tablet_manager.cc:1403] T b0fec06ad37d4c12a7045d424b88cc5e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.035s user 0.000s sys 0.003s
I20251029 02:52:20.172389 1166 raft_consensus.cc:359] T b0fec06ad37d4c12a7045d424b88cc5e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:20.172617 1166 raft_consensus.cc:385] T b0fec06ad37d4c12a7045d424b88cc5e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:20.172708 1166 raft_consensus.cc:740] T b0fec06ad37d4c12a7045d424b88cc5e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:20.173399 1174 ts_tablet_manager.cc:1929] T e65eaaae43ac435ebd665546de9d68dd P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:20.173514 1174 log.cc:1199] T e65eaaae43ac435ebd665546de9d68dd P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/e65eaaae43ac435ebd665546de9d68dd
I20251029 02:52:20.173903 1174 ts_tablet_manager.cc:1950] T e65eaaae43ac435ebd665546de9d68dd P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:20.172981 1166 consensus_queue.cc:260] T b0fec06ad37d4c12a7045d424b88cc5e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:20.175701 1166 ts_tablet_manager.cc:1434] T b0fec06ad37d4c12a7045d424b88cc5e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20251029 02:52:20.176095 1166 tablet_bootstrap.cc:492] T abf18e06df9b482a8b25c078b9495272 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:20.178894 900 tablet_service.cc:1505] Processing CreateTablet for tablet 0c94bd8cf035428280cc7449dcc036b7 (DEFAULT_TABLE table=default.Table_b3a7b65a549244a6af331747d706ecf3 [id=c21ccfd7b1964426938b032b019df906]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:20.183323 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 1040) took 7076 ms. Trace:
I20251029 02:52:20.183480 1172 tablet_bootstrap.cc:492] T 35d9b8dfd61e4d8db441427a49fb9dca P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:20.183815 1172 ts_tablet_manager.cc:1403] T 35d9b8dfd61e4d8db441427a49fb9dca P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.191s user 0.003s sys 0.001s
I20251029 02:52:20.183697 1174 rpcz_store.cc:276] 1029 02:52:13.107028 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:13.107076 (+ 48us) service_pool.cc:225] Handling call
1029 02:52:20.128568 (+7021492us) ts_tablet_manager.cc:1151] Deleting tablet e65eaaae43ac435ebd665546de9d68dd
1029 02:52:20.128580 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:20.135032 (+ 6452us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:20.173277 (+ 38245us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:20.183296 (+ 10019us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7021328}
I20251029 02:52:20.184437 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet e65eaaae43ac435ebd665546de9d68dd (table default.Table_d9fe74a49f4843b089aae7fca9e4a826 [id=152882c742b2459ba863fd41083284c1]) successfully deleted
I20251029 02:52:20.184865 1174 tablet_replica.cc:333] T 2f198c3259864f7ab3625cb7bfae20b0 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:20.184820 1172 raft_consensus.cc:359] T 35d9b8dfd61e4d8db441427a49fb9dca P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:20.185132 1172 raft_consensus.cc:385] T 35d9b8dfd61e4d8db441427a49fb9dca P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:20.185222 1172 raft_consensus.cc:740] T 35d9b8dfd61e4d8db441427a49fb9dca P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:20.185302 1174 raft_consensus.cc:2243] T 2f198c3259864f7ab3625cb7bfae20b0 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:20.185570 1172 consensus_queue.cc:260] T 35d9b8dfd61e4d8db441427a49fb9dca P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:20.185747 1174 raft_consensus.cc:2272] T 2f198c3259864f7ab3625cb7bfae20b0 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:20.186276 1172 ts_tablet_manager.cc:1434] T 35d9b8dfd61e4d8db441427a49fb9dca P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 02:52:20.186498 1174 ts_tablet_manager.cc:1916] T 2f198c3259864f7ab3625cb7bfae20b0 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:20.194636 739 tablet_service.cc:1505] Processing CreateTablet for tablet dac429c108ea4e069ae5f62ac8d2a09f (DEFAULT_TABLE table=default.Table_3bb798717ff0482d97c9c76ab35a0214 [id=47eb725b918043809e07f792dfc29c5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:20.195163 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dac429c108ea4e069ae5f62ac8d2a09f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:20.198659 1172 tablet_bootstrap.cc:492] T dc33fd3404ec41c3935b02c84a12971b P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:20.198724 1028 tablet_service.cc:1505] Processing CreateTablet for tablet dac429c108ea4e069ae5f62ac8d2a09f (DEFAULT_TABLE table=default.Table_3bb798717ff0482d97c9c76ab35a0214 [id=47eb725b918043809e07f792dfc29c5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:20.199227 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dac429c108ea4e069ae5f62ac8d2a09f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:20.200176 1172 tablet_bootstrap.cc:654] T dc33fd3404ec41c3935b02c84a12971b P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
W20251029 02:52:20.203889 968 tablet.cc:2378] T 2b278db5b2b04344b864797055a5f665 P c756528642a543eeacd93cef3e335a7d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251029 02:52:20.207893 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0c94bd8cf035428280cc7449dcc036b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:20.209590 2656 raft_consensus.cc:493] T e3000183230e4350b47f5085952f2db6 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:20.209729 2656 raft_consensus.cc:515] T e3000183230e4350b47f5085952f2db6 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:20.210116 526 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 266aaea633bf4c059b9c02f092a474be
I20251029 02:52:20.210274 2656 leader_election.cc:290] T e3000183230e4350b47f5085952f2db6 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:20.210788 888 tablet_service.cc:1552] Processing DeleteTablet for tablet 266aaea633bf4c059b9c02f092a474be with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:20 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:20.210882 1062 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e3000183230e4350b47f5085952f2db6" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:52:20.211459 849 leader_election.cc:343] T e3000183230e4350b47f5085952f2db6 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:20.212038 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e3000183230e4350b47f5085952f2db6" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:20.212229 772 raft_consensus.cc:2468] T e3000183230e4350b47f5085952f2db6 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:20.212637 847 leader_election.cc:304] T e3000183230e4350b47f5085952f2db6 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5
I20251029 02:52:20.213037 2656 raft_consensus.cc:2804] T e3000183230e4350b47f5085952f2db6 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:20.213107 2656 raft_consensus.cc:493] T e3000183230e4350b47f5085952f2db6 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:20.213160 2656 raft_consensus.cc:3060] T e3000183230e4350b47f5085952f2db6 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:20.213707 748 tablet_service.cc:1552] Processing DeleteTablet for tablet 266aaea633bf4c059b9c02f092a474be with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:20 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:20.213899 524 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet aef3021f7d684df197eb1cfd5c6b8772
I20251029 02:52:20.214778 748 tablet_service.cc:1552] Processing DeleteTablet for tablet aef3021f7d684df197eb1cfd5c6b8772 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:20 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:20.214973 888 tablet_service.cc:1552] Processing DeleteTablet for tablet aef3021f7d684df197eb1cfd5c6b8772 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:20 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:20.216122 2656 raft_consensus.cc:515] T e3000183230e4350b47f5085952f2db6 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:20.217413 1062 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e3000183230e4350b47f5085952f2db6" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
W20251029 02:52:20.217892 849 leader_election.cc:343] T e3000183230e4350b47f5085952f2db6 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:20.217916 517 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet d8c66a892c90475c834f08c0734f41db
I20251029 02:52:20.218930 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e3000183230e4350b47f5085952f2db6" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:20.219071 748 tablet_service.cc:1552] Processing DeleteTablet for tablet d8c66a892c90475c834f08c0734f41db with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:20 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:20.219108 772 raft_consensus.cc:3060] T e3000183230e4350b47f5085952f2db6 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:20.219607 1031 tablet_service.cc:1552] Processing DeleteTablet for tablet d8c66a892c90475c834f08c0734f41db with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:20 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:20.221850 1166 tablet_bootstrap.cc:654] T abf18e06df9b482a8b25c078b9495272 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:20.221818 772 raft_consensus.cc:2468] T e3000183230e4350b47f5085952f2db6 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:20.222496 847 leader_election.cc:304] T e3000183230e4350b47f5085952f2db6 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5
I20251029 02:52:20.222833 2655 raft_consensus.cc:2804] T e3000183230e4350b47f5085952f2db6 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:20.222965 2655 raft_consensus.cc:697] T e3000183230e4350b47f5085952f2db6 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:52:20.223263 2655 consensus_queue.cc:237] T e3000183230e4350b47f5085952f2db6 P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:20.224666 2656 leader_election.cc:290] T e3000183230e4350b47f5085952f2db6 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
W20251029 02:52:20.226918 847 consensus_peers.cc:597] T c081d2eacfac4ca8a0048e2a1c4f8f83 P c756528642a543eeacd93cef3e335a7d -> Peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Couldn't send request to peer b495a49349324f7d81908f2da700f0d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c081d2eacfac4ca8a0048e2a1c4f8f83. This is attempt 1: this message will repeat every 5th retry.
W20251029 02:52:20.229300 521 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b6436a020e3542fbb25ac2846ed42bef consensus state has no leader
I20251029 02:52:20.229684 1062 raft_consensus.cc:3060] T 24f07b1efac64b099b0b5ed467463749 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:20.229880 1166 tablet_bootstrap.cc:492] T abf18e06df9b482a8b25c078b9495272 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:20.230028 1166 ts_tablet_manager.cc:1403] T abf18e06df9b482a8b25c078b9495272 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.054s user 0.005s sys 0.001s
I20251029 02:52:20.231173 1166 raft_consensus.cc:359] T abf18e06df9b482a8b25c078b9495272 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:20.231421 1166 raft_consensus.cc:385] T abf18e06df9b482a8b25c078b9495272 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:20.231490 1166 raft_consensus.cc:740] T abf18e06df9b482a8b25c078b9495272 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:20.234684 1166 consensus_queue.cc:260] T abf18e06df9b482a8b25c078b9495272 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:20.235811 1174 ts_tablet_manager.cc:1929] T 2f198c3259864f7ab3625cb7bfae20b0 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:20.236510 1166 ts_tablet_manager.cc:1434] T abf18e06df9b482a8b25c078b9495272 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20251029 02:52:20.236927 1166 tablet_bootstrap.cc:492] T cc802bda2f6a4e439b605f2cafa37967 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:20.237188 1174 log.cc:1199] T 2f198c3259864f7ab3625cb7bfae20b0 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/2f198c3259864f7ab3625cb7bfae20b0
I20251029 02:52:20.237680 1174 ts_tablet_manager.cc:1950] T 2f198c3259864f7ab3625cb7bfae20b0 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:20.238643 525 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet 055184c641ba45a588bca611fb24d130 (table default.Table_1922b14d4cfb4671a6224fcd6ace2d4e [id=aec3db766dee438c89fb2f1939ee7b7b]). Expected version 2 got 0
I20251029 02:52:20.242380 1031 tablet_service.cc:1552] Processing DeleteTablet for tablet 24f07b1efac64b099b0b5ed467463749 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:59 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:20.243446 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 1041) took 7136 ms. Trace:
I20251029 02:52:20.243643 1034 tablet_service.cc:1552] Processing DeleteTablet for tablet 266aaea633bf4c059b9c02f092a474be with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:20 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:20.244334 1042 tablet_service.cc:1552] Processing DeleteTablet for tablet aef3021f7d684df197eb1cfd5c6b8772 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:20 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:20.245199 1166 tablet_bootstrap.cc:654] T cc802bda2f6a4e439b605f2cafa37967 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:20.246351 1174 rpcz_store.cc:276] 1029 02:52:13.107150 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:13.107279 (+ 129us) service_pool.cc:225] Handling call
1029 02:52:20.184810 (+7077531us) ts_tablet_manager.cc:1151] Deleting tablet 2f198c3259864f7ab3625cb7bfae20b0
1029 02:52:20.184822 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:20.190367 (+ 5545us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:20.235672 (+ 45305us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:20.243421 (+ 7749us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7077409}
I20251029 02:52:20.246763 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_200223c02b1440608d88186f1a099283"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:20.247171 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 2f198c3259864f7ab3625cb7bfae20b0 (table default.Table_a3377614e4a54266a4457944d48b0060 [id=991a151cc5d84421bf50858b1fa864e6]) successfully deleted
W20251029 02:52:20.247391 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_200223c02b1440608d88186f1a099283 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:20.247630 1167 tablet_bootstrap.cc:492] T c45ffc304e73490387ce800b702b5881 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:20.247824 1167 ts_tablet_manager.cc:1403] T c45ffc304e73490387ce800b702b5881 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.078s user 0.006s sys 0.006s
I20251029 02:52:20.248571 1167 raft_consensus.cc:359] T c45ffc304e73490387ce800b702b5881 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:20.248692 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_d12a80a21baa4785a3a404f8d594851f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:20.248843 1167 raft_consensus.cc:385] T c45ffc304e73490387ce800b702b5881 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:20.248903 1167 raft_consensus.cc:740] T c45ffc304e73490387ce800b702b5881 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:20.249131 1167 consensus_queue.cc:260] T c45ffc304e73490387ce800b702b5881 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
W20251029 02:52:20.249285 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d12a80a21baa4785a3a404f8d594851f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:20.249634 1167 ts_tablet_manager.cc:1434] T c45ffc304e73490387ce800b702b5881 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251029 02:52:20.249912 1167 tablet_bootstrap.cc:492] T e3ac6597d52441569a7aeef051087ad2 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:20.250897 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 1055) took 6832 ms. Trace:
I20251029 02:52:20.251048 1174 rpcz_store.cc:276] 1029 02:52:13.418834 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:13.419095 (+ 261us) service_pool.cc:225] Handling call
1029 02:52:20.250728 (+6831633us) ts_tablet_manager.cc:1151] Deleting tablet 15cadd681ceb4daba178a9cc2b7aec08
1029 02:52:20.250744 (+ 16us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:20.250876 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6830646}
I20251029 02:52:20.251385 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_93591a471a3946cebf99e3160fb195d4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:20.251672 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 15cadd681ceb4daba178a9cc2b7aec08 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 15cadd681ceb4daba178a9cc2b7aec08 already in progress: creating tablet
W20251029 02:52:20.251871 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_93591a471a3946cebf99e3160fb195d4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:20.252094 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 1059) took 6832 ms. Trace:
I20251029 02:52:20.252178 1174 rpcz_store.cc:276] 1029 02:52:13.419837 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:13.420918 (+ 1081us) service_pool.cc:225] Handling call
1029 02:52:20.251951 (+6831033us) ts_tablet_manager.cc:1151] Deleting tablet 31ca9012ddf6401ba78888a0c82b4ef2
1029 02:52:20.251965 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:20.252074 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6830837}
W20251029 02:52:20.252588 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 31ca9012ddf6401ba78888a0c82b4ef2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 31ca9012ddf6401ba78888a0c82b4ef2 already in progress: creating tablet
I20251029 02:52:20.252945 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 1056) took 6833 ms. Trace:
I20251029 02:52:20.253039 1174 rpcz_store.cc:276] 1029 02:52:13.419074 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:13.421190 (+ 2116us) service_pool.cc:225] Handling call
1029 02:52:20.252836 (+6831646us) ts_tablet_manager.cc:1151] Deleting tablet 21c73cb2f1084827b581fb09cf7239a1
1029 02:52:20.252845 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:20.252931 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6831489}
I20251029 02:52:20.253307 1167 tablet_bootstrap.cc:654] T e3ac6597d52441569a7aeef051087ad2 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
W20251029 02:52:20.253419 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 21c73cb2f1084827b581fb09cf7239a1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 21c73cb2f1084827b581fb09cf7239a1 already in progress: creating tablet
I20251029 02:52:20.253371 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_f513d9b3495a4a1f9350bfd5a8e396d4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:20.253813 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 1058) took 6834 ms. Trace:
I20251029 02:52:20.253911 1174 rpcz_store.cc:276] 1029 02:52:13.419508 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:13.427255 (+ 7747us) service_pool.cc:225] Handling call
1029 02:52:20.253719 (+6826464us) ts_tablet_manager.cc:1151] Deleting tablet 416e7c48daa849cd9635656529488922
1029 02:52:20.253728 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:20.253797 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6825882}
W20251029 02:52:20.253966 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f513d9b3495a4a1f9350bfd5a8e396d4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:52:20.254321 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 416e7c48daa849cd9635656529488922 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 416e7c48daa849cd9635656529488922 already in progress: creating tablet
I20251029 02:52:20.254729 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 1057) took 6835 ms. Trace:
I20251029 02:52:20.256678 1174 rpcz_store.cc:276] 1029 02:52:13.419299 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:13.428034 (+ 8735us) service_pool.cc:225] Handling call
1029 02:52:20.254592 (+6826558us) ts_tablet_manager.cc:1151] Deleting tablet eb3c20822d684bea89eb358251e21e33
1029 02:52:20.254605 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:20.254709 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6826180}
W20251029 02:52:20.257170 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet eb3c20822d684bea89eb358251e21e33 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eb3c20822d684bea89eb358251e21e33 already in progress: creating tablet
I20251029 02:52:20.257522 1174 tablet_replica.cc:333] T 82289e0f823c4d43bfd8c4e878572b2b P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:20.257653 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_7112531047164b6ca1129374fc46904d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:20.257972 1174 raft_consensus.cc:2243] T 82289e0f823c4d43bfd8c4e878572b2b P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:20.258111 1174 pending_rounds.cc:70] T 82289e0f823c4d43bfd8c4e878572b2b P c756528642a543eeacd93cef3e335a7d: Trying to abort 1 pending ops.
I20251029 02:52:20.258144 1174 pending_rounds.cc:77] T 82289e0f823c4d43bfd8c4e878572b2b P c756528642a543eeacd93cef3e335a7d: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7215949155507691520 op_type: ALTER_SCHEMA_OP alter_schema_request { tablet_id: "82289e0f823c4d43bfd8c4e878572b2b" schema { columns { id: 10 name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } schema_version: 1 new_table_name: "default.Table_e7bb3e262a894c2f84b6edfc3917b30d" dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" new_extra_config { } }
W20251029 02:52:20.258441 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7112531047164b6ca1129374fc46904d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:20.261420 748 tablet_service.cc:1552] Processing DeleteTablet for tablet 35d9b8dfd61e4d8db441427a49fb9dca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:07 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:20.261675 740 tablet_service.cc:1552] Processing DeleteTablet for tablet 877e278deb4a4b01ac66b06be0890baf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:09 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:20.262125 1174 raft_consensus.cc:2272] T 82289e0f823c4d43bfd8c4e878572b2b P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:20.262954 1174 ts_tablet_manager.cc:1916] T 82289e0f823c4d43bfd8c4e878572b2b P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:20.263123 888 tablet_service.cc:1552] Processing DeleteTablet for tablet d8c66a892c90475c834f08c0734f41db with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:20 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:20.272239 1175 ts_tablet_manager.cc:1929] T 946f65014d97425aab3851009b742807 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:20.272377 1175 log.cc:1199] T 946f65014d97425aab3851009b742807 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/946f65014d97425aab3851009b742807
I20251029 02:52:20.272732 1175 ts_tablet_manager.cc:1950] T 946f65014d97425aab3851009b742807 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:20.273514 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 1101) took 5551 ms. Trace:
I20251029 02:52:20.273618 1175 rpcz_store.cc:276] 1029 02:52:14.722265 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:14.722334 (+ 69us) service_pool.cc:225] Handling call
1029 02:52:19.600024 (+4877690us) ts_tablet_manager.cc:1151] Deleting tablet 946f65014d97425aab3851009b742807
1029 02:52:19.600035 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:20.174927 (+574892us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:20.272113 (+ 97186us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:20.273490 (+ 1377us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4865067}
I20251029 02:52:20.274212 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 946f65014d97425aab3851009b742807 (table default.Table_7dd1cd855bb74080b6e242847e5faac6 [id=8a3311a64d244193985716c908a42f27]) successfully deleted
I20251029 02:52:20.275643 1172 tablet_bootstrap.cc:492] T dc33fd3404ec41c3935b02c84a12971b P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:20.275846 1172 ts_tablet_manager.cc:1403] T dc33fd3404ec41c3935b02c84a12971b P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.077s user 0.003s sys 0.000s
I20251029 02:52:20.276576 1172 raft_consensus.cc:359] T dc33fd3404ec41c3935b02c84a12971b P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:20.276816 1172 raft_consensus.cc:385] T dc33fd3404ec41c3935b02c84a12971b P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:20.276903 1172 raft_consensus.cc:740] T dc33fd3404ec41c3935b02c84a12971b P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:20.277153 1172 consensus_queue.cc:260] T dc33fd3404ec41c3935b02c84a12971b P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:20.280704 1172 ts_tablet_manager.cc:1434] T dc33fd3404ec41c3935b02c84a12971b P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20251029 02:52:20.283218 521 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet dc33fd3404ec41c3935b02c84a12971b (table default.Table_46e2f96551d3497c919e1cab3521d63b [id=09ee821707fa4082889bfccb8667b0b7]). Expected version 1 got 0
I20251029 02:52:20.287046 1172 tablet_bootstrap.cc:492] T da904606986f452cadd06187fff06785 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:20.289372 1172 tablet_bootstrap.cc:654] T da904606986f452cadd06187fff06785 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:20.289743 521 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet dc33fd3404ec41c3935b02c84a12971b (table default.Table_46e2f96551d3497c919e1cab3521d63b [id=09ee821707fa4082889bfccb8667b0b7]). Expected version 1 got 0
I20251029 02:52:20.290221 2656 raft_consensus.cc:493] T 294afb9b012c40c0b05b73795d6f0ac8 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:20.290416 2656 raft_consensus.cc:515] T 294afb9b012c40c0b05b73795d6f0ac8 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:20.293298 1052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "294afb9b012c40c0b05b73795d6f0ac8" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:52:20.293848 849 leader_election.cc:343] T 294afb9b012c40c0b05b73795d6f0ac8 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:20.294677 2656 leader_election.cc:290] T 294afb9b012c40c0b05b73795d6f0ac8 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:20.295469 519 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_75325372892242518c3f0a4b8b657c48" } new_table_name: "default.Table_60c18fa14f6e470fac7134a843e4bcd8" modify_external_catalogs: true
I20251029 02:52:20.296422 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "294afb9b012c40c0b05b73795d6f0ac8" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:20.299147 772 raft_consensus.cc:2468] T 294afb9b012c40c0b05b73795d6f0ac8 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:20.299626 847 leader_election.cc:304] T 294afb9b012c40c0b05b73795d6f0ac8 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5
I20251029 02:52:20.300173 2656 raft_consensus.cc:2804] T 294afb9b012c40c0b05b73795d6f0ac8 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:20.300292 2656 raft_consensus.cc:493] T 294afb9b012c40c0b05b73795d6f0ac8 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:20.300416 2656 raft_consensus.cc:3060] T 294afb9b012c40c0b05b73795d6f0ac8 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
W20251029 02:52:20.302088 824 tablet.cc:2378] T 35d9b8dfd61e4d8db441427a49fb9dca P b495a49349324f7d81908f2da700f0d9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251029 02:52:20.303296 2656 raft_consensus.cc:515] T 294afb9b012c40c0b05b73795d6f0ac8 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:20.304612 1052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "294afb9b012c40c0b05b73795d6f0ac8" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:52:20.304610 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "294afb9b012c40c0b05b73795d6f0ac8" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:20.304788 772 raft_consensus.cc:3060] T 294afb9b012c40c0b05b73795d6f0ac8 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
W20251029 02:52:20.305230 849 leader_election.cc:343] T 294afb9b012c40c0b05b73795d6f0ac8 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:20.307641 772 raft_consensus.cc:2468] T 294afb9b012c40c0b05b73795d6f0ac8 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:20.308149 847 leader_election.cc:304] T 294afb9b012c40c0b05b73795d6f0ac8 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5
I20251029 02:52:20.308496 2747 raft_consensus.cc:2804] T 294afb9b012c40c0b05b73795d6f0ac8 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:20.308607 2747 raft_consensus.cc:697] T 294afb9b012c40c0b05b73795d6f0ac8 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:52:20.308837 2747 consensus_queue.cc:237] T 294afb9b012c40c0b05b73795d6f0ac8 P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:20.317023 2656 leader_election.cc:290] T 294afb9b012c40c0b05b73795d6f0ac8 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:20.318202 2986 raft_consensus.cc:493] T 3a2b95997454452bb985c51fc02d77e2 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:20.318387 2986 raft_consensus.cc:515] T 3a2b95997454452bb985c51fc02d77e2 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:20.319130 2986 leader_election.cc:290] T 3a2b95997454452bb985c51fc02d77e2 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:52:20.319888 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a2b95997454452bb985c51fc02d77e2" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:20.320655 711 leader_election.cc:343] T 3a2b95997454452bb985c51fc02d77e2 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:20.321830 1062 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "3a2b95997454452bb985c51fc02d77e2" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:52:20.322444 712 leader_election.cc:343] T 3a2b95997454452bb985c51fc02d77e2 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:20.323168 712 leader_election.cc:304] T 3a2b95997454452bb985c51fc02d77e2 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:20.323493 2656 raft_consensus.cc:493] T 023a4094b424464c8f49d6eb9cc15da4 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:20.323625 2656 raft_consensus.cc:515] T 023a4094b424464c8f49d6eb9cc15da4 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:20.323781 2986 raft_consensus.cc:2749] T 3a2b95997454452bb985c51fc02d77e2 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:20.325218 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "023a4094b424464c8f49d6eb9cc15da4" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:20.325397 772 raft_consensus.cc:2468] T 023a4094b424464c8f49d6eb9cc15da4 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:20.325798 847 leader_election.cc:304] T 023a4094b424464c8f49d6eb9cc15da4 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:20.326025 2747 raft_consensus.cc:2804] T 023a4094b424464c8f49d6eb9cc15da4 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:20.326097 2747 raft_consensus.cc:493] T 023a4094b424464c8f49d6eb9cc15da4 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:20.326164 2747 raft_consensus.cc:3060] T 023a4094b424464c8f49d6eb9cc15da4 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:20.327544 1062 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "023a4094b424464c8f49d6eb9cc15da4" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:52:20.328079 849 leader_election.cc:343] T 023a4094b424464c8f49d6eb9cc15da4 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:20.328325 2656 leader_election.cc:290] T 023a4094b424464c8f49d6eb9cc15da4 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:20.328840 2747 raft_consensus.cc:515] T 023a4094b424464c8f49d6eb9cc15da4 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:20.330221 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 1102) took 5566 ms. Trace:
I20251029 02:52:20.330312 1175 rpcz_store.cc:276] 1029 02:52:14.763635 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:14.766574 (+ 2939us) service_pool.cc:225] Handling call
1029 02:52:20.330069 (+5563495us) ts_tablet_manager.cc:1151] Deleting tablet df669cf8da7f4c98a3f16423c55c15eb
1029 02:52:20.330082 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:20.330199 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5563332}
I20251029 02:52:20.330441 2747 leader_election.cc:290] T 023a4094b424464c8f49d6eb9cc15da4 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:20.335110 518 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_75f0822a7de44b8ea84235f6790a2140" } new_table_name: "default.Table_e51f77418e95447f9839662efc11d340" modify_external_catalogs: true
I20251029 02:52:20.338778 1062 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "023a4094b424464c8f49d6eb9cc15da4" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:52:20.340713 1174 ts_tablet_manager.cc:1929] T 82289e0f823c4d43bfd8c4e878572b2b P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20251029 02:52:20.341979 1110 tablet.cc:2378] T abf18e06df9b482a8b25c078b9495272 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251029 02:52:20.341279 849 leader_election.cc:343] T 023a4094b424464c8f49d6eb9cc15da4 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:20.342566 888 tablet_service.cc:1505] Processing CreateTablet for tablet dac429c108ea4e069ae5f62ac8d2a09f (DEFAULT_TABLE table=default.Table_3bb798717ff0482d97c9c76ab35a0214 [id=47eb725b918043809e07f792dfc29c5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:20.344517 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dac429c108ea4e069ae5f62ac8d2a09f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:20.345175 1042 tablet_service.cc:1505] Processing CreateTablet for tablet 5e47387d598c41baad846d4ac3439bb0 (DEFAULT_TABLE table=default.Table_a3ea3760372c4e8f88ef9506f9426e28 [id=b64a733c30b84aeeba1ebed23af13823]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:20.345536 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e47387d598c41baad846d4ac3439bb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:20.345757 891 tablet_service.cc:1505] Processing CreateTablet for tablet 5e47387d598c41baad846d4ac3439bb0 (DEFAULT_TABLE table=default.Table_a3ea3760372c4e8f88ef9506f9426e28 [id=b64a733c30b84aeeba1ebed23af13823]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:20.346203 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e47387d598c41baad846d4ac3439bb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:20.350905 2649 raft_consensus.cc:493] T cc802bda2f6a4e439b605f2cafa37967 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:20.351047 2649 raft_consensus.cc:515] T cc802bda2f6a4e439b605f2cafa37967 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:20.351648 2649 leader_election.cc:290] T cc802bda2f6a4e439b605f2cafa37967 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:20.363379 517 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_1aa03bfd183b40188824e6368f5746ee" } modify_external_catalogs: true
I20251029 02:52:20.363592 517 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_1aa03bfd183b40188824e6368f5746ee" } modify_external_catalogs: true
I20251029 02:52:20.366796 525 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_daad6fda420f4659bc87d5991d60cf2f" } new_table_name: "default.Table_bb3184a4784b40ea89c3a46b8b9f02f3" modify_external_catalogs: true
I20251029 02:52:20.367749 1052 raft_consensus.cc:3060] T b0fec06ad37d4c12a7045d424b88cc5e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:20.368104 522 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_0ae102f09fdc41bcbead0148653df9e2" } modify_external_catalogs: true
I20251029 02:52:20.368274 522 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_0ae102f09fdc41bcbead0148653df9e2" } modify_external_catalogs: true
I20251029 02:52:20.369033 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 1103) took 5600 ms. Trace:
I20251029 02:52:20.369128 1175 rpcz_store.cc:276] 1029 02:52:14.768770 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:14.772816 (+ 4046us) service_pool.cc:225] Handling call
1029 02:52:20.368872 (+5596056us) ts_tablet_manager.cc:1151] Deleting tablet fe9e660de57a46c7ba1d82f25e12b4b0
1029 02:52:20.368880 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:20.369011 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5594844}
I20251029 02:52:20.369437 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 1105) took 5593 ms. Trace:
I20251029 02:52:20.369493 1175 rpcz_store.cc:276] 1029 02:52:14.776287 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:14.780329 (+ 4042us) service_pool.cc:225] Handling call
1029 02:52:20.369347 (+5589018us) ts_tablet_manager.cc:1151] Deleting tablet e1d00ea3dd3744ffa5258bbe0095851c
1029 02:52:20.369355 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:20.369425 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5588785}
I20251029 02:52:20.369585 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_9a18834476fa412dbbd1ff3e8d70fc5f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:20.369752 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 1106) took 5511 ms. Trace:
I20251029 02:52:20.369808 1175 rpcz_store.cc:276] 1029 02:52:14.857964 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:14.858267 (+ 303us) service_pool.cc:225] Handling call
1029 02:52:20.369656 (+5511389us) ts_tablet_manager.cc:1151] Deleting tablet d48d9178108b4eb59062d29b40ba0c11
1029 02:52:20.369663 (+ 7us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:20.369739 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5511152}
I20251029 02:52:20.370018 1175 tablet_replica.cc:333] T fcb3db8399aa4cf188b07dda220f7f5b P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:20.370204 1175 raft_consensus.cc:2243] T fcb3db8399aa4cf188b07dda220f7f5b P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus shutting down.
W20251029 02:52:20.370221 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9a18834476fa412dbbd1ff3e8d70fc5f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:20.370358 1175 raft_consensus.cc:2272] T fcb3db8399aa4cf188b07dda220f7f5b P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:20.370970 1175 ts_tablet_manager.cc:1916] T fcb3db8399aa4cf188b07dda220f7f5b P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 02:52:20.371971 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet df669cf8da7f4c98a3f16423c55c15eb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet df669cf8da7f4c98a3f16423c55c15eb already in progress: creating tablet
W20251029 02:52:20.372294 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet fe9e660de57a46c7ba1d82f25e12b4b0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fe9e660de57a46c7ba1d82f25e12b4b0 already in progress: creating tablet
W20251029 02:52:20.372510 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet e1d00ea3dd3744ffa5258bbe0095851c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e1d00ea3dd3744ffa5258bbe0095851c already in progress: creating tablet
W20251029 02:52:20.372732 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet d48d9178108b4eb59062d29b40ba0c11 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d48d9178108b4eb59062d29b40ba0c11 already in progress: creating tablet
I20251029 02:52:20.373905 523 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_e75db5cf8f5f49ea837cb6d63f4b7ebf" } new_table_name: "default.Table_5667e55a7de6480d8f65fb4cdc120779" modify_external_catalogs: true
I20251029 02:52:20.375828 1176 ts_tablet_manager.cc:1929] T 44b518c3a0964a6fa863b2784d80c8a4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20251029 02:52:20.375828 2426 consensus_queue.cc:1048] T b0fec06ad37d4c12a7045d424b88cc5e P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:20.381232 2833 log.cc:826] T 00000000000000000000000000000000 P e9672114a6a742e2864c16ff60ba246e: Log is configured to *not* fsync() on all Append() calls
I20251029 02:52:20.383400 1176 log.cc:1199] T 44b518c3a0964a6fa863b2784d80c8a4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/44b518c3a0964a6fa863b2784d80c8a4
I20251029 02:52:20.383761 1176 ts_tablet_manager.cc:1950] T 44b518c3a0964a6fa863b2784d80c8a4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:20.383289 1166 tablet_bootstrap.cc:492] T cc802bda2f6a4e439b605f2cafa37967 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:20.384120 1166 ts_tablet_manager.cc:1403] T cc802bda2f6a4e439b605f2cafa37967 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.147s user 0.000s sys 0.011s
I20251029 02:52:20.384902 1166 raft_consensus.cc:359] T cc802bda2f6a4e439b605f2cafa37967 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:20.385169 1166 raft_consensus.cc:385] T cc802bda2f6a4e439b605f2cafa37967 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:20.385277 1166 raft_consensus.cc:740] T cc802bda2f6a4e439b605f2cafa37967 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:20.385529 1166 consensus_queue.cc:260] T cc802bda2f6a4e439b605f2cafa37967 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:20.386402 1166 ts_tablet_manager.cc:1434] T cc802bda2f6a4e439b605f2cafa37967 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251029 02:52:20.353456 767 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cc802bda2f6a4e439b605f2cafa37967" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:20.393200 767 raft_consensus.cc:2468] T cc802bda2f6a4e439b605f2cafa37967 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:20.393680 847 leader_election.cc:304] T cc802bda2f6a4e439b605f2cafa37967 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:20.393994 2649 raft_consensus.cc:2804] T cc802bda2f6a4e439b605f2cafa37967 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:20.394086 2649 raft_consensus.cc:493] T cc802bda2f6a4e439b605f2cafa37967 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:20.394166 2649 raft_consensus.cc:3060] T cc802bda2f6a4e439b605f2cafa37967 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:20.360050 2986 consensus_queue.cc:1048] T b125cc9aaf4248b8bfa48fb152b832a6 P b495a49349324f7d81908f2da700f0d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:20.373158 739 tablet_service.cc:1505] Processing CreateTablet for tablet 5e47387d598c41baad846d4ac3439bb0 (DEFAULT_TABLE table=default.Table_a3ea3760372c4e8f88ef9506f9426e28 [id=b64a733c30b84aeeba1ebed23af13823]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:20.395643 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e47387d598c41baad846d4ac3439bb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:20.396871 2649 raft_consensus.cc:515] T cc802bda2f6a4e439b605f2cafa37967 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:20.397914 2649 leader_election.cc:290] T cc802bda2f6a4e439b605f2cafa37967 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:20.398001 767 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cc802bda2f6a4e439b605f2cafa37967" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:20.398167 767 raft_consensus.cc:3060] T cc802bda2f6a4e439b605f2cafa37967 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:20.401291 767 raft_consensus.cc:2468] T cc802bda2f6a4e439b605f2cafa37967 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:20.401714 847 leader_election.cc:304] T cc802bda2f6a4e439b605f2cafa37967 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:20.403949 965 heartbeater.cc:344] Connected to a master server at 127.0.104.62:42343
I20251029 02:52:20.404763 1059 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cc802bda2f6a4e439b605f2cafa37967" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:52:20.404961 1059 raft_consensus.cc:3060] T cc802bda2f6a4e439b605f2cafa37967 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:20.405295 1166 tablet_bootstrap.cc:492] T 1df0d944717d487a98ea83a8ec04bd46 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:20.407011 1166 tablet_bootstrap.cc:654] T 1df0d944717d487a98ea83a8ec04bd46 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:20.407991 1059 raft_consensus.cc:2468] T cc802bda2f6a4e439b605f2cafa37967 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:20.409790 2649 raft_consensus.cc:2804] T cc802bda2f6a4e439b605f2cafa37967 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:20.409950 2649 raft_consensus.cc:697] T cc802bda2f6a4e439b605f2cafa37967 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:52:20.410290 2649 consensus_queue.cc:237] T cc802bda2f6a4e439b605f2cafa37967 P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:20.404835 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1338) took 1239 ms. Trace:
I20251029 02:52:20.414685 1176 rpcz_store.cc:276] 1029 02:52:19.165529 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:19.165582 (+ 53us) service_pool.cc:225] Handling call
1029 02:52:19.581450 (+415868us) ts_tablet_manager.cc:1151] Deleting tablet 44b518c3a0964a6fa863b2784d80c8a4
1029 02:52:19.581458 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:19.588986 (+ 7528us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:20.375713 (+786727us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:20.404811 (+ 29098us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":415707}
I20251029 02:52:20.415459 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 44b518c3a0964a6fa863b2784d80c8a4 (table default.Table_9e67f2d45e8f44c6bff1b72166c092a2 [id=e0c0cf3d456444c191a32bb180e83798]) successfully deleted
I20251029 02:52:20.418742 1042 tablet_service.cc:1505] Processing CreateTablet for tablet bb2a0b13fddd40d4b6b90e845f8c874c (DEFAULT_TABLE table=default.Table_8c6777c5195142f8b08409b19267c331 [id=e29e88f07ca24a4987cf76b8e3de7087]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:20.419391 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb2a0b13fddd40d4b6b90e845f8c874c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:20.330694 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "023a4094b424464c8f49d6eb9cc15da4" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:20.421093 772 raft_consensus.cc:3060] T 023a4094b424464c8f49d6eb9cc15da4 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:20.421123 891 tablet_service.cc:1505] Processing CreateTablet for tablet bb2a0b13fddd40d4b6b90e845f8c874c (DEFAULT_TABLE table=default.Table_8c6777c5195142f8b08409b19267c331 [id=e29e88f07ca24a4987cf76b8e3de7087]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:20.422295 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb2a0b13fddd40d4b6b90e845f8c874c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:20.427804 1176 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37626 (request call id 1340) took 1010 ms. Trace:
I20251029 02:52:20.427950 1176 rpcz_store.cc:276] 1029 02:52:19.417089 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:19.417144 (+ 55us) service_pool.cc:225] Handling call
1029 02:52:20.427606 (+1010462us) ts_tablet_manager.cc:1151] Deleting tablet 2c424a54c9714fe490677e64e3dd2931
1029 02:52:20.427622 (+ 16us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:20.427769 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1010266}
I20251029 02:52:20.428331 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_c6b8c213691347269a21926a10a5a051"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:20.428871 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c6b8c213691347269a21926a10a5a051 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:20.431039 1176 tablet_replica.cc:333] T 628a7d0419cb434f8171faf500d585d8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
W20251029 02:52:20.431037 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 2c424a54c9714fe490677e64e3dd2931 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2c424a54c9714fe490677e64e3dd2931 already in progress: creating tablet
I20251029 02:52:20.431267 1176 raft_consensus.cc:2243] T 628a7d0419cb434f8171faf500d585d8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus shutting down.
W20251029 02:52:20.431270 513 catalog_manager.cc:4952] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 2d01627906b444dc8386191004e650e2 because the tablet was not found. No further retry: Not found: Tablet not found: 2d01627906b444dc8386191004e650e2
I20251029 02:52:20.431424 1176 raft_consensus.cc:2272] T 628a7d0419cb434f8171faf500d585d8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251029 02:52:20.431434 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 4c17d21145e54cc78134cab37059ce1e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4c17d21145e54cc78134cab37059ce1e already in progress: creating tablet
W20251029 02:52:20.431598 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 902188c70bbc4d6c9cd4c1ba2c2fe295 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 902188c70bbc4d6c9cd4c1ba2c2fe295 already in progress: creating tablet
W20251029 02:52:20.431756 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet b79327b8c1354810a8b1b0d9e62fe3c8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b79327b8c1354810a8b1b0d9e62fe3c8 already in progress: creating tablet
W20251029 02:52:20.431906 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet b22929678f734288829c53fffadfa8a9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b22929678f734288829c53fffadfa8a9 already in progress: creating tablet
W20251029 02:52:20.432065 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 2d33bc6dafb64d948cd62feb50193b26 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2d33bc6dafb64d948cd62feb50193b26 already in progress: creating tablet
W20251029 02:52:20.432217 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 09567e75ea1140e5ba506a49fbb1d77e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 09567e75ea1140e5ba506a49fbb1d77e already in progress: creating tablet
W20251029 02:52:20.432370 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 78c7cbd6bde646c69eae1698d1b0f7a9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 78c7cbd6bde646c69eae1698d1b0f7a9 already in progress: creating tablet
W20251029 02:52:20.432569 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet d22b01dc872b4ebb8f65b586b29eea00 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d22b01dc872b4ebb8f65b586b29eea00 already in progress: creating tablet
I20251029 02:52:20.432654 1176 ts_tablet_manager.cc:1916] T 628a7d0419cb434f8171faf500d585d8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:20.451669 1034 tablet_service.cc:1505] Processing CreateTablet for tablet e42548161c3641f6b446821c5f946348 (DEFAULT_TABLE table=default.Table_3e74debf297b4d7cb2f35a8a9bdf9155 [id=a990d28876df46d39f6071d0542e1474]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:20.452126 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e42548161c3641f6b446821c5f946348. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:20.453655 1031 tablet_service.cc:1505] Processing CreateTablet for tablet 2d01627906b444dc8386191004e650e2 (DEFAULT_TABLE table=default.Table_a68b9a5d4e444e579e08cc644d3aee8c [id=c040966044af42d0a3715df461398ed5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:20.454058 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d01627906b444dc8386191004e650e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:20.462805 887 tablet_service.cc:1505] Processing CreateTablet for tablet e42548161c3641f6b446821c5f946348 (DEFAULT_TABLE table=default.Table_3e74debf297b4d7cb2f35a8a9bdf9155 [id=a990d28876df46d39f6071d0542e1474]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:20.463126 3009 raft_consensus.cc:493] T abf18e06df9b482a8b25c078b9495272 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:20.463208 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e42548161c3641f6b446821c5f946348. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:20.463255 3009 raft_consensus.cc:515] T abf18e06df9b482a8b25c078b9495272 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:20.464102 3009 leader_election.cc:290] T abf18e06df9b482a8b25c078b9495272 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:52:20.468055 1166 tablet_bootstrap.cc:492] T 1df0d944717d487a98ea83a8ec04bd46 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:20.468223 1166 ts_tablet_manager.cc:1403] T 1df0d944717d487a98ea83a8ec04bd46 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.063s user 0.003s sys 0.000s
I20251029 02:52:20.468740 520 catalog_manager.cc:5649] T 08371b3daa184a0e9f4eb3d313b52c56 P c756528642a543eeacd93cef3e335a7d reported cstate change: term changed from 0 to 1, leader changed from <none> to c756528642a543eeacd93cef3e335a7d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "c756528642a543eeacd93cef3e335a7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } health_report { overall_health: UNKNOWN } } }
I20251029 02:52:20.469000 1166 raft_consensus.cc:359] T 1df0d944717d487a98ea83a8ec04bd46 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:20.469170 1166 raft_consensus.cc:385] T 1df0d944717d487a98ea83a8ec04bd46 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:20.469213 1166 raft_consensus.cc:740] T 1df0d944717d487a98ea83a8ec04bd46 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:20.469236 520 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet 08371b3daa184a0e9f4eb3d313b52c56 (table default.Table_1453b5168e64406688903f1d6927f549 [id=40f1beecaad24a64b4348663c9ae9afd]). Expected version 1 got 0
I20251029 02:52:20.469439 1166 consensus_queue.cc:260] T 1df0d944717d487a98ea83a8ec04bd46 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:20.470114 1166 ts_tablet_manager.cc:1434] T 1df0d944717d487a98ea83a8ec04bd46 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251029 02:52:20.470425 1166 tablet_bootstrap.cc:492] T 2b278db5b2b04344b864797055a5f665 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:20.470423 888 tablet_service.cc:1505] Processing CreateTablet for tablet 95ec3582881445ed8a4f09494666463c (DEFAULT_TABLE table=default.Table_57d5826b7f1a4a71923282eae27c2e14 [id=708a216641144e178513efc0357e031e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:20.470911 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95ec3582881445ed8a4f09494666463c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:20.471359 1167 tablet_bootstrap.cc:492] T e3ac6597d52441569a7aeef051087ad2 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:20.471644 1031 tablet_service.cc:1552] Processing DeleteTablet for tablet b0fec06ad37d4c12a7045d424b88cc5e with delete_type TABLET_DATA_DELETED (replaced by tablet 4eec207d19f540518bb567d7ab21d048) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:20.472090 1167 ts_tablet_manager.cc:1403] T e3ac6597d52441569a7aeef051087ad2 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.222s user 0.022s sys 0.026s
I20251029 02:52:20.472915 1167 raft_consensus.cc:359] T e3ac6597d52441569a7aeef051087ad2 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:20.473094 1167 raft_consensus.cc:385] T e3ac6597d52441569a7aeef051087ad2 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:20.474535 1166 tablet_bootstrap.cc:654] T 2b278db5b2b04344b864797055a5f665 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:20.479600 1167 raft_consensus.cc:740] T e3ac6597d52441569a7aeef051087ad2 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:20.480063 1167 consensus_queue.cc:260] T e3ac6597d52441569a7aeef051087ad2 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:20.483410 916 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "abf18e06df9b482a8b25c078b9495272" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:20.486609 1172 tablet_bootstrap.cc:492] T da904606986f452cadd06187fff06785 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:20.486920 767 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "abf18e06df9b482a8b25c078b9495272" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:20.487511 989 leader_election.cc:304] T abf18e06df9b482a8b25c078b9495272 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5; no voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:20.487849 3009 raft_consensus.cc:2749] T abf18e06df9b482a8b25c078b9495272 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:20.491274 1062 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "cc802bda2f6a4e439b605f2cafa37967" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:20.491565 1062 raft_consensus.cc:2376] T cc802bda2f6a4e439b605f2cafa37967 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1. Re-sending same reply.
I20251029 02:52:20.492916 1167 ts_tablet_manager.cc:1434] T e3ac6597d52441569a7aeef051087ad2 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.021s user 0.001s sys 0.001s
I20251029 02:52:20.492547 739 tablet_service.cc:1505] Processing CreateTablet for tablet bb2a0b13fddd40d4b6b90e845f8c874c (DEFAULT_TABLE table=default.Table_8c6777c5195142f8b08409b19267c331 [id=e29e88f07ca24a4987cf76b8e3de7087]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:20.493310 1167 tablet_bootstrap.cc:492] T 688af6580c4d4637a39f817b4e324618 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:20.493541 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb2a0b13fddd40d4b6b90e845f8c874c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:20.495544 748 tablet_service.cc:1505] Processing CreateTablet for tablet 95ec3582881445ed8a4f09494666463c (DEFAULT_TABLE table=default.Table_57d5826b7f1a4a71923282eae27c2e14 [id=708a216641144e178513efc0357e031e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:20.495985 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95ec3582881445ed8a4f09494666463c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:20.496732 1167 tablet_bootstrap.cc:654] T 688af6580c4d4637a39f817b4e324618 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:20.498715 754 tablet_service.cc:1505] Processing CreateTablet for tablet e42548161c3641f6b446821c5f946348 (DEFAULT_TABLE table=default.Table_3e74debf297b4d7cb2f35a8a9bdf9155 [id=a990d28876df46d39f6071d0542e1474]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:20.499104 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e42548161c3641f6b446821c5f946348. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:20.503568 1028 tablet_service.cc:1505] Processing CreateTablet for tablet 95ec3582881445ed8a4f09494666463c (DEFAULT_TABLE table=default.Table_57d5826b7f1a4a71923282eae27c2e14 [id=708a216641144e178513efc0357e031e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:20.504105 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95ec3582881445ed8a4f09494666463c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:20.508008 1172 ts_tablet_manager.cc:1403] T da904606986f452cadd06187fff06785 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.221s user 0.002s sys 0.002s
W20251029 02:52:20.510207 519 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9426756257c64db982d1e879c0c0f8a9 consensus state has no leader
W20251029 02:52:20.510785 518 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ffe664eab28c4aa88ac299c5b5e66410 consensus state has no leader
I20251029 02:52:20.511160 518 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_126f4f49037f46628fa5e1d33b4fcf24" } modify_external_catalogs: true
I20251029 02:52:20.511351 518 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_126f4f49037f46628fa5e1d33b4fcf24" } modify_external_catalogs: true
I20251029 02:52:20.513399 517 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 9e0d410678594cbaa78bd7f56c3cb40e
I20251029 02:52:20.514212 517 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_c873cfe841d04574a044e4796fcb3ea7" } modify_external_catalogs: true
I20251029 02:52:20.514492 517 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_c873cfe841d04574a044e4796fcb3ea7" } modify_external_catalogs: true
I20251029 02:52:20.515713 1172 raft_consensus.cc:359] T da904606986f452cadd06187fff06785 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:20.515995 1172 raft_consensus.cc:385] T da904606986f452cadd06187fff06785 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:20.516070 1172 raft_consensus.cc:740] T da904606986f452cadd06187fff06785 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:20.516285 1172 consensus_queue.cc:260] T da904606986f452cadd06187fff06785 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:20.518229 1172 ts_tablet_manager.cc:1434] T da904606986f452cadd06187fff06785 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.010s user 0.001s sys 0.000s
I20251029 02:52:20.518412 772 raft_consensus.cc:2468] T 023a4094b424464c8f49d6eb9cc15da4 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
W20251029 02:52:20.518924 525 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fb63ec8e38fe4cbb8df2065b760c9112 consensus state has no leader
I20251029 02:52:20.519248 525 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_93495adef2ff4b039f9aebdb6c748b49" } modify_external_catalogs: true
I20251029 02:52:20.519383 525 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_93495adef2ff4b039f9aebdb6c748b49" } modify_external_catalogs: true
I20251029 02:52:20.521400 1042 tablet_service.cc:1552] Processing DeleteTablet for tablet 9e0d410678594cbaa78bd7f56c3cb40e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:20 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:20.525799 1175 ts_tablet_manager.cc:1929] T fcb3db8399aa4cf188b07dda220f7f5b P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:20.526026 1175 log.cc:1199] T fcb3db8399aa4cf188b07dda220f7f5b P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/fcb3db8399aa4cf188b07dda220f7f5b
I20251029 02:52:20.529481 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_fc1bf10095e44b669ac62ca092fb511c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:20.530160 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fc1bf10095e44b669ac62ca092fb511c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:20.531217 750 tablet_service.cc:1552] Processing DeleteTablet for tablet 9e0d410678594cbaa78bd7f56c3cb40e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:20 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:20.531957 1175 ts_tablet_manager.cc:1950] T fcb3db8399aa4cf188b07dda220f7f5b P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:20.534435 3041 raft_consensus.cc:493] T 7fb5b908db77453da5cfb32e2c955f41 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:20.534663 3041 raft_consensus.cc:515] T 7fb5b908db77453da5cfb32e2c955f41 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:20.535284 3041 leader_election.cc:290] T 7fb5b908db77453da5cfb32e2c955f41 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:20.536787 847 leader_election.cc:304] T 023a4094b424464c8f49d6eb9cc15da4 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5
I20251029 02:52:20.537118 3039 raft_consensus.cc:2804] T 023a4094b424464c8f49d6eb9cc15da4 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:20.537312 3039 raft_consensus.cc:697] T 023a4094b424464c8f49d6eb9cc15da4 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:52:20.537689 1174 log.cc:1199] T 82289e0f823c4d43bfd8c4e878572b2b P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/82289e0f823c4d43bfd8c4e878572b2b
I20251029 02:52:20.537632 3039 consensus_queue.cc:237] T 023a4094b424464c8f49d6eb9cc15da4 P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:20.538080 1174 ts_tablet_manager.cc:1950] T 82289e0f823c4d43bfd8c4e878572b2b P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:20.538929 887 tablet_service.cc:1505] Processing CreateTablet for tablet 2d01627906b444dc8386191004e650e2 (DEFAULT_TABLE table=default.Table_a68b9a5d4e444e579e08cc644d3aee8c [id=c040966044af42d0a3715df461398ed5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:20.539183 1172 tablet_bootstrap.cc:492] T 68b1c4f88aee4e4795397d1c796ff948 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:20.539533 1042 tablet_service.cc:1505] Processing CreateTablet for tablet a13c01e7b336457c9eadf9601f75cdb3 (DEFAULT_TABLE table=default.Table_52063b7a3e27479caf9d097f7eecd731 [id=dcecf4dc973d4c31b732eddc0a34f93b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:20.539935 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a13c01e7b336457c9eadf9601f75cdb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:20.540043 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 1107) took 5674 ms. Trace:
I20251029 02:52:20.540130 1175 rpcz_store.cc:276] 1029 02:52:14.865198 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:14.865265 (+ 67us) service_pool.cc:225] Handling call
1029 02:52:20.369970 (+5504705us) ts_tablet_manager.cc:1151] Deleting tablet fcb3db8399aa4cf188b07dda220f7f5b
1029 02:52:20.369979 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:20.503663 (+133684us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:20.525682 (+ 22019us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:20.540025 (+ 14343us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5504491}
I20251029 02:52:20.540364 522 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 6dff036b32904ddb9598be4b79504bb6
I20251029 02:52:20.540462 1175 tablet_replica.cc:333] T 628a7d0419cb434f8171faf500d585d8 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:20.540666 1175 raft_consensus.cc:2243] T 628a7d0419cb434f8171faf500d585d8 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:20.540814 1175 raft_consensus.cc:2272] T 628a7d0419cb434f8171faf500d585d8 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:20.541061 522 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_735539ed48814bbdb84a3c3e6390191d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:20.541339 1175 ts_tablet_manager.cc:1916] T 628a7d0419cb434f8171faf500d585d8 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 02:52:20.541579 522 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_735539ed48814bbdb84a3c3e6390191d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:52:20.542623 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.3:45003, user_credentials={real_user=slave}} blocked reactor thread for 52628.9us
I20251029 02:52:20.544477 1031 tablet_service.cc:1552] Processing DeleteTablet for tablet 6dff036b32904ddb9598be4b79504bb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:20 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:20.545800 895 tablet_service.cc:1552] Processing DeleteTablet for tablet 6dff036b32904ddb9598be4b79504bb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:20 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:20.548964 1031 tablet_service.cc:1505] Processing CreateTablet for tablet a823b9682e424451a2f15b94ea990441 (DEFAULT_TABLE table=default.Table_46f6e47207184dedad719ac68774c0b7 [id=c51d93e10fe4423a8cea0360059a8418]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:20.549508 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a823b9682e424451a2f15b94ea990441. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:20.550074 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet fcb3db8399aa4cf188b07dda220f7f5b (table default.Table_33eacbf00740465185d70b46a3782893 [id=22e43c4483ae488b81fb94a7d221bdce]) successfully deleted
I20251029 02:52:20.550364 895 tablet_service.cc:1505] Processing CreateTablet for tablet a13c01e7b336457c9eadf9601f75cdb3 (DEFAULT_TABLE table=default.Table_52063b7a3e27479caf9d097f7eecd731 [id=dcecf4dc973d4c31b732eddc0a34f93b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:20.550997 879 tablet_service.cc:1505] Processing CreateTablet for tablet a823b9682e424451a2f15b94ea990441 (DEFAULT_TABLE table=default.Table_46f6e47207184dedad719ac68774c0b7 [id=c51d93e10fe4423a8cea0360059a8418]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:20.553184 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d01627906b444dc8386191004e650e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:20.555207 1166 tablet_bootstrap.cc:492] T 2b278db5b2b04344b864797055a5f665 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:20.555362 1166 ts_tablet_manager.cc:1403] T 2b278db5b2b04344b864797055a5f665 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.085s user 0.006s sys 0.000s
I20251029 02:52:20.556139 1166 raft_consensus.cc:359] T 2b278db5b2b04344b864797055a5f665 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:20.556324 1166 raft_consensus.cc:385] T 2b278db5b2b04344b864797055a5f665 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:20.556384 1166 raft_consensus.cc:740] T 2b278db5b2b04344b864797055a5f665 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:20.556607 1166 consensus_queue.cc:260] T 2b278db5b2b04344b864797055a5f665 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
W20251029 02:52:20.556989 849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:45003, user_credentials={real_user=slave}} blocked reactor thread for 73244.6us
I20251029 02:52:20.557089 1166 ts_tablet_manager.cc:1434] T 2b278db5b2b04344b864797055a5f665 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251029 02:52:20.557529 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a13c01e7b336457c9eadf9601f75cdb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:20.561798 1062 raft_consensus.cc:3060] T abf18e06df9b482a8b25c078b9495272 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:20.562631 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a823b9682e424451a2f15b94ea990441. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:20.567121 1052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7fb5b908db77453da5cfb32e2c955f41" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:20.567442 1176 ts_tablet_manager.cc:1929] T 628a7d0419cb434f8171faf500d585d8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:20.567622 1176 log.cc:1199] T 628a7d0419cb434f8171faf500d585d8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/628a7d0419cb434f8171faf500d585d8
I20251029 02:52:20.568063 1176 ts_tablet_manager.cc:1950] T 628a7d0419cb434f8171faf500d585d8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:20.568517 1176 tablet_replica.cc:333] T 5815626b05204b578cc4a110f246bf10 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:20.568799 1176 raft_consensus.cc:2243] T 5815626b05204b578cc4a110f246bf10 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:20.569023 1176 raft_consensus.cc:2272] T 5815626b05204b578cc4a110f246bf10 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:20.569715 1176 ts_tablet_manager.cc:1916] T 5815626b05204b578cc4a110f246bf10 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:20.562945 750 tablet_service.cc:1505] Processing CreateTablet for tablet 2d01627906b444dc8386191004e650e2 (DEFAULT_TABLE table=default.Table_a68b9a5d4e444e579e08cc644d3aee8c [id=c040966044af42d0a3715df461398ed5]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 02:52:20.571045 849 leader_election.cc:343] T 7fb5b908db77453da5cfb32e2c955f41 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:20.572057 2992 consensus_queue.cc:1048] T b125cc9aaf4248b8bfa48fb152b832a6 P b495a49349324f7d81908f2da700f0d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:20.572237 2987 raft_consensus.cc:493] T 35d9b8dfd61e4d8db441427a49fb9dca P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:20.572367 2987 raft_consensus.cc:515] T 35d9b8dfd61e4d8db441427a49fb9dca P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:20.572556 526 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_46f6e47207184dedad719ac68774c0b7" } new_table_name: "default.Table_c9e9e020cae84cf29ad75a4bf57692b0" modify_external_catalogs: true
I20251029 02:52:20.573428 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d01627906b444dc8386191004e650e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:20.574966 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 628a7d0419cb434f8171faf500d585d8 (table default.Table_31aa9d49c83b4ab29823acea7480bc96 [id=d6b287a41f2c46868c3f207b7d4359ba]) successfully deleted
I20251029 02:52:20.577606 767 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7fb5b908db77453da5cfb32e2c955f41" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:20.577800 767 raft_consensus.cc:2468] T 7fb5b908db77453da5cfb32e2c955f41 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:20.579722 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 1070) took 6953 ms. Trace:
I20251029 02:52:20.579887 1174 rpcz_store.cc:276] 1029 02:52:13.626097 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:13.626395 (+ 298us) service_pool.cc:225] Handling call
1029 02:52:20.257472 (+6631077us) ts_tablet_manager.cc:1151] Deleting tablet 82289e0f823c4d43bfd8c4e878572b2b
1029 02:52:20.257484 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:20.295032 (+ 37548us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:20.340579 (+ 45547us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:20.579702 (+239123us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6630771}
I20251029 02:52:20.580358 1174 tablet_replica.cc:333] T d041e87ceb11441983d0e38b7929c3a4 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:20.580588 1174 raft_consensus.cc:2243] T d041e87ceb11441983d0e38b7929c3a4 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:20.581043 1174 raft_consensus.cc:2272] T d041e87ceb11441983d0e38b7929c3a4 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:20.581705 1174 ts_tablet_manager.cc:1916] T d041e87ceb11441983d0e38b7929c3a4 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:20.583503 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 82289e0f823c4d43bfd8c4e878572b2b (table default.Table_e7bb3e262a894c2f84b6edfc3917b30d [id=2cca3b7c50b243b5883b8ac1f353f253]) successfully deleted
I20251029 02:52:20.586894 2987 leader_election.cc:290] T 35d9b8dfd61e4d8db441427a49fb9dca P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:20.587945 1166 tablet_bootstrap.cc:492] T 506f8c112a3b43e1a3d6bbafb8cb81e8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:20.589692 1166 tablet_bootstrap.cc:654] T 506f8c112a3b43e1a3d6bbafb8cb81e8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:20.597071 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_c0fb39915dbb4a9bb77c515f63ad8135"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:20.597874 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c0fb39915dbb4a9bb77c515f63ad8135 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:20.599303 1052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "35d9b8dfd61e4d8db441427a49fb9dca" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:52:20.599855 712 leader_election.cc:343] T 35d9b8dfd61e4d8db441427a49fb9dca P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:20.600566 524 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet da904606986f452cadd06187fff06785 (table default.Table_3c0a0c360c4545dab0a663809e2d0acc [id=24f8ffe28d81427d9e9c5beb9c938d02]). Expected version 1 got 0
I20251029 02:52:20.601325 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_3dcd8654fea94e34856d42f72f2e8716"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:20.601707 754 tablet_service.cc:1505] Processing CreateTablet for tablet a13c01e7b336457c9eadf9601f75cdb3 (DEFAULT_TABLE table=default.Table_52063b7a3e27479caf9d097f7eecd731 [id=dcecf4dc973d4c31b732eddc0a34f93b]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 02:52:20.601890 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3dcd8654fea94e34856d42f72f2e8716 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:20.602104 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a13c01e7b336457c9eadf9601f75cdb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:20.602705 518 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 93440d660eac4986aeb17abc18144700
I20251029 02:52:20.603367 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_ff9a5e3946f14e29a5fa8aef8480d77b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:20.603914 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ff9a5e3946f14e29a5fa8aef8480d77b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:20.607530 748 tablet_service.cc:1552] Processing DeleteTablet for tablet 6dff036b32904ddb9598be4b79504bb6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:20 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:20.611246 1034 tablet_service.cc:1552] Processing DeleteTablet for tablet 93440d660eac4986aeb17abc18144700 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:20 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:20.611889 519 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet b125cc9aaf4248b8bfa48fb152b832a6 (table default.Table_110e1b64737c4040ada22360d5a373cb [id=3513c69f6b3941849ac5b8ae0a5846d3]). Expected version 5 got 0
I20251029 02:52:20.614065 915 raft_consensus.cc:1275] T 055184c641ba45a588bca611fb24d130 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Refusing update from remote peer b495a49349324f7d81908f2da700f0d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251029 02:52:20.613976 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_662ccc59d7dc44e09289a2ec8fccf684"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:20.614997 756 tablet_service.cc:1505] Processing CreateTablet for tablet a823b9682e424451a2f15b94ea990441 (DEFAULT_TABLE table=default.Table_46f6e47207184dedad719ac68774c0b7 [id=c51d93e10fe4423a8cea0360059a8418]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 02:52:20.615072 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_662ccc59d7dc44e09289a2ec8fccf684 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:20.616750 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a823b9682e424451a2f15b94ea990441. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:20.617648 1034 tablet_service.cc:1552] Processing DeleteTablet for tablet cc802bda2f6a4e439b605f2cafa37967 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:57 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:20.621006 1166 tablet_bootstrap.cc:492] T 506f8c112a3b43e1a3d6bbafb8cb81e8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:20.621137 1166 ts_tablet_manager.cc:1403] T 506f8c112a3b43e1a3d6bbafb8cb81e8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.033s user 0.003s sys 0.000s
I20251029 02:52:20.621156 1039 tablet_service.cc:1552] Processing DeleteTablet for tablet 1df0d944717d487a98ea83a8ec04bd46 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:01 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:20.621518 900 tablet_service.cc:1552] Processing DeleteTablet for tablet 93440d660eac4986aeb17abc18144700 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:20 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:20.621776 891 tablet_service.cc:1552] Processing DeleteTablet for tablet 2b278db5b2b04344b864797055a5f665 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:00 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:20.621867 1166 raft_consensus.cc:359] T 506f8c112a3b43e1a3d6bbafb8cb81e8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:20.621953 888 tablet_service.cc:1552] Processing DeleteTablet for tablet 9e0d410678594cbaa78bd7f56c3cb40e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:20 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:20.622023 1166 raft_consensus.cc:385] T 506f8c112a3b43e1a3d6bbafb8cb81e8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:20.622071 1166 raft_consensus.cc:740] T 506f8c112a3b43e1a3d6bbafb8cb81e8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:20.622310 1166 consensus_queue.cc:260] T 506f8c112a3b43e1a3d6bbafb8cb81e8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:20.622951 1166 ts_tablet_manager.cc:1434] T 506f8c112a3b43e1a3d6bbafb8cb81e8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251029 02:52:20.623235 1166 tablet_bootstrap.cc:492] T f93b0c39d7174dabbe018b1546dbc198 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:20.624882 1166 tablet_bootstrap.cc:654] T f93b0c39d7174dabbe018b1546dbc198 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:20.625627 748 tablet_service.cc:1552] Processing DeleteTablet for tablet 93440d660eac4986aeb17abc18144700 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:20 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:20.627552 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "35d9b8dfd61e4d8db441427a49fb9dca" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:20.628412 849 consensus_peers.cc:597] T 08371b3daa184a0e9f4eb3d313b52c56 P c756528642a543eeacd93cef3e335a7d -> Peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Couldn't send request to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:20.628551 2986 consensus_queue.cc:1048] T 055184c641ba45a588bca611fb24d130 P b495a49349324f7d81908f2da700f0d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:20.630769 2986 raft_consensus.cc:493] T da904606986f452cadd06187fff06785 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:20.630905 2986 raft_consensus.cc:515] T da904606986f452cadd06187fff06785 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:20.632184 1176 ts_tablet_manager.cc:1929] T 5815626b05204b578cc4a110f246bf10 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:20.633846 1176 log.cc:1199] T 5815626b05204b578cc4a110f246bf10 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/5815626b05204b578cc4a110f246bf10
W20251029 02:52:20.634598 711 leader_election.cc:343] T 35d9b8dfd61e4d8db441427a49fb9dca P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:20.634716 711 leader_election.cc:304] T 35d9b8dfd61e4d8db441427a49fb9dca P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:20.635159 2987 raft_consensus.cc:2749] T 35d9b8dfd61e4d8db441427a49fb9dca P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:20.635634 900 tablet_service.cc:1552] Processing DeleteTablet for tablet e3000183230e4350b47f5085952f2db6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:01 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:20.636022 888 tablet_service.cc:1552] Processing DeleteTablet for tablet 294afb9b012c40c0b05b73795d6f0ac8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:59 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:20.636319 891 tablet_service.cc:1552] Processing DeleteTablet for tablet 023a4094b424464c8f49d6eb9cc15da4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:51:58 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:20.638366 2986 leader_election.cc:290] T da904606986f452cadd06187fff06785 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:52:20.638847 1176 ts_tablet_manager.cc:1950] T 5815626b05204b578cc4a110f246bf10 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:20.639025 1052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da904606986f452cadd06187fff06785" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:20.638865 1031 tablet_service.cc:1505] Processing CreateTablet for tablet 960c8df2b26d4d3381906e770cfc9f04 (DEFAULT_TABLE table=default.Table_0599b691b9134e749b5d3b952686b828 [id=898d292c25dc45a0a601506c5876513d]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 02:52:20.643187 712 leader_election.cc:343] T da904606986f452cadd06187fff06785 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:20.643309 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 960c8df2b26d4d3381906e770cfc9f04. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:20.643482 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 5815626b05204b578cc4a110f246bf10 (table default.Table_796f8f19849d4b01922da96435e6833c [id=45dc803b4e1041e78f40a3e2e1369041]) successfully deleted
I20251029 02:52:20.643839 1176 tablet_replica.cc:333] T 24f07b1efac64b099b0b5ed467463749 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
W20251029 02:52:20.643858 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 8c972c88f64e486bb2b75306549bb23f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8c972c88f64e486bb2b75306549bb23f already in progress: creating tablet
W20251029 02:52:20.644276 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet f72dae5a0e604eb384c1f5f06fe680bb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f72dae5a0e604eb384c1f5f06fe680bb already in progress: creating tablet
W20251029 02:52:20.644652 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet d8c66a892c90475c834f08c0734f41db because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d8c66a892c90475c834f08c0734f41db already in progress: creating tablet
I20251029 02:52:20.648133 1176 raft_consensus.cc:2243] T 24f07b1efac64b099b0b5ed467463749 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:20.648397 1176 raft_consensus.cc:2272] T 24f07b1efac64b099b0b5ed467463749 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:20.648644 919 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da904606986f452cadd06187fff06785" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:20.649328 711 leader_election.cc:343] T da904606986f452cadd06187fff06785 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:20.649468 711 leader_election.cc:304] T da904606986f452cadd06187fff06785 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:20.649886 2986 raft_consensus.cc:2749] T da904606986f452cadd06187fff06785 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:20.650980 1176 ts_tablet_manager.cc:1916] T 24f07b1efac64b099b0b5ed467463749 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:20.651391 740 tablet_service.cc:1505] Processing CreateTablet for tablet 960c8df2b26d4d3381906e770cfc9f04 (DEFAULT_TABLE table=default.Table_0599b691b9134e749b5d3b952686b828 [id=898d292c25dc45a0a601506c5876513d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:20.651823 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 960c8df2b26d4d3381906e770cfc9f04. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:20.652784 748 tablet_service.cc:1505] Processing CreateTablet for tablet 746d3895ddd6413da478641a68950876 (DEFAULT_TABLE table=default.Table_46e4796809474bca91dc44012fa5f9f6 [id=abfae208d0484c07ac4c2d2af7703f69]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:20.654747 754 tablet_service.cc:1505] Processing CreateTablet for tablet c7ff396a667a4a10a34c111806bf5aa7 (DEFAULT_TABLE table=default.Table_08c8b82cbf9a424c8514bea1a6324fb1 [id=de904db373d742ea9fcdd7aac63357bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:20.650336 520 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet da904606986f452cadd06187fff06785 (table default.Table_3c0a0c360c4545dab0a663809e2d0acc [id=24f8ffe28d81427d9e9c5beb9c938d02]). Expected version 1 got 0
I20251029 02:52:20.657796 900 tablet_service.cc:1505] Processing CreateTablet for tablet c7ff396a667a4a10a34c111806bf5aa7 (DEFAULT_TABLE table=default.Table_08c8b82cbf9a424c8514bea1a6324fb1 [id=de904db373d742ea9fcdd7aac63357bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:20.658274 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7ff396a667a4a10a34c111806bf5aa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:20.661641 517 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 44c5224ef3444516a675c23662d34bec
I20251029 02:52:20.661775 1172 tablet_bootstrap.cc:654] T 68b1c4f88aee4e4795397d1c796ff948 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:20.661826 888 tablet_service.cc:1505] Processing CreateTablet for tablet 746d3895ddd6413da478641a68950876 (DEFAULT_TABLE table=default.Table_46e4796809474bca91dc44012fa5f9f6 [id=abfae208d0484c07ac4c2d2af7703f69]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:20.662182 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 746d3895ddd6413da478641a68950876. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:20.662331 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_9f561a4a08bf48ccb6cf792e9d1a1b8a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:20.662878 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9f561a4a08bf48ccb6cf792e9d1a1b8a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:20.663928 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 746d3895ddd6413da478641a68950876. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:20.664539 891 tablet_service.cc:1505] Processing CreateTablet for tablet 960c8df2b26d4d3381906e770cfc9f04 (DEFAULT_TABLE table=default.Table_0599b691b9134e749b5d3b952686b828 [id=898d292c25dc45a0a601506c5876513d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:20.665304 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 960c8df2b26d4d3381906e770cfc9f04. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:20.665930 887 tablet_service.cc:1552] Processing DeleteTablet for tablet 44c5224ef3444516a675c23662d34bec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:20 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:20.666848 756 tablet_service.cc:1552] Processing DeleteTablet for tablet 44c5224ef3444516a675c23662d34bec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:20 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:20.668251 3044 raft_consensus.cc:493] T 372d93914abc41b38a702742b7db0bc2 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0cfd4bee16f84b7c8b0ff3ce14db20d5)
I20251029 02:52:20.668387 3044 raft_consensus.cc:515] T 372d93914abc41b38a702742b7db0bc2 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:20.667702 520 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet b125cc9aaf4248b8bfa48fb152b832a6 (table default.Table_110e1b64737c4040ada22360d5a373cb [id=3513c69f6b3941849ac5b8ae0a5846d3]). Expected version 5 got 0
I20251029 02:52:20.669891 756 tablet_service.cc:1505] Processing CreateTablet for tablet daf4973adcce44d7a1d3d3489926a868 (DEFAULT_TABLE table=default.Table_5a4b5d7211d4473f9bc57d3198c1e87d [id=d76bd4acdbc54d5b82f2257cf9fa9861]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:20.670234 3049 raft_consensus.cc:493] T c45ffc304e73490387ce800b702b5881 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:20.670359 3049 raft_consensus.cc:515] T c45ffc304e73490387ce800b702b5881 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:20.671335 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7ff396a667a4a10a34c111806bf5aa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:20.670650 3044 leader_election.cc:290] T 372d93914abc41b38a702742b7db0bc2 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:20.672255 887 tablet_service.cc:1505] Processing CreateTablet for tablet daf4973adcce44d7a1d3d3489926a868 (DEFAULT_TABLE table=default.Table_5a4b5d7211d4473f9bc57d3198c1e87d [id=d76bd4acdbc54d5b82f2257cf9fa9861]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:20.674249 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet daf4973adcce44d7a1d3d3489926a868. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:20.680459 525 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 96e7e1b0f14e4e738e7b2d0eaf8d53bb
I20251029 02:52:20.681667 750 tablet_service.cc:1552] Processing DeleteTablet for tablet 96e7e1b0f14e4e738e7b2d0eaf8d53bb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:20 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:20.683259 3049 leader_election.cc:290] T c45ffc304e73490387ce800b702b5881 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:20.684391 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet daf4973adcce44d7a1d3d3489926a868. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:20.685781 895 tablet_service.cc:1552] Processing DeleteTablet for tablet 96e7e1b0f14e4e738e7b2d0eaf8d53bb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:20 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:20.686029 3049 raft_consensus.cc:493] T 2b278db5b2b04344b864797055a5f665 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:20.686841 3049 raft_consensus.cc:515] T 2b278db5b2b04344b864797055a5f665 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:20.687485 3049 leader_election.cc:290] T 2b278db5b2b04344b864797055a5f665 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:20.688469 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_fb7c42a3ad74451d99aa21aa08c6845c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:20.689105 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fb7c42a3ad74451d99aa21aa08c6845c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:20.690627 847 leader_election.cc:304] T 7fb5b908db77453da5cfb32e2c955f41 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5
I20251029 02:52:20.691063 3049 raft_consensus.cc:2804] T 7fb5b908db77453da5cfb32e2c955f41 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:20.691236 3049 raft_consensus.cc:493] T 7fb5b908db77453da5cfb32e2c955f41 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:20.691650 3049 raft_consensus.cc:3060] T 7fb5b908db77453da5cfb32e2c955f41 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:20.691215 3044 consensus_queue.cc:1048] T b0fec06ad37d4c12a7045d424b88cc5e P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:20.692962 1167 tablet_bootstrap.cc:492] T 688af6580c4d4637a39f817b4e324618 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:20.693576 1052 raft_consensus.cc:3060] T 055184c641ba45a588bca611fb24d130 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:20.694209 1028 tablet_service.cc:1505] Processing CreateTablet for tablet 746d3895ddd6413da478641a68950876 (DEFAULT_TABLE table=default.Table_46e4796809474bca91dc44012fa5f9f6 [id=abfae208d0484c07ac4c2d2af7703f69]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:20.695897 520 master_service.cc:954] ReplaceTablet: received request to replace tablet 6fcf3e12810d41ddb96d9f2cf52f2c5d from {username='slave'} at 127.0.0.1:35314
I20251029 02:52:20.696172 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 746d3895ddd6413da478641a68950876. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:20.697103 1039 tablet_service.cc:1552] Processing DeleteTablet for tablet 44c5224ef3444516a675c23662d34bec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:20 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:20.697480 1039 tablet_service.cc:1505] Processing CreateTablet for tablet daf4973adcce44d7a1d3d3489926a868 (DEFAULT_TABLE table=default.Table_5a4b5d7211d4473f9bc57d3198c1e87d [id=d76bd4acdbc54d5b82f2257cf9fa9861]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:20.697844 1034 tablet_service.cc:1552] Processing DeleteTablet for tablet 96e7e1b0f14e4e738e7b2d0eaf8d53bb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:20 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:20.699020 1052 raft_consensus.cc:1275] T 055184c641ba45a588bca611fb24d130 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Refusing update from remote peer b495a49349324f7d81908f2da700f0d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251029 02:52:20.699075 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet daf4973adcce44d7a1d3d3489926a868. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:20.696666 1031 tablet_service.cc:1505] Processing CreateTablet for tablet c7ff396a667a4a10a34c111806bf5aa7 (DEFAULT_TABLE table=default.Table_08c8b82cbf9a424c8514bea1a6324fb1 [id=de904db373d742ea9fcdd7aac63357bc]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 02:52:20.700417 849 leader_election.cc:343] T 372d93914abc41b38a702742b7db0bc2 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Not found: Tablet not found: 372d93914abc41b38a702742b7db0bc2
I20251029 02:52:20.701344 1167 ts_tablet_manager.cc:1403] T 688af6580c4d4637a39f817b4e324618 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.208s user 0.006s sys 0.010s
I20251029 02:52:20.703173 2992 consensus_queue.cc:1048] T 055184c641ba45a588bca611fb24d130 P b495a49349324f7d81908f2da700f0d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:20.704169 766 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c45ffc304e73490387ce800b702b5881" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:20.704342 766 raft_consensus.cc:2468] T c45ffc304e73490387ce800b702b5881 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:20.704674 767 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b278db5b2b04344b864797055a5f665" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:20.704804 767 raft_consensus.cc:2468] T 2b278db5b2b04344b864797055a5f665 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:20.707837 3049 raft_consensus.cc:515] T 7fb5b908db77453da5cfb32e2c955f41 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:20.708714 3049 leader_election.cc:290] T 7fb5b908db77453da5cfb32e2c955f41 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:20.709978 754 tablet_service.cc:1505] Processing CreateTablet for tablet c91d8ff1ae0e47bbb7328ca89af39d8a (DEFAULT_TABLE table=default.Table_06ef187658064f3d9f471ac2b1d40245 [id=7336a2cea1234935a024155fe46d6b75]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:20.710860 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c91d8ff1ae0e47bbb7328ca89af39d8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:20.711380 1174 ts_tablet_manager.cc:1929] T d041e87ceb11441983d0e38b7929c3a4 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:20.711498 1174 log.cc:1199] T d041e87ceb11441983d0e38b7929c3a4 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/d041e87ceb11441983d0e38b7929c3a4
I20251029 02:52:20.711848 1174 ts_tablet_manager.cc:1950] T d041e87ceb11441983d0e38b7929c3a4 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:20.712154 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 1072) took 6992 ms. Trace:
I20251029 02:52:20.712240 1174 rpcz_store.cc:276] 1029 02:52:13.719433 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:13.719644 (+ 211us) service_pool.cc:225] Handling call
1029 02:52:20.580311 (+6860667us) ts_tablet_manager.cc:1151] Deleting tablet d041e87ceb11441983d0e38b7929c3a4
1029 02:52:20.580321 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:20.625366 (+ 45045us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:20.711265 (+ 85899us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:20.712139 (+ 874us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6860511}
I20251029 02:52:20.712620 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet d041e87ceb11441983d0e38b7929c3a4 (table default.Table_6a78c7ef966c490aaa5e73f56d6d194e [id=c54b92541c3047c0887c36d31743e71c]) successfully deleted
I20251029 02:52:20.700011 1059 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c45ffc304e73490387ce800b702b5881" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:20.714866 1034 tablet_service.cc:1505] Processing CreateTablet for tablet c91d8ff1ae0e47bbb7328ca89af39d8a (DEFAULT_TABLE table=default.Table_06ef187658064f3d9f471ac2b1d40245 [id=7336a2cea1234935a024155fe46d6b75]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 02:52:20.724090 849 leader_election.cc:343] T c45ffc304e73490387ce800b702b5881 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:20.724771 521 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_5557b4fcc4f34c0cacd4af39e9d24e35" } new_table_name: "default.Table_92d77f2b41c14d11afd04421adcdea45" modify_external_catalogs: true
I20251029 02:52:20.726310 772 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7fb5b908db77453da5cfb32e2c955f41" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:20.726652 772 raft_consensus.cc:3060] T 7fb5b908db77453da5cfb32e2c955f41 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:20.727591 1027 tablet_service.cc:1552] Processing DeleteTablet for tablet 2b278db5b2b04344b864797055a5f665 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:00 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:20.727820 1062 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b278db5b2b04344b864797055a5f665" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:20.728036 1062 raft_consensus.cc:2468] T 2b278db5b2b04344b864797055a5f665 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:20.729672 1062 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7fb5b908db77453da5cfb32e2c955f41" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:52:20.731439 1027 tablet_service.cc:1505] Processing CreateTablet for tablet b5d4557ad3ec41d28f5c9429ce0b9cdf (DEFAULT_TABLE table=default.Table_ae6b1788f8824096bd72ee8ec2ddcb15 [id=f9a30ed259fa4ce08ba390aa7e2d27ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:20.730499 1172 tablet_bootstrap.cc:492] T 68b1c4f88aee4e4795397d1c796ff948 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:20.735963 1172 ts_tablet_manager.cc:1403] T 68b1c4f88aee4e4795397d1c796ff948 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.197s user 0.004s sys 0.002s
I20251029 02:52:20.738734 849 leader_election.cc:304] T 2b278db5b2b04344b864797055a5f665 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters:
W20251029 02:52:20.739319 849 leader_election.cc:343] T 7fb5b908db77453da5cfb32e2c955f41 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:20.739578 522 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_cb67eff3fb7c4cd9bc24d6bf31c3cd8f" } new_table_name: "default.Table_855a024bc2b846699debfe6131aa3bc0" modify_external_catalogs: true
I20251029 02:52:20.739874 3044 raft_consensus.cc:2804] T 2b278db5b2b04344b864797055a5f665 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:20.739953 3044 raft_consensus.cc:493] T 2b278db5b2b04344b864797055a5f665 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:20.740010 3044 raft_consensus.cc:3060] T 2b278db5b2b04344b864797055a5f665 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:20.741971 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5d4557ad3ec41d28f5c9429ce0b9cdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:20.742210 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c91d8ff1ae0e47bbb7328ca89af39d8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:20.742688 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7ff396a667a4a10a34c111806bf5aa7. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:20.744340 756 tablet_service.cc:1505] Processing CreateTablet for tablet b5d4557ad3ec41d28f5c9429ce0b9cdf (DEFAULT_TABLE table=default.Table_ae6b1788f8824096bd72ee8ec2ddcb15 [id=f9a30ed259fa4ce08ba390aa7e2d27ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:20.745246 895 tablet_service.cc:1505] Processing CreateTablet for tablet b5d4557ad3ec41d28f5c9429ce0b9cdf (DEFAULT_TABLE table=default.Table_ae6b1788f8824096bd72ee8ec2ddcb15 [id=f9a30ed259fa4ce08ba390aa7e2d27ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:20.745616 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5d4557ad3ec41d28f5c9429ce0b9cdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:20.749732 1176 ts_tablet_manager.cc:1929] T 24f07b1efac64b099b0b5ed467463749 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:20.750326 1176 log.cc:1199] T 24f07b1efac64b099b0b5ed467463749 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/24f07b1efac64b099b0b5ed467463749
I20251029 02:52:20.750991 1176 ts_tablet_manager.cc:1950] T 24f07b1efac64b099b0b5ed467463749 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
W20251029 02:52:20.752184 526 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a823b9682e424451a2f15b94ea990441 consensus state has no leader
I20251029 02:52:20.752579 1176 tablet_replica.cc:333] T b0fec06ad37d4c12a7045d424b88cc5e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:20.752759 1176 raft_consensus.cc:2243] T b0fec06ad37d4c12a7045d424b88cc5e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:20.752907 1176 raft_consensus.cc:2272] T b0fec06ad37d4c12a7045d424b88cc5e P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:20.754813 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5d4557ad3ec41d28f5c9429ce0b9cdf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:20.755219 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 24f07b1efac64b099b0b5ed467463749 (table default.Table_21b8ae2debb241bda65fe387a4a3190f [id=f3fa2dbbef7e47359492d5a5eef9b8fd]) successfully deleted
W20251029 02:52:20.755445 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 266aaea633bf4c059b9c02f092a474be because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 266aaea633bf4c059b9c02f092a474be already in progress: creating tablet
W20251029 02:52:20.755623 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet aef3021f7d684df197eb1cfd5c6b8772 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet aef3021f7d684df197eb1cfd5c6b8772 already in progress: creating tablet
I20251029 02:52:20.757005 526 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_bab353e338ed483e9f685de2637cc326" } modify_external_catalogs: true
I20251029 02:52:20.757185 526 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_bab353e338ed483e9f685de2637cc326" } modify_external_catalogs: true
I20251029 02:52:20.762777 1027 tablet_service.cc:1505] Processing CreateTablet for tablet 0a9e0fc46f4740f297ef25d186a97161 (DEFAULT_TABLE table=default.Table_adbbcf20c8ea4fb691d28bdee6541552 [id=29bee9a25bca41d6ae14a4b423122b71]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:20.763271 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a9e0fc46f4740f297ef25d186a97161. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:20.764057 772 raft_consensus.cc:2468] T 7fb5b908db77453da5cfb32e2c955f41 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:20.765348 879 tablet_service.cc:1505] Processing CreateTablet for tablet 0a9e0fc46f4740f297ef25d186a97161 (DEFAULT_TABLE table=default.Table_adbbcf20c8ea4fb691d28bdee6541552 [id=29bee9a25bca41d6ae14a4b423122b71]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:20.765690 1060 raft_consensus.cc:3060] T 1df0d944717d487a98ea83a8ec04bd46 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:20.765755 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a9e0fc46f4740f297ef25d186a97161. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:20.766204 712 consensus_peers.cc:597] T 5815626b05204b578cc4a110f246bf10 P b495a49349324f7d81908f2da700f0d9 -> Peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Couldn't send request to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5815626b05204b578cc4a110f246bf10. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:20.766647 821 heartbeater.cc:344] Connected to a master server at 127.0.104.62:42343
I20251029 02:52:20.770726 887 tablet_service.cc:1505] Processing CreateTablet for tablet c91d8ff1ae0e47bbb7328ca89af39d8a (DEFAULT_TABLE table=default.Table_06ef187658064f3d9f471ac2b1d40245 [id=7336a2cea1234935a024155fe46d6b75]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:20.771070 1172 raft_consensus.cc:359] T 68b1c4f88aee4e4795397d1c796ff948 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:20.776598 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c91d8ff1ae0e47bbb7328ca89af39d8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:20.777045 1172 raft_consensus.cc:385] T 68b1c4f88aee4e4795397d1c796ff948 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:20.777313 1172 raft_consensus.cc:740] T 68b1c4f88aee4e4795397d1c796ff948 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:20.777808 1172 consensus_queue.cc:260] T 68b1c4f88aee4e4795397d1c796ff948 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:20.780794 1172 ts_tablet_manager.cc:1434] T 68b1c4f88aee4e4795397d1c796ff948 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.043s user 0.002s sys 0.000s
I20251029 02:52:20.784446 518 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_2598d9a14a4e46539f1f775098559c84" } new_table_name: "default.Table_6a50b398719142b396b2165f6981016f" modify_external_catalogs: true
I20251029 02:52:20.799165 2836 mvcc.cc:204] Tried to move back new op lower bound from 7215949170992205824 to 7215949167674966016. Current Snapshot: MvccSnapshot[applied={T|T < 7215949170992205824}]
W20251029 02:52:20.786682 847 leader_election.cc:343] T 372d93914abc41b38a702742b7db0bc2 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 372d93914abc41b38a702742b7db0bc2
I20251029 02:52:20.799990 847 leader_election.cc:304] T 372d93914abc41b38a702742b7db0bc2 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9
I20251029 02:52:20.800412 3039 raft_consensus.cc:2749] T 372d93914abc41b38a702742b7db0bc2 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251029 02:52:20.802831 847 leader_election.cc:304] T c45ffc304e73490387ce800b702b5881 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5
I20251029 02:52:20.803216 3039 raft_consensus.cc:2804] T c45ffc304e73490387ce800b702b5881 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:20.803367 3039 raft_consensus.cc:493] T c45ffc304e73490387ce800b702b5881 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:20.803504 3039 raft_consensus.cc:3060] T c45ffc304e73490387ce800b702b5881 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:20.805318 1174 tablet_replica.cc:333] T 67651306f3b44e7eb14dcee9a2007a81 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:20.806751 1174 raft_consensus.cc:2243] T 67651306f3b44e7eb14dcee9a2007a81 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:20.808064 1174 raft_consensus.cc:2272] T 67651306f3b44e7eb14dcee9a2007a81 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:20.808730 1174 ts_tablet_manager.cc:1916] T 67651306f3b44e7eb14dcee9a2007a81 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:20.778715 754 tablet_service.cc:1505] Processing CreateTablet for tablet 0a9e0fc46f4740f297ef25d186a97161 (DEFAULT_TABLE table=default.Table_adbbcf20c8ea4fb691d28bdee6541552 [id=29bee9a25bca41d6ae14a4b423122b71]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:20.788254 2992 consensus_queue.cc:1048] T 1df0d944717d487a98ea83a8ec04bd46 P b495a49349324f7d81908f2da700f0d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:20.798478 3066 tablet.cc:1722] T 055184c641ba45a588bca611fb24d130 P b495a49349324f7d81908f2da700f0d9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 02:52:20.809923 2986 raft_consensus.cc:493] T 68b1c4f88aee4e4795397d1c796ff948 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:20.813951 2986 raft_consensus.cc:515] T 68b1c4f88aee4e4795397d1c796ff948 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:20.815485 2986 leader_election.cc:290] T 68b1c4f88aee4e4795397d1c796ff948 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:52:20.815606 3068 tablet.cc:1722] T 055184c641ba45a588bca611fb24d130 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 02:52:20.815433 1062 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "68b1c4f88aee4e4795397d1c796ff948" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:52:20.816552 712 leader_election.cc:343] T 68b1c4f88aee4e4795397d1c796ff948 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:20.817289 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0a9e0fc46f4740f297ef25d186a97161. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:20.817595 1172 tablet_bootstrap.cc:492] T c64d6761ef924ccaa9c2579f8ba47e33 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:20.818909 913 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "68b1c4f88aee4e4795397d1c796ff948" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:20.819463 711 leader_election.cc:343] T 68b1c4f88aee4e4795397d1c796ff948 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:20.819599 711 leader_election.cc:304] T 68b1c4f88aee4e4795397d1c796ff948 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:20.821799 2986 raft_consensus.cc:2749] T 68b1c4f88aee4e4795397d1c796ff948 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:20.817327 1175 ts_tablet_manager.cc:1929] T 628a7d0419cb434f8171faf500d585d8 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:20.822674 1175 log.cc:1199] T 628a7d0419cb434f8171faf500d585d8 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/628a7d0419cb434f8171faf500d585d8
I20251029 02:52:20.823253 1175 ts_tablet_manager.cc:1950] T 628a7d0419cb434f8171faf500d585d8 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:20.832499 2992 consensus_queue.cc:1048] T 1df0d944717d487a98ea83a8ec04bd46 P b495a49349324f7d81908f2da700f0d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:20.833055 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 1108) took 5956 ms. Trace:
I20251029 02:52:20.834062 1027 tablet_service.cc:1505] Processing CreateTablet for tablet 2d51e583bfed4277aca102ebb413ec10 (DEFAULT_TABLE table=default.Table_82605703457f430e9edcf160c498ad0c [id=d32947045dd44c518e9819e829b826af]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:20.834563 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d51e583bfed4277aca102ebb413ec10. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:20.840358 1166 tablet_bootstrap.cc:492] T f93b0c39d7174dabbe018b1546dbc198 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:20.840512 1166 ts_tablet_manager.cc:1403] T f93b0c39d7174dabbe018b1546dbc198 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.217s user 0.010s sys 0.007s
I20251029 02:52:20.840898 1175 rpcz_store.cc:276] 1029 02:52:14.876137 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:14.876323 (+ 186us) service_pool.cc:225] Handling call
1029 02:52:20.540422 (+5664099us) ts_tablet_manager.cc:1151] Deleting tablet 628a7d0419cb434f8171faf500d585d8
1029 02:52:20.540430 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:20.709756 (+169326us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:20.817223 (+107467us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:20.833036 (+ 15813us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5663871}
I20251029 02:52:20.841362 1166 raft_consensus.cc:359] T f93b0c39d7174dabbe018b1546dbc198 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:20.841584 1166 raft_consensus.cc:385] T f93b0c39d7174dabbe018b1546dbc198 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:20.841638 1166 raft_consensus.cc:740] T f93b0c39d7174dabbe018b1546dbc198 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:20.841907 1166 consensus_queue.cc:260] T f93b0c39d7174dabbe018b1546dbc198 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:20.842486 1166 ts_tablet_manager.cc:1434] T f93b0c39d7174dabbe018b1546dbc198 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 02:52:20.842830 1166 tablet_bootstrap.cc:492] T 7816556521144c928d7a02694e22e79f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:20.843080 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 628a7d0419cb434f8171faf500d585d8 (table default.Table_31aa9d49c83b4ab29823acea7480bc96 [id=d6b287a41f2c46868c3f207b7d4359ba]) successfully deleted
I20251029 02:52:20.844411 1166 tablet_bootstrap.cc:654] T 7816556521144c928d7a02694e22e79f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:20.846738 523 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet 055184c641ba45a588bca611fb24d130 (table default.Table_1922b14d4cfb4671a6224fcd6ace2d4e [id=aec3db766dee438c89fb2f1939ee7b7b]). Expected version 2 got 0
I20251029 02:52:20.850698 1175 tablet_replica.cc:333] T 078ace5eada044c88d2462efb2b57aa8 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:20.851213 1031 tablet_service.cc:1552] Processing DeleteTablet for tablet 506f8c112a3b43e1a3d6bbafb8cb81e8 with delete_type TABLET_DATA_DELETED (replaced by tablet ef2e9dd8f90a47389d9062d1f99a1da1) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:20.851918 1166 tablet_bootstrap.cc:492] T 7816556521144c928d7a02694e22e79f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:20.852479 1175 raft_consensus.cc:2243] T 078ace5eada044c88d2462efb2b57aa8 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:20.852427 523 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_dbf30facf56548e1b20d6fec42080e05" } modify_external_catalogs: true
I20251029 02:52:20.852603 523 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_dbf30facf56548e1b20d6fec42080e05" } modify_external_catalogs: true
I20251029 02:52:20.855083 1175 raft_consensus.cc:2272] T 078ace5eada044c88d2462efb2b57aa8 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:20.854787 2992 raft_consensus.cc:493] T 877e278deb4a4b01ac66b06be0890baf P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:20.855623 1166 ts_tablet_manager.cc:1403] T 7816556521144c928d7a02694e22e79f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.003s
I20251029 02:52:20.855561 2992 raft_consensus.cc:515] T 877e278deb4a4b01ac66b06be0890baf P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:20.856388 2992 leader_election.cc:290] T 877e278deb4a4b01ac66b06be0890baf P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:20.857156 918 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "877e278deb4a4b01ac66b06be0890baf" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:20.857303 1166 raft_consensus.cc:359] T 7816556521144c928d7a02694e22e79f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:20.857437 1166 raft_consensus.cc:385] T 7816556521144c928d7a02694e22e79f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:20.857718 1167 raft_consensus.cc:359] T 688af6580c4d4637a39f817b4e324618 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:20.857898 1167 raft_consensus.cc:385] T 688af6580c4d4637a39f817b4e324618 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:20.857952 1167 raft_consensus.cc:740] T 688af6580c4d4637a39f817b4e324618 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:20.857488 1166 raft_consensus.cc:740] T 7816556521144c928d7a02694e22e79f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
W20251029 02:52:20.858063 711 leader_election.cc:343] T 877e278deb4a4b01ac66b06be0890baf P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:20.858045 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "877e278deb4a4b01ac66b06be0890baf" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:20.858137 1167 consensus_queue.cc:260] T 688af6580c4d4637a39f817b4e324618 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:20.858351 1166 consensus_queue.cc:260] T 7816556521144c928d7a02694e22e79f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
W20251029 02:52:20.858599 712 leader_election.cc:343] T 877e278deb4a4b01ac66b06be0890baf P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:20.858695 1167 ts_tablet_manager.cc:1434] T 688af6580c4d4637a39f817b4e324618 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.157s user 0.001s sys 0.001s
I20251029 02:52:20.858742 712 leader_election.cc:304] T 877e278deb4a4b01ac66b06be0890baf P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:20.859047 1167 tablet_bootstrap.cc:492] T 9492c1027aba4455ad8f539addaed5a7 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:20.859124 2992 raft_consensus.cc:2749] T 877e278deb4a4b01ac66b06be0890baf P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:20.852720 1172 tablet_bootstrap.cc:654] T c64d6761ef924ccaa9c2579f8ba47e33 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:20.855264 754 tablet_service.cc:1505] Processing CreateTablet for tablet 2d51e583bfed4277aca102ebb413ec10 (DEFAULT_TABLE table=default.Table_82605703457f430e9edcf160c498ad0c [id=d32947045dd44c518e9819e829b826af]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:20.860311 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d51e583bfed4277aca102ebb413ec10. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:20.861282 1175 ts_tablet_manager.cc:1916] T 078ace5eada044c88d2462efb2b57aa8 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:20.861651 1167 tablet_bootstrap.cc:654] T 9492c1027aba4455ad8f539addaed5a7 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:20.862488 847 leader_election.cc:304] T 7fb5b908db77453da5cfb32e2c955f41 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5
I20251029 02:52:20.863560 3048 raft_consensus.cc:2804] T 7fb5b908db77453da5cfb32e2c955f41 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:20.863682 3048 raft_consensus.cc:697] T 7fb5b908db77453da5cfb32e2c955f41 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:52:20.863911 3048 consensus_queue.cc:237] T 7fb5b908db77453da5cfb32e2c955f41 P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:20.865986 1166 ts_tablet_manager.cc:1434] T 7816556521144c928d7a02694e22e79f P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20251029 02:52:20.866708 1166 tablet_bootstrap.cc:492] T c45ffc304e73490387ce800b702b5881 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:20.869789 1166 tablet_bootstrap.cc:654] T c45ffc304e73490387ce800b702b5881 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:20.879010 1166 tablet_bootstrap.cc:492] T c45ffc304e73490387ce800b702b5881 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:20.879712 1166 ts_tablet_manager.cc:1403] T c45ffc304e73490387ce800b702b5881 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.001s
I20251029 02:52:20.880723 1166 raft_consensus.cc:359] T c45ffc304e73490387ce800b702b5881 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:20.881939 1166 raft_consensus.cc:385] T c45ffc304e73490387ce800b702b5881 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:20.882154 1166 raft_consensus.cc:740] T c45ffc304e73490387ce800b702b5881 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:20.882611 1166 consensus_queue.cc:260] T c45ffc304e73490387ce800b702b5881 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:20.883545 1166 ts_tablet_manager.cc:1434] T c45ffc304e73490387ce800b702b5881 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20251029 02:52:20.884001 3049 consensus_queue.cc:1048] T 08371b3daa184a0e9f4eb3d313b52c56 P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:20.884611 1166 tablet_bootstrap.cc:492] T 9492c1027aba4455ad8f539addaed5a7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:20.886996 1166 tablet_bootstrap.cc:654] T 9492c1027aba4455ad8f539addaed5a7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:20.897045 1166 tablet_bootstrap.cc:492] T 9492c1027aba4455ad8f539addaed5a7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:20.897699 1166 ts_tablet_manager.cc:1403] T 9492c1027aba4455ad8f539addaed5a7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.000s
W20251029 02:52:20.897881 847 consensus_peers.cc:597] T 628a7d0419cb434f8171faf500d585d8 P c756528642a543eeacd93cef3e335a7d -> Peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Couldn't send request to peer b495a49349324f7d81908f2da700f0d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 628a7d0419cb434f8171faf500d585d8. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:20.898725 1166 raft_consensus.cc:359] T 9492c1027aba4455ad8f539addaed5a7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:20.899941 1166 raft_consensus.cc:385] T 9492c1027aba4455ad8f539addaed5a7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:20.900146 1166 raft_consensus.cc:740] T 9492c1027aba4455ad8f539addaed5a7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:20.900563 1166 consensus_queue.cc:260] T 9492c1027aba4455ad8f539addaed5a7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:20.901520 1166 ts_tablet_manager.cc:1434] T 9492c1027aba4455ad8f539addaed5a7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20251029 02:52:20.902225 1166 tablet_bootstrap.cc:492] T fcb3db8399aa4cf188b07dda220f7f5b P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:20.902191 3039 raft_consensus.cc:515] T c45ffc304e73490387ce800b702b5881 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:20.904538 3039 leader_election.cc:290] T c45ffc304e73490387ce800b702b5881 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:20.905550 1166 tablet_bootstrap.cc:654] T fcb3db8399aa4cf188b07dda220f7f5b P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:20.906881 767 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c45ffc304e73490387ce800b702b5881" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:20.907111 767 raft_consensus.cc:3060] T c45ffc304e73490387ce800b702b5881 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:20.910106 767 raft_consensus.cc:2468] T c45ffc304e73490387ce800b702b5881 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:20.910952 847 leader_election.cc:304] T c45ffc304e73490387ce800b702b5881 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:20.911592 3049 raft_consensus.cc:2804] T c45ffc304e73490387ce800b702b5881 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:20.914812 3049 raft_consensus.cc:697] T c45ffc304e73490387ce800b702b5881 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:52:20.915622 1062 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c45ffc304e73490387ce800b702b5881" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:52:20.915822 1062 raft_consensus.cc:3060] T c45ffc304e73490387ce800b702b5881 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:20.917625 1166 tablet_bootstrap.cc:492] T fcb3db8399aa4cf188b07dda220f7f5b P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:20.917752 1166 ts_tablet_manager.cc:1403] T fcb3db8399aa4cf188b07dda220f7f5b P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.002s
I20251029 02:52:20.918493 1062 raft_consensus.cc:2468] T c45ffc304e73490387ce800b702b5881 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:20.918576 1166 raft_consensus.cc:359] T fcb3db8399aa4cf188b07dda220f7f5b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:20.920130 1166 raft_consensus.cc:385] T fcb3db8399aa4cf188b07dda220f7f5b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:20.920329 1166 raft_consensus.cc:740] T fcb3db8399aa4cf188b07dda220f7f5b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:20.920967 1166 consensus_queue.cc:260] T fcb3db8399aa4cf188b07dda220f7f5b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:20.921888 1166 ts_tablet_manager.cc:1434] T fcb3db8399aa4cf188b07dda220f7f5b P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251029 02:52:20.923002 1166 tablet_bootstrap.cc:492] T 15137fc1c70347b381ffdfe5653a5f22 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:20.923640 3044 raft_consensus.cc:515] T 2b278db5b2b04344b864797055a5f665 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:20.924739 1174 ts_tablet_manager.cc:1929] T 67651306f3b44e7eb14dcee9a2007a81 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:20.924902 1174 log.cc:1199] T 67651306f3b44e7eb14dcee9a2007a81 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/67651306f3b44e7eb14dcee9a2007a81
I20251029 02:52:20.925071 1166 tablet_bootstrap.cc:654] T 15137fc1c70347b381ffdfe5653a5f22 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:20.925328 1174 ts_tablet_manager.cc:1950] T 67651306f3b44e7eb14dcee9a2007a81 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:20.925724 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 1073) took 7206 ms. Trace:
I20251029 02:52:20.925840 1174 rpcz_store.cc:276] 1029 02:52:13.719508 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:13.720329 (+ 821us) service_pool.cc:225] Handling call
1029 02:52:20.805250 (+7084921us) ts_tablet_manager.cc:1151] Deleting tablet 67651306f3b44e7eb14dcee9a2007a81
1029 02:52:20.805263 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:20.904189 (+ 98926us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:20.924629 (+ 20440us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:20.925706 (+ 1077us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7084749}
I20251029 02:52:20.926249 1174 tablet_replica.cc:333] T 59eee86f249e4d6a9d09d30276dcd659 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:20.926467 1174 raft_consensus.cc:2243] T 59eee86f249e4d6a9d09d30276dcd659 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:20.926705 1174 raft_consensus.cc:2272] T 59eee86f249e4d6a9d09d30276dcd659 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:20.930928 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 67651306f3b44e7eb14dcee9a2007a81 (table default.Table_ff796c36eb2c4bb1809f39e1cb68c223 [id=95bbd9adfe994fecab10c9f23c0bedb0]) successfully deleted
I20251029 02:52:20.915102 3049 consensus_queue.cc:237] T c45ffc304e73490387ce800b702b5881 P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:20.932904 3044 leader_election.cc:290] T 2b278db5b2b04344b864797055a5f665 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:20.933018 1062 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b278db5b2b04344b864797055a5f665" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:52:20.933230 767 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2b278db5b2b04344b864797055a5f665" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:20.933313 1062 raft_consensus.cc:3060] T 2b278db5b2b04344b864797055a5f665 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:20.933455 767 raft_consensus.cc:3060] T 2b278db5b2b04344b864797055a5f665 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:20.935020 1166 tablet_bootstrap.cc:492] T 15137fc1c70347b381ffdfe5653a5f22 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:20.935284 2986 raft_consensus.cc:493] T dc33fd3404ec41c3935b02c84a12971b P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:20.935474 2986 raft_consensus.cc:515] T dc33fd3404ec41c3935b02c84a12971b P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:20.935819 1166 ts_tablet_manager.cc:1403] T 15137fc1c70347b381ffdfe5653a5f22 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.004s
I20251029 02:52:20.936116 2986 leader_election.cc:290] T dc33fd3404ec41c3935b02c84a12971b P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:20.936335 767 raft_consensus.cc:2468] T 2b278db5b2b04344b864797055a5f665 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:20.936559 1062 raft_consensus.cc:2468] T 2b278db5b2b04344b864797055a5f665 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:20.936830 847 leader_election.cc:304] T 2b278db5b2b04344b864797055a5f665 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:20.937458 1166 raft_consensus.cc:359] T 15137fc1c70347b381ffdfe5653a5f22 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:20.937584 1174 ts_tablet_manager.cc:1916] T 59eee86f249e4d6a9d09d30276dcd659 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:20.938230 1166 raft_consensus.cc:385] T 15137fc1c70347b381ffdfe5653a5f22 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:20.938441 1166 raft_consensus.cc:740] T 15137fc1c70347b381ffdfe5653a5f22 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:20.938800 3044 raft_consensus.cc:2804] T 2b278db5b2b04344b864797055a5f665 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:20.939049 3044 raft_consensus.cc:697] T 2b278db5b2b04344b864797055a5f665 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:52:20.938863 1166 consensus_queue.cc:260] T 15137fc1c70347b381ffdfe5653a5f22 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:20.941314 1166 ts_tablet_manager.cc:1434] T 15137fc1c70347b381ffdfe5653a5f22 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20251029 02:52:20.942109 1166 tablet_bootstrap.cc:492] T c7b78fd458e4414b9fc3aa064a80b206 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:20.943302 918 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc33fd3404ec41c3935b02c84a12971b" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:20.943908 711 leader_election.cc:343] T dc33fd3404ec41c3935b02c84a12971b P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:20.944293 1062 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "dc33fd3404ec41c3935b02c84a12971b" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:52:20.944896 712 leader_election.cc:343] T dc33fd3404ec41c3935b02c84a12971b P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:20.945031 712 leader_election.cc:304] T dc33fd3404ec41c3935b02c84a12971b P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:20.945518 2986 raft_consensus.cc:2749] T dc33fd3404ec41c3935b02c84a12971b P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:20.946971 1166 tablet_bootstrap.cc:654] T c7b78fd458e4414b9fc3aa064a80b206 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
W20251029 02:52:20.959517 849 consensus_peers.cc:597] T 628a7d0419cb434f8171faf500d585d8 P c756528642a543eeacd93cef3e335a7d -> Peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Couldn't send request to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 628a7d0419cb434f8171faf500d585d8. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:20.961848 754 tablet_service.cc:1505] Processing CreateTablet for tablet 3a1c62fe831b4887a479d0794efbe81d (DEFAULT_TABLE table=default.Table_1fe2eb342d984f98ae67e78d577827e8 [id=3bb4d1d8901e49a1888345fbdc1343aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:20.961939 895 tablet_service.cc:1505] Processing CreateTablet for tablet 3a1c62fe831b4887a479d0794efbe81d (DEFAULT_TABLE table=default.Table_1fe2eb342d984f98ae67e78d577827e8 [id=3bb4d1d8901e49a1888345fbdc1343aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:20.962383 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a1c62fe831b4887a479d0794efbe81d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:20.963765 756 tablet_service.cc:1505] Processing CreateTablet for tablet 1c24ed8c046c4119ba84dcae5d2154bf (DEFAULT_TABLE table=default.Table_4b3a575375ab4bb19ef4d9f7d9ab74cb [id=9bc3968738ca426f8ac8dec3b513b085]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:20.964151 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c24ed8c046c4119ba84dcae5d2154bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:20.967772 891 tablet_service.cc:1505] Processing CreateTablet for tablet 1c24ed8c046c4119ba84dcae5d2154bf (DEFAULT_TABLE table=default.Table_4b3a575375ab4bb19ef4d9f7d9ab74cb [id=9bc3968738ca426f8ac8dec3b513b085]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:20.968223 1027 tablet_service.cc:1505] Processing CreateTablet for tablet 3a1c62fe831b4887a479d0794efbe81d (DEFAULT_TABLE table=default.Table_1fe2eb342d984f98ae67e78d577827e8 [id=3bb4d1d8901e49a1888345fbdc1343aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:20.968776 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a1c62fe831b4887a479d0794efbe81d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:20.969563 885 tablet_service.cc:1505] Processing CreateTablet for tablet 472fc991ba8945f2a599836e973e2bae (DEFAULT_TABLE table=default.Table_ea69b281d8ca4cf5a9536a17ce80e7d2 [id=c0222db11f2e43ec966336b2f66e94ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:20.970537 1031 tablet_service.cc:1505] Processing CreateTablet for tablet 1c24ed8c046c4119ba84dcae5d2154bf (DEFAULT_TABLE table=default.Table_4b3a575375ab4bb19ef4d9f7d9ab74cb [id=9bc3968738ca426f8ac8dec3b513b085]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:20.970960 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c24ed8c046c4119ba84dcae5d2154bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:20.974879 1034 tablet_service.cc:1505] Processing CreateTablet for tablet 472fc991ba8945f2a599836e973e2bae (DEFAULT_TABLE table=default.Table_ea69b281d8ca4cf5a9536a17ce80e7d2 [id=c0222db11f2e43ec966336b2f66e94ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:20.975304 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 472fc991ba8945f2a599836e973e2bae. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:20.980410 739 tablet_service.cc:1505] Processing CreateTablet for tablet 472fc991ba8945f2a599836e973e2bae (DEFAULT_TABLE table=default.Table_ea69b281d8ca4cf5a9536a17ce80e7d2 [id=c0222db11f2e43ec966336b2f66e94ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:20.982986 3068 tablet.cc:1722] T cedfb53078d94c81b5e294ce7438325e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 02:52:20.987602 3066 tablet.cc:1722] T cedfb53078d94c81b5e294ce7438325e P b495a49349324f7d81908f2da700f0d9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 02:52:20.987928 1107 heartbeater.cc:344] Connected to a master server at 127.0.104.62:42343
I20251029 02:52:20.994809 519 catalog_manager.cc:5741] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481) does not have the latest schema for tablet 08371b3daa184a0e9f4eb3d313b52c56 (table default.Table_1453b5168e64406688903f1d6927f549 [id=40f1beecaad24a64b4348663c9ae9afd]). Expected version 1 got 0
I20251029 02:52:20.995368 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 472fc991ba8945f2a599836e973e2bae. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:21.006212 879 tablet_service.cc:1505] Processing CreateTablet for tablet 2d51e583bfed4277aca102ebb413ec10 (DEFAULT_TABLE table=default.Table_82605703457f430e9edcf160c498ad0c [id=d32947045dd44c518e9819e829b826af]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:21.009748 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3a1c62fe831b4887a479d0794efbe81d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:21.009970 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c24ed8c046c4119ba84dcae5d2154bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:21.010929 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 472fc991ba8945f2a599836e973e2bae. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:21.012404 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d51e583bfed4277aca102ebb413ec10. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:21.014038 847 consensus_peers.cc:597] T 078ace5eada044c88d2462efb2b57aa8 P c756528642a543eeacd93cef3e335a7d -> Peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Couldn't send request to peer b495a49349324f7d81908f2da700f0d9. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:20.939334 3044 consensus_queue.cc:237] T 2b278db5b2b04344b864797055a5f665 P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
W20251029 02:52:21.021122 3074 log.cc:927] Time spent T 1df0d944717d487a98ea83a8ec04bd46 P c756528642a543eeacd93cef3e335a7d: Append to log took a long time: real 0.073s user 0.000s sys 0.000s
I20251029 02:52:21.025118 3067 tablet.cc:1722] T 055184c641ba45a588bca611fb24d130 P c756528642a543eeacd93cef3e335a7d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 02:52:21.025594 1174 ts_tablet_manager.cc:1929] T 59eee86f249e4d6a9d09d30276dcd659 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:21.025703 1174 log.cc:1199] T 59eee86f249e4d6a9d09d30276dcd659 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/59eee86f249e4d6a9d09d30276dcd659
I20251029 02:52:21.040227 1166 tablet_bootstrap.cc:492] T c7b78fd458e4414b9fc3aa064a80b206 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:21.040175 1175 ts_tablet_manager.cc:1929] T 078ace5eada044c88d2462efb2b57aa8 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:21.040319 1175 log.cc:1199] T 078ace5eada044c88d2462efb2b57aa8 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/078ace5eada044c88d2462efb2b57aa8
I20251029 02:52:21.040366 1166 ts_tablet_manager.cc:1403] T c7b78fd458e4414b9fc3aa064a80b206 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.098s user 0.025s sys 0.039s
I20251029 02:52:21.040695 1175 ts_tablet_manager.cc:1950] T 078ace5eada044c88d2462efb2b57aa8 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:21.041165 1166 raft_consensus.cc:359] T c7b78fd458e4414b9fc3aa064a80b206 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:21.041329 1166 raft_consensus.cc:385] T c7b78fd458e4414b9fc3aa064a80b206 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:21.041385 1166 raft_consensus.cc:740] T c7b78fd458e4414b9fc3aa064a80b206 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:21.042284 1172 tablet_bootstrap.cc:492] T c64d6761ef924ccaa9c2579f8ba47e33 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:21.042496 1172 ts_tablet_manager.cc:1403] T c64d6761ef924ccaa9c2579f8ba47e33 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.225s user 0.006s sys 0.001s
I20251029 02:52:21.044036 1166 consensus_queue.cc:260] T c7b78fd458e4414b9fc3aa064a80b206 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:21.044682 1166 ts_tablet_manager.cc:1434] T c7b78fd458e4414b9fc3aa064a80b206 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20251029 02:52:21.047299 1172 raft_consensus.cc:359] T c64d6761ef924ccaa9c2579f8ba47e33 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:21.047703 1172 raft_consensus.cc:385] T c64d6761ef924ccaa9c2579f8ba47e33 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:21.047837 1172 raft_consensus.cc:740] T c64d6761ef924ccaa9c2579f8ba47e33 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:21.048584 1172 consensus_queue.cc:260] T c64d6761ef924ccaa9c2579f8ba47e33 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:21.049685 1172 ts_tablet_manager.cc:1434] T c64d6761ef924ccaa9c2579f8ba47e33 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20251029 02:52:21.052919 1166 tablet_bootstrap.cc:492] T 786d9148fb1d417a8c2c8b8c53665899 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:21.057308 517 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_68193ad4def64cf98025107373b5c3d0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:21.057937 517 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_68193ad4def64cf98025107373b5c3d0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:21.061743 1166 tablet_bootstrap.cc:654] T 786d9148fb1d417a8c2c8b8c53665899 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:21.063946 1176 ts_tablet_manager.cc:1916] T b0fec06ad37d4c12a7045d424b88cc5e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:21.072961 3009 raft_consensus.cc:493] T 9492c1027aba4455ad8f539addaed5a7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:21.073154 3009 raft_consensus.cc:515] T 9492c1027aba4455ad8f539addaed5a7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:21.074126 3009 leader_election.cc:290] T 9492c1027aba4455ad8f539addaed5a7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:21.074823 767 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9492c1027aba4455ad8f539addaed5a7" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:21.075066 767 raft_consensus.cc:2468] T 9492c1027aba4455ad8f539addaed5a7 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 0.
I20251029 02:52:21.075277 918 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9492c1027aba4455ad8f539addaed5a7" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:21.075893 990 leader_election.cc:343] T 9492c1027aba4455ad8f539addaed5a7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
W20251029 02:52:21.077733 849 consensus_peers.cc:597] T 24f07b1efac64b099b0b5ed467463749 P c756528642a543eeacd93cef3e335a7d -> Peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Couldn't send request to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 24f07b1efac64b099b0b5ed467463749. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:21.079382 989 leader_election.cc:304] T 9492c1027aba4455ad8f539addaed5a7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters: c756528642a543eeacd93cef3e335a7d
I20251029 02:52:21.079803 3009 raft_consensus.cc:2804] T 9492c1027aba4455ad8f539addaed5a7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:21.079877 3009 raft_consensus.cc:493] T 9492c1027aba4455ad8f539addaed5a7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:21.079932 3009 raft_consensus.cc:3060] T 9492c1027aba4455ad8f539addaed5a7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:21.082754 3009 raft_consensus.cc:515] T 9492c1027aba4455ad8f539addaed5a7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:21.084488 767 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9492c1027aba4455ad8f539addaed5a7" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:21.084699 754 tablet_service.cc:1505] Processing CreateTablet for tablet b5396faf0afe4963a99447811f06b65c (DEFAULT_TABLE table=default.Table_77922304a486493b8d07d408c24815ef [id=2415e8da9501413c91490a302551c3aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:21.084698 767 raft_consensus.cc:3060] T 9492c1027aba4455ad8f539addaed5a7 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:21.087992 767 raft_consensus.cc:2468] T 9492c1027aba4455ad8f539addaed5a7 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 1.
I20251029 02:52:21.089288 756 tablet_service.cc:1505] Processing CreateTablet for tablet fb4989e9035b459b8a0b206133d49bff (DEFAULT_TABLE table=default.Table_c938c99be4f0482b94fc22d82972bb4a [id=694834b73ccb44f2be556af73c4de3ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:21.089663 748 tablet_service.cc:1505] Processing CreateTablet for tablet e095f00f34a145b1bbd22deecb899ecc (DEFAULT_TABLE table=default.Table_4098c28776dd4f41864c61d2299b90cd [id=47ee5993b95a477db52f9c51d034e549]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:21.086629 1034 tablet_service.cc:1505] Processing CreateTablet for tablet fb4989e9035b459b8a0b206133d49bff (DEFAULT_TABLE table=default.Table_c938c99be4f0482b94fc22d82972bb4a [id=694834b73ccb44f2be556af73c4de3ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:21.090432 1031 tablet_service.cc:1505] Processing CreateTablet for tablet e095f00f34a145b1bbd22deecb899ecc (DEFAULT_TABLE table=default.Table_4098c28776dd4f41864c61d2299b90cd [id=47ee5993b95a477db52f9c51d034e549]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:21.090575 989 leader_election.cc:304] T 9492c1027aba4455ad8f539addaed5a7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters:
I20251029 02:52:21.091715 3054 raft_consensus.cc:2804] T 9492c1027aba4455ad8f539addaed5a7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:21.084241 3009 leader_election.cc:290] T 9492c1027aba4455ad8f539addaed5a7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Requested vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:21.096751 1027 tablet_service.cc:1505] Processing CreateTablet for tablet b5396faf0afe4963a99447811f06b65c (DEFAULT_TABLE table=default.Table_77922304a486493b8d07d408c24815ef [id=2415e8da9501413c91490a302551c3aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:21.097159 888 tablet_service.cc:1505] Processing CreateTablet for tablet e095f00f34a145b1bbd22deecb899ecc (DEFAULT_TABLE table=default.Table_4098c28776dd4f41864c61d2299b90cd [id=47ee5993b95a477db52f9c51d034e549]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:21.097668 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e095f00f34a145b1bbd22deecb899ecc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:21.099486 887 tablet_service.cc:1505] Processing CreateTablet for tablet fb4989e9035b459b8a0b206133d49bff (DEFAULT_TABLE table=default.Table_c938c99be4f0482b94fc22d82972bb4a [id=694834b73ccb44f2be556af73c4de3ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:21.098160 900 tablet_service.cc:1505] Processing CreateTablet for tablet b5396faf0afe4963a99447811f06b65c (DEFAULT_TABLE table=default.Table_77922304a486493b8d07d408c24815ef [id=2415e8da9501413c91490a302551c3aa]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:21.100605 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5396faf0afe4963a99447811f06b65c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:21.100700 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5396faf0afe4963a99447811f06b65c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:21.100934 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e095f00f34a145b1bbd22deecb899ecc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:21.102435 525 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_e6fd174aebf746e28bfa0c2ef9a699c2" } new_table_name: "default.Table_d24087e911214aaeb2f794f5c60489a4" modify_external_catalogs: true
I20251029 02:52:21.106899 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb4989e9035b459b8a0b206133d49bff. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:21.109320 521 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4b29426988b043e29783fecc16e7f13e consensus state has no leader
I20251029 02:52:21.112241 3044 consensus_queue.cc:1048] T e3000183230e4350b47f5085952f2db6 P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:21.113286 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb4989e9035b459b8a0b206133d49bff. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:21.116896 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_5ac1286c336242caa90eda31d79ff9b2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:21.117655 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5ac1286c336242caa90eda31d79ff9b2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:21.124972 520 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 6fcf3e12810d41ddb96d9f2cf52f2c5d
I20251029 02:52:21.100210 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9492c1027aba4455ad8f539addaed5a7" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
I20251029 02:52:21.125483 520 catalog_manager.cc:6719] ReplaceTablet: tablet 6fcf3e12810d41ddb96d9f2cf52f2c5d deleted and replaced by tablet c2fd0123bc1b48beb778da519a4a933d
I20251029 02:52:21.125793 740 tablet_service.cc:1552] Processing DeleteTablet for tablet 6fcf3e12810d41ddb96d9f2cf52f2c5d with delete_type TABLET_DATA_DELETED (replaced by tablet c2fd0123bc1b48beb778da519a4a933d) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:21.125856 1176 ts_tablet_manager.cc:1929] T b0fec06ad37d4c12a7045d424b88cc5e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:21.125954 1176 log.cc:1199] T b0fec06ad37d4c12a7045d424b88cc5e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/b0fec06ad37d4c12a7045d424b88cc5e
I20251029 02:52:21.126341 1176 ts_tablet_manager.cc:1950] T b0fec06ad37d4c12a7045d424b88cc5e P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
W20251029 02:52:21.127317 990 leader_election.cc:343] T 9492c1027aba4455ad8f539addaed5a7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:21.127907 1176 tablet_replica.cc:333] T 6dff036b32904ddb9598be4b79504bb6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:21.128152 1176 raft_consensus.cc:2243] T 6dff036b32904ddb9598be4b79504bb6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:21.128338 1176 raft_consensus.cc:2272] T 6dff036b32904ddb9598be4b79504bb6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:21.128348 894 tablet_service.cc:1552] Processing DeleteTablet for tablet 6fcf3e12810d41ddb96d9f2cf52f2c5d with delete_type TABLET_DATA_DELETED (replaced by tablet c2fd0123bc1b48beb778da519a4a933d) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:21.131124 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet b0fec06ad37d4c12a7045d424b88cc5e (table default.Table_c4333d8c61d346b1ac1a809d3ccb1915 [id=6d576d3499e649d78000c31fc5dab58c]) successfully deleted
I20251029 02:52:21.131687 1166 tablet_bootstrap.cc:492] T 786d9148fb1d417a8c2c8b8c53665899 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
W20251029 02:52:21.131851 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 9e0d410678594cbaa78bd7f56c3cb40e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9e0d410678594cbaa78bd7f56c3cb40e already in progress: creating tablet
I20251029 02:52:21.132952 1166 ts_tablet_manager.cc:1403] T 786d9148fb1d417a8c2c8b8c53665899 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.080s user 0.002s sys 0.009s
I20251029 02:52:21.134384 1166 raft_consensus.cc:359] T 786d9148fb1d417a8c2c8b8c53665899 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:21.134670 1174 ts_tablet_manager.cc:1950] T 59eee86f249e4d6a9d09d30276dcd659 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:21.135068 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 1074) took 7415 ms. Trace:
I20251029 02:52:21.135226 1174 rpcz_store.cc:276] 1029 02:52:13.719596 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:13.720525 (+ 929us) service_pool.cc:225] Handling call
1029 02:52:20.926201 (+7205676us) ts_tablet_manager.cc:1151] Deleting tablet 59eee86f249e4d6a9d09d30276dcd659
1029 02:52:20.926211 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:20.943806 (+ 17595us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:21.025499 (+ 81693us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:21.135045 (+109546us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7205544}
I20251029 02:52:21.136816 3044 consensus_queue.cc:1048] T 294afb9b012c40c0b05b73795d6f0ac8 P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:21.137188 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 59eee86f249e4d6a9d09d30276dcd659 (table default.Table_8f1e631e578a4010927f77c77cc29cb0 [id=edac8dce6fa747bda675efdf71adb1d6]) successfully deleted
I20251029 02:52:21.137600 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 1071) took 7418 ms. Trace:
I20251029 02:52:21.137744 1174 rpcz_store.cc:276] 1029 02:52:13.719329 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:13.721557 (+ 2228us) service_pool.cc:225] Handling call
1029 02:52:21.137492 (+7415935us) ts_tablet_manager.cc:1151] Deleting tablet 777992acb1bd4765886b802940db74bb
1029 02:52:21.137507 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:21.137581 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7415730}
W20251029 02:52:21.138698 512 catalog_manager.cc:4952] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 777992acb1bd4765886b802940db74bb because the tablet was not found. No further retry: Not found: Tablet not found: 777992acb1bd4765886b802940db74bb
I20251029 02:52:21.139559 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 1075) took 7404 ms. Trace:
I20251029 02:52:21.140200 1166 raft_consensus.cc:385] T 786d9148fb1d417a8c2c8b8c53665899 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:21.139945 1174 rpcz_store.cc:276] 1029 02:52:13.735521 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:13.735575 (+ 54us) service_pool.cc:225] Handling call
1029 02:52:21.139292 (+7403717us) ts_tablet_manager.cc:1151] Deleting tablet dab6ed2266ed42508738399d388c4da3
1029 02:52:21.139307 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:21.139510 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7403470}
I20251029 02:52:21.140512 3054 raft_consensus.cc:697] T 9492c1027aba4455ad8f539addaed5a7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Becoming Leader. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Running, Role: LEADER
W20251029 02:52:21.140651 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet dab6ed2266ed42508738399d388c4da3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dab6ed2266ed42508738399d388c4da3 already in progress: creating tablet
I20251029 02:52:21.141033 3054 consensus_queue.cc:237] T 9492c1027aba4455ad8f539addaed5a7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:21.141407 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 1076) took 7343 ms. Trace:
I20251029 02:52:21.138206 1027 tablet_service.cc:1552] Processing DeleteTablet for tablet 6fcf3e12810d41ddb96d9f2cf52f2c5d with delete_type TABLET_DATA_DELETED (replaced by tablet c2fd0123bc1b48beb778da519a4a933d) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:21.140583 1166 raft_consensus.cc:740] T 786d9148fb1d417a8c2c8b8c53665899 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:21.142264 1166 consensus_queue.cc:260] T 786d9148fb1d417a8c2c8b8c53665899 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:21.142956 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5396faf0afe4963a99447811f06b65c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:21.142952 1172 tablet_bootstrap.cc:492] T 38f83ebcb2a440bc8e6a8d36607e6666 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:21.143062 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 1125) took 5866 ms. Trace:
I20251029 02:52:21.143127 1175 rpcz_store.cc:276] 1029 02:52:15.276957 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:15.283305 (+ 6348us) service_pool.cc:225] Handling call
1029 02:52:20.850636 (+5567331us) ts_tablet_manager.cc:1151] Deleting tablet 078ace5eada044c88d2462efb2b57aa8
1029 02:52:20.850648 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:20.914351 (+ 63703us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:21.040061 (+125710us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:21.143042 (+102981us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5567126}
I20251029 02:52:21.143612 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 078ace5eada044c88d2462efb2b57aa8 (table default.Table_3fae9f42aaba465bb13918661650e73a [id=9173815e55cc4e7a95b8b663e30c622c]) successfully deleted
I20251029 02:52:21.145773 1172 tablet_bootstrap.cc:654] T 38f83ebcb2a440bc8e6a8d36607e6666 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:21.132992 1176 ts_tablet_manager.cc:1916] T 6dff036b32904ddb9598be4b79504bb6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 02:52:21.150707 511 catalog_manager.cc:4767] Async tablet task AlterTable RPC for tablet 4b29426988b043e29783fecc16e7f13e (table default.Table_92d77f2b41c14d11afd04421adcdea45 [id=adde6f5f632b447f9f1037e4e14ada8d], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 4b29426988b043e29783fecc16e7f13e consensus state has no leader
I20251029 02:52:21.151324 1166 ts_tablet_manager.cc:1434] T 786d9148fb1d417a8c2c8b8c53665899 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.018s user 0.002s sys 0.000s
I20251029 02:52:21.151727 1166 tablet_bootstrap.cc:492] T 294afb9b012c40c0b05b73795d6f0ac8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:21.153191 1166 tablet_bootstrap.cc:654] T 294afb9b012c40c0b05b73795d6f0ac8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
W20251029 02:52:21.156852 849 consensus_peers.cc:597] T e3000183230e4350b47f5085952f2db6 P c756528642a543eeacd93cef3e335a7d -> Peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Couldn't send request to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:21.164526 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 1127) took 5832 ms. Trace:
I20251029 02:52:21.164666 1175 rpcz_store.cc:276] 1029 02:52:15.331711 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:15.331777 (+ 66us) service_pool.cc:225] Handling call
1029 02:52:21.143375 (+5811598us) ts_tablet_manager.cc:1151] Deleting tablet 7ff2399833e546448b85b9caeb5200fb
1029 02:52:21.164351 (+ 20976us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:21.164497 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5811393}
I20251029 02:52:21.165067 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 1129) took 5830 ms. Trace:
I20251029 02:52:21.165160 1175 rpcz_store.cc:276] 1029 02:52:15.334827 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:15.334929 (+ 102us) service_pool.cc:225] Handling call
1029 02:52:21.164952 (+5830023us) ts_tablet_manager.cc:1151] Deleting tablet 2c4e78fbd7e645eb8b3be8ac4bfbb425
1029 02:52:21.164962 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:21.165049 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5829821}
I20251029 02:52:21.165313 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e095f00f34a145b1bbd22deecb899ecc. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:21.165490 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 1128) took 5830 ms. Trace:
I20251029 02:52:21.165560 1175 rpcz_store.cc:276] 1029 02:52:15.334732 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:15.334960 (+ 228us) service_pool.cc:225] Handling call
1029 02:52:21.165360 (+5830400us) ts_tablet_manager.cc:1151] Deleting tablet f66a7af499b446c7a33965c9e37ce460
1029 02:52:21.165369 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:21.165474 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5826657}
I20251029 02:52:21.165843 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 1130) took 5789 ms. Trace:
I20251029 02:52:21.165911 1175 rpcz_store.cc:276] 1029 02:52:15.376318 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:15.376739 (+ 421us) service_pool.cc:225] Handling call
1029 02:52:21.165744 (+5789005us) ts_tablet_manager.cc:1151] Deleting tablet 59fd5608e4c04fa293ca04595a20d858
1029 02:52:21.165752 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:21.165828 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5788301}
I20251029 02:52:21.166260 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 1151) took 5562 ms. Trace:
I20251029 02:52:21.166323 1175 rpcz_store.cc:276] 1029 02:52:15.603469 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:15.604747 (+ 1278us) service_pool.cc:225] Handling call
1029 02:52:21.166129 (+5561382us) ts_tablet_manager.cc:1151] Deleting tablet e91d6bdca3ad40c48d93e6196644462c
1029 02:52:21.166138 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:21.166242 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5561209}
W20251029 02:52:21.167543 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 7ff2399833e546448b85b9caeb5200fb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7ff2399833e546448b85b9caeb5200fb already in progress: creating tablet
W20251029 02:52:21.167834 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 2c4e78fbd7e645eb8b3be8ac4bfbb425 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2c4e78fbd7e645eb8b3be8ac4bfbb425 already in progress: creating tablet
W20251029 02:52:21.167986 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet f66a7af499b446c7a33965c9e37ce460 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f66a7af499b446c7a33965c9e37ce460 already in progress: creating tablet
W20251029 02:52:21.168133 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 59fd5608e4c04fa293ca04595a20d858 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 59fd5608e4c04fa293ca04595a20d858 already in progress: creating tablet
W20251029 02:52:21.168292 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet e91d6bdca3ad40c48d93e6196644462c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e91d6bdca3ad40c48d93e6196644462c already in progress: creating tablet
I20251029 02:52:21.168874 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 1154) took 5480 ms. Trace:
I20251029 02:52:21.168973 1175 rpcz_store.cc:276] 1029 02:52:15.688627 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:15.688890 (+ 263us) service_pool.cc:225] Handling call
1029 02:52:21.168757 (+5479867us) ts_tablet_manager.cc:1151] Deleting tablet 6dc30c614be64b8ba629d9884d450290
1029 02:52:21.168774 (+ 17us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:21.168855 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5477420}
I20251029 02:52:21.169509 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 1155) took 5480 ms. Trace:
I20251029 02:52:21.169656 1175 rpcz_store.cc:276] 1029 02:52:15.688725 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:15.689136 (+ 411us) service_pool.cc:225] Handling call
1029 02:52:21.169417 (+5480281us) ts_tablet_manager.cc:1151] Deleting tablet e60b029eb3ec466cbdced20d536f4f47
1029 02:52:21.169425 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:21.169494 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5480150}
I20251029 02:52:21.170135 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 1153) took 5481 ms. Trace:
I20251029 02:52:21.170428 1175 rpcz_store.cc:276] 1029 02:52:15.688528 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:15.690097 (+ 1569us) service_pool.cc:225] Handling call
1029 02:52:21.170036 (+5479939us) ts_tablet_manager.cc:1151] Deleting tablet a3c8b25e6f4849d6aadca322d5365af1
1029 02:52:21.170044 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:21.170122 (+ 78us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5479689}
I20251029 02:52:21.170804 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb4989e9035b459b8a0b206133d49bff. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:21.170897 1176 ts_tablet_manager.cc:1929] T 6dff036b32904ddb9598be4b79504bb6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251029 02:52:21.171002 1176 log.cc:1199] T 6dff036b32904ddb9598be4b79504bb6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/6dff036b32904ddb9598be4b79504bb6
I20251029 02:52:21.171335 1176 ts_tablet_manager.cc:1950] T 6dff036b32904ddb9598be4b79504bb6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:21.172235 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 1152) took 5483 ms. Trace:
I20251029 02:52:21.172328 1175 rpcz_store.cc:276] 1029 02:52:15.688404 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:15.698618 (+ 10214us) service_pool.cc:225] Handling call
1029 02:52:21.172090 (+5473472us) ts_tablet_manager.cc:1151] Deleting tablet 18632de3bbbd491a86460497544580c1
1029 02:52:21.172116 (+ 26us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:21.172214 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5471708}
W20251029 02:52:21.172792 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 6dc30c614be64b8ba629d9884d450290 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6dc30c614be64b8ba629d9884d450290 already in progress: creating tablet
W20251029 02:52:21.173074 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet e60b029eb3ec466cbdced20d536f4f47 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e60b029eb3ec466cbdced20d536f4f47 already in progress: creating tablet
W20251029 02:52:21.173260 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet a3c8b25e6f4849d6aadca322d5365af1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a3c8b25e6f4849d6aadca322d5365af1 already in progress: creating tablet
I20251029 02:52:21.173727 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 1156) took 5419 ms. Trace:
I20251029 02:52:21.173805 1175 rpcz_store.cc:276] 1029 02:52:15.753945 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:15.754117 (+ 172us) service_pool.cc:225] Handling call
1029 02:52:21.173631 (+5419514us) ts_tablet_manager.cc:1151] Deleting tablet f1e975b267464e64a5bbe24ef4e308a3
1029 02:52:21.173642 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:21.173712 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5419335}
I20251029 02:52:21.174682 2986 raft_consensus.cc:493] T c7b78fd458e4414b9fc3aa064a80b206 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:21.174847 2986 raft_consensus.cc:515] T c7b78fd458e4414b9fc3aa064a80b206 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:21.175668 2986 leader_election.cc:290] T c7b78fd458e4414b9fc3aa064a80b206 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
W20251029 02:52:21.176610 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet f1e975b267464e64a5bbe24ef4e308a3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f1e975b267464e64a5bbe24ef4e308a3 already in progress: creating tablet
I20251029 02:52:21.174665 1175 tablet_replica.cc:333] T 688af6580c4d4637a39f817b4e324618 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:21.177330 1175 raft_consensus.cc:2243] T 688af6580c4d4637a39f817b4e324618 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:21.177549 1175 raft_consensus.cc:2272] T 688af6580c4d4637a39f817b4e324618 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Raft consensus is shut down!
W20251029 02:52:21.178539 849 consensus_peers.cc:597] T 294afb9b012c40c0b05b73795d6f0ac8 P c756528642a543eeacd93cef3e335a7d -> Peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Couldn't send request to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:21.178629 1176 tablet_replica.cc:333] T cc802bda2f6a4e439b605f2cafa37967 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:21.178608 3046 consensus_queue.cc:1048] T 506f8c112a3b43e1a3d6bbafb8cb81e8 P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:21.178838 1176 raft_consensus.cc:2243] T cc802bda2f6a4e439b605f2cafa37967 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:21.178992 1176 raft_consensus.cc:2272] T cc802bda2f6a4e439b605f2cafa37967 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:21.180301 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 6dff036b32904ddb9598be4b79504bb6 (table default.Table_0ae102f09fdc41bcbead0148653df9e2 [id=65ec61bf400a4628b98e64e7d43c025c]) successfully deleted
I20251029 02:52:21.180366 1176 ts_tablet_manager.cc:1916] T cc802bda2f6a4e439b605f2cafa37967 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 02:52:21.180716 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 93440d660eac4986aeb17abc18144700 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 93440d660eac4986aeb17abc18144700 already in progress: creating tablet
I20251029 02:52:21.181342 1062 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c7b78fd458e4414b9fc3aa064a80b206" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:21.181560 1062 raft_consensus.cc:2468] T c7b78fd458e4414b9fc3aa064a80b206 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b495a49349324f7d81908f2da700f0d9 in term 0.
I20251029 02:52:21.181989 712 leader_election.cc:304] T c7b78fd458e4414b9fc3aa064a80b206 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters:
I20251029 02:52:21.182621 2986 raft_consensus.cc:2804] T c7b78fd458e4414b9fc3aa064a80b206 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:21.182720 2986 raft_consensus.cc:493] T c7b78fd458e4414b9fc3aa064a80b206 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:21.182791 2986 raft_consensus.cc:3060] T c7b78fd458e4414b9fc3aa064a80b206 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:21.186198 2986 raft_consensus.cc:515] T c7b78fd458e4414b9fc3aa064a80b206 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:21.189738 2986 leader_election.cc:290] T c7b78fd458e4414b9fc3aa064a80b206 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 election: Requested vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701)
I20251029 02:52:21.190991 1062 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c7b78fd458e4414b9fc3aa064a80b206" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:52:21.191231 1062 raft_consensus.cc:3060] T c7b78fd458e4414b9fc3aa064a80b206 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:21.191603 1166 tablet_bootstrap.cc:492] T 294afb9b012c40c0b05b73795d6f0ac8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:21.191732 1166 ts_tablet_manager.cc:1403] T 294afb9b012c40c0b05b73795d6f0ac8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.040s user 0.004s sys 0.003s
I20251029 02:52:21.192337 1166 raft_consensus.cc:359] T 294afb9b012c40c0b05b73795d6f0ac8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:21.192476 1166 raft_consensus.cc:385] T 294afb9b012c40c0b05b73795d6f0ac8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:21.192517 1166 raft_consensus.cc:740] T 294afb9b012c40c0b05b73795d6f0ac8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:21.194079 1176 ts_tablet_manager.cc:1929] T cc802bda2f6a4e439b605f2cafa37967 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:21.194183 1176 log.cc:1199] T cc802bda2f6a4e439b605f2cafa37967 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/cc802bda2f6a4e439b605f2cafa37967
I20251029 02:52:21.194458 1176 ts_tablet_manager.cc:1950] T cc802bda2f6a4e439b605f2cafa37967 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:21.197476 1175 ts_tablet_manager.cc:1916] T 688af6580c4d4637a39f817b4e324618 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:21.199445 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet cc802bda2f6a4e439b605f2cafa37967 (table default.Table_6e96405b009143d2bd1c0942cb98c033 [id=67905be94563417c8713eadd51eb94cc]) successfully deleted
I20251029 02:52:21.199890 1176 tablet_replica.cc:333] T 1df0d944717d487a98ea83a8ec04bd46 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:21.196149 1062 raft_consensus.cc:2468] T c7b78fd458e4414b9fc3aa064a80b206 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b495a49349324f7d81908f2da700f0d9 in term 1.
I20251029 02:52:21.196532 3009 raft_consensus.cc:493] T 506f8c112a3b43e1a3d6bbafb8cb81e8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:21.200124 1176 raft_consensus.cc:2243] T 1df0d944717d487a98ea83a8ec04bd46 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:21.200186 3009 raft_consensus.cc:515] T 506f8c112a3b43e1a3d6bbafb8cb81e8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:21.200300 1176 raft_consensus.cc:2272] T 1df0d944717d487a98ea83a8ec04bd46 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:21.200881 3009 leader_election.cc:290] T 506f8c112a3b43e1a3d6bbafb8cb81e8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:21.201476 767 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "506f8c112a3b43e1a3d6bbafb8cb81e8" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:21.201944 913 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "506f8c112a3b43e1a3d6bbafb8cb81e8" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:21.202791 1176 ts_tablet_manager.cc:1916] T 1df0d944717d487a98ea83a8ec04bd46 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:21.202925 990 leader_election.cc:304] T 506f8c112a3b43e1a3d6bbafb8cb81e8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5; no voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:21.203810 520 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_b90a97409b234cbd97e863c7235041ed" } modify_external_catalogs: true
I20251029 02:52:21.203989 520 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_b90a97409b234cbd97e863c7235041ed" } modify_external_catalogs: true
I20251029 02:52:21.206224 3009 raft_consensus.cc:2749] T 506f8c112a3b43e1a3d6bbafb8cb81e8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:21.207088 712 leader_election.cc:304] T c7b78fd458e4414b9fc3aa064a80b206 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters:
I20251029 02:52:21.207530 2986 raft_consensus.cc:2804] T c7b78fd458e4414b9fc3aa064a80b206 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:21.207759 2986 raft_consensus.cc:697] T c7b78fd458e4414b9fc3aa064a80b206 P b495a49349324f7d81908f2da700f0d9 [term 1 LEADER]: Becoming Leader. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Running, Role: LEADER
I20251029 02:52:21.208113 2986 consensus_queue.cc:237] T c7b78fd458e4414b9fc3aa064a80b206 P b495a49349324f7d81908f2da700f0d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
W20251029 02:52:21.210768 522 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 85159079149241adae174fc0453ed0e7 consensus state has no leader
I20251029 02:52:21.211313 526 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 0f97ec729cac41efa1bd2a8990cb7380
I20251029 02:52:21.227996 1176 ts_tablet_manager.cc:1929] T 1df0d944717d487a98ea83a8ec04bd46 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:21.229415 1176 log.cc:1199] T 1df0d944717d487a98ea83a8ec04bd46 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/1df0d944717d487a98ea83a8ec04bd46
I20251029 02:52:21.229791 1176 ts_tablet_manager.cc:1950] T 1df0d944717d487a98ea83a8ec04bd46 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:21.230566 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 1df0d944717d487a98ea83a8ec04bd46 (table default.Table_d477b4de8ad4449aa5d53db48d45273f [id=513d017e315a49d0a0fbe1fb12649f82]) successfully deleted
W20251029 02:52:21.232300 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 44c5224ef3444516a675c23662d34bec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 44c5224ef3444516a675c23662d34bec already in progress: creating tablet
W20251029 02:52:21.233110 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 96e7e1b0f14e4e738e7b2d0eaf8d53bb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 96e7e1b0f14e4e738e7b2d0eaf8d53bb already in progress: creating tablet
I20251029 02:52:21.233557 1176 tablet_replica.cc:333] T 2b278db5b2b04344b864797055a5f665 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:21.233733 1176 raft_consensus.cc:2243] T 2b278db5b2b04344b864797055a5f665 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:21.233866 1176 raft_consensus.cc:2272] T 2b278db5b2b04344b864797055a5f665 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:21.238693 891 tablet_service.cc:1505] Processing CreateTablet for tablet 2d0a63df94364a19963005071b7395b2 (DEFAULT_TABLE table=default.Table_ca88b32b14364d2aab4e1b1480676c12 [id=8aacd9b9d75346419887f8be922803ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:21.238960 1176 ts_tablet_manager.cc:1916] T 2b278db5b2b04344b864797055a5f665 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:21.239162 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d0a63df94364a19963005071b7395b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:21.251776 2992 consensus_queue.cc:1048] T 5815626b05204b578cc4a110f246bf10 P b495a49349324f7d81908f2da700f0d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:21.259496 887 tablet_service.cc:1505] Processing CreateTablet for tablet f032a2e1c1d54ef59b89874dbb274771 (DEFAULT_TABLE table=default.Table_e80c2a94126d48cf94c30936cdca8281 [id=2071412cb5604909a34e8699f6fbd337]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:21.259990 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f032a2e1c1d54ef59b89874dbb274771. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:21.260535 1027 tablet_service.cc:1505] Processing CreateTablet for tablet 2d0a63df94364a19963005071b7395b2 (DEFAULT_TABLE table=default.Table_ca88b32b14364d2aab4e1b1480676c12 [id=8aacd9b9d75346419887f8be922803ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:21.260919 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d0a63df94364a19963005071b7395b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:21.262823 1034 tablet_service.cc:1505] Processing CreateTablet for tablet f032a2e1c1d54ef59b89874dbb274771 (DEFAULT_TABLE table=default.Table_e80c2a94126d48cf94c30936cdca8281 [id=2071412cb5604909a34e8699f6fbd337]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:21.263149 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f032a2e1c1d54ef59b89874dbb274771. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:21.266847 748 tablet_service.cc:1505] Processing CreateTablet for tablet 2d0a63df94364a19963005071b7395b2 (DEFAULT_TABLE table=default.Table_ca88b32b14364d2aab4e1b1480676c12 [id=8aacd9b9d75346419887f8be922803ea]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 02:52:21.267230 849 consensus_peers.cc:597] T b0fec06ad37d4c12a7045d424b88cc5e P c756528642a543eeacd93cef3e335a7d -> Peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Couldn't send request to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b0fec06ad37d4c12a7045d424b88cc5e. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:21.267688 756 tablet_service.cc:1505] Processing CreateTablet for tablet f032a2e1c1d54ef59b89874dbb274771 (DEFAULT_TABLE table=default.Table_e80c2a94126d48cf94c30936cdca8281 [id=2071412cb5604909a34e8699f6fbd337]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:21.268373 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f032a2e1c1d54ef59b89874dbb274771. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:21.270866 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d0a63df94364a19963005071b7395b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:21.275007 1062 raft_consensus.cc:3060] T 506f8c112a3b43e1a3d6bbafb8cb81e8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:21.276837 1176 ts_tablet_manager.cc:1929] T 2b278db5b2b04344b864797055a5f665 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:21.276961 1176 log.cc:1199] T 2b278db5b2b04344b864797055a5f665 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/2b278db5b2b04344b864797055a5f665
I20251029 02:52:21.277263 1176 ts_tablet_manager.cc:1950] T 2b278db5b2b04344b864797055a5f665 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:21.277721 1176 tablet_replica.cc:333] T 506f8c112a3b43e1a3d6bbafb8cb81e8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:21.277851 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 2b278db5b2b04344b864797055a5f665 (table default.Table_73bc152b9f224b07ae2524e0e18f7df0 [id=61b689e02284411a9d7cd0795682791a]) successfully deleted
I20251029 02:52:21.278934 2747 consensus_queue.cc:1048] T 506f8c112a3b43e1a3d6bbafb8cb81e8 P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251029 02:52:21.280190 849 consensus_peers.cc:597] T 506f8c112a3b43e1a3d6bbafb8cb81e8 P c756528642a543eeacd93cef3e335a7d -> Peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Couldn't send request to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:21.285115 1176 raft_consensus.cc:2243] T 506f8c112a3b43e1a3d6bbafb8cb81e8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:21.285482 1176 raft_consensus.cc:2272] T 506f8c112a3b43e1a3d6bbafb8cb81e8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:21.286033 1176 ts_tablet_manager.cc:1916] T 506f8c112a3b43e1a3d6bbafb8cb81e8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:21.141636 1174 rpcz_store.cc:276] 1029 02:52:13.798084 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:13.798784 (+ 700us) service_pool.cc:225] Handling call
1029 02:52:21.141023 (+7342239us) ts_tablet_manager.cc:1151] Deleting tablet b4621bf9e6324eaeaa64a6ca51cd5ca9
1029 02:52:21.141033 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:21.141357 (+ 324us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7342052}
I20251029 02:52:21.287606 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 1078) took 7459 ms. Trace:
I20251029 02:52:21.287693 1174 rpcz_store.cc:276] 1029 02:52:13.828042 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:13.828154 (+ 112us) service_pool.cc:225] Handling call
1029 02:52:21.287476 (+7459322us) ts_tablet_manager.cc:1151] Deleting tablet e40d4910365f45e9a45d9b3bc5037c95
1029 02:52:21.287489 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:21.287588 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7459153}
I20251029 02:52:21.288045 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 1079) took 7383 ms. Trace:
I20251029 02:52:21.288123 1174 rpcz_store.cc:276] 1029 02:52:13.904888 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:13.904965 (+ 77us) service_pool.cc:225] Handling call
1029 02:52:21.287935 (+7382970us) ts_tablet_manager.cc:1151] Deleting tablet 3e9a076e086141ef8ed59d4c8633f961
1029 02:52:21.287946 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:21.288030 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7382779}
W20251029 02:52:21.288434 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet b4621bf9e6324eaeaa64a6ca51cd5ca9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b4621bf9e6324eaeaa64a6ca51cd5ca9 already in progress: creating tablet
W20251029 02:52:21.288793 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet e40d4910365f45e9a45d9b3bc5037c95 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e40d4910365f45e9a45d9b3bc5037c95 already in progress: creating tablet
W20251029 02:52:21.289381 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 3e9a076e086141ef8ed59d4c8633f961 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3e9a076e086141ef8ed59d4c8633f961 already in progress: creating tablet
I20251029 02:52:21.289829 915 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c7b78fd458e4414b9fc3aa064a80b206" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
W20251029 02:52:21.290369 711 leader_election.cc:343] T c7b78fd458e4414b9fc3aa064a80b206 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:21.292305 1167 tablet_bootstrap.cc:492] T 9492c1027aba4455ad8f539addaed5a7 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:21.292498 1167 ts_tablet_manager.cc:1403] T 9492c1027aba4455ad8f539addaed5a7 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.434s user 0.009s sys 0.006s
I20251029 02:52:21.293284 1167 raft_consensus.cc:359] T 9492c1027aba4455ad8f539addaed5a7 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:21.293519 1167 raft_consensus.cc:385] T 9492c1027aba4455ad8f539addaed5a7 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20251029 02:52:21.294164 917 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.512s user 0.009s sys 0.008s
W20251029 02:52:21.294308 917 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 0.513s user 0.009s sys 0.008s
I20251029 02:52:21.303298 3067 tablet.cc:1722] T cedfb53078d94c81b5e294ce7438325e P c756528642a543eeacd93cef3e335a7d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20251029 02:52:21.314702 1027 tablet_service.cc:1505] Processing CreateTablet for tablet d12a2ead07b146b6a8ae7446277ac0e5 (DEFAULT_TABLE table=default.Table_e6fd174aebf746e28bfa0c2ef9a699c2 [id=ccf67bcaae1d4d949da9d60fcb2d5e64]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:21.315213 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d12a2ead07b146b6a8ae7446277ac0e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:21.317092 888 tablet_service.cc:1505] Processing CreateTablet for tablet 912a3e75820245469322c8ecff1ab5d0 (DEFAULT_TABLE table=default.Table_dc881c73ba4f46bf92afb390195d2b04 [id=50283aa86f87412c9506504150a0a902]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:21.317451 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 912a3e75820245469322c8ecff1ab5d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:21.318912 1034 tablet_service.cc:1505] Processing CreateTablet for tablet 912a3e75820245469322c8ecff1ab5d0 (DEFAULT_TABLE table=default.Table_dc881c73ba4f46bf92afb390195d2b04 [id=50283aa86f87412c9506504150a0a902]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:21.319312 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 912a3e75820245469322c8ecff1ab5d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:21.322712 748 tablet_service.cc:1505] Processing CreateTablet for tablet 912a3e75820245469322c8ecff1ab5d0 (DEFAULT_TABLE table=default.Table_dc881c73ba4f46bf92afb390195d2b04 [id=50283aa86f87412c9506504150a0a902]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:21.323082 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 912a3e75820245469322c8ecff1ab5d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:21.323814 3048 raft_consensus.cc:493] T e3ac6597d52441569a7aeef051087ad2 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:21.323937 3048 raft_consensus.cc:515] T e3ac6597d52441569a7aeef051087ad2 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:21.324575 1176 ts_tablet_manager.cc:1929] T 506f8c112a3b43e1a3d6bbafb8cb81e8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:21.324671 1176 log.cc:1199] T 506f8c112a3b43e1a3d6bbafb8cb81e8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/506f8c112a3b43e1a3d6bbafb8cb81e8
I20251029 02:52:21.324931 1176 ts_tablet_manager.cc:1950] T 506f8c112a3b43e1a3d6bbafb8cb81e8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:21.325067 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e3ac6597d52441569a7aeef051087ad2" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:21.325495 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet 506f8c112a3b43e1a3d6bbafb8cb81e8 (table default.Table_d7cbd7b4c4bf4886a4ccf4c89da3cbce [id=9107fd53f5c94d2d80be50ef1f7c8e9f]) successfully deleted
W20251029 02:52:21.325523 849 leader_election.cc:343] T e3ac6597d52441569a7aeef051087ad2 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:21.326200 3048 leader_election.cc:290] T e3ac6597d52441569a7aeef051087ad2 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:21.326285 769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e3ac6597d52441569a7aeef051087ad2" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:21.326471 769 raft_consensus.cc:2468] T e3ac6597d52441569a7aeef051087ad2 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:21.326611 756 tablet_service.cc:1505] Processing CreateTablet for tablet d12a2ead07b146b6a8ae7446277ac0e5 (DEFAULT_TABLE table=default.Table_e6fd174aebf746e28bfa0c2ef9a699c2 [id=ccf67bcaae1d4d949da9d60fcb2d5e64]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:21.326892 847 leader_election.cc:304] T e3ac6597d52441569a7aeef051087ad2 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5
I20251029 02:52:21.327031 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d12a2ead07b146b6a8ae7446277ac0e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:21.327538 3048 raft_consensus.cc:2804] T e3ac6597d52441569a7aeef051087ad2 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:21.327608 3048 raft_consensus.cc:493] T e3ac6597d52441569a7aeef051087ad2 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:21.327661 3048 raft_consensus.cc:3060] T e3ac6597d52441569a7aeef051087ad2 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:21.331247 3048 raft_consensus.cc:515] T e3ac6597d52441569a7aeef051087ad2 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:21.332227 1060 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e3ac6597d52441569a7aeef051087ad2" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
W20251029 02:52:21.332934 849 leader_election.cc:343] T e3ac6597d52441569a7aeef051087ad2 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:21.333608 3048 leader_election.cc:290] T e3ac6597d52441569a7aeef051087ad2 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:21.333704 769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e3ac6597d52441569a7aeef051087ad2" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:21.333886 769 raft_consensus.cc:3060] T e3ac6597d52441569a7aeef051087ad2 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
W20251029 02:52:21.334213 849 consensus_peers.cc:597] T cc802bda2f6a4e439b605f2cafa37967 P c756528642a543eeacd93cef3e335a7d -> Peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Couldn't send request to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cc802bda2f6a4e439b605f2cafa37967. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:21.334775 887 tablet_service.cc:1505] Processing CreateTablet for tablet d12a2ead07b146b6a8ae7446277ac0e5 (DEFAULT_TABLE table=default.Table_e6fd174aebf746e28bfa0c2ef9a699c2 [id=ccf67bcaae1d4d949da9d60fcb2d5e64]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:21.335163 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d12a2ead07b146b6a8ae7446277ac0e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:21.336635 769 raft_consensus.cc:2468] T e3ac6597d52441569a7aeef051087ad2 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:21.337057 847 leader_election.cc:304] T e3ac6597d52441569a7aeef051087ad2 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5
W20251029 02:52:21.337626 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 6fcf3e12810d41ddb96d9f2cf52f2c5d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6fcf3e12810d41ddb96d9f2cf52f2c5d already in progress: creating tablet
W20251029 02:52:21.337953 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.3:45003, user_credentials={real_user=slave}} blocked reactor thread for 48302.7us
I20251029 02:52:21.338292 1172 tablet_bootstrap.cc:492] T 38f83ebcb2a440bc8e6a8d36607e6666 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:21.338542 1172 ts_tablet_manager.cc:1403] T 38f83ebcb2a440bc8e6a8d36607e6666 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.196s user 0.004s sys 0.002s
I20251029 02:52:21.339529 1172 raft_consensus.cc:359] T 38f83ebcb2a440bc8e6a8d36607e6666 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:21.339788 1172 raft_consensus.cc:385] T 38f83ebcb2a440bc8e6a8d36607e6666 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:21.339926 1172 raft_consensus.cc:740] T 38f83ebcb2a440bc8e6a8d36607e6666 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:21.198642 1166 consensus_queue.cc:260] T 294afb9b012c40c0b05b73795d6f0ac8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:21.344357 1166 ts_tablet_manager.cc:1434] T 294afb9b012c40c0b05b73795d6f0ac8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.153s user 0.001s sys 0.001s
I20251029 02:52:21.344529 1167 raft_consensus.cc:740] T 9492c1027aba4455ad8f539addaed5a7 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:21.344874 1167 consensus_queue.cc:260] T 9492c1027aba4455ad8f539addaed5a7 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:21.348081 526 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_1cda56d119414969969012027b7e7671" } modify_external_catalogs: true
I20251029 02:52:21.348259 526 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_1cda56d119414969969012027b7e7671" } modify_external_catalogs: true
I20251029 02:52:21.349659 522 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_e83f9801f1db4c51bddc74af9f67fe30" } modify_external_catalogs: true
I20251029 02:52:21.349824 522 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_e83f9801f1db4c51bddc74af9f67fe30" } modify_external_catalogs: true
W20251029 02:52:21.360738 518 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b41e114ed6e94f5895bdec9bee836d2d consensus state has no leader
I20251029 02:52:21.362169 523 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 75aa2dd1ed1b49649f3257df5d45cb32
I20251029 02:52:21.362892 1031 tablet_service.cc:1552] Processing DeleteTablet for tablet 75aa2dd1ed1b49649f3257df5d45cb32 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:20 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:21.362893 1034 tablet_service.cc:1505] Processing CreateTablet for tablet ccb824368be0485dadb80adbda10e523 (DEFAULT_TABLE table=default.Table_1cda56d119414969969012027b7e7671 [id=20574d69a3694b2baf7675761bbb44d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:21.364960 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_a5501e95805a4c3c92789ea6dcce7900"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:21.365507 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a5501e95805a4c3c92789ea6dcce7900 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:21.367887 1031 tablet_service.cc:1505] Processing CreateTablet for tablet ef6e206fa5454f658df7204f58800cdd (DEFAULT_TABLE table=default.Table_a9246d0b85c04d90a2531912329b8366 [id=d8c3965d712f4be6b27033f2da535da2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:21.371435 518 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet b125cc9aaf4248b8bfa48fb152b832a6 (table default.Table_110e1b64737c4040ada22360d5a373cb [id=3513c69f6b3941849ac5b8ae0a5846d3]). Expected version 5 got 0
I20251029 02:52:21.371793 518 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet e3ac6597d52441569a7aeef051087ad2 (table default.Table_2c15df0a33124ce69282c48d6455d1b2 [id=242159edd3bc4771ac206d4e450b9475]). Expected version 2 got 0
I20251029 02:52:21.372295 518 catalog_manager.cc:5649] T c45ffc304e73490387ce800b702b5881 P c756528642a543eeacd93cef3e335a7d reported cstate change: term changed from 0 to 1, leader changed from <none> to c756528642a543eeacd93cef3e335a7d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "c756528642a543eeacd93cef3e335a7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } health_report { overall_health: UNKNOWN } } }
I20251029 02:52:21.372776 518 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet c45ffc304e73490387ce800b702b5881 (table default.Table_075076404d494b29b2cb90e9f10219e5 [id=97e8edf0d4df452d96c9e13e55aa0736]). Expected version 1 got 0
I20251029 02:52:21.373060 518 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet f4729ecd182042c2a10f67ca25224101 (table default.Table_973ef856041949598a23da3dab8b40db [id=1dd2d5488bc647c8b82c53886583b28d]). Expected version 4 got 0
I20251029 02:52:21.373556 1042 tablet_service.cc:1505] Processing CreateTablet for tablet c8fe5669dcff40368679e077108c9fcd (DEFAULT_TABLE table=default.Table_2ef80417992a45b09f93d266693e6f5b [id=d04465f38a8c4cdbb7d642ed1871dcea]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:21.377862 524 catalog_manager.cc:6194] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: default.Table_1922b14d4cfb4671a6224fcd6ace2d4e [id=aec3db766dee438c89fb2f1939ee7b7b] alter complete (version 2)
I20251029 02:52:21.378876 1028 tablet_service.cc:1552] Processing DeleteTablet for tablet f93b0c39d7174dabbe018b1546dbc198 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:14 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:21.379164 1039 tablet_service.cc:1552] Processing DeleteTablet for tablet 1df0d944717d487a98ea83a8ec04bd46 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:01 UTC) from {username='slave'} at 127.0.0.1:37626
W20251029 02:52:21.379612 525 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d12a2ead07b146b6a8ae7446277ac0e5 consensus state has no leader
I20251029 02:52:21.381845 1039 tablet_service.cc:1505] Processing CreateTablet for tablet e7af7aeff3e546b698de519b3ae5562c (DEFAULT_TABLE table=default.Table_d164b5f77ca746b28adb0849b5fb65fb [id=268ba5073a6d4f8e9b0d4c0ffa17ece7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:21.382179 1028 tablet_service.cc:1505] Processing CreateTablet for tablet 6bc9536b54c44ae4816d01264c058369 (DEFAULT_TABLE table=default.Table_f1a523ffbf3c47b0bfabb752b8c2f5a2 [id=a4744b434b4a4df295a4a6036423cfea]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:21.395685 1040 tablet_service.cc:1505] Processing CreateTablet for tablet a248268e16654a2890d11ce815b06bda (DEFAULT_TABLE table=default.Table_fd4e20d865b640bd818b71d13226aa26 [id=352e90b0914549b2a3ece35022e20621]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:21.399828 1033 tablet_service.cc:1505] Processing CreateTablet for tablet 6ca34c3ac2f94987bfbaadd4bccca519 (DEFAULT_TABLE table=default.Table_eda0a7e4e3a940b0b6e8169f8f36b0b9 [id=63af5638c18a463ead31d01ac0486f66]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:21.402618 1024 tablet_service.cc:1505] Processing CreateTablet for tablet 7a7682158cac4008a6043b357f4edc73 (DEFAULT_TABLE table=default.Table_015f00d002fd4196b1f126905dcaec9f [id=12ba8b4239f240379fd656b68713a5d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:21.407765 3068 tablet.cc:1722] T b125cc9aaf4248b8bfa48fb152b832a6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20251029 02:52:21.408073 1055 raft_consensus.cc:3060] T f93b0c39d7174dabbe018b1546dbc198 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:21.411518 1042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8fe5669dcff40368679e077108c9fcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:21.411908 1025 tablet_service.cc:1505] Processing CreateTablet for tablet d444cb9413f34abea1a7a95ebccc3a3a (DEFAULT_TABLE table=default.Table_af9e4be8b41a49ec84d8f5d5cd347b4a [id=ec93ec4625ff4816a602e8c8198ff4a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:21.412206 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d444cb9413f34abea1a7a95ebccc3a3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:21.412672 1029 tablet_service.cc:1505] Processing CreateTablet for tablet fee646615c1447eb8ecb37b749a7aeba (DEFAULT_TABLE table=default.Table_0847f9152d7041d7a2e814ad64f9d06b [id=12d5cf9f472e49ff98fd62314893a8b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:21.413034 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fee646615c1447eb8ecb37b749a7aeba. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:21.347453 3066 tablet.cc:1722] T b125cc9aaf4248b8bfa48fb152b832a6 P b495a49349324f7d81908f2da700f0d9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20251029 02:52:21.364995 756 tablet_service.cc:1552] Processing DeleteTablet for tablet 75aa2dd1ed1b49649f3257df5d45cb32 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:20 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:21.371412 748 tablet_service.cc:1505] Processing CreateTablet for tablet ccb824368be0485dadb80adbda10e523 (DEFAULT_TABLE table=default.Table_1cda56d119414969969012027b7e7671 [id=20574d69a3694b2baf7675761bbb44d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:21.414839 1176 tablet_replica.cc:333] T f93b0c39d7174dabbe018b1546dbc198 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: stopping tablet replica
I20251029 02:52:21.415067 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ccb824368be0485dadb80adbda10e523. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:21.415315 1172 consensus_queue.cc:260] T 38f83ebcb2a440bc8e6a8d36607e6666 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:21.350630 3047 raft_consensus.cc:2804] T e3ac6597d52441569a7aeef051087ad2 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:21.416093 1167 ts_tablet_manager.cc:1434] T 9492c1027aba4455ad8f539addaed5a7 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.123s user 0.001s sys 0.001s
I20251029 02:52:21.416250 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ca34c3ac2f94987bfbaadd4bccca519. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:21.416831 1034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ccb824368be0485dadb80adbda10e523. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:21.374363 754 tablet_service.cc:1505] Processing CreateTablet for tablet ef6e206fa5454f658df7204f58800cdd (DEFAULT_TABLE table=default.Table_a9246d0b85c04d90a2531912329b8366 [id=d8c3965d712f4be6b27033f2da535da2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:21.418828 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a7682158cac4008a6043b357f4edc73. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:21.418871 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef6e206fa5454f658df7204f58800cdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:21.422868 1166 tablet_bootstrap.cc:492] T 688af6580c4d4637a39f817b4e324618 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:21.423367 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7af7aeff3e546b698de519b3ae5562c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:21.423686 3047 raft_consensus.cc:697] T e3ac6597d52441569a7aeef051087ad2 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:52:21.423979 3047 consensus_queue.cc:237] T e3ac6597d52441569a7aeef051087ad2 P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:21.425441 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef6e206fa5454f658df7204f58800cdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:21.426681 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6bc9536b54c44ae4816d01264c058369. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:21.426879 919 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c7b78fd458e4414b9fc3aa064a80b206" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:21.430735 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 75aa2dd1ed1b49649f3257df5d45cb32 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 75aa2dd1ed1b49649f3257df5d45cb32 already in progress: creating tablet
I20251029 02:52:21.431005 1040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a248268e16654a2890d11ce815b06bda. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:21.432003 1167 tablet_bootstrap.cc:492] T 786d9148fb1d417a8c2c8b8c53665899 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:21.351567 3067 tablet.cc:1722] T b125cc9aaf4248b8bfa48fb152b832a6 P c756528642a543eeacd93cef3e335a7d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20251029 02:52:21.354624 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 1080) took 7449 ms. Trace:
I20251029 02:52:21.432452 1174 rpcz_store.cc:276] 1029 02:52:13.905020 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:13.905624 (+ 604us) service_pool.cc:225] Handling call
1029 02:52:21.288338 (+7382714us) ts_tablet_manager.cc:1151] Deleting tablet 2cea2f2fe68545db804a732d0c01a876
1029 02:52:21.288346 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:21.354594 (+ 66248us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7382568}
I20251029 02:52:21.432864 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 1077) took 7615 ms. Trace:
I20251029 02:52:21.432938 1174 rpcz_store.cc:276] 1029 02:52:13.817607 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:13.910111 (+ 92504us) service_pool.cc:225] Handling call
1029 02:52:21.432727 (+7522616us) ts_tablet_manager.cc:1151] Deleting tablet eaf4018c0d624b1cb8c6528eea7b689e
1029 02:52:21.432739 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:21.432846 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7521527}
I20251029 02:52:21.433286 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 1081) took 7521 ms. Trace:
I20251029 02:52:21.433346 1174 rpcz_store.cc:276] 1029 02:52:13.911416 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:13.913366 (+ 1950us) service_pool.cc:225] Handling call
1029 02:52:21.433166 (+7519800us) ts_tablet_manager.cc:1151] Deleting tablet 786d9148fb1d417a8c2c8b8c53665899
1029 02:52:21.433179 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:21.433270 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7519556}
I20251029 02:52:21.433580 1174 tablet_replica.cc:333] T 58dde9397b434b089ade9968f369006a P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:21.433779 1174 raft_consensus.cc:2243] T 58dde9397b434b089ade9968f369006a P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:21.434144 1174 raft_consensus.cc:2272] T 58dde9397b434b089ade9968f369006a P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:21.434749 1174 ts_tablet_manager.cc:1916] T 58dde9397b434b089ade9968f369006a P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 02:52:21.434791 711 leader_election.cc:343] T c7b78fd458e4414b9fc3aa064a80b206 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:21.375411 739 tablet_service.cc:1505] Processing CreateTablet for tablet c8fe5669dcff40368679e077108c9fcd (DEFAULT_TABLE table=default.Table_2ef80417992a45b09f93d266693e6f5b [id=d04465f38a8c4cdbb7d642ed1871dcea]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:21.435393 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8fe5669dcff40368679e077108c9fcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:21.436465 1172 ts_tablet_manager.cc:1434] T 38f83ebcb2a440bc8e6a8d36607e6666 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.098s user 0.002s sys 0.000s
I20251029 02:52:21.362918 891 tablet_service.cc:1552] Processing DeleteTablet for tablet 75aa2dd1ed1b49649f3257df5d45cb32 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:20 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:21.437533 1175 ts_tablet_manager.cc:1929] T 688af6580c4d4637a39f817b4e324618 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:21.437624 1175 log.cc:1199] T 688af6580c4d4637a39f817b4e324618 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/688af6580c4d4637a39f817b4e324618
I20251029 02:52:21.437881 1175 ts_tablet_manager.cc:1950] T 688af6580c4d4637a39f817b4e324618 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:21.366889 900 tablet_service.cc:1505] Processing CreateTablet for tablet ccb824368be0485dadb80adbda10e523 (DEFAULT_TABLE table=default.Table_1cda56d119414969969012027b7e7671 [id=20574d69a3694b2baf7675761bbb44d5]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:21.438340 900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ccb824368be0485dadb80adbda10e523. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:21.384399 740 tablet_service.cc:1505] Processing CreateTablet for tablet e7af7aeff3e546b698de519b3ae5562c (DEFAULT_TABLE table=default.Table_d164b5f77ca746b28adb0849b5fb65fb [id=268ba5073a6d4f8e9b0d4c0ffa17ece7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:21.438648 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 1157) took 5684 ms. Trace:
I20251029 02:52:21.438824 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7af7aeff3e546b698de519b3ae5562c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:21.438732 1175 rpcz_store.cc:276] 1029 02:52:15.754067 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:15.754327 (+ 260us) service_pool.cc:225] Handling call
1029 02:52:21.174608 (+5420281us) ts_tablet_manager.cc:1151] Deleting tablet 688af6580c4d4637a39f817b4e324618
1029 02:52:21.174621 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:21.295287 (+120666us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:21.437440 (+142153us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:21.438628 (+ 1188us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5420151}
I20251029 02:52:21.439042 1175 tablet_replica.cc:333] T 5815626b05204b578cc4a110f246bf10 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:21.439225 1175 raft_consensus.cc:2243] T 5815626b05204b578cc4a110f246bf10 P b495a49349324f7d81908f2da700f0d9 [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:21.439524 1175 raft_consensus.cc:2272] T 5815626b05204b578cc4a110f246bf10 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:21.392570 750 tablet_service.cc:1505] Processing CreateTablet for tablet 6bc9536b54c44ae4816d01264c058369 (DEFAULT_TABLE table=default.Table_f1a523ffbf3c47b0bfabb752b8c2f5a2 [id=a4744b434b4a4df295a4a6036423cfea]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:21.442363 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6bc9536b54c44ae4816d01264c058369. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:21.443413 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 2cea2f2fe68545db804a732d0c01a876 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2cea2f2fe68545db804a732d0c01a876 already in progress: creating tablet
I20251029 02:52:21.443542 520 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 5171c8aa52ac497d9858bb6d0ea8dfee
W20251029 02:52:21.443645 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet eaf4018c0d624b1cb8c6528eea7b689e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eaf4018c0d624b1cb8c6528eea7b689e already in progress: creating tablet
W20251029 02:52:21.443811 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 786d9148fb1d417a8c2c8b8c53665899 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 786d9148fb1d417a8c2c8b8c53665899 already in progress: creating tablet
I20251029 02:52:21.444377 526 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet ccb824368be0485dadb80adbda10e523
I20251029 02:52:21.444895 891 tablet_service.cc:1552] Processing DeleteTablet for tablet 5171c8aa52ac497d9858bb6d0ea8dfee with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:21 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:21.397171 742 tablet_service.cc:1505] Processing CreateTablet for tablet a248268e16654a2890d11ce815b06bda (DEFAULT_TABLE table=default.Table_fd4e20d865b640bd818b71d13226aa26 [id=352e90b0914549b2a3ece35022e20621]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:21.445606 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a248268e16654a2890d11ce815b06bda. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:21.401033 744 tablet_service.cc:1505] Processing CreateTablet for tablet 6ca34c3ac2f94987bfbaadd4bccca519 (DEFAULT_TABLE table=default.Table_eda0a7e4e3a940b0b6e8169f8f36b0b9 [id=63af5638c18a463ead31d01ac0486f66]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:21.446847 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ca34c3ac2f94987bfbaadd4bccca519. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:21.446763 523 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_b1481652c89a48339dff0a29f019207c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:21.447263 523 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b1481652c89a48339dff0a29f019207c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:21.448776 891 tablet_service.cc:1552] Processing DeleteTablet for tablet ccb824368be0485dadb80adbda10e523 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:21 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:21.372444 888 tablet_service.cc:1505] Processing CreateTablet for tablet ef6e206fa5454f658df7204f58800cdd (DEFAULT_TABLE table=default.Table_a9246d0b85c04d90a2531912329b8366 [id=d8c3965d712f4be6b27033f2da535da2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:21.448825 526 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_ee31f1e11459439d94caf01db7e77312"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:21.449187 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef6e206fa5454f658df7204f58800cdd. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:21.449293 526 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ee31f1e11459439d94caf01db7e77312 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:21.450330 517 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_e51f77418e95447f9839662efc11d340" } new_table_name: "default.Table_bcd2fce2c88e49028d2377e355d6bf9c" modify_external_catalogs: true
I20251029 02:52:21.404170 738 tablet_service.cc:1505] Processing CreateTablet for tablet 7a7682158cac4008a6043b357f4edc73 (DEFAULT_TABLE table=default.Table_015f00d002fd4196b1f126905dcaec9f [id=12ba8b4239f240379fd656b68713a5d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:21.376187 879 tablet_service.cc:1505] Processing CreateTablet for tablet c8fe5669dcff40368679e077108c9fcd (DEFAULT_TABLE table=default.Table_2ef80417992a45b09f93d266693e6f5b [id=d04465f38a8c4cdbb7d642ed1871dcea]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:21.450713 1175 ts_tablet_manager.cc:1916] T 5815626b05204b578cc4a110f246bf10 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:21.450747 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a7682158cac4008a6043b357f4edc73. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:21.450790 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8fe5669dcff40368679e077108c9fcd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:21.451490 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_c904766cee9342b6a37e3a43ac3762ac"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:21.451901 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c904766cee9342b6a37e3a43ac3762ac in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251029 02:52:21.409188 712 consensus_peers.cc:597] T 6dff036b32904ddb9598be4b79504bb6 P b495a49349324f7d81908f2da700f0d9 -> Peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Couldn't send request to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6dff036b32904ddb9598be4b79504bb6. This is attempt 1: this message will repeat every 5th retry.
W20251029 02:52:21.452791 712 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:45003, user_credentials={real_user=slave}} blocked reactor thread for 43639.5us
I20251029 02:52:21.453145 524 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_2b3547dd03dd4b7499a9d44977670e0c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:21.453526 524 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2b3547dd03dd4b7499a9d44977670e0c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:21.383352 895 tablet_service.cc:1505] Processing CreateTablet for tablet e7af7aeff3e546b698de519b3ae5562c (DEFAULT_TABLE table=default.Table_d164b5f77ca746b28adb0849b5fb65fb [id=268ba5073a6d4f8e9b0d4c0ffa17ece7]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:21.454566 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_3dee083fd8914e56a459aae142a944f8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251029 02:52:21.454770 895 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7af7aeff3e546b698de519b3ae5562c. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:21.454968 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3dee083fd8914e56a459aae142a944f8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:21.455987 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_a4ae3b12ee95487daebc2af160055bc1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:21.456379 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a4ae3b12ee95487daebc2af160055bc1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:21.457355 520 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_0e24ef50ea804278ab30f6ca2212cb0c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:21.457719 520 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0e24ef50ea804278ab30f6ca2212cb0c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:21.392716 885 tablet_service.cc:1505] Processing CreateTablet for tablet 6bc9536b54c44ae4816d01264c058369 (DEFAULT_TABLE table=default.Table_f1a523ffbf3c47b0bfabb752b8c2f5a2 [id=a4744b434b4a4df295a4a6036423cfea]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:21.458876 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6bc9536b54c44ae4816d01264c058369. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:21.459292 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 688af6580c4d4637a39f817b4e324618 (table default.Table_a75c04341d7244f3ba161a938bb401b3 [id=0dc6f8e6205d4e3997946747bbc86e2c]) successfully deleted
I20251029 02:52:21.459662 756 tablet_service.cc:1552] Processing DeleteTablet for tablet ccb824368be0485dadb80adbda10e523 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:21 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:21.459951 522 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 68422ab54c474246b6c7f8cc3c668bcd
I20251029 02:52:21.460119 522 master_service.cc:954] ReplaceTablet: received request to replace tablet 09e3ce70eb9a4a6c9035dabbf7434636 from {username='slave'} at 127.0.0.1:35314
I20251029 02:52:21.397570 894 tablet_service.cc:1505] Processing CreateTablet for tablet a248268e16654a2890d11ce815b06bda (DEFAULT_TABLE table=default.Table_fd4e20d865b640bd818b71d13226aa26 [id=352e90b0914549b2a3ece35022e20621]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:21.462802 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a248268e16654a2890d11ce815b06bda. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:21.411530 749 tablet_service.cc:1505] Processing CreateTablet for tablet d444cb9413f34abea1a7a95ebccc3a3a (DEFAULT_TABLE table=default.Table_af9e4be8b41a49ec84d8f5d5cd347b4a [id=ec93ec4625ff4816a602e8c8198ff4a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:21.466929 1175 ts_tablet_manager.cc:1929] T 5815626b05204b578cc4a110f246bf10 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:21.467005 1167 tablet_bootstrap.cc:654] T 786d9148fb1d417a8c2c8b8c53665899 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:21.467041 1175 log.cc:1199] T 5815626b05204b578cc4a110f246bf10 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/5815626b05204b578cc4a110f246bf10
I20251029 02:52:21.466948 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d444cb9413f34abea1a7a95ebccc3a3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:21.467382 1175 ts_tablet_manager.cc:1950] T 5815626b05204b578cc4a110f246bf10 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:21.467689 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 1158) took 5698 ms. Trace:
I20251029 02:52:21.467763 1175 rpcz_store.cc:276] 1029 02:52:15.769522 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:15.769744 (+ 222us) service_pool.cc:225] Handling call
1029 02:52:21.439003 (+5669259us) ts_tablet_manager.cc:1151] Deleting tablet 5815626b05204b578cc4a110f246bf10
1029 02:52:21.439011 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:21.463913 (+ 24902us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:21.466852 (+ 2939us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:21.467671 (+ 819us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5669042}
I20251029 02:52:21.468093 1175 tablet_replica.cc:333] T b0fec06ad37d4c12a7045d424b88cc5e P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:21.468144 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 5815626b05204b578cc4a110f246bf10 (table default.Table_796f8f19849d4b01922da96435e6833c [id=45dc803b4e1041e78f40a3e2e1369041]) successfully deleted
I20251029 02:52:21.468777 1166 tablet_bootstrap.cc:654] T 688af6580c4d4637a39f817b4e324618 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:21.469237 1052 raft_consensus.cc:3060] T 7816556521144c928d7a02694e22e79f P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:21.473127 3085 raft_consensus.cc:493] T 15137fc1c70347b381ffdfe5653a5f22 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:21.473266 3085 raft_consensus.cc:515] T 15137fc1c70347b381ffdfe5653a5f22 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:21.473831 3085 leader_election.cc:290] T 15137fc1c70347b381ffdfe5653a5f22 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:21.475052 769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15137fc1c70347b381ffdfe5653a5f22" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
W20251029 02:52:21.475764 989 leader_election.cc:343] T 15137fc1c70347b381ffdfe5653a5f22 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Illegal state: must be running to vote when last-logged opid is not known
W20251029 02:52:21.482676 847 consensus_peers.cc:597] T fcb3db8399aa4cf188b07dda220f7f5b P c756528642a543eeacd93cef3e335a7d -> Peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Couldn't send request to peer b495a49349324f7d81908f2da700f0d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fcb3db8399aa4cf188b07dda220f7f5b. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:21.494333 1175 raft_consensus.cc:2243] T b0fec06ad37d4c12a7045d424b88cc5e P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:21.496551 1059 raft_consensus.cc:3060] T fcb3db8399aa4cf188b07dda220f7f5b P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:21.498802 1175 raft_consensus.cc:2272] T b0fec06ad37d4c12a7045d424b88cc5e P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:21.500247 1175 ts_tablet_manager.cc:1916] T b0fec06ad37d4c12a7045d424b88cc5e P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:21.500897 1039 tablet_service.cc:1552] Processing DeleteTablet for tablet 5171c8aa52ac497d9858bb6d0ea8dfee with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:21 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:21.501181 1042 tablet_service.cc:1552] Processing DeleteTablet for tablet ccb824368be0485dadb80adbda10e523 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:21 UTC) from {username='slave'} at 127.0.0.1:37626
I20251029 02:52:21.501401 750 tablet_service.cc:1552] Processing DeleteTablet for tablet 5171c8aa52ac497d9858bb6d0ea8dfee with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:21 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:21.503182 919 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "15137fc1c70347b381ffdfe5653a5f22" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
W20251029 02:52:21.505128 990 leader_election.cc:343] T 15137fc1c70347b381ffdfe5653a5f22 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:21.505254 990 leader_election.cc:304] T 15137fc1c70347b381ffdfe5653a5f22 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5; no voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:21.505592 3054 raft_consensus.cc:2749] T 15137fc1c70347b381ffdfe5653a5f22 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:21.505897 1174 ts_tablet_manager.cc:1929] T 58dde9397b434b089ade9968f369006a P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:21.510314 518 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_110e1b64737c4040ada22360d5a373cb" } new_table_name: "default.Table_b609086feb2640449e2493c5c4b16992" modify_external_catalogs: true
I20251029 02:52:21.404513 886 tablet_service.cc:1505] Processing CreateTablet for tablet 7a7682158cac4008a6043b357f4edc73 (DEFAULT_TABLE table=default.Table_015f00d002fd4196b1f126905dcaec9f [id=12ba8b4239f240379fd656b68713a5d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:21.412056 884 tablet_service.cc:1505] Processing CreateTablet for tablet d444cb9413f34abea1a7a95ebccc3a3a (DEFAULT_TABLE table=default.Table_af9e4be8b41a49ec84d8f5d5cd347b4a [id=ec93ec4625ff4816a602e8c8198ff4a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:21.515094 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d444cb9413f34abea1a7a95ebccc3a3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:21.515241 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a7682158cac4008a6043b357f4edc73. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:21.514463 885 tablet_service.cc:1552] Processing DeleteTablet for tablet 67651306f3b44e7eb14dcee9a2007a81 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:03 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:21.514649 879 tablet_service.cc:1552] Processing DeleteTablet for tablet d041e87ceb11441983d0e38b7929c3a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:01 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:21.401396 887 tablet_service.cc:1505] Processing CreateTablet for tablet 6ca34c3ac2f94987bfbaadd4bccca519 (DEFAULT_TABLE table=default.Table_eda0a7e4e3a940b0b6e8169f8f36b0b9 [id=63af5638c18a463ead31d01ac0486f66]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:21.527642 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ca34c3ac2f94987bfbaadd4bccca519. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:21.530571 1174 log.cc:1199] T 58dde9397b434b089ade9968f369006a P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/58dde9397b434b089ade9968f369006a
I20251029 02:52:21.531004 1174 ts_tablet_manager.cc:1950] T 58dde9397b434b089ade9968f369006a P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:21.531302 1172 tablet_bootstrap.cc:492] T ff4c38150df04a75855e0e60ff417ddb P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:21.531610 1166 tablet_bootstrap.cc:492] T 688af6580c4d4637a39f817b4e324618 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:21.531821 1166 ts_tablet_manager.cc:1403] T 688af6580c4d4637a39f817b4e324618 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.109s user 0.003s sys 0.002s
I20251029 02:52:21.532687 1166 raft_consensus.cc:359] T 688af6580c4d4637a39f817b4e324618 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:21.535681 750 tablet_service.cc:1505] Processing CreateTablet for tablet fee646615c1447eb8ecb37b749a7aeba (DEFAULT_TABLE table=default.Table_0847f9152d7041d7a2e814ad64f9d06b [id=12d5cf9f472e49ff98fd62314893a8b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:21.536194 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fee646615c1447eb8ecb37b749a7aeba. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:21.533646 1176 raft_consensus.cc:2243] T f93b0c39d7174dabbe018b1546dbc198 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus shutting down.
W20251029 02:52:21.542469 849 consensus_peers.cc:597] T f93b0c39d7174dabbe018b1546dbc198 P c756528642a543eeacd93cef3e335a7d -> Peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Couldn't send request to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:21.542822 1176 pending_rounds.cc:70] T f93b0c39d7174dabbe018b1546dbc198 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Trying to abort 1 pending ops.
I20251029 02:52:21.542884 1176 pending_rounds.cc:77] T f93b0c39d7174dabbe018b1546dbc198 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7215949159913771008 op_type: NO_OP noop_request { }
I20251029 02:52:21.542976 1176 raft_consensus.cc:2889] T f93b0c39d7174dabbe018b1546dbc198 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20251029 02:52:21.543064 1176 raft_consensus.cc:2272] T f93b0c39d7174dabbe018b1546dbc198 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:21.543439 1166 raft_consensus.cc:385] T 688af6580c4d4637a39f817b4e324618 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:21.543555 1166 raft_consensus.cc:740] T 688af6580c4d4637a39f817b4e324618 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:21.543849 1166 consensus_queue.cc:260] T 688af6580c4d4637a39f817b4e324618 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:21.544495 1166 ts_tablet_manager.cc:1434] T 688af6580c4d4637a39f817b4e324618 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.013s user 0.001s sys 0.000s
I20251029 02:52:21.544600 526 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet c45ffc304e73490387ce800b702b5881 (table default.Table_075076404d494b29b2cb90e9f10219e5 [id=97e8edf0d4df452d96c9e13e55aa0736]). Expected version 1 got 0
I20251029 02:52:21.544837 1166 tablet_bootstrap.cc:492] T ff4c38150df04a75855e0e60ff417ddb P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:21.547128 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 1082) took 7619 ms. Trace:
I20251029 02:52:21.547333 1174 rpcz_store.cc:276] 1029 02:52:13.927261 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:13.927345 (+ 84us) service_pool.cc:225] Handling call
1029 02:52:21.433537 (+7506192us) ts_tablet_manager.cc:1151] Deleting tablet 58dde9397b434b089ade9968f369006a
1029 02:52:21.433545 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:21.502709 (+ 69164us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:21.505801 (+ 3092us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:21.547109 (+ 41308us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7505988}
I20251029 02:52:21.549541 739 tablet_service.cc:1505] Processing CreateTablet for tablet 07c6c2679b804fc09e2bcf19554e7c7d (DEFAULT_TABLE table=default.Table_2c1c156e87f540629b084beda3c5768d [id=49391c3be8b041fdb903757b57d7b851]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:21.549722 1024 tablet_service.cc:1505] Processing CreateTablet for tablet 07c6c2679b804fc09e2bcf19554e7c7d (DEFAULT_TABLE table=default.Table_2c1c156e87f540629b084beda3c5768d [id=49391c3be8b041fdb903757b57d7b851]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:21.550199 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07c6c2679b804fc09e2bcf19554e7c7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:21.550343 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07c6c2679b804fc09e2bcf19554e7c7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:21.557569 1172 tablet_bootstrap.cc:654] T ff4c38150df04a75855e0e60ff417ddb P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:21.557888 1174 tablet_replica.cc:333] T 7bb85a12f573494cbb3118d1947eda27 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:21.558167 1174 raft_consensus.cc:2243] T 7bb85a12f573494cbb3118d1947eda27 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:21.558642 1174 raft_consensus.cc:2272] T 7bb85a12f573494cbb3118d1947eda27 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:21.560350 1174 ts_tablet_manager.cc:1916] T 7bb85a12f573494cbb3118d1947eda27 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:21.561547 1176 ts_tablet_manager.cc:1916] T f93b0c39d7174dabbe018b1546dbc198 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:21.557540 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 58dde9397b434b089ade9968f369006a (table default.Table_b9f2a2e3d9f8446fbd46ca6c48155ccf [id=233cd457a662483a84d93d4243a978b1]) successfully deleted
I20251029 02:52:21.566591 526 catalog_manager.cc:5649] T e3ac6597d52441569a7aeef051087ad2 P c756528642a543eeacd93cef3e335a7d reported cstate change: term changed from 0 to 1, leader changed from <none> to c756528642a543eeacd93cef3e335a7d (127.0.104.2). New cstate: current_term: 1 leader_uuid: "c756528642a543eeacd93cef3e335a7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } health_report { overall_health: UNKNOWN } } }
I20251029 02:52:21.567010 526 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet e3ac6597d52441569a7aeef051087ad2 (table default.Table_2c15df0a33124ce69282c48d6455d1b2 [id=242159edd3bc4771ac206d4e450b9475]). Expected version 2 got 0
I20251029 02:52:21.567329 526 catalog_manager.cc:5741] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701) does not have the latest schema for tablet 08371b3daa184a0e9f4eb3d313b52c56 (table default.Table_1453b5168e64406688903f1d6927f549 [id=40f1beecaad24a64b4348663c9ae9afd]). Expected version 1 got 0
I20251029 02:52:21.568636 1176 ts_tablet_manager.cc:1929] T f93b0c39d7174dabbe018b1546dbc198 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:21.568817 1176 log.cc:1199] T f93b0c39d7174dabbe018b1546dbc198 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-2/wal/wals/f93b0c39d7174dabbe018b1546dbc198
I20251029 02:52:21.568835 886 tablet_service.cc:1505] Processing CreateTablet for tablet 07c6c2679b804fc09e2bcf19554e7c7d (DEFAULT_TABLE table=default.Table_2c1c156e87f540629b084beda3c5768d [id=49391c3be8b041fdb903757b57d7b851]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:21.569324 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07c6c2679b804fc09e2bcf19554e7c7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:21.569550 1176 ts_tablet_manager.cc:1950] T f93b0c39d7174dabbe018b1546dbc198 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Deleting consensus metadata
I20251029 02:52:21.570381 513 catalog_manager.cc:4985] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): tablet f93b0c39d7174dabbe018b1546dbc198 (table default.Table_a2eda0beb1fb4fd3aa2a64eded1e17ad [id=61fac9a7d5b049e08485ce8695ba5a48]) successfully deleted
W20251029 02:52:21.571115 513 catalog_manager.cc:4952] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 1df0d944717d487a98ea83a8ec04bd46 because the tablet was not found. No further retry: Not found: Tablet not found: 1df0d944717d487a98ea83a8ec04bd46
I20251029 02:52:21.571736 750 tablet_service.cc:1505] Processing CreateTablet for tablet 64e5e789102e4067a1c3c551784cdef0 (DEFAULT_TABLE table=default.Table_570d77b4786e4f80a120744a8e94abf0 [id=c2354f113a854046a5cffc064a3ef21b]), partition=RANGE (key) PARTITION UNBOUNDED
W20251029 02:52:21.571985 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 5171c8aa52ac497d9858bb6d0ea8dfee because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5171c8aa52ac497d9858bb6d0ea8dfee already in progress: creating tablet
W20251029 02:52:21.572295 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet ccb824368be0485dadb80adbda10e523 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ccb824368be0485dadb80adbda10e523 already in progress: creating tablet
I20251029 02:52:21.572458 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64e5e789102e4067a1c3c551784cdef0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:21.573014 1166 tablet_bootstrap.cc:654] T ff4c38150df04a75855e0e60ff417ddb P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:21.574740 1175 ts_tablet_manager.cc:1929] T b0fec06ad37d4c12a7045d424b88cc5e P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:21.574901 1175 log.cc:1199] T b0fec06ad37d4c12a7045d424b88cc5e P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/b0fec06ad37d4c12a7045d424b88cc5e
I20251029 02:52:21.575331 1175 ts_tablet_manager.cc:1950] T b0fec06ad37d4c12a7045d424b88cc5e P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:21.576218 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 1159) took 5806 ms. Trace:
I20251029 02:52:21.576380 1175 rpcz_store.cc:276] 1029 02:52:15.769691 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:15.770058 (+ 367us) service_pool.cc:225] Handling call
1029 02:52:21.468051 (+5697993us) ts_tablet_manager.cc:1151] Deleting tablet b0fec06ad37d4c12a7045d424b88cc5e
1029 02:52:21.468061 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:21.540044 (+ 71983us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:21.574632 (+ 34588us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:21.576195 (+ 1563us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5697831}
I20251029 02:52:21.576946 1175 tablet_replica.cc:333] T cc802bda2f6a4e439b605f2cafa37967 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:21.577183 1175 raft_consensus.cc:2243] T cc802bda2f6a4e439b605f2cafa37967 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:21.577401 1175 raft_consensus.cc:2272] T cc802bda2f6a4e439b605f2cafa37967 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:21.577986 1175 ts_tablet_manager.cc:1916] T cc802bda2f6a4e439b605f2cafa37967 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:21.579705 1167 tablet_bootstrap.cc:492] T 786d9148fb1d417a8c2c8b8c53665899 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:21.579902 1167 ts_tablet_manager.cc:1403] T 786d9148fb1d417a8c2c8b8c53665899 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.164s user 0.003s sys 0.003s
I20251029 02:52:21.580632 1167 raft_consensus.cc:359] T 786d9148fb1d417a8c2c8b8c53665899 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:21.580958 1167 raft_consensus.cc:385] T 786d9148fb1d417a8c2c8b8c53665899 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:21.581065 1167 raft_consensus.cc:740] T 786d9148fb1d417a8c2c8b8c53665899 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:21.581379 1167 consensus_queue.cc:260] T 786d9148fb1d417a8c2c8b8c53665899 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:21.582074 1167 ts_tablet_manager.cc:1434] T 786d9148fb1d417a8c2c8b8c53665899 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 02:52:21.584127 1031 tablet_service.cc:1505] Processing CreateTablet for tablet 135bb898a4f24ee4b659be0b47fa690e (DEFAULT_TABLE table=default.Table_bdff4e69029249389def3ceac417dfbc [id=e05669b2913f4e09903138c3e2da3794]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:21.584607 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 135bb898a4f24ee4b659be0b47fa690e. 1 dirs total, 0 dirs full, 0 dirs failed
W20251029 02:52:21.588011 517 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ffe664eab28c4aa88ac299c5b5e66410 consensus state has no leader
I20251029 02:52:21.590776 879 tablet_service.cc:1505] Processing CreateTablet for tablet 64e5e789102e4067a1c3c551784cdef0 (DEFAULT_TABLE table=default.Table_570d77b4786e4f80a120744a8e94abf0 [id=c2354f113a854046a5cffc064a3ef21b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:21.593421 1166 tablet_bootstrap.cc:492] T ff4c38150df04a75855e0e60ff417ddb P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:21.593427 525 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_00585488392141a58ecfc968a6f3ca61" } new_table_name: "default.Table_4e27afb943a9491b9b1b08e2fd933a35" modify_external_catalogs: true
I20251029 02:52:21.593645 1166 ts_tablet_manager.cc:1403] T ff4c38150df04a75855e0e60ff417ddb P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.049s user 0.004s sys 0.000s
I20251029 02:52:21.594370 1166 raft_consensus.cc:359] T ff4c38150df04a75855e0e60ff417ddb P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:21.594554 1166 raft_consensus.cc:385] T ff4c38150df04a75855e0e60ff417ddb P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:21.594862 1166 raft_consensus.cc:740] T ff4c38150df04a75855e0e60ff417ddb P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:21.594871 1167 tablet_bootstrap.cc:492] T c64d6761ef924ccaa9c2579f8ba47e33 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:21.595144 1166 consensus_queue.cc:260] T ff4c38150df04a75855e0e60ff417ddb P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:21.595371 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64e5e789102e4067a1c3c551784cdef0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:21.595669 524 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_2f9f5508a47d486ca4004b3c2b50013d" } modify_external_catalogs: true
I20251029 02:52:21.595813 1166 ts_tablet_manager.cc:1434] T ff4c38150df04a75855e0e60ff417ddb P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251029 02:52:21.595839 524 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_2f9f5508a47d486ca4004b3c2b50013d" } modify_external_catalogs: true
I20251029 02:52:21.596073 1166 tablet_bootstrap.cc:492] T e3000183230e4350b47f5085952f2db6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:21.596654 1167 tablet_bootstrap.cc:654] T c64d6761ef924ccaa9c2579f8ba47e33 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:21.597561 1166 tablet_bootstrap.cc:654] T e3000183230e4350b47f5085952f2db6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
W20251029 02:52:21.599468 847 consensus_peers.cc:597] T cc802bda2f6a4e439b605f2cafa37967 P c756528642a543eeacd93cef3e335a7d -> Peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Couldn't send request to peer b495a49349324f7d81908f2da700f0d9. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:21.601745 3047 raft_consensus.cc:493] T 688af6580c4d4637a39f817b4e324618 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:21.601929 3047 raft_consensus.cc:515] T 688af6580c4d4637a39f817b4e324618 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:21.602998 1052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "688af6580c4d4637a39f817b4e324618" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:21.603348 1052 raft_consensus.cc:2468] T 688af6580c4d4637a39f817b4e324618 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:21.603765 849 leader_election.cc:304] T 688af6580c4d4637a39f817b4e324618 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:21.604123 2426 raft_consensus.cc:2804] T 688af6580c4d4637a39f817b4e324618 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:21.604195 2426 raft_consensus.cc:493] T 688af6580c4d4637a39f817b4e324618 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:21.604254 2426 raft_consensus.cc:3060] T 688af6580c4d4637a39f817b4e324618 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:21.607177 2426 raft_consensus.cc:515] T 688af6580c4d4637a39f817b4e324618 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:21.607583 1172 tablet_bootstrap.cc:492] T ff4c38150df04a75855e0e60ff417ddb P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:21.607707 1172 ts_tablet_manager.cc:1403] T ff4c38150df04a75855e0e60ff417ddb P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.077s user 0.004s sys 0.000s
I20251029 02:52:21.608654 1172 raft_consensus.cc:359] T ff4c38150df04a75855e0e60ff417ddb P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:21.608794 2426 leader_election.cc:290] T 688af6580c4d4637a39f817b4e324618 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:21.608965 1172 raft_consensus.cc:385] T ff4c38150df04a75855e0e60ff417ddb P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:21.609088 1172 raft_consensus.cc:740] T ff4c38150df04a75855e0e60ff417ddb P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
W20251029 02:52:21.609177 847 leader_election.cc:343] T 688af6580c4d4637a39f817b4e324618 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 688af6580c4d4637a39f817b4e324618
I20251029 02:52:21.609315 1172 consensus_queue.cc:260] T ff4c38150df04a75855e0e60ff417ddb P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:21.610134 1055 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "688af6580c4d4637a39f817b4e324618" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:52:21.610328 1055 raft_consensus.cc:3060] T 688af6580c4d4637a39f817b4e324618 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
W20251029 02:52:21.610954 847 leader_election.cc:343] T 688af6580c4d4637a39f817b4e324618 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 688af6580c4d4637a39f817b4e324618
I20251029 02:52:21.611734 3047 leader_election.cc:290] T 688af6580c4d4637a39f817b4e324618 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
W20251029 02:52:21.612424 849 consensus_peers.cc:597] T 023a4094b424464c8f49d6eb9cc15da4 P c756528642a543eeacd93cef3e335a7d -> Peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Couldn't send request to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:21.614331 1055 raft_consensus.cc:2468] T 688af6580c4d4637a39f817b4e324618 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:21.614956 739 tablet_service.cc:1505] Processing CreateTablet for tablet 135bb898a4f24ee4b659be0b47fa690e (DEFAULT_TABLE table=default.Table_bdff4e69029249389def3ceac417dfbc [id=e05669b2913f4e09903138c3e2da3794]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:21.615566 879 tablet_service.cc:1505] Processing CreateTablet for tablet 135bb898a4f24ee4b659be0b47fa690e (DEFAULT_TABLE table=default.Table_bdff4e69029249389def3ceac417dfbc [id=e05669b2913f4e09903138c3e2da3794]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:21.616917 1172 ts_tablet_manager.cc:1434] T ff4c38150df04a75855e0e60ff417ddb P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.009s user 0.002s sys 0.001s
I20251029 02:52:21.618310 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 135bb898a4f24ee4b659be0b47fa690e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:21.620240 3044 consensus_queue.cc:1048] T 023a4094b424464c8f49d6eb9cc15da4 P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:21.621165 849 leader_election.cc:304] T 688af6580c4d4637a39f817b4e324618 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters: b495a49349324f7d81908f2da700f0d9
I20251029 02:52:21.621524 3044 raft_consensus.cc:2804] T 688af6580c4d4637a39f817b4e324618 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:21.621630 3044 raft_consensus.cc:697] T 688af6580c4d4637a39f817b4e324618 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:52:21.621886 3044 consensus_queue.cc:237] T 688af6580c4d4637a39f817b4e324618 P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:21.624892 1175 ts_tablet_manager.cc:1929] T cc802bda2f6a4e439b605f2cafa37967 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20251029 02:52:21.626938 1024 tablet_service.cc:1505] Processing CreateTablet for tablet 64e5e789102e4067a1c3c551784cdef0 (DEFAULT_TABLE table=default.Table_570d77b4786e4f80a120744a8e94abf0 [id=c2354f113a854046a5cffc064a3ef21b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:21.627480 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 64e5e789102e4067a1c3c551784cdef0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:21.630842 1175 log.cc:1199] T cc802bda2f6a4e439b605f2cafa37967 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/cc802bda2f6a4e439b605f2cafa37967
I20251029 02:52:21.631388 1175 ts_tablet_manager.cc:1950] T cc802bda2f6a4e439b605f2cafa37967 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:21.638144 522 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 09e3ce70eb9a4a6c9035dabbf7434636
I20251029 02:52:21.638676 522 catalog_manager.cc:6719] ReplaceTablet: tablet 09e3ce70eb9a4a6c9035dabbf7434636 deleted and replaced by tablet 8373bfd07e1a48b49164a72cca6ce4df
I20251029 02:52:21.638933 885 tablet_service.cc:1552] Processing DeleteTablet for tablet 09e3ce70eb9a4a6c9035dabbf7434636 with delete_type TABLET_DATA_DELETED (replaced by tablet 8373bfd07e1a48b49164a72cca6ce4df) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:21.639482 738 tablet_service.cc:1552] Processing DeleteTablet for tablet 09e3ce70eb9a4a6c9035dabbf7434636 with delete_type TABLET_DATA_DELETED (replaced by tablet 8373bfd07e1a48b49164a72cca6ce4df) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:21.640492 1031 tablet_service.cc:1552] Processing DeleteTablet for tablet 09e3ce70eb9a4a6c9035dabbf7434636 with delete_type TABLET_DATA_DELETED (replaced by tablet 8373bfd07e1a48b49164a72cca6ce4df) from {username='slave'} at 127.0.0.1:37626
W20251029 02:52:21.641248 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 09e3ce70eb9a4a6c9035dabbf7434636 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 09e3ce70eb9a4a6c9035dabbf7434636 already in progress: creating tablet
I20251029 02:52:21.641461 2426 raft_consensus.cc:493] T 20a762344c5f46f88481c4ac7ad6e43b P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:21.641645 2426 raft_consensus.cc:515] T 20a762344c5f46f88481c4ac7ad6e43b P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:21.642287 1166 tablet_bootstrap.cc:492] T e3000183230e4350b47f5085952f2db6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:21.642481 1166 ts_tablet_manager.cc:1403] T e3000183230e4350b47f5085952f2db6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.046s user 0.004s sys 0.004s
W20251029 02:52:21.643030 847 leader_election.cc:343] T 20a762344c5f46f88481c4ac7ad6e43b P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 20a762344c5f46f88481c4ac7ad6e43b
I20251029 02:52:21.643590 2426 leader_election.cc:290] T 20a762344c5f46f88481c4ac7ad6e43b P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:21.643908 1166 raft_consensus.cc:359] T e3000183230e4350b47f5085952f2db6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:21.644085 1166 raft_consensus.cc:385] T e3000183230e4350b47f5085952f2db6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:21.644143 1166 raft_consensus.cc:740] T e3000183230e4350b47f5085952f2db6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:21.644344 1166 consensus_queue.cc:260] T e3000183230e4350b47f5085952f2db6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:21.644836 1166 ts_tablet_manager.cc:1434] T e3000183230e4350b47f5085952f2db6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251029 02:52:21.645113 1166 tablet_bootstrap.cc:492] T 08371b3daa184a0e9f4eb3d313b52c56 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:21.645336 1167 tablet_bootstrap.cc:492] T c64d6761ef924ccaa9c2579f8ba47e33 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:21.645467 1167 ts_tablet_manager.cc:1403] T c64d6761ef924ccaa9c2579f8ba47e33 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.051s user 0.002s sys 0.002s
I20251029 02:52:21.646791 1166 tablet_bootstrap.cc:654] T 08371b3daa184a0e9f4eb3d313b52c56 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:21.646948 1167 raft_consensus.cc:359] T c64d6761ef924ccaa9c2579f8ba47e33 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
W20251029 02:52:21.647117 849 leader_election.cc:343] T 20a762344c5f46f88481c4ac7ad6e43b P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Not found: Tablet not found: 20a762344c5f46f88481c4ac7ad6e43b
I20251029 02:52:21.647264 849 leader_election.cc:304] T 20a762344c5f46f88481c4ac7ad6e43b P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9
I20251029 02:52:21.647694 2426 raft_consensus.cc:2749] T 20a762344c5f46f88481c4ac7ad6e43b P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251029 02:52:21.655611 1174 ts_tablet_manager.cc:1929] T 7bb85a12f573494cbb3118d1947eda27 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:21.657194 1174 log.cc:1199] T 7bb85a12f573494cbb3118d1947eda27 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/7bb85a12f573494cbb3118d1947eda27
I20251029 02:52:21.657711 1174 ts_tablet_manager.cc:1950] T 7bb85a12f573494cbb3118d1947eda27 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:21.659313 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 135bb898a4f24ee4b659be0b47fa690e. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:21.662987 1172 tablet_bootstrap.cc:492] T e74eda48dca647efacabe525ef316944 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:21.663319 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 1160) took 5796 ms. Trace:
I20251029 02:52:21.663520 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 1083) took 7736 ms. Trace:
I20251029 02:52:21.663457 1175 rpcz_store.cc:276] 1029 02:52:15.867221 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:15.867283 (+ 62us) service_pool.cc:225] Handling call
1029 02:52:21.576896 (+5709613us) ts_tablet_manager.cc:1151] Deleting tablet cc802bda2f6a4e439b605f2cafa37967
1029 02:52:21.576906 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:21.606370 (+ 29464us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:21.624779 (+ 18409us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:21.663296 (+ 38517us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5709432}
I20251029 02:52:21.663969 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet cc802bda2f6a4e439b605f2cafa37967 (table default.Table_6e96405b009143d2bd1c0942cb98c033 [id=67905be94563417c8713eadd51eb94cc]) successfully deleted
I20251029 02:52:21.664079 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 1165) took 5683 ms. Trace:
I20251029 02:52:21.664167 1175 rpcz_store.cc:276] 1029 02:52:15.980609 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:15.984946 (+ 4337us) service_pool.cc:225] Handling call
1029 02:52:21.663927 (+5678981us) ts_tablet_manager.cc:1151] Deleting tablet 2849e856dd714da88fca5a9fafe74c22
1029 02:52:21.663941 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:21.664058 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5678722}
W20251029 02:52:21.664680 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 2849e856dd714da88fca5a9fafe74c22 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2849e856dd714da88fca5a9fafe74c22 already in progress: creating tablet
I20251029 02:52:21.664741 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 1164) took 5684 ms. Trace:
I20251029 02:52:21.664918 1172 tablet_bootstrap.cc:654] T e74eda48dca647efacabe525ef316944 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:21.664820 1175 rpcz_store.cc:276] 1029 02:52:15.980308 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:15.985557 (+ 5249us) service_pool.cc:225] Handling call
1029 02:52:21.664643 (+5679086us) ts_tablet_manager.cc:1151] Deleting tablet 19e15a6df77f4e0c9b40ae4f49fdd06f
1029 02:52:21.664651 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:21.664725 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5678800}
W20251029 02:52:21.665251 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 19e15a6df77f4e0c9b40ae4f49fdd06f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 19e15a6df77f4e0c9b40ae4f49fdd06f already in progress: creating tablet
I20251029 02:52:21.665400 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 1168) took 5627 ms. Trace:
I20251029 02:52:21.665529 1175 rpcz_store.cc:276] 1029 02:52:16.038296 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:16.040208 (+ 1912us) service_pool.cc:225] Handling call
1029 02:52:21.665297 (+5625089us) ts_tablet_manager.cc:1151] Deleting tablet 46e054ff563f448d86d8b4f84810291b
1029 02:52:21.665312 (+ 15us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:21.665386 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5624888}
W20251029 02:52:21.666003 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 46e054ff563f448d86d8b4f84810291b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 46e054ff563f448d86d8b4f84810291b already in progress: creating tablet
I20251029 02:52:21.666126 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 1169) took 5622 ms. Trace:
I20251029 02:52:21.666185 1175 rpcz_store.cc:276] 1029 02:52:16.043574 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:16.048024 (+ 4450us) service_pool.cc:225] Handling call
1029 02:52:21.666025 (+5618001us) ts_tablet_manager.cc:1151] Deleting tablet 5bcadc0b68e84ae290690edef392a45b
1029 02:52:21.666036 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:21.666112 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5617790}
W20251029 02:52:21.666594 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 5bcadc0b68e84ae290690edef392a45b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5bcadc0b68e84ae290690edef392a45b already in progress: creating tablet
I20251029 02:52:21.666694 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 1170) took 5612 ms. Trace:
I20251029 02:52:21.666759 1175 rpcz_store.cc:276] 1029 02:52:16.054134 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:16.057434 (+ 3300us) service_pool.cc:225] Handling call
1029 02:52:21.666606 (+5609172us) ts_tablet_manager.cc:1151] Deleting tablet 60f0eadc074a4872b6638314fed98e0e
1029 02:52:21.666614 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:21.666681 (+ 67us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5608982}
W20251029 02:52:21.667270 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 60f0eadc074a4872b6638314fed98e0e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 60f0eadc074a4872b6638314fed98e0e already in progress: creating tablet
I20251029 02:52:21.667363 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 1187) took 5374 ms. Trace:
I20251029 02:52:21.667426 1175 rpcz_store.cc:276] 1029 02:52:16.292537 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:16.292626 (+ 89us) service_pool.cc:225] Handling call
1029 02:52:21.667274 (+5374648us) ts_tablet_manager.cc:1151] Deleting tablet 90c2ed692a954a25bdf79b9884f6c5ce
1029 02:52:21.667283 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:21.667351 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5374447}
I20251029 02:52:21.667443 1174 rpcz_store.cc:276] 1029 02:52:13.927446 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:13.927494 (+ 48us) service_pool.cc:225] Handling call
1029 02:52:21.557828 (+7630334us) ts_tablet_manager.cc:1151] Deleting tablet 7bb85a12f573494cbb3118d1947eda27
1029 02:52:21.557841 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:21.630959 (+ 73118us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:21.655505 (+ 24546us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:21.663496 (+ 7991us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7630149}
W20251029 02:52:21.667778 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 90c2ed692a954a25bdf79b9884f6c5ce because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 90c2ed692a954a25bdf79b9884f6c5ce already in progress: creating tablet
I20251029 02:52:21.667876 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 1188) took 5373 ms. Trace:
I20251029 02:52:21.667987 1175 rpcz_store.cc:276] 1029 02:52:16.293959 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:16.294006 (+ 47us) service_pool.cc:225] Handling call
1029 02:52:21.667791 (+5373785us) ts_tablet_manager.cc:1151] Deleting tablet 6408901d1e5f4ac8a7bff0de708660eb
1029 02:52:21.667799 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:21.667862 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5373606}
W20251029 02:52:21.668380 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 6408901d1e5f4ac8a7bff0de708660eb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6408901d1e5f4ac8a7bff0de708660eb already in progress: creating tablet
I20251029 02:52:21.668460 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 1186) took 5406 ms. Trace:
I20251029 02:52:21.668521 1175 rpcz_store.cc:276] 1029 02:52:16.262119 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:16.262181 (+ 62us) service_pool.cc:225] Handling call
1029 02:52:21.668375 (+5406194us) ts_tablet_manager.cc:1151] Deleting tablet 7fba607c566c4fa1a803cba8d86ca646
1029 02:52:21.668384 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:21.668448 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5353230}
W20251029 02:52:21.669238 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 7fba607c566c4fa1a803cba8d86ca646 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7fba607c566c4fa1a803cba8d86ca646 already in progress: creating tablet
I20251029 02:52:21.669320 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 1191) took 5247 ms. Trace:
I20251029 02:52:21.669385 1175 rpcz_store.cc:276] 1029 02:52:16.421922 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:16.422881 (+ 959us) service_pool.cc:225] Handling call
1029 02:52:21.669227 (+5246346us) ts_tablet_manager.cc:1151] Deleting tablet 818262aee43c4343b864e01f836e76b8
1029 02:52:21.669235 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:21.669308 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5246141}
I20251029 02:52:21.669819 1175 tablet_replica.cc:333] T e3000183230e4350b47f5085952f2db6 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
W20251029 02:52:21.669811 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 818262aee43c4343b864e01f836e76b8 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 818262aee43c4343b864e01f836e76b8 already in progress: creating tablet
I20251029 02:52:21.670069 1174 tablet_replica.cc:333] T 72f9d4408f384e7093316d85f092e1b2 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:21.670078 1175 raft_consensus.cc:2243] T e3000183230e4350b47f5085952f2db6 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:21.670159 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 7bb85a12f573494cbb3118d1947eda27 (table default.Table_0100f32eabf24d88a739a3cdac3597e1 [id=f22450cafadd404d8d224cdee646b5a2]) successfully deleted
I20251029 02:52:21.647176 1167 raft_consensus.cc:385] T c64d6761ef924ccaa9c2579f8ba47e33 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:21.670367 1175 raft_consensus.cc:2272] T e3000183230e4350b47f5085952f2db6 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:21.670493 1167 raft_consensus.cc:740] T c64d6761ef924ccaa9c2579f8ba47e33 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:21.670778 1167 consensus_queue.cc:260] T c64d6761ef924ccaa9c2579f8ba47e33 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:21.671398 1167 ts_tablet_manager.cc:1434] T c64d6761ef924ccaa9c2579f8ba47e33 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.026s user 0.001s sys 0.001s
I20251029 02:52:21.671451 1175 ts_tablet_manager.cc:1916] T e3000183230e4350b47f5085952f2db6 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:21.671734 1167 tablet_bootstrap.cc:492] T 2b0988f736a7402997ca3ba323a59e18 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:21.673808 1167 tablet_bootstrap.cc:654] T 2b0988f736a7402997ca3ba323a59e18 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:21.670310 1174 raft_consensus.cc:2243] T 72f9d4408f384e7093316d85f092e1b2 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:21.674978 1174 raft_consensus.cc:2272] T 72f9d4408f384e7093316d85f092e1b2 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:21.675669 1174 ts_tablet_manager.cc:1916] T 72f9d4408f384e7093316d85f092e1b2 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:21.685259 1174 ts_tablet_manager.cc:1929] T 72f9d4408f384e7093316d85f092e1b2 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:21.689582 520 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_eb6ed8b2aee44b00a5ac4f8d2a357f28" } new_table_name: "default.Table_31ef1a308d11493fa97badc050340802" modify_external_catalogs: true
I20251029 02:52:21.691098 521 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_3472aa6b45a94900a36ced17aa190e9e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:21.691685 521 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3472aa6b45a94900a36ced17aa190e9e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:21.691881 519 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_d80394d599864f1fb36d8008be9d70c2" } modify_external_catalogs: true
I20251029 02:52:21.692018 519 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_d80394d599864f1fb36d8008be9d70c2" } modify_external_catalogs: true
I20251029 02:52:21.691699 522 catalog_manager.cc:3527] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_7533d0093ad94ff6b065550375df858a" } new_table_name: "default.Table_a02e90ab9ae549939eafb32ea466863e" modify_external_catalogs: true
I20251029 02:52:21.696974 1172 tablet_bootstrap.cc:492] T e74eda48dca647efacabe525ef316944 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:21.697238 1172 ts_tablet_manager.cc:1403] T e74eda48dca647efacabe525ef316944 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.034s user 0.003s sys 0.003s
I20251029 02:52:21.698240 1172 raft_consensus.cc:359] T e74eda48dca647efacabe525ef316944 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:21.698522 1172 raft_consensus.cc:385] T e74eda48dca647efacabe525ef316944 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:21.698864 1172 raft_consensus.cc:740] T e74eda48dca647efacabe525ef316944 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:21.699254 1172 consensus_queue.cc:260] T e74eda48dca647efacabe525ef316944 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:21.700474 1172 ts_tablet_manager.cc:1434] T e74eda48dca647efacabe525ef316944 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20251029 02:52:21.704726 1172 tablet_bootstrap.cc:492] T 15137fc1c70347b381ffdfe5653a5f22 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:21.707763 1174 log.cc:1199] T 72f9d4408f384e7093316d85f092e1b2 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/72f9d4408f384e7093316d85f092e1b2
I20251029 02:52:21.708245 1174 ts_tablet_manager.cc:1950] T 72f9d4408f384e7093316d85f092e1b2 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:21.708669 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 1086) took 7707 ms. Trace:
I20251029 02:52:21.708820 1174 rpcz_store.cc:276] 1029 02:52:14.001244 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:14.001303 (+ 59us) service_pool.cc:225] Handling call
1029 02:52:21.670007 (+7668704us) ts_tablet_manager.cc:1151] Deleting tablet 72f9d4408f384e7093316d85f092e1b2
1029 02:52:21.670023 (+ 16us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:21.679165 (+ 9142us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:21.685149 (+ 5984us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:21.708651 (+ 23502us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7668502}
I20251029 02:52:21.709285 1174 tablet_replica.cc:333] T 372d93914abc41b38a702742b7db0bc2 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:21.709543 1174 raft_consensus.cc:2243] T 372d93914abc41b38a702742b7db0bc2 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:21.709568 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 72f9d4408f384e7093316d85f092e1b2 (table default.Table_8a277d5e05fb437096e36f29a6307d74 [id=e5c07dcc74d1493fa9e1ef153b10d87b]) successfully deleted
I20251029 02:52:21.709784 1174 raft_consensus.cc:2272] T 372d93914abc41b38a702742b7db0bc2 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:21.709800 2992 raft_consensus.cc:493] T 38f83ebcb2a440bc8e6a8d36607e6666 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:21.709931 2992 raft_consensus.cc:515] T 38f83ebcb2a440bc8e6a8d36607e6666 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:21.710348 1166 tablet_bootstrap.cc:492] T 08371b3daa184a0e9f4eb3d313b52c56 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:21.710426 1174 ts_tablet_manager.cc:1916] T 372d93914abc41b38a702742b7db0bc2 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:21.710494 1166 ts_tablet_manager.cc:1403] T 08371b3daa184a0e9f4eb3d313b52c56 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.065s user 0.003s sys 0.000s
I20251029 02:52:21.701373 1167 tablet_bootstrap.cc:492] T 2b0988f736a7402997ca3ba323a59e18 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:21.710587 2992 leader_election.cc:290] T 38f83ebcb2a440bc8e6a8d36607e6666 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:21.710741 1167 ts_tablet_manager.cc:1403] T 2b0988f736a7402997ca3ba323a59e18 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.039s user 0.006s sys 0.002s
I20251029 02:52:21.711289 1166 raft_consensus.cc:359] T 08371b3daa184a0e9f4eb3d313b52c56 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:21.711448 1166 raft_consensus.cc:385] T 08371b3daa184a0e9f4eb3d313b52c56 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:21.711500 1166 raft_consensus.cc:740] T 08371b3daa184a0e9f4eb3d313b52c56 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:21.711488 919 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "38f83ebcb2a440bc8e6a8d36607e6666" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:21.711537 1167 raft_consensus.cc:359] T 2b0988f736a7402997ca3ba323a59e18 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:21.711649 1167 raft_consensus.cc:385] T 2b0988f736a7402997ca3ba323a59e18 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:21.711691 1167 raft_consensus.cc:740] T 2b0988f736a7402997ca3ba323a59e18 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:21.711706 1166 consensus_queue.cc:260] T 08371b3daa184a0e9f4eb3d313b52c56 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:21.711870 1167 consensus_queue.cc:260] T 2b0988f736a7402997ca3ba323a59e18 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:21.712386 1167 ts_tablet_manager.cc:1434] T 2b0988f736a7402997ca3ba323a59e18 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 02:52:21.712772 1167 tablet_bootstrap.cc:492] T d24fe99a1fb54476b90da2c22ff5d383 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:21.712956 1052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "38f83ebcb2a440bc8e6a8d36607e6666" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
W20251029 02:52:21.713477 712 leader_election.cc:343] T 38f83ebcb2a440bc8e6a8d36607e6666 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
W20251029 02:52:21.714078 711 leader_election.cc:343] T 38f83ebcb2a440bc8e6a8d36607e6666 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): Illegal state: must be running to vote when last-logged opid is not known
I20251029 02:52:21.714560 1167 tablet_bootstrap.cc:654] T d24fe99a1fb54476b90da2c22ff5d383 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:21.714733 1166 ts_tablet_manager.cc:1434] T 08371b3daa184a0e9f4eb3d313b52c56 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20251029 02:52:21.715116 1166 tablet_bootstrap.cc:492] T d1866d6dcb0349bba5db86163375cada P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:21.714187 711 leader_election.cc:304] T 38f83ebcb2a440bc8e6a8d36607e6666 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:21.716333 2987 raft_consensus.cc:2749] T 38f83ebcb2a440bc8e6a8d36607e6666 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:21.716737 1166 tablet_bootstrap.cc:654] T d1866d6dcb0349bba5db86163375cada P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:21.722162 1172 tablet_bootstrap.cc:654] T 15137fc1c70347b381ffdfe5653a5f22 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:21.724334 887 tablet_service.cc:1505] Processing CreateTablet for tablet fee646615c1447eb8ecb37b749a7aeba (DEFAULT_TABLE table=default.Table_0847f9152d7041d7a2e814ad64f9d06b [id=12d5cf9f472e49ff98fd62314893a8b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20251029 02:52:21.725503 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fee646615c1447eb8ecb37b749a7aeba. 1 dirs total, 0 dirs full, 0 dirs failed
I20251029 02:52:21.726959 518 master_service.cc:954] ReplaceTablet: received request to replace tablet 358bdfbcb4ee440c899fb6eedef68f44 from {username='slave'} at 127.0.0.1:35314
I20251029 02:52:21.727591 518 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_cac1c6f3bbf54a0d93c23dafc0b7d83c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:21.728087 518 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cac1c6f3bbf54a0d93c23dafc0b7d83c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:21.733847 3066 tablet.cc:1722] T b125cc9aaf4248b8bfa48fb152b832a6 P b495a49349324f7d81908f2da700f0d9: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 6
I20251029 02:52:21.736675 1174 ts_tablet_manager.cc:1929] T 372d93914abc41b38a702742b7db0bc2 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:21.738452 1167 tablet_bootstrap.cc:492] T d24fe99a1fb54476b90da2c22ff5d383 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:21.738720 1167 ts_tablet_manager.cc:1403] T d24fe99a1fb54476b90da2c22ff5d383 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.000s
I20251029 02:52:21.738829 3067 tablet.cc:1722] T b125cc9aaf4248b8bfa48fb152b832a6 P c756528642a543eeacd93cef3e335a7d: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 6
I20251029 02:52:21.739543 1167 raft_consensus.cc:359] T d24fe99a1fb54476b90da2c22ff5d383 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:21.739775 1167 raft_consensus.cc:385] T d24fe99a1fb54476b90da2c22ff5d383 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:21.739858 1167 raft_consensus.cc:740] T d24fe99a1fb54476b90da2c22ff5d383 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:21.740106 1167 consensus_queue.cc:260] T d24fe99a1fb54476b90da2c22ff5d383 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:21.740653 1167 ts_tablet_manager.cc:1434] T d24fe99a1fb54476b90da2c22ff5d383 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 02:52:21.740957 1167 tablet_bootstrap.cc:492] T da904606986f452cadd06187fff06785 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:21.742673 1167 tablet_bootstrap.cc:654] T da904606986f452cadd06187fff06785 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:21.742801 1174 log.cc:1199] T 372d93914abc41b38a702742b7db0bc2 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/372d93914abc41b38a702742b7db0bc2
I20251029 02:52:21.744062 1174 ts_tablet_manager.cc:1950] T 372d93914abc41b38a702742b7db0bc2 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:21.744443 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 1087) took 7743 ms. Trace:
I20251029 02:52:21.744585 1175 ts_tablet_manager.cc:1929] T e3000183230e4350b47f5085952f2db6 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:21.744769 1175 log.cc:1199] T e3000183230e4350b47f5085952f2db6 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/e3000183230e4350b47f5085952f2db6
I20251029 02:52:21.744570 1174 rpcz_store.cc:276] 1029 02:52:14.001366 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:14.001416 (+ 50us) service_pool.cc:225] Handling call
1029 02:52:21.709234 (+7707818us) ts_tablet_manager.cc:1151] Deleting tablet 372d93914abc41b38a702742b7db0bc2
1029 02:52:21.709245 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:21.732378 (+ 23133us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:21.736582 (+ 4204us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:21.744428 (+ 7846us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7707697}
I20251029 02:52:21.745148 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 372d93914abc41b38a702742b7db0bc2 (table default.Table_4e9d94609b4a4d1f94615dcc9830d717 [id=761ebdd0095c414488aa221cbc99d3bc]) successfully deleted
I20251029 02:52:21.745211 1175 ts_tablet_manager.cc:1950] T e3000183230e4350b47f5085952f2db6 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:21.745539 1174 tablet_replica.cc:333] T 6142018e44af4d32818adefa2ee43d02 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:21.745770 1174 raft_consensus.cc:2243] T 6142018e44af4d32818adefa2ee43d02 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:21.745996 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 1190) took 5324 ms. Trace:
I20251029 02:52:21.746141 1174 raft_consensus.cc:2272] T 6142018e44af4d32818adefa2ee43d02 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:21.746712 1174 ts_tablet_manager.cc:1916] T 6142018e44af4d32818adefa2ee43d02 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:21.747674 3054 raft_consensus.cc:493] T 294afb9b012c40c0b05b73795d6f0ac8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:21.747805 3054 raft_consensus.cc:515] T 294afb9b012c40c0b05b73795d6f0ac8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:21.748405 3054 leader_election.cc:290] T 294afb9b012c40c0b05b73795d6f0ac8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:21.749042 919 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "294afb9b012c40c0b05b73795d6f0ac8" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:21.757964 1174 ts_tablet_manager.cc:1929] T 6142018e44af4d32818adefa2ee43d02 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:21.749639 769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "294afb9b012c40c0b05b73795d6f0ac8" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:21.759496 1172 tablet_bootstrap.cc:492] T 15137fc1c70347b381ffdfe5653a5f22 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:21.762676 1172 ts_tablet_manager.cc:1403] T 15137fc1c70347b381ffdfe5653a5f22 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.058s user 0.003s sys 0.003s
I20251029 02:52:21.762816 989 leader_election.cc:304] T 294afb9b012c40c0b05b73795d6f0ac8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5; no voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:21.763259 3054 raft_consensus.cc:2749] T 294afb9b012c40c0b05b73795d6f0ac8 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:21.762609 1175 rpcz_store.cc:276] 1029 02:52:16.421756 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:16.423554 (+ 1798us) service_pool.cc:225] Handling call
1029 02:52:21.669779 (+5246225us) ts_tablet_manager.cc:1151] Deleting tablet e3000183230e4350b47f5085952f2db6
1029 02:52:21.669787 (+ 8us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:21.677551 (+ 7764us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:21.744490 (+ 66939us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:21.745978 (+ 1488us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5245995}
I20251029 02:52:21.763660 1172 raft_consensus.cc:359] T 15137fc1c70347b381ffdfe5653a5f22 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:21.763855 1172 raft_consensus.cc:385] T 15137fc1c70347b381ffdfe5653a5f22 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:21.763947 1172 raft_consensus.cc:740] T 15137fc1c70347b381ffdfe5653a5f22 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:21.764209 1172 consensus_queue.cc:260] T 15137fc1c70347b381ffdfe5653a5f22 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:21.764524 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet e3000183230e4350b47f5085952f2db6 (table default.Table_5069071caedd4971a57a438fba4fac4f [id=79d22e7fddb24e68a0db1aeacbdfb6e3]) successfully deleted
I20251029 02:52:21.764489 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 1189) took 5342 ms. Trace:
I20251029 02:52:21.764849 1172 ts_tablet_manager.cc:1434] T 15137fc1c70347b381ffdfe5653a5f22 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20251029 02:52:21.764825 1166 tablet_bootstrap.cc:492] T d1866d6dcb0349bba5db86163375cada P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:21.765206 1172 tablet_bootstrap.cc:492] T a530fe8de8ae4be5b0fd9a5f7e94ea9e P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:21.759814 1167 tablet_bootstrap.cc:492] T da904606986f452cadd06187fff06785 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:21.765946 1167 ts_tablet_manager.cc:1403] T da904606986f452cadd06187fff06785 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.025s user 0.000s sys 0.004s
I20251029 02:52:21.765234 1166 ts_tablet_manager.cc:1403] T d1866d6dcb0349bba5db86163375cada P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.050s user 0.004s sys 0.000s
I20251029 02:52:21.767069 1167 raft_consensus.cc:359] T da904606986f452cadd06187fff06785 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:21.767344 1167 raft_consensus.cc:385] T da904606986f452cadd06187fff06785 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:21.767436 1167 raft_consensus.cc:740] T da904606986f452cadd06187fff06785 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:21.767536 517 catalog_manager.cc:5649] T 9492c1027aba4455ad8f539addaed5a7 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } health_report { overall_health: UNKNOWN } } }
I20251029 02:52:21.767733 1167 consensus_queue.cc:260] T da904606986f452cadd06187fff06785 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:21.768139 517 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet c45ffc304e73490387ce800b702b5881 (table default.Table_075076404d494b29b2cb90e9f10219e5 [id=97e8edf0d4df452d96c9e13e55aa0736]). Expected version 1 got 0
I20251029 02:52:21.768307 1167 ts_tablet_manager.cc:1434] T da904606986f452cadd06187fff06785 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 02:52:21.768373 517 catalog_manager.cc:5741] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003) does not have the latest schema for tablet 15137fc1c70347b381ffdfe5653a5f22 (table default.Table_41530f589e9749fb8d7f21460e63db86 [id=4bacb12c51d34f78bf4b1aba964c38ce]). Expected version 2 got 0
I20251029 02:52:21.768570 1167 tablet_bootstrap.cc:492] T ff4c38150df04a75855e0e60ff417ddb P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:21.769080 1166 raft_consensus.cc:359] T d1866d6dcb0349bba5db86163375cada P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:21.769636 1166 raft_consensus.cc:385] T d1866d6dcb0349bba5db86163375cada P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:21.769906 1166 raft_consensus.cc:740] T d1866d6dcb0349bba5db86163375cada P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:21.771095 1172 tablet_bootstrap.cc:654] T a530fe8de8ae4be5b0fd9a5f7e94ea9e P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:21.770684 1166 consensus_queue.cc:260] T d1866d6dcb0349bba5db86163375cada P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:21.772161 1167 tablet_bootstrap.cc:654] T ff4c38150df04a75855e0e60ff417ddb P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:21.764743 1175 rpcz_store.cc:276] 1029 02:52:16.421614 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:16.423652 (+ 2038us) service_pool.cc:225] Handling call
1029 02:52:21.764063 (+5340411us) ts_tablet_manager.cc:1151] Deleting tablet 62f4a04568804c3da03393c08676c5da
1029 02:52:21.764076 (+ 13us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:21.764473 (+ 397us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5340055}
W20251029 02:52:21.772497 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet 62f4a04568804c3da03393c08676c5da because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 62f4a04568804c3da03393c08676c5da already in progress: creating tablet
I20251029 02:52:21.772702 1166 ts_tablet_manager.cc:1434] T d1866d6dcb0349bba5db86163375cada P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20251029 02:52:21.773068 1174 log.cc:1199] T 6142018e44af4d32818adefa2ee43d02 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/6142018e44af4d32818adefa2ee43d02
I20251029 02:52:21.773527 1174 ts_tablet_manager.cc:1950] T 6142018e44af4d32818adefa2ee43d02 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:21.773716 1166 tablet_bootstrap.cc:492] T 7fb5b908db77453da5cfb32e2c955f41 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:21.774205 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 1085) took 7773 ms. Trace:
I20251029 02:52:21.774328 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 1193) took 5144 ms. Trace:
I20251029 02:52:21.774467 1175 rpcz_store.cc:276] 1029 02:52:16.628961 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:16.629414 (+ 453us) service_pool.cc:225] Handling call
1029 02:52:21.772830 (+5143416us) ts_tablet_manager.cc:1151] Deleting tablet c73059366f4b45528f8b907263c4d085
1029 02:52:21.772839 (+ 9us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:21.772936 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5143191}
W20251029 02:52:21.774997 511 catalog_manager.cc:4965] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): delete failed for tablet c73059366f4b45528f8b907263c4d085 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c73059366f4b45528f8b907263c4d085 already in progress: creating tablet
I20251029 02:52:21.778985 1174 rpcz_store.cc:276] 1029 02:52:14.001130 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:14.001414 (+ 284us) service_pool.cc:225] Handling call
1029 02:52:21.745487 (+7744073us) ts_tablet_manager.cc:1151] Deleting tablet 6142018e44af4d32818adefa2ee43d02
1029 02:52:21.745497 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:21.754787 (+ 9290us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:21.757868 (+ 3081us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:21.774188 (+ 16320us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7743852}
I20251029 02:52:21.779490 512 catalog_manager.cc:4985] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): tablet 6142018e44af4d32818adefa2ee43d02 (table default.Table_a45ccb3fef5d4947ab7a8709b7c32119 [id=b3e01f7d5a5f4675a998eaad985347ca]) successfully deleted
I20251029 02:52:21.779747 3047 raft_consensus.cc:493] T 786d9148fb1d417a8c2c8b8c53665899 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:21.775359 1175 tablet_replica.cc:333] T 506f8c112a3b43e1a3d6bbafb8cb81e8 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:21.779922 3047 raft_consensus.cc:515] T 786d9148fb1d417a8c2c8b8c53665899 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:21.780207 1175 raft_consensus.cc:2243] T 506f8c112a3b43e1a3d6bbafb8cb81e8 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:21.780395 1175 raft_consensus.cc:2272] T 506f8c112a3b43e1a3d6bbafb8cb81e8 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:21.780658 1166 tablet_bootstrap.cc:654] T 7fb5b908db77453da5cfb32e2c955f41 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:21.780673 3047 leader_election.cc:290] T 786d9148fb1d417a8c2c8b8c53665899 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
W20251029 02:52:21.781631 847 leader_election.cc:343] T 786d9148fb1d417a8c2c8b8c53665899 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 786d9148fb1d417a8c2c8b8c53665899
I20251029 02:52:21.781739 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 1089) took 7734 ms. Trace:
I20251029 02:52:21.781901 1174 rpcz_store.cc:276] 1029 02:52:14.047409 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:14.047469 (+ 60us) service_pool.cc:225] Handling call
1029 02:52:21.781622 (+7734153us) ts_tablet_manager.cc:1151] Deleting tablet 96427e60edeb45bdab9666a5055b5a99
1029 02:52:21.781634 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:21.781724 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7733982}
W20251029 02:52:21.782361 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 96427e60edeb45bdab9666a5055b5a99 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 96427e60edeb45bdab9666a5055b5a99 already in progress: creating tablet
I20251029 02:52:21.782917 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 1093) took 7695 ms. Trace:
I20251029 02:52:21.782835 1052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "786d9148fb1d417a8c2c8b8c53665899" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:21.783051 1174 rpcz_store.cc:276] 1029 02:52:14.087613 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:14.127127 (+ 39514us) service_pool.cc:225] Handling call
1029 02:52:21.782796 (+7655669us) ts_tablet_manager.cc:1151] Deleting tablet 7751599f5dfc43baa20dc3487c6eddd9
1029 02:52:21.782808 (+ 12us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:21.782901 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7655419}
I20251029 02:52:21.783238 1052 raft_consensus.cc:2468] T 786d9148fb1d417a8c2c8b8c53665899 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:21.783802 849 leader_election.cc:304] T 786d9148fb1d417a8c2c8b8c53665899 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters: b495a49349324f7d81908f2da700f0d9
I20251029 02:52:21.784603 2426 raft_consensus.cc:2804] T 786d9148fb1d417a8c2c8b8c53665899 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:21.784685 2426 raft_consensus.cc:493] T 786d9148fb1d417a8c2c8b8c53665899 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:21.784741 2426 raft_consensus.cc:3060] T 786d9148fb1d417a8c2c8b8c53665899 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:21.787712 2426 raft_consensus.cc:515] T 786d9148fb1d417a8c2c8b8c53665899 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:21.788206 1175 ts_tablet_manager.cc:1916] T 506f8c112a3b43e1a3d6bbafb8cb81e8 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:21.788586 2426 leader_election.cc:290] T 786d9148fb1d417a8c2c8b8c53665899 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
W20251029 02:52:21.789057 847 leader_election.cc:343] T 786d9148fb1d417a8c2c8b8c53665899 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Not found: Tablet not found: 786d9148fb1d417a8c2c8b8c53665899
W20251029 02:52:21.789364 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 7751599f5dfc43baa20dc3487c6eddd9 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7751599f5dfc43baa20dc3487c6eddd9 already in progress: creating tablet
I20251029 02:52:21.789495 1055 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "786d9148fb1d417a8c2c8b8c53665899" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:52:21.789652 1055 raft_consensus.cc:3060] T 786d9148fb1d417a8c2c8b8c53665899 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:21.789755 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 1099) took 7606 ms. Trace:
I20251029 02:52:21.789829 1174 rpcz_store.cc:276] 1029 02:52:14.183538 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:14.183600 (+ 62us) service_pool.cc:225] Handling call
1029 02:52:21.789639 (+7606039us) ts_tablet_manager.cc:1151] Deleting tablet e2c8983cded84a1c8b4205c48a4eac33
1029 02:52:21.789649 (+ 10us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:21.789736 (+ 87us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7605856}
W20251029 02:52:21.790194 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet e2c8983cded84a1c8b4205c48a4eac33 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e2c8983cded84a1c8b4205c48a4eac33 already in progress: creating tablet
I20251029 02:52:21.790860 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 1098) took 7612 ms. Trace:
I20251029 02:52:21.790930 1174 rpcz_store.cc:276] 1029 02:52:14.178749 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:14.191734 (+ 12985us) service_pool.cc:225] Handling call
1029 02:52:21.790732 (+7598998us) ts_tablet_manager.cc:1151] Deleting tablet 595b4073cc0446cea041838097af7471
1029 02:52:21.790743 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:21.790844 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7598734}
I20251029 02:52:21.792496 1055 raft_consensus.cc:2468] T 786d9148fb1d417a8c2c8b8c53665899 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
W20251029 02:52:21.793339 512 catalog_manager.cc:4965] TS c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701): delete failed for tablet 595b4073cc0446cea041838097af7471 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 595b4073cc0446cea041838097af7471 already in progress: creating tablet
W20251029 02:52:21.796084 525 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c5b3edb898544f9eb6b501e20adbdd4b consensus state has no leader
I20251029 02:52:21.796552 525 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_00fa1479cf2e46cc9bb8b66ecafb6778"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:21.799692 525 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_00fa1479cf2e46cc9bb8b66ecafb6778 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:21.804879 2987 raft_consensus.cc:493] T 44b518c3a0964a6fa863b2784d80c8a4 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0cfd4bee16f84b7c8b0ff3ce14db20d5)
I20251029 02:52:21.805039 2987 raft_consensus.cc:515] T 44b518c3a0964a6fa863b2784d80c8a4 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } }
I20251029 02:52:21.805823 2987 leader_election.cc:290] T 44b518c3a0964a6fa863b2784d80c8a4 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
W20251029 02:52:21.807720 712 leader_election.cc:343] T 44b518c3a0964a6fa863b2784d80c8a4 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Not found: Tablet not found: 44b518c3a0964a6fa863b2784d80c8a4
I20251029 02:52:21.801414 3048 raft_consensus.cc:493] T c64d6761ef924ccaa9c2579f8ba47e33 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:21.809531 3048 raft_consensus.cc:515] T c64d6761ef924ccaa9c2579f8ba47e33 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:21.810680 3048 leader_election.cc:290] T c64d6761ef924ccaa9c2579f8ba47e33 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:21.806946 919 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "44b518c3a0964a6fa863b2784d80c8a4" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:21.811607 711 leader_election.cc:304] T 44b518c3a0964a6fa863b2784d80c8a4 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b495a49349324f7d81908f2da700f0d9; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d
I20251029 02:52:21.811688 769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c64d6761ef924ccaa9c2579f8ba47e33" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:21.811820 769 raft_consensus.cc:2468] T c64d6761ef924ccaa9c2579f8ba47e33 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 0.
I20251029 02:52:21.812000 2987 raft_consensus.cc:2749] T 44b518c3a0964a6fa863b2784d80c8a4 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251029 02:52:21.803369 849 leader_election.cc:304] T 786d9148fb1d417a8c2c8b8c53665899 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters: b495a49349324f7d81908f2da700f0d9
I20251029 02:52:21.812587 847 leader_election.cc:304] T c64d6761ef924ccaa9c2579f8ba47e33 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:21.812788 3048 raft_consensus.cc:2804] T 786d9148fb1d417a8c2c8b8c53665899 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:21.812901 3048 raft_consensus.cc:697] T 786d9148fb1d417a8c2c8b8c53665899 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:52:21.800088 1174 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:35702 (request call id 1104) took 7498 ms. Trace:
I20251029 02:52:21.813490 3046 raft_consensus.cc:2804] T c64d6761ef924ccaa9c2579f8ba47e33 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:21.813577 3046 raft_consensus.cc:493] T c64d6761ef924ccaa9c2579f8ba47e33 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:21.813650 3046 raft_consensus.cc:3060] T c64d6761ef924ccaa9c2579f8ba47e33 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:21.814476 1172 tablet_bootstrap.cc:492] T a530fe8de8ae4be5b0fd9a5f7e94ea9e P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:21.814633 1172 ts_tablet_manager.cc:1403] T a530fe8de8ae4be5b0fd9a5f7e94ea9e P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.049s user 0.003s sys 0.001s
I20251029 02:52:21.815434 1172 raft_consensus.cc:359] T a530fe8de8ae4be5b0fd9a5f7e94ea9e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:21.815577 1172 raft_consensus.cc:385] T a530fe8de8ae4be5b0fd9a5f7e94ea9e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:21.815620 1172 raft_consensus.cc:740] T a530fe8de8ae4be5b0fd9a5f7e94ea9e P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:21.816071 1172 consensus_queue.cc:260] T a530fe8de8ae4be5b0fd9a5f7e94ea9e P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:21.813165 3048 consensus_queue.cc:237] T 786d9148fb1d417a8c2c8b8c53665899 P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:21.819041 3046 raft_consensus.cc:515] T c64d6761ef924ccaa9c2579f8ba47e33 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:21.819785 3046 leader_election.cc:290] T c64d6761ef924ccaa9c2579f8ba47e33 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Requested vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:21.820570 769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c64d6761ef924ccaa9c2579f8ba47e33" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:21.820734 769 raft_consensus.cc:3060] T c64d6761ef924ccaa9c2579f8ba47e33 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:21.818312 1172 ts_tablet_manager.cc:1434] T a530fe8de8ae4be5b0fd9a5f7e94ea9e P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251029 02:52:21.822188 1172 tablet_bootstrap.cc:492] T 78662eeae70b4148a4ad0d47be5c566d P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:21.823191 1167 tablet_bootstrap.cc:492] T ff4c38150df04a75855e0e60ff417ddb P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:21.823307 1167 ts_tablet_manager.cc:1403] T ff4c38150df04a75855e0e60ff417ddb P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.055s user 0.002s sys 0.002s
I20251029 02:52:21.823663 769 raft_consensus.cc:2468] T c64d6761ef924ccaa9c2579f8ba47e33 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c756528642a543eeacd93cef3e335a7d in term 1.
I20251029 02:52:21.824183 1172 tablet_bootstrap.cc:654] T 78662eeae70b4148a4ad0d47be5c566d P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:21.824108 1167 raft_consensus.cc:359] T ff4c38150df04a75855e0e60ff417ddb P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:21.824251 1167 raft_consensus.cc:385] T ff4c38150df04a75855e0e60ff417ddb P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:21.824283 847 leader_election.cc:304] T c64d6761ef924ccaa9c2579f8ba47e33 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b495a49349324f7d81908f2da700f0d9, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:21.824812 3046 raft_consensus.cc:2804] T c64d6761ef924ccaa9c2579f8ba47e33 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:21.824975 3046 raft_consensus.cc:697] T c64d6761ef924ccaa9c2579f8ba47e33 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Becoming Leader. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Running, Role: LEADER
I20251029 02:52:21.825259 3046 consensus_queue.cc:237] T c64d6761ef924ccaa9c2579f8ba47e33 P c756528642a543eeacd93cef3e335a7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:21.824357 1167 raft_consensus.cc:740] T ff4c38150df04a75855e0e60ff417ddb P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:21.832392 1167 consensus_queue.cc:260] T ff4c38150df04a75855e0e60ff417ddb P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:21.833292 1167 ts_tablet_manager.cc:1434] T ff4c38150df04a75855e0e60ff417ddb P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20251029 02:52:21.833410 1175 ts_tablet_manager.cc:1929] T 506f8c112a3b43e1a3d6bbafb8cb81e8 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:21.833659 1167 tablet_bootstrap.cc:492] T 38f83ebcb2a440bc8e6a8d36607e6666 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:21.833838 1175 log.cc:1199] T 506f8c112a3b43e1a3d6bbafb8cb81e8 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/506f8c112a3b43e1a3d6bbafb8cb81e8
I20251029 02:52:21.835372 1167 tablet_bootstrap.cc:654] T 38f83ebcb2a440bc8e6a8d36607e6666 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:21.834899 1172 tablet_bootstrap.cc:492] T 78662eeae70b4148a4ad0d47be5c566d P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:21.836131 1172 ts_tablet_manager.cc:1403] T 78662eeae70b4148a4ad0d47be5c566d P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20251029 02:52:21.837422 1172 raft_consensus.cc:359] T 78662eeae70b4148a4ad0d47be5c566d P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:21.846465 1172 raft_consensus.cc:385] T 78662eeae70b4148a4ad0d47be5c566d P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:21.847091 1172 raft_consensus.cc:740] T 78662eeae70b4148a4ad0d47be5c566d P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:21.848184 1172 consensus_queue.cc:260] T 78662eeae70b4148a4ad0d47be5c566d P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:21.849578 3044 raft_consensus.cc:493] T 9492c1027aba4455ad8f539addaed5a7 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:21.836215 1175 ts_tablet_manager.cc:1950] T 506f8c112a3b43e1a3d6bbafb8cb81e8 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:21.849720 3044 raft_consensus.cc:515] T 9492c1027aba4455ad8f539addaed5a7 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:21.850939 769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9492c1027aba4455ad8f539addaed5a7" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:21.851120 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 1194) took 5213 ms. Trace:
I20251029 02:52:21.851176 769 raft_consensus.cc:2393] T 9492c1027aba4455ad8f539addaed5a7 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c756528642a543eeacd93cef3e335a7d in current term 1: Already voted for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in this term.
I20251029 02:52:21.851209 1175 rpcz_store.cc:276] 1029 02:52:16.637479 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:16.637718 (+ 239us) service_pool.cc:225] Handling call
1029 02:52:21.775310 (+5137592us) ts_tablet_manager.cc:1151] Deleting tablet 506f8c112a3b43e1a3d6bbafb8cb81e8
1029 02:52:21.775321 (+ 11us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:21.809886 (+ 34565us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:21.833280 (+ 23394us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:21.851098 (+ 17818us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5137309}
I20251029 02:52:21.851704 3044 leader_election.cc:290] T 9492c1027aba4455ad8f539addaed5a7 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:21.852330 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 506f8c112a3b43e1a3d6bbafb8cb81e8 (table default.Table_d7cbd7b4c4bf4886a4ccf4c89da3cbce [id=9107fd53f5c94d2d80be50ef1f7c8e9f]) successfully deleted
I20251029 02:52:21.852658 1175 tablet_replica.cc:333] T 24f07b1efac64b099b0b5ed467463749 P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:21.853442 1175 raft_consensus.cc:2243] T 24f07b1efac64b099b0b5ed467463749 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:21.855480 1175 raft_consensus.cc:2272] T 24f07b1efac64b099b0b5ed467463749 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:21.856119 1175 ts_tablet_manager.cc:1916] T 24f07b1efac64b099b0b5ed467463749 P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:21.859638 524 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e9261eb85ea443d5a3beb36bd0c549f0: Sending DeleteTablet for 3 replicas of tablet 8c028683f8db4c02a10a37780e20f6dc
I20251029 02:52:21.861397 879 tablet_service.cc:1552] Processing DeleteTablet for tablet 8c028683f8db4c02a10a37780e20f6dc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:21 UTC) from {username='slave'} at 127.0.0.1:35702
I20251029 02:52:21.861442 1024 tablet_service.cc:1552] Processing DeleteTablet for tablet 8c028683f8db4c02a10a37780e20f6dc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:21 UTC) from {username='slave'} at 127.0.0.1:37626
W20251029 02:52:21.862035 513 catalog_manager.cc:4965] TS 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): delete failed for tablet 8c028683f8db4c02a10a37780e20f6dc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8c028683f8db4c02a10a37780e20f6dc already in progress: creating tablet
I20251029 02:52:21.862188 3068 tablet.cc:1722] T b125cc9aaf4248b8bfa48fb152b832a6 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 6
I20251029 02:52:21.867344 1166 tablet_bootstrap.cc:492] T 7fb5b908db77453da5cfb32e2c955f41 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:21.867604 1166 ts_tablet_manager.cc:1403] T 7fb5b908db77453da5cfb32e2c955f41 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.094s user 0.005s sys 0.008s
I20251029 02:52:21.868490 1166 raft_consensus.cc:359] T 7fb5b908db77453da5cfb32e2c955f41 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:21.868801 1166 raft_consensus.cc:385] T 7fb5b908db77453da5cfb32e2c955f41 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:21.868906 1166 raft_consensus.cc:740] T 7fb5b908db77453da5cfb32e2c955f41 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:21.869222 1166 consensus_queue.cc:260] T 7fb5b908db77453da5cfb32e2c955f41 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:21.869872 1166 ts_tablet_manager.cc:1434] T 7fb5b908db77453da5cfb32e2c955f41 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251029 02:52:21.813416 1174 rpcz_store.cc:276] 1029 02:52:14.301437 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:14.301486 (+ 49us) service_pool.cc:225] Handling call
1029 02:52:21.799923 (+7498437us) ts_tablet_manager.cc:1151] Deleting tablet c0936e54f1d246388a2d82dbc52aac42
1029 02:52:21.799940 (+ 17us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:21.800065 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7498239}
I20251029 02:52:21.871084 1166 tablet_bootstrap.cc:492] T da904606986f452cadd06187fff06785 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:21.872840 1174 tablet_replica.cc:333] T f93b0c39d7174dabbe018b1546dbc198 P c756528642a543eeacd93cef3e335a7d: stopping tablet replica
I20251029 02:52:21.854250 1172 ts_tablet_manager.cc:1434] T 78662eeae70b4148a4ad0d47be5c566d P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.018s user 0.003s sys 0.000s
I20251029 02:52:21.861001 750 tablet_service.cc:1552] Processing DeleteTablet for tablet 8c028683f8db4c02a10a37780e20f6dc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-29 02:52:21 UTC) from {username='slave'} at 127.0.0.1:44580
I20251029 02:52:21.874749 1166 tablet_bootstrap.cc:654] T da904606986f452cadd06187fff06785 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:21.877048 1172 tablet_bootstrap.cc:492] T 897a9a75d13f493e90526da4ca7f54d2 P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:21.878814 1172 tablet_bootstrap.cc:654] T 897a9a75d13f493e90526da4ca7f54d2 P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:21.886737 1172 tablet_bootstrap.cc:492] T 897a9a75d13f493e90526da4ca7f54d2 P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:21.886778 1174 raft_consensus.cc:2243] T f93b0c39d7174dabbe018b1546dbc198 P c756528642a543eeacd93cef3e335a7d [term 1 LEADER]: Raft consensus shutting down.
I20251029 02:52:21.887086 1172 ts_tablet_manager.cc:1403] T 897a9a75d13f493e90526da4ca7f54d2 P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.002s
I20251029 02:52:21.887167 1174 raft_consensus.cc:2272] T f93b0c39d7174dabbe018b1546dbc198 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:21.888471 1172 raft_consensus.cc:359] T 897a9a75d13f493e90526da4ca7f54d2 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:21.888793 1172 raft_consensus.cc:385] T 897a9a75d13f493e90526da4ca7f54d2 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:21.888932 1172 raft_consensus.cc:740] T 897a9a75d13f493e90526da4ca7f54d2 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:21.889341 1172 consensus_queue.cc:260] T 897a9a75d13f493e90526da4ca7f54d2 P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:21.892153 1172 ts_tablet_manager.cc:1434] T 897a9a75d13f493e90526da4ca7f54d2 P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251029 02:52:21.893584 1172 tablet_bootstrap.cc:492] T f65fe9a5137b48669afef51ad7dd76ae P b495a49349324f7d81908f2da700f0d9: Bootstrap starting.
I20251029 02:52:21.892994 1174 ts_tablet_manager.cc:1916] T f93b0c39d7174dabbe018b1546dbc198 P c756528642a543eeacd93cef3e335a7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251029 02:52:21.896546 524 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_913e2dbb7d5d4692886f576858021b92" } modify_external_catalogs: true
I20251029 02:52:21.896715 524 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_913e2dbb7d5d4692886f576858021b92" } modify_external_catalogs: true
I20251029 02:52:21.899444 1166 tablet_bootstrap.cc:492] T da904606986f452cadd06187fff06785 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:21.899652 1166 ts_tablet_manager.cc:1403] T da904606986f452cadd06187fff06785 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.002s
I20251029 02:52:21.900444 1166 raft_consensus.cc:359] T da904606986f452cadd06187fff06785 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:21.900677 1166 raft_consensus.cc:385] T da904606986f452cadd06187fff06785 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:21.900776 1166 raft_consensus.cc:740] T da904606986f452cadd06187fff06785 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:21.901043 1166 consensus_queue.cc:260] T da904606986f452cadd06187fff06785 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:21.904436 1166 ts_tablet_manager.cc:1434] T da904606986f452cadd06187fff06785 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20251029 02:52:21.904856 1166 tablet_bootstrap.cc:492] T 023a4094b424464c8f49d6eb9cc15da4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
I20251029 02:52:21.906625 1166 tablet_bootstrap.cc:654] T 023a4094b424464c8f49d6eb9cc15da4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
W20251029 02:52:21.909626 520 catalog_manager.cc:5909] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f3794a0d2f4f421f8b2d0c5649bcb2ec consensus state has no leader
I20251029 02:52:21.910166 1172 tablet_bootstrap.cc:654] T f65fe9a5137b48669afef51ad7dd76ae P b495a49349324f7d81908f2da700f0d9: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:21.912807 1052 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c64d6761ef924ccaa9c2579f8ba47e33" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:21.913587 1055 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c64d6761ef924ccaa9c2579f8ba47e33" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5"
I20251029 02:52:21.914409 1062 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9492c1027aba4455ad8f539addaed5a7" candidate_uuid: "c756528642a543eeacd93cef3e335a7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" is_pre_election: true
I20251029 02:52:21.915172 1174 ts_tablet_manager.cc:1929] T f93b0c39d7174dabbe018b1546dbc198 P c756528642a543eeacd93cef3e335a7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:21.915302 1174 log.cc:1199] T f93b0c39d7174dabbe018b1546dbc198 P c756528642a543eeacd93cef3e335a7d: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-1/wal/wals/f93b0c39d7174dabbe018b1546dbc198
I20251029 02:52:21.915650 1174 ts_tablet_manager.cc:1950] T f93b0c39d7174dabbe018b1546dbc198 P c756528642a543eeacd93cef3e335a7d: Deleting consensus metadata
I20251029 02:52:21.915663 1062 raft_consensus.cc:3060] T 7fb5b908db77453da5cfb32e2c955f41 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:21.919876 1175 ts_tablet_manager.cc:1929] T 24f07b1efac64b099b0b5ed467463749 P b495a49349324f7d81908f2da700f0d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20251029 02:52:21.920059 1175 log.cc:1199] T 24f07b1efac64b099b0b5ed467463749 P b495a49349324f7d81908f2da700f0d9: Deleting WAL directory at /tmp/dist-test-taskYVokEG/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1761706306080299-416-0/minicluster-data/ts-0/wal/wals/24f07b1efac64b099b0b5ed467463749
I20251029 02:52:21.920529 1175 ts_tablet_manager.cc:1950] T 24f07b1efac64b099b0b5ed467463749 P b495a49349324f7d81908f2da700f0d9: Deleting consensus metadata
I20251029 02:52:21.921461 1175 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44580 (request call id 1195) took 5015 ms. Trace:
I20251029 02:52:21.921629 1175 rpcz_store.cc:276] 1029 02:52:16.906347 (+ 0us) service_pool.cc:168] Inserting onto call queue
1029 02:52:16.906410 (+ 63us) service_pool.cc:225] Handling call
1029 02:52:21.852604 (+4946194us) ts_tablet_manager.cc:1151] Deleting tablet 24f07b1efac64b099b0b5ed467463749
1029 02:52:21.852618 (+ 14us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
1029 02:52:21.891560 (+ 38942us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:21.919763 (+ 28203us) tablet_metadata.cc:641] Metadata flushed
1029 02:52:21.921435 (+ 1672us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":64,"tablet-delete.queue_time_us":4878288}
I20251029 02:52:21.922242 511 catalog_manager.cc:4985] TS b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): tablet 24f07b1efac64b099b0b5ed467463749 (table default.Table_21b8ae2debb241bda65fe387a4a3190f [id=f3fa2dbbef7e47359492d5a5eef9b8fd]) successfully deleted
I20251029 02:52:21.922705 1175 tablet_replica.cc:333] T 7816556521144c928d7a02694e22e79f P b495a49349324f7d81908f2da700f0d9: stopping tablet replica
I20251029 02:52:21.922963 1175 raft_consensus.cc:2243] T 7816556521144c928d7a02694e22e79f P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251029 02:52:21.923213 1175 raft_consensus.cc:2272] T 7816556521144c928d7a02694e22e79f P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251029 02:52:21.923866 1175 ts_tablet_manager.cc:1916] T 7816556521144c928d7a02694e22e79f P b495a49349324f7d81908f2da700f0d9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20251029 02:52:21.915784 849 leader_election.cc:343] T c64d6761ef924ccaa9c2579f8ba47e33 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
W20251029 02:52:21.924579 849 leader_election.cc:343] T c64d6761ef924ccaa9c2579f8ba47e33 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Illegal state: must be running to vote when last-logged opid is not known
W20251029 02:52:21.924867 849 consensus_peers.cc:597] T 2b278db5b2b04344b864797055a5f665 P c756528642a543eeacd93cef3e335a7d -> Peer 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003): Couldn't send request to peer 0cfd4bee16f84b7c8b0ff3ce14db20d5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2b278db5b2b04344b864797055a5f665. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:21.925029 849 leader_election.cc:304] T 9492c1027aba4455ad8f539addaed5a7 P c756528642a543eeacd93cef3e335a7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c756528642a543eeacd93cef3e335a7d; no voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9
I20251029 02:52:21.925570 520 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_5db0b0e77f584976b6253ec87f386bb8" } modify_external_catalogs: true
I20251029 02:52:21.925725 520 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_5db0b0e77f584976b6253ec87f386bb8" } modify_external_catalogs: true
I20251029 02:52:21.927767 3048 raft_consensus.cc:3060] T 9492c1027aba4455ad8f539addaed5a7 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:21.930406 3048 raft_consensus.cc:2749] T 9492c1027aba4455ad8f539addaed5a7 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251029 02:52:21.933982 521 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_da047830460443a29c5269c98fc859e9" } modify_external_catalogs: true
I20251029 02:52:21.934170 521 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35314:
table { table_name: "default.Table_da047830460443a29c5269c98fc859e9" } modify_external_catalogs: true
I20251029 02:52:21.935611 3046 consensus_queue.cc:1048] T 7fb5b908db77453da5cfb32e2c955f41 P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:21.936017 3046 consensus_queue.cc:1048] T c45ffc304e73490387ce800b702b5881 P c756528642a543eeacd93cef3e335a7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251029 02:52:21.938990 3054 raft_consensus.cc:493] T da904606986f452cadd06187fff06785 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:21.939142 3054 raft_consensus.cc:515] T da904606986f452cadd06187fff06785 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:21.939612 1167 tablet_bootstrap.cc:492] T 38f83ebcb2a440bc8e6a8d36607e6666 P c756528642a543eeacd93cef3e335a7d: No bootstrap required, opened a new log
I20251029 02:52:21.939823 1167 ts_tablet_manager.cc:1403] T 38f83ebcb2a440bc8e6a8d36607e6666 P c756528642a543eeacd93cef3e335a7d: Time spent bootstrapping tablet: real 0.106s user 0.003s sys 0.000s
I20251029 02:52:21.939883 519 catalog_manager.cc:5931] Not sending DeleteTablet requests; no consensus state for tablet 17e68d2da6014603b87dcacc39fb86c0
I20251029 02:52:21.939914 3054 leader_election.cc:290] T da904606986f452cadd06187fff06785 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:21.940696 769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da904606986f452cadd06187fff06785" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9" is_pre_election: true
I20251029 02:52:21.940888 769 raft_consensus.cc:2468] T da904606986f452cadd06187fff06785 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 0.
I20251029 02:52:21.941015 919 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da904606986f452cadd06187fff06785" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d" is_pre_election: true
I20251029 02:52:21.941031 1167 raft_consensus.cc:359] T 38f83ebcb2a440bc8e6a8d36607e6666 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:21.941231 919 raft_consensus.cc:2468] T da904606986f452cadd06187fff06785 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 0.
I20251029 02:52:21.941359 989 leader_election.cc:304] T da904606986f452cadd06187fff06785 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, b495a49349324f7d81908f2da700f0d9; no voters:
I20251029 02:52:21.941224 1167 raft_consensus.cc:385] T 38f83ebcb2a440bc8e6a8d36607e6666 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:21.941850 1167 raft_consensus.cc:740] T 38f83ebcb2a440bc8e6a8d36607e6666 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c756528642a543eeacd93cef3e335a7d, State: Initialized, Role: FOLLOWER
I20251029 02:52:21.941864 3054 raft_consensus.cc:2804] T da904606986f452cadd06187fff06785 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251029 02:52:21.941933 3054 raft_consensus.cc:493] T da904606986f452cadd06187fff06785 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251029 02:52:21.941999 3054 raft_consensus.cc:3060] T da904606986f452cadd06187fff06785 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:21.942180 1167 consensus_queue.cc:260] T 38f83ebcb2a440bc8e6a8d36607e6666 P c756528642a543eeacd93cef3e335a7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:21.942679 1167 ts_tablet_manager.cc:1434] T 38f83ebcb2a440bc8e6a8d36607e6666 P c756528642a543eeacd93cef3e335a7d: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20251029 02:52:21.944814 3054 raft_consensus.cc:515] T da904606986f452cadd06187fff06785 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:21.945736 919 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da904606986f452cadd06187fff06785" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"
I20251029 02:52:21.945879 919 raft_consensus.cc:3060] T da904606986f452cadd06187fff06785 P c756528642a543eeacd93cef3e335a7d [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:21.946061 1166 tablet_bootstrap.cc:492] T 023a4094b424464c8f49d6eb9cc15da4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:21.946183 1166 ts_tablet_manager.cc:1403] T 023a4094b424464c8f49d6eb9cc15da4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.041s user 0.001s sys 0.002s
I20251029 02:52:21.946928 1166 raft_consensus.cc:359] T 023a4094b424464c8f49d6eb9cc15da4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:21.947106 1166 raft_consensus.cc:385] T 023a4094b424464c8f49d6eb9cc15da4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:21.947166 1166 raft_consensus.cc:740] T 023a4094b424464c8f49d6eb9cc15da4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Initialized, Role: FOLLOWER
I20251029 02:52:21.947407 1166 consensus_queue.cc:260] T 023a4094b424464c8f49d6eb9cc15da4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:21.948613 919 raft_consensus.cc:2468] T da904606986f452cadd06187fff06785 P c756528642a543eeacd93cef3e335a7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 1.
I20251029 02:52:21.949831 990 leader_election.cc:304] T da904606986f452cadd06187fff06785 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cfd4bee16f84b7c8b0ff3ce14db20d5, c756528642a543eeacd93cef3e335a7d; no voters:
I20251029 02:52:21.950381 3085 raft_consensus.cc:2804] T da904606986f452cadd06187fff06785 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 FOLLOWER]: Leader election won for term 1
I20251029 02:52:21.950526 3085 raft_consensus.cc:697] T da904606986f452cadd06187fff06785 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 1 LEADER]: Becoming Leader. State: Replica: 0cfd4bee16f84b7c8b0ff3ce14db20d5, State: Running, Role: LEADER
I20251029 02:52:21.950803 3085 consensus_queue.cc:237] T da904606986f452cadd06187fff06785 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
W20251029 02:52:21.951803 847 consensus_peers.cc:597] T b0fec06ad37d4c12a7045d424b88cc5e P c756528642a543eeacd93cef3e335a7d -> Peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Couldn't send request to peer b495a49349324f7d81908f2da700f0d9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b0fec06ad37d4c12a7045d424b88cc5e. This is attempt 1: this message will repeat every 5th retry.
I20251029 02:52:21.958287 769 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "da904606986f452cadd06187fff06785" candidate_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b495a49349324f7d81908f2da700f0d9"
I20251029 02:52:21.958452 769 raft_consensus.cc:3060] T da904606986f452cadd06187fff06785 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Advancing to term 1
I20251029 02:52:21.961360 769 raft_consensus.cc:2468] T da904606986f452cadd06187fff06785 P b495a49349324f7d81908f2da700f0d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cfd4bee16f84b7c8b0ff3ce14db20d5 in term 1.
I20251029 02:52:21.961963 1166 ts_tablet_manager.cc:1434] T 023a4094b424464c8f49d6eb9cc15da4 P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20251029 02:52:21.962152 3054 leader_election.cc:290] T da904606986f452cadd06187fff06785 P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [CANDIDATE]: Term 1 election: Requested vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481)
I20251029 02:52:21.963661 1166 tablet_bootstrap.cc:492] T f65fe9a5137b48669afef51ad7dd76ae P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Bootstrap starting.
W20251029 02:52:21.967181 847 consensus_peers.cc:597] T 7816556521144c928d7a02694e22e79f P c756528642a543eeacd93cef3e335a7d -> Peer b495a49349324f7d81908f2da700f0d9 (127.0.104.1:38481): Couldn't send request to peer b495a49349324f7d81908f2da700f0d9. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20251029 02:52:21.971668 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.3:45003, user_credentials={real_user=slave}} blocked reactor thread for 167006us
I20251029 02:52:21.973059 1172 tablet_bootstrap.cc:492] T f65fe9a5137b48669afef51ad7dd76ae P b495a49349324f7d81908f2da700f0d9: No bootstrap required, opened a new log
I20251029 02:52:21.974049 1172 ts_tablet_manager.cc:1403] T f65fe9a5137b48669afef51ad7dd76ae P b495a49349324f7d81908f2da700f0d9: Time spent bootstrapping tablet: real 0.081s user 0.008s sys 0.012s
I20251029 02:52:21.978811 1172 raft_consensus.cc:359] T f65fe9a5137b48669afef51ad7dd76ae P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:21.980242 1172 raft_consensus.cc:385] T f65fe9a5137b48669afef51ad7dd76ae P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251029 02:52:21.980487 1166 tablet_bootstrap.cc:654] T f65fe9a5137b48669afef51ad7dd76ae P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:21.980829 1172 raft_consensus.cc:740] T f65fe9a5137b48669afef51ad7dd76ae P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b495a49349324f7d81908f2da700f0d9, State: Initialized, Role: FOLLOWER
I20251029 02:52:21.983162 1167 tablet_bootstrap.cc:492] T c172417d1d444c439e9cda611a2ed551 P c756528642a543eeacd93cef3e335a7d: Bootstrap starting.
I20251029 02:52:21.982367 1172 consensus_queue.cc:260] T f65fe9a5137b48669afef51ad7dd76ae P b495a49349324f7d81908f2da700f0d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:21.984211 519 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35314:
name: "default.Table_cc77a9a6e3984ed0a001f74514de2741"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251029 02:52:21.984804 519 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cc77a9a6e3984ed0a001f74514de2741 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251029 02:52:21.984932 1167 tablet_bootstrap.cc:654] T c172417d1d444c439e9cda611a2ed551 P c756528642a543eeacd93cef3e335a7d: Neither blocks nor log segments found. Creating new log.
I20251029 02:52:21.987306 1166 tablet_bootstrap.cc:492] T f65fe9a5137b48669afef51ad7dd76ae P 0cfd4bee16f84b7c8b0ff3ce14db20d5: No bootstrap required, opened a new log
I20251029 02:52:21.987999 1166 ts_tablet_manager.cc:1403] T f65fe9a5137b48669afef51ad7dd76ae P 0cfd4bee16f84b7c8b0ff3ce14db20d5: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.001s
I20251029 02:52:21.988927 2987 raft_consensus.cc:493] T e74eda48dca647efacabe525ef316944 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251029 02:52:21.989099 2987 raft_consensus.cc:515] T e74eda48dca647efacabe525ef316944 P b495a49349324f7d81908f2da700f0d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:21.989732 2987 leader_election.cc:290] T e74eda48dca647efacabe525ef316944 P b495a49349324f7d81908f2da700f0d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c756528642a543eeacd93cef3e335a7d (127.0.104.2:41701), 0cfd4bee16f84b7c8b0ff3ce14db20d5 (127.0.104.3:45003)
I20251029 02:52:21.990136 1166 raft_consensus.cc:359] T f65fe9a5137b48669afef51ad7dd76ae P 0cfd4bee16f84b7c8b0ff3ce14db20d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0cfd4bee16f84b7c8b0ff3ce14db20d5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45003 } } peers { permanent_uuid: "c756528642a543eeacd93cef3e335a7d" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41701 } } peers { permanent_uuid: "b495a49349324f7d81908f2da700f0d9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38481 } }
I20251029 02:52:21.990487 1172 ts_tablet_manager.cc:1434] T f65fe9a5137b48669afef51ad7dd76ae P b495a49349324f7d81908f2da700f0d9: Time spent starting tablet: real 0.016s user 0.003s sys 0.000s
I20251029 02:52:21.990983 919 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "e74eda48dca647efacabe525ef316944" candidate_uuid: "b495a49349324f7d81908f2da700f0d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c756528642a543eeacd93cef3e335a7d"